Well, when it is a mortal sin, yes. When it is a venial sin, no. When it is a mortal sin, I think it should be confessed. If the priest says it is not a mortal sin, I would say it is better to go to another priest and explain the situation to your conscience.
Of course, I have to say that if you repent of this sin in view of confession, it is pardoned. The Lord pardons this sin because of your intention to confess it.
Follow-up Question: But it's like it's a matter of closure and actually making sure that you brought that mortal sin to Jesus in the sacrament to a priest that actually will absolve it in a proper way.
Yes, it should be done.