It can be very difficult, but we must give no opportunity to the devil. (Ephesians 4:27 ESV).
The body is not meant for sexual immorality, but for the Lord, and the Lord for the body....Flee from sexual immorality. Every other sin a person commits is outside the body, but the sexually immoral person sins against his own body. Or do you not know that your body is a temple of the Holy Spirit within you, whom you have from God? You are not your own, for you were bought with a price. So glorify God in your body.
1 Corinthians 6:13b, 18-20 ESV
Any sex outside the covenant of marriage is sexual immorality.
Blessed is the man who remains steadfast under trial, for when he has stood the test he will receive the crown of life, which God has promised to those who love him. Let no one say when he is tempted, “I am being tempted by God,” for God cannot be tempted with evil, and he himself tempts no one. But each person is tempted when he is lured and enticed by his own desire. Then desire when it has conceived gives birth to sin, and sin when it is fully grown brings forth death.
James 1:12-15 ESV
"It just so happened..." See, you've already experienced a little of how a little leaven leavens the whole lump. (Galatians 5:9 ESV) When we give in just a little, we find it easier to give in a little more and a little more and a little more..... Then we begin to deceive ourselves into justifying it one way or another.
A person steals a candy bar from the local convenience store and discovers that it wasn't that difficult. So, he does it again and again and pretty soon he starts to get more and more confident about it and he starts stealing larger and more valuable items and as he gets away with that it continues to escalate. He even begins to take pride in his ability to do it. This is a story from my own past experience.
The same is true with sexuality. I was a virgin until I was 23 years old. I felt I could control myself as I was making out with girls and for quite few years, I did manage to not go any further than some petting as you have done. Yet, fate eventually caught up with me and then one day it happened that I gave in. And then I gave in again and again and before long, I was informed by my girlfriend that she had missed her cycle. Yup, she was pregnant.
Give a little and the leaven grows.