Should I Step Back?

Hello. A 22-year-old girl I am. I have a "crash" with a 38-year-old man, who, however, has a wife and 2 children. He started looking for me about a year and a half ago, a general company of friends we go on vacation every year for a week. Then his wife had just given birth to their second child and did not come. He was with only one child. He started with nice things at first, but there was always a hint of sex. I struggled for a long time because I didn't think it was right and I wouldn't take advantage of someone's happiness to satisfy him or myself. Many times he invited me to them, as his wife and children were not here, I refused, but it so happened that I had to go to them 2-3 times for work. The first time we just talked, the second kissed me once, the third too. He's wanted sex for a long time, but I don't want to make that mistake and then get sick of it. Recently, shortly after the last time I was in them, began to distance himself, did not respond to messages, did not even reply to the name day card. We ran outside and it was as if nothing had happened. I don't want to "own" him and be his mistress, but I care about him a lot as a person and a friend. Should I try to talk to him or just back off Thanks. Good evening!