I need help with a guy problem how do I tell him I'm done?

I've been talking to this guy lately and I've been wanting to hang out and he said he does but I don't know if that's true because when I'm not busy he says he's working or some stuff. And i can't take not seeing him. Monday night we were at the same part but he didn't come up to me because I was with my friends and I heard a rumor bout something. I want to tell him "I'm done" but I don't wanna make the wrong choice. I lost my virginity to him so it's hard. And he was dating my friend when she passed away/: I'm just scared.