I just can't make myself do it...?

You are building this up in your mind to be a huge deal, when it's not. First of all, not going is just telling everyone (including the ex) that you are insecure. Second, if you pick an event that is pretty big, like a birthday party, there will be so many people there that you'll just be one of a group. Seriously, you're nervousness is making this a whole lot worse then it will be once you actually do it.