need help with who to invite please....(sorry, kinda long)

Im trying to get a general idea of how many people we want to invite since I am just starting to look at reception halls. Here is my dilemma: I go to grad school out of state, and I have a group of about 15-20 people that I hang out with. Now most of the time we hang out, its the entire group. There are two or three people that I hang out with outside of the group, and these are friends that I would definitely want at the wedding. However, Im afraid if I invite them and not the others, then everyone else would be pissed at me and it would ruin our whole group dynamic. And honestly, these are the only people I hang out with out here, so I don't want to stop being included in things. I was thinking of maybe just not inviting anyone from out here at all and doing a separate little party here after the wedding and not have to worry about any of this. But the couple of good friends are always asking about plans and are kind of excited to come to NY. So now I just don't know what to do. If I am going to have a lot more people than I originally planned (Im trying to keep it relatively small), then I would have to change the halls that I was planning on looking at, and it could add up to a lot more $$. What would you do?
Sorry if this is a lame question, I've been debating about it for a while now and figured you guys are always helpful in these situations.