Opinions Re: Invite Question (long) POLL

Near the end of our college career in CT, FI and I were introduced to 2 guys through a mutual friend. These guys became a part of our group that partied and hung out all the time. In fact, we partied very frequently at their place (they were roommates and close friends). In the beginning, we were closer to 'Gary' than to 'Jay,' but wouldn't have considered either of them really close friends. About a year after college, FI and I moved to San Diego. During the 3 years that we lived there, we didn't talk to either of them. Also during that time, 'Gary' got married, and we never received an invite, though we didn't really expect one either. Then we left San Diego and moved to NY. 'Jay' got married shortly after our move and invited us (to our surprise) to the wedding. We attended his wedding, as well as the weddings of 3 other mutual friends. The only time we saw or spoke to 'Jay' or 'Gary' was at these weddings.

Now it's time to finalize our guest list. Even though we aren't really close to either of them, we feel we need to invite 'Jay' since we were invited to his wedding. I also argue that we should invite Gary because I feel like he would be hurt if we invited 'Jay' and not him, and that he most likely would have invited us to the wedding if we were on the east coast at the time. FI doesn't think we should invite 'Gary.' He says, 'We weren't invited to his wedding, we never talk to him, and at $115 per plate ($230 including his wife), I don't want to invite him.'

What do you ladies and gentlemen think?