Ring bearer/flower girl

So I'm being plagued with an issue...

We have a ring bearer already in mind (he'll be 3 at the time of our wedding). Do we really have to have a flower girl? I feel like it might be too much between our families.
Here's the reason why I'm asking, we don't plan to have children at our wedding -ceremony and reception. The only issue then is that the ring bearer will be there, his cousins (parents are brother and sister), have two twin girls who will be 2 years old by the wedding. If the ring bearer is there, are we obligated to invite the twins? Then in that case, I'll need to invite the children on my side. What the situation in this case (formalities, etc.)? I don't want to aggravate any side of the family.
Is it just something I'm thinking too much about?
Should I just be like, 'we'll he's in the bridal party and just the exception,' and have others deal with it?

There's a lot of questions jumbled up in here! My apologies but it's something I just have no idea about.