Would this annoy you? (vent)

Ok. So here is my story.

When FH and I decided to move in together he was living with his brother (FBIL). This was back in June. They were locked in a lease through this March. FH told FBIL way in advance that we were going to move in together, and asked FBIL what he wanted to do: FBIL could keep the apartment and find someone to move in, and FH would move out, or FH could keep the apartment and FBIL could move out. At first FBIL decided to keep the apartment. After 2 months of delaying us he changed his mind and decided to move out, and move home to LI to FILs to save money. So I moved in. Then, for approx. 4 months FBIL was working in the city so he would stay over in our tinry guest room/my office 2-3 times per week, during which time I had to let FH know in advance my plans every morning so that FBIL wouldnt be using the bathroom for his routine half hour when I needed to get ready for work. Then FBIL went traveling for a bit. Now he is back, but not working in Manhattan. Already this week he ahs stayed over twice at the last minute, and wants to stay over again. We have a TINY apartment with 1 bathroom, and the guest room is where I do my work (I'm a grad student). Now don't get me wrong I like FBIL, although he is a bit flaky, but this is our first 6 months of living together and I feel like we have a 3rd roommate who doesnt walk the dog, leaves his shoes everywhere and doesnt pay rent.

Am I wrong to be a little annoyed by this? What should I do?

Sorry for the length!