I have attended almost two dozen weddings. The best weddings were the smaller weddings because it was so much more intimate and personal. Huge weddings are not all that much fun because I feel like we are all like a herd of cattle; filing in to be seated, fed and let loose.
One hundred people is not a small wedding. It is a perfect size, in my opinion. This way you can actually greet your guests and spend time with them if just to say hello. The larger weddings I have attended, I did not even get to talk to the bride or groom other than giving them a hug as we left the church.
I think you have an odd way of thinking and I do not quite understand why you think 100 people in attendance at your wedding is too small of a wedding. Please rethink your internal dialogue and change it to see that whether 95 or 115 people show up, it will still be a great wedding. You'll see.
On the other hand, if you keep brainwashing yourself into believing that 100 or so people are not enough guests, then you are just talking yourself into having not as good of a time, for no real good reason.
Even if 20 people decline, so what? You are so overthinking this one. Relax.