I just went through this. Except that I had most people say they were coming, but then only 7 showed up - starting about 2 hours after it was supposed to start.
I had leftovers galore. I was worried that the ones that did show up would think I was lame. I was worried that no one had a good time.
Then I realized:
I love leftovers. Especially finger foods.
The ones that didn't show up were almost to a person sick - bad sick. They didn't come and get the rest of us sick.
No one thought I was lame. They had a good time.
Your party will be fine even if you get only 3 other people showing up. You'll have fun, everyone will have plenty to eat and your friends are not going to judge you for the others not showing.
Deep breaths and just HAVE FUN.