Dear Katie:
Understandably, you care what other people say about you, what they think about you and what they say. We all do. A good aim would be to care less, that is to not rush to accommodate what other people think you should do.
When your boyfriend criticizes you for having fewer friends than he does, it is unkind of him. The fault is not with you having fewer friends but with him, for criticizing you for it.
I think it is okay to have the number of friends that you are having. It is okay. No need to accommodate what other people think, how many friends other people think you should have. Your boyfriend (and anyone else) can have as many friends as he wants to have, and so can you.
If you agree with what I just wrote, let him know that, assert yourself with him.
anita