Here's a WAG.
Minimum 3. 1/guards when 1/sleeps and 1/hunts or gathers food du jour.
Allow me to opine...,
I think it depends on land. Location, location, location.
Easy access to a river. Water means fish and farming are possible along the river.
Forested area near by for building materials.
Motte and Bailey style fort overlooking the area at large. It may look like a civil war breastwork at first. The more people that come seeking food/security the bigger it can be.
You put up a big sign "you don't work you don't eat" Riff Raff filter.
As food/safety is increased, so will the population. If you build it they will come.
Sit back and relax in your fiefdom while enjoying the last diet Coke.
I don't see any correct answer here.