Many post apocalyptic books contain some sort of "Goon Squad" that seeks to destroy the hero and leave the public in disarray. Some have the goons be trained, others have them as inept thugs. Either way you slice this poop sandwich, it's going to stink. Is this a realistic scenario? From reading blow by blow details from war torn and folded countries like Liberia, Congo and Venezuela, we can begin to see what happens in a power vacuum or war torn area. Factions will exist, they will become territorial and only lash out when cornered or forced to by dwindling supplies.
The trained and motivated aren't going to be your first priority. It's the desperate and hungry that will be thrust upon your property looking for supplies, if things devolve to that level. Your best bet it to stay anonymous. Keep yourself as innocuous as possible. Grey man all the time, without drawing any attention to yourself or your actions.
Buying bulk items at non commercial stores will get you noticed. Same with getting a pallet of food delivered to your home. Your neighbors will notice. If you don't have a basement, build shelves near the ceiling of your garage to house your stores so that people driving by can't see 20 food buckets piled about.
If you have disregarded all thoughts of remaining anonymous and have decided to advertise your abilities and discontent, you will need to have a method of early warning and detection for your adversaries. I'm not going to go into the specifics, but you will need a few minutes notice that the boogey man is coming.
A trained and determined enemy will do surveillance on you and yours in order to find your vulnerabilities. If they find that you have minimal security or personnel, they may opt to just execute you from long range to keep their exposure to danger low. Counter surveillance will be key in that situation. That art takes people and time, so make sure you have trained trackers to interdict outside assets. Drone technology is another issue to deal with. Modern drones can house sophisticated optics and infrared technology to track your outpost.
The moral of the story is that it's easier to remain hidden than it is to protect a known target. Due diligence staying hidden will be much easier than expecting a fight.