I'm not gonna waste to many words on this. Looks like you visited The Village at a bad time, or simply didn't listen to the Guards.
And yes, we don't have Guards online 24/7. So there always is a risk of bandits being inside.
If you're visiting The Village: Call out when you are coming up to the walls, don't have a gun out and all should be fine