Bandit ambush in towns

[Daddy]

Sometimes when you try going into the tavern or marketplace you're ambushed and a battle ensues, sometimes during this (since there can be up to 10 enemies) they spawn in some locations that you have to first find and go to. Sometimes it takes 10 minutes to find the last enemy and they just stand there like 400 meters at the other side of 8 buildings AND you can't skip the battle.
Maybe there is a way to make the enemies go to your location instead of just standing still, or making them spawn closer?