Or can you somehow kill them in their castles?
They need to be taken prisoner.
Oh, you should add an option where you can refuse to give the guard your weapons, then attack the guard and take the keys, then you would come in the lord's castle but his entire guard would fight you. That would be awesome.