Towns captured from other factions demonstrate a degeneration in relationship (over that of the effect of the initial capture), without any logical or rational explanation.
Free Folk: Thenn, Frostfang's Camp (Captured from Free Folk - faction defeated)
It's because they are not the same religion as yourself.
Is it possible that the dynamics of the Religion functionality might not seem logical in respect to castles? Especially in the context of the associated village/s not being owned by the player (having been allocated to other faction lords) and therefore precluding the option of building a religious building to evangalise the residents of the village and the associated castle pr town? Alternatively, would it be possible for the sovereign to build any (relevant) building in any faction settlement to ensure prosperity, loyalty (to the builder), and religious ferver throughout the conquered realm?