Happens in the base game and other mods too. The best I could figure out was that the further away the lord is from you the more likely he is to ignore your summons regardless of relation score, which seems to jibe with your issue. In that case you have to go find them and have them follow you.
