Lord starting siege alone

[MnBL]

I'm enlisted on one of the lord's of stormland. My lord proceed to follow the marshal to siege a town. When the siege began only my lord's party is in the battlefield against the whole garrison.

I'm thinking this is caused by my lord begin at the center of the town on campaign map and makes it first to enter the siege scene. Sometimes after retreating from battle other parties will join but it seems only 1 or 2 other lord's party max. Is there a workaround for this? It is definitely a sure lose situation as the town has like 600+ garrison and battle advantage is -12 atleast