The lazy vague Bethesda way of addressing this is to only feature a small contingent of Nord “volunteers”, maybe as corpses in a dungeon with some journal entries that could be read as them being sent by the Stormcloaks or just coming of their own accord.