"Why did he come with you? He's not SHIELD-- he wasn't assigned to you."

It's actually kind of a stupid question, he realizes as he hears it said aloud. An ex-soldier or even a random civilian might have chosen to accompany Rogers for any number of reasons, up to and including self-preservation or sheer utility.