A useful case to start with in Virginia is Thomas v. Commonwealth, 57 Va.App. 267 (2010). In this case, the Virginia Court of Appeals was kind enough give us a list of all the things that an officer can do during a legitimate stop:
• to obtain the registration for the vehicle and request the identities of its occupants,
• to seek radio dispatch confirmation of the information obtained from the vehicle occupants,
• to detain Thomas, a passenger, during the duration of the stop,• to ask questions unrelated to the traffic violation,
• to order the driver and Thomas out of the vehicle,
• to walk a drug-sniffing dog around the vehicle,
• to seize Thomas's handgun the moment they saw it
Of course, this wasn't the first case in which the Virginian appellate courts had ruled passengers could be detained. As early as 1998 the Virginia Court of Appeals ruled that "police officers may also detain passengers beside an automobile until the completion of a lawful traffic stop." Harris v. Commonwealth, 27 Va. App. 554, 562 (1998); see also McCain v. Commonwealth, 275 Va. 546, 553 (2008).
Ultimately, however, the basis of the ability of the officer to detain passengers during the stop of a car comes from a US Supreme Court opinion, Brendlin v. California, 551 U.S. 249 (2007):
An officer who orders one particular car to pull over acts with an implicit claim of right based on fault of some sort, and a sensible person would not expect a police officer to allow people to come and go freely from the physical focal point of an investigation into faulty behavior or wrongdoing. If the likely wrongdoing is not the driving, the passenger will reasonably feel subject to suspicion owing to close association; but even when the wrongdoing is only bad driving, the passenger will expect to be subject to some scrutiny, and his attempt to leave the scene would be so obviously likely to prompt an objection from the officer that no passenger would feel free to leave in the first place.. . . . . .What we have said in [prior] opinions probably reflects a societal expectation of unquestioned police command at odds with any notion that a passenger would feel free to leave, or to terminate the personal encounter any other way, without advance permission.
Reading through the various cases, it seems fairly obvious that the officer's ability to detain passengers was long assumed, because at first the argument seemed to be whether the officer could make them leave the vehicle while it was legally stopped and hold them outside the vehicle. But, of course, nothing can exist only in common sense. Eventually, there have to be court opinions to tell us what we already know.