Predicting future motions of road participants is an important task for driving autonomously in urban scenes. Existing models excel at predicting marginal trajectories for single agents, yet it remains an open question to jointly predict scene compliant trajectories over multiple agents. The challenge is due to exponentially increasing prediction space as a function of the number of agents. In this work, we exploit the underlying relations between interacting agents and decouple the joint prediction problem into marginal prediction problems. Our proposed approach M2I first classifies interacting agents as pairs of influencers and reactors, and then leverages a marginal prediction model and a conditional prediction model to predict trajectories for the influencers and reactors, respectively. The predictions from interacting agents are combined and selected according to their joint likelihoods. Experiments show that our simple but effective approach achieves state-of-the-art performance on the Waymo Open Motion Dataset interactive prediction benchmark.

Marginal to Conditional

M2I includes three models that share the same context encoder. The relation predictor includes a relation prediction head to predict distribution over relation types. The marginal predictor adopts a trajectory prediction head to produce multi-modal prediction samples. The conditional trajectory predictor takes an augmented scene context input, including the future trajectory of influencers, to produce conditional predictions.

Relationship Prediction Demos

Here are some of our relationship prediction results against some very challenging scenarios from the Waymo Open Motion Dataset. Our relation predictor can successfuly predict relations involving interactive vehicles marked in blue. Detected relationships were marked as red arrows from the influencers to the reactors.

Merging into high speed traffic

The relationship predictor chooses the best influencer to follow while merging.

Merging into crowded traffic

The relationship predictor correctly sorts the crowd before the green light at an intersection.

Multiple lane changings

The relationship predictor selects reasonable influencers for both blue agents against complex scenarios.

Yielding to a jaywalker

The relationship predictor learns to yield to pedestrians when nessesary.

Yielding to other vehicles

The relationship predictor learns the right of the way.

Yielding with detection failure

The relationship predictor can make correct predictions with detection failures on the traffic lights.

Conditional Trajectory Prediction Demos

Here are some of our conditional trajectory prediction results for the reactors from the Waymo Open Motion Dataset. Our conditional trajectory predictor outperforms the marginal trajectory predictor on solving conflicts under complex multi-agent interactions.

Yielding to pedestrians

The conditional predictor predicts a yielding trajectory after several pedestrians in a busy intersection.

Yielding to a cyclist

The conditional predictor predicts a yielding trajectory after a cyclist in a busy intersection.

Yielding to a jaywalker

The conditional predictor predicts a yielding trajectory after a jaywalker. A clear slow down and speed up can be spotted before and after the interaction.

Turning into the main road

The conditional trajectory predictor predicts the turning vehicle to wait until it is clear of traffic.

Right turns yield to continous left turns

The conditional trajectory predictor predicts a turning trajectory until previous traffic is clear.

Multiple turnings

The conditional predictor follows the predicted relations and generates consistant trajectories.

Cutting in before an intersection

The conditional predictor generates shorter and slower trajectory for the reactor than the influencer.

Following another vehicle

The conditional predictor generates trajectories for the vehicle to follow the leading vehicle when turning right.


If you find our work intriguing, inspiring or useful to your research, please consider citing:

title={{M2I}: From Factored Marginal Trajectory Prediction to Interactive Prediction},
author={Sun, Qiao and Huang, Xin and Gu, Junru and Williams, Brian and Zhao, Hang},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},