Towards Motion Forecasting with Real-World Perception Inputs: Are End-to-End Approaches Competitive?

Yihong Xu    Loïck Chambon    Éloi Zablocki    Mickaël Chen    Alexandre Alahi    Matthieu Cord    Patrick Pérez

ICRA 2024

Abstract

Motion forecasting is crucial in enabling autonomous vehicles to anticipate the future trajectories of surrounding agents. To do so, it requires solving mapping, detection, tracking, and then forecasting problems, in a multi-step pipeline. In this complex system, advances in conventional forecasting methods have been made using curated data, i.e., with the assumption of perfect maps, detection, and tracking. This paradigm, however, ignores any errors from upstream modules. Meanwhile, an emerging end-to-end paradigm, that tightly integrates the perception and forecasting architectures into joint training, promises to solve this issue. So far, however, the evaluation protocols between the two methods were incompatible and their comparison was not possible. In fact, and perhaps surprisingly, conventional forecasting methods are usually not trained nor tested in real-world pipelines (e.g., with upstream detection, tracking, and mapping modules). In this work, we aim to bring forecasting models closer to real-world deployment. First, we propose a unified evaluation pipeline for forecasting methods with real-world perception inputs, allowing us to compare the performance of conventional and end-to-end methods for the first time. Second, our in-depth study uncovers a substantial performance gap when transitioning from curated to perception-based data. In particular, we show that this gap (1) stems not only from differences in precision but also from the nature of imperfect inputs provided by perception modules, and that (2) is not trivially reduced by simply finetuning on perception outputs. Based on extensive experiments, we provide recommendations for critical areas that require improvement and guidance towards more robust motion forecasting in the real world. We release an evaluation library to benchmark models under standardized and practical conditions.

BibTeX

@inproceedings{xu2024realworldforecasting,
  title      = {Towards Motion Forecasting with Real-World Perception Inputs: Are End-to-End Approaches Competitive?},
  author     = {Yihong Xu and
                  Lo{\"{\i}}ck Chambon and
                  {\'{E}}loi Zablocki and
                  Micka{\"{e}}l Chen and
                  Alexandre Alahi and
                  Matthieu Cord and
                  Patrick P{\'{e}}rez},
  booktitle = {ICRA},
  year      = {2024}
}