Yandex launches MADrive technology for virtual testing of self-driving cars

Yandex researchers have unveiled MADrive and the MAD-Cars dataset, tools designed to create photorealistic road scenes by replacing real cars in videos with 3D models and simulating dangerous scenarios – like sudden lane changes, overtakes, and hard braking – enabling safe training of autonomous driving algorithms.

• MADrive recreates road scenes and swaps out real vehicles with photorealistic 3D models that account for angle, depth, and lighting.
• The MAD-Cars dataset includes around 70,000 video clips featuring over 150 car brands – one of the largest open datasets of its kind.
• The technology can generate new traffic scenarios by altering trajectories and simulating critical maneuvers impossible to reproduce safely on real roads.

Yandex has made MADrive available along with the MAD-Cars dataset to the public, enabling researchers and engineers worldwide to safely test rare and hazardous driving situations to improve the safety and reliability of autonomous vehicles.

Replacing real cars with 3D models in video

MADrive reconstructs entire road scenes by swapping out real cars in videos with photorealistic 3D models. These models are carefully matched to the original vehicles in shape and color and are seamlessly integrated into the scene, taking into account perspective, depth, and lighting – making virtual drives look natural and ideal for training and testing autonomous driving systems.

Beyond replaying actual trips, MADrive allows the creation of entirely new traffic scenarios, such as altering car trajectories, modeling sudden lane changes, unexpected braking, overtakes, and other complex maneuvers.

The engine behind this is the open MAD-Cars dataset – an anonymized collection of videos showing vehicles of various makes, models, and colors, captured from multiple angles under real-world conditions. With about 70,000 clips covering over 150 car brands, it stands as one of the largest open datasets available for this purpose.

Solving the simulator data shortage

“Simulators are a crucial part of autonomous vehicle development: algorithms learn and are tested without ever hitting real roads. But simulators that reproduce real drives face a data shortage problem. For example, if a car turned left in the original footage, we can’t show how it would look if it overtook or made a sudden lane change. Our approach removes this limitation by letting us create new road scenes that appear just as natural as real videos. This opens the door to safely testing rare and critical scenarios that can’t be staged on real roads,” said Viktor Yurchenko, head of Yandex’s autonomous vehicle sensor simulation group.

In tests, MADrive has demonstrated higher quality results compared to existing scene reconstruction methods, especially when not only replicating footage but generating continuations or alternative scenarios.

The Yandex autonomous driving engineering team and researchers at Yandex Research published a detailed technical article on Habr outlining their approach and experiment results. This technology paves the way for next-gen simulators offering more photorealistic and varied scenario modeling. Plans are underway to expand the vehicle database and improve lighting algorithms to make simulations even more lifelike.