An Open-Source Probabilistic Programming System for Data Generation and Safety in AI-Based Autonomy

CVPR 2024 Tutorial


Autonomous systems, such as self-driving cars or robotics in general, are increasingly operating in more complex environments, dynamically interacting with multiple entities (e.g. humans or robots). Yet, how can we formally model and generate such complex environments in simulations? The use cases may span synthetic training data generation and rigorous evaluation of an autonomous systems in particular environments, or scenarios. Scenic has been used in variety of domains such as self-driving, aviation, indoor robotics, reinforcement learning, and augmented/virtual reality. We provide a tutorial on Scenic which is a simulator-agnostic, domain-specific probabilistic programming language to model and generate distributions of scenarios. Scenic provides domain-specific supports to model and sample from distributions with spatial and temporal constraints of the real physical world. We will provide a tutorial on not only the syntax and the semantics of the language but also how to interface the language to your simulator of choice. For more information on Scenic, please visit our documentation.


Sanjit Seshia

University of California, Berkeley

Daniel Fremont

University of California, Santa Cruz

Edward Kim

University of California, Berkeley


Edward Kim

University of California, Berkeley

Jinkyu Kim

Korea University, South Korea

Necmiye Ozay

University of Michigan, Ann Arbor

Parasara Sridhar Duggirala

University of North Carolina, Chapel Hill

Hazem Torfah

Chalmers University of Technology, Sweden

Kimin Lee

School of AI, KAIST, South Korea

Marcell Vazquez-Chanlatte

Nissan's Alliance Innovation Lab, Silicon Valley


Today's autonomous systems rely heavily on the use of machine learning components trained on large amounts of data. Even so, it is expensive to collect relevant data and test these systems in the real world in a manner that captures typical data distributions and also covers edge cases. Therefore, simulators are widely adopted in the robotics and computer vision community to train, test, and debug autonomous and semi-autonomous systems. However, working directly with simulators can be too low-level and problem-specific. To support the design lifecycle of autonomous/semi-autonomous systems, one needs to raise the level of abstraction above individual simulators and provide a formal framework for world modeling. Such a world model can help reason about the safety of a system and facilitate data generation and sim-to-real validation, as well as help to interpret, validate, share, or re-use training and test scenarios across the community.

The objective of this tutorial is to introduce SCENIC, an open-source, domain-specific probabilistic programming language for world modeling that addresses the above needs. SCENIC is designed to model and generate interactive (or reactive), multi-agent scenarios in a manner portable to any simulator. In SCENIC, users can precisely model a stochastic environment in which an autonomous/semi-autonomous system operates, can perform a variety of design and analysis tasks, and can communicate them as interpretable programs. SCENIC has a variety of demonstrated use cases, including synthetic data generation, data augmentation, debugging and retraining and redesign of perception components, sim-to-real validation, testing safety of autonomous system both in simulation and in the real world, training reinforcement learning agents in multiplayer settings, and more. To achieve these goals, SCENIC has been designed to be (i) intuitive to learn, (ii) probabilistic to capture the uncertainty and stochasticity in the real world, (iii) simulator-agnostic, and (iv) open-source and in the public domain for external members to contribute.

Details & Schedule

Time: TBD

Time zone: PDT (Pacific Daylight Time)
Location: Seattle Convention Center, TBD
More details: CVPR tutorial page (TBD)

For details, please contact Edward Kim.
Website template by Oriane Siméoni from the Object localization for free CVPR '23 tutorial.
Last updated: 11th of March 2024