CS422: Interactive and Embodied Learning

Winter 2021 (Note: Winter 2022 content coming soon!)

Instructors: Nick Haber & Fei-Fei Li

Meeting times: Tuesday, Thursday 2:30 - 3:50 PM (Zoom details provided on Canvas.)

Image of AI agents learning interactively

Course Description

Most successful machine learning algorithms of today use either carefully curated, human-labeled datasets, or large amounts of experience aimed at achieving well-defined goals within specific environments. In contrast, people learn through their agency: they interact with their environments, exploring and building complex mental models of their world so as to be able to flexibly adapt to a wide variety of tasks. One crucial next direction in artificial intelligence is to create artificial agents that learn in this flexible and robust way. Class will be a seminar, with discussion. Students will read and take turns presenting current works, and they will produce a proposal of a feasible next research direction. Topics build upon reinforcement learning techniques and include intrinsic motivation, model-based RL, hierarchical RL, physical simulation, self-supervised representation learning, multi-agent RL, grounded language learning, emergent communication, and optimal experiment design, with an eye towards learning complementary topics we see as fertile ground for new research.

The primary goal of this course is to provide a survey of techniques related to agents learning rich models of their world through interaction. Readings and discussions have been planned with an eye towards germinating new work, and the project component should serve as a way to get started. The course is suitable for students looking to dive into recent works (from e.g. NeurIPS, ICML, and ICLR). Students should walk away from the course understanding not only relevant state-of-the-art approaches, but also its limitations, gaps in our understanding, benchmarks waiting to be created, and useful methods waiting to be fruitfully combined. This is an exciting, rapidly-growing field with many opportunities to quickly get started!

Prerequisites

Willing to read current AI literature (from e.g. NeurIPS, ICML, ICLR)
If not experienced in this, at least at a point where you're willing to learn this skill.
Proficiency in Python, and ideally Tensorflow or PyTorch
No assignments require this, but you will get a great deal more out of the course (and potentially make more compelling presentations and project) if you can e.g. look over a paper's github repo and play.
Knowledge in linear algebra, calculus, and statistics (MATH 19, 41, or 51, CS 109, or equivalent)
Again, no assignments require this, but you will get a great deal more out of readings if you are comfortable with these concepts.
CS 229, 231N, 234, or equivalent
Deep reinforcement learning will play a key role.

Class time

Winter quarter, 2021
Tu, Th 2:30 - 3:50

Office hours

By appointment
nhaber@stanford.edu

Forum, Zoom link, project submissions

Visit our Canvas page.

Readings

Detailed readings & syllabus page.
Required readings in bold.
Contents subject to change.

Grading

Project 40%
Presentations 30%
Discussion participation 30%

Resources

Hints/expectations for giving presentations.
Project timeline.
Have a look at some helpful resources
for relevant blog posts,
environments, and codebases.
This will grow throughout the quarter
as we find new ones!

Student responsibilities

Students will read required papers for each class, and students/groups of students (depending on enrollment) will take turns presenting. Note: each class has a required paper as well as a number of supplementary papers. The supplementary lists can get quite long -- this is not meant to be imposing! We simply think that each represents a handy constellation of papers for those interested in getting more acquainted with each subarea. We will try to provide a roadmap as we go, and feel free to ask for more context!

Students will produce a 6 page research proposal exploring a feasible next direction in depth, or some working demonstration of an idea. These will be “peer reviewed” by fellow students. For details, see our project timeline.

Topics at-a-glance

For full details including required papers and supplemental sources, please see readings.
Topics (particularly the later, more special-topic-oriented classes) subject to modification.

Framing and basic computational considerations

Class 1: conceptual framing, reading and project organization
Class 2: Early attempts: world models, intrinsic motivations, behavioral & performance metrics

Learning about embodiment and environment through interaction

Exploring intrinsic motivation signals

Class 3: First deep RL self-supervised intrinsic motivation methods, failure modes of intrinsic motivation methods (e.g. “white noise problem”), evaluating benchmarks
Class 4: Successive deep RL intrinsic motivation methods, MuJoCo & real robotics benchmarks, continued analysis of failure modes

World model learning and model-based RL

Class 5: Learning world models, planning with world models, and pitfalls of model-based approaches
Class 6: Intrinsic motivation, exploration, and model-based RL, DeepMind Control Suite benchmark

Goal-based approaches and active inference

Class 7: Goal-based intrinsic motivation and hierarchical RL, procedurally-generated maze environments
Class 8: Active inference

Learning about other agents through interaction

Class 9: Self-supervised prediction of other agents, theory of mind, and multi-agent RL with cooperation and competition
Class 10: Intrinsic motivation and multi-agent reinforcement learning

Complementary methods

Class 11: Agent57, catastrophic forgetting, hindsight experience replay, environment design
Class 12: Compositionality and planning, nonstationarity

Learning about objects and physics through interaction

Class 13: Self-supervised models of real-world objects and physics, challenges in pixel space, towards end-to-end training
Class 14: Learning self-supervised models of real-world objects and physics through interaction, benchmarking interactive learning

Learning sensory representations through interaction

Class 15: First approaches to self-supervised visual representation learning, how to benchmark visual representations, “leave info out” approaches.
Class 16: State-of-the-art self-supervised visual representation learning, first attempts at learning visual representations through interaction

Learning language through interaction

Class 17: Grounded language learning
Class 18: Language as a cognitive tool, emergent communication

Higher-level inquiry through interaction

Class 19: Optimal experimental design, active learning, and their relation to intrinsically-motivated RL
Class 20: Human curiosity and internet search