CS422: Interactive and Embodied Learning

Syllabus and readings

Required readings in bold; others supplementary. As an alternative to covering the one or two assigned papers, presenters should feel free to present on the general topic domain using the primary and supplementary readings (or even suggest their own!). The supplementary lists can get quite long -- this is not meant to be imposing! We simply think that each represents a handy constellation of papers for those interested in getting more acquainted with each subarea. We will provide a roadmap as we go with overviews in class. Feel free to ask for more context at any time.

Please see these guidelines for giving a good talk. Every student is responsible for doing all required readings. Please come prepared to discuss, and come being able to describe background, methodology, experimental setups, and key results (for experimental papers). For non-experimental papers, be prepared to discuss the background and each key idea.

Framing

Class 1 (Jan 8): Conceptual framing, logistics.

Class 2 (Jan 10): An early relevant on curiosity and world model building (instructor presents to model presentations).

Intrinsic Motivation Systems for Autonomous Mental Development
Supplementary:
A Developmental Approach to Machine Learning?
From Needs to Goals and Representations: Foundations for a Unified Theory of Motivation, Personality, and Development
Core Knowledge

Class 3 (Jan 17): Introduction: open-ended learning, autotelic agents, and curiosity (Instructor lecture). Projects discussion.

Learning to Play with Intrinsically-Motivated Self-Aware Agents
Active World Model Learning with Progress Curiosity
Supplementary:
Developmental Curiosity and Social Interaction in Virtual Agents

Class 4 (Jan 22): Project pitches.

Open-ended learning, adaptation, concept learning

Class 5 (Jan 24) Adaptive agents (Student presentation 1)

Human-Timescale Adaptation in an Open-Ended Task Space
First-Explore, then Exploit: Meta-Learning Intelligent Exploration
Supplementary:
Bigger, Better, Faster: Human-level Atari with human-level efficiency
ALAN : Autonomously Exploring Robotic Agents in the Real World
Internet Explorer: Targeted Representation Learning on the Open Web
AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agents

Class 6 (Jan 29): Concept discovery and AI for science (Student presentation 2)

Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZero
Supplementary:
AI-driven Automated Discovery Tools Reveal Diverse Behavioral Competencies of Biological Networks

Class 7 (Jan 31): The future of open-ended learning (Student presentation 3)

General intelligence requires rethinking exploration
Supplementary:
Autotelic Agents with Intrinsically Motivated Goal-Conditioned Reinforcement Learning: a Short Survey

Integrating language -- background

Class 8 (Feb 5): Thinking step-by-step (Student presentation 4)

Graph of Thoughts: Solving Elaborate Problems with Large Language Models
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Class 9 (Feb 7) Can LLMs reason, though? (Student presentation 5)

Can Large Language Models Really Improve by Self-critiquing Their Own Plans?
GPT-4 Doesn't Know It's Wrong: An Analysis of Iterative Prompting for Reasoning Problems
Supplementary:
Large Language Models Cannot Self-Correct Reasoning Yet

Class 10 (Feb 12) Code for agency? (Instructor lecture)

Parsel: Algorithmic Reasoning with Language Models by Composing Decompositions
Hypothesis Search: Inductive Reasoning with Language Models

Class 11 (Feb 14) Project midpoint feedback session.

Integrating language -- first attempts

Class 12 (Feb 21) Language-based agents, 1 (Student presentation 6)

RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Generative Agents: Interactive Simulacra of Human Behavior
Supplementary:
CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization
Lyfe Agents: Generative agents for low-cost real-time social interactions
Learning to Model the World with Language

Class 13 (Feb 26) Language-based agents, 2 (Student presentation 7)

Voyager: An Open-Ended Embodied Agent with Large Language Models
Augmenting Autotelic Agents with Large Language Models
Supplementary:
Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents
Thinker: Learning to Plan and Act

Class 14 (Feb 28) Language-based agents, 3 (Student presentation 8)

Reflexion: Language Agents with Verbal Reinforcement Learning
STaR: Bootstrapping Reasoning With Reasoning
Supplementary:
Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation

Class 15 (Mar 4) New benchmarks (Student presentation 9)

Benchmarking Large Language Models As AI Research Agents
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Supplementary:
RoboHive: A Unified Framework for Robot Learning
Gigastep - One Billion Steps per Second Multi-agent Reinforcement Learning
AgentBench: Evaluating LLMs as Agents