Specification gaming: the flip side of AI ingenuity

Source: Data-Efficient Deep Reinforcement Learning for Dexterous Manipulation (Popov et al, 2017)
Source: Faulty Reward Functions in the Wild (Amodei & Clark, 2016)
Source: Deep Reinforcement Learning From Human Preferences (Christiano et al, 2017)
Source: AI Learns to Walk (Code Bullet, 2019)
  • How do we faithfully capture the human concept of a given task in a reward function?
  • How do we avoid making mistakes in our implicit assumptions about the domain, or design agents that correct mistaken assumptions instead of gaming them?
  • How do we avoid reward tampering?
Sources: Montezuma, Hero, Private Eye — Reward learning from human preferences and demonstrations in Atari (Ibarz et al, 2018). Gripper — Learning a high diversity of object manipulations through an evolutionary-based babbling (Ecarlat et al, 2015). Qbert — Back to Basics: Benchmarking Canonical Evolution Strategies for Playing Atari (Chrabaszcz et al, 2018). Pong, Robot hand — Deep Reinforcement Learning From Human Preferences (Christiano et al, 2017). Ceiling — Genetic Algorithm Physics Exploiting (Higueras, 2015). Pole-vaulting — Towards efficient evolutionary design of autonomous robots (Krcah, 2008). Self-driving car — tweet by Mat Kelcey (Udacity, 2017). Montezuma — Go-Explore: a New Approach for Hard-Exploration Problems (Ecoffet et al, 2019). Somersaulting — Evolved Virtual Creatures (Sims, 1994).

--

--

--

We research and build safe AI systems that learn how to solve problems and advance scientific discovery for all. Explore our work: deepmind.com

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

How does an AI velociraptor know what anything is?

What Do Jobs of the Future Look Like?

My First Step Into Machine Learning

A Peek into the Black Box

Odin Health — Providing Clarity During COVID-19 to Hospitals at No Charge.

Which Speech Recognition API to choose for your project?

AI in Dating App: The Unexpected Love Affair between the Two

Wear a Mask, Wash Your Hands, Protect Your Data:

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
DeepMind Safety Research

DeepMind Safety Research

We research and build safe AI systems that learn how to solve problems and advance scientific discovery for all. Explore our work: deepmind.com

More from Medium

OpenAI’s GPT-3 Inspired Model can Solve Problems from the Math Olympiads

Generative modeling

What is AI ( RL ) & Benign AI | AI — Friend / Foe ? | ft Moonfall 2022

Experiences with Marie, the Replika AI agent — the beginning