Designing agent incentives to avoid side effects
By Victoria Krakovna (DeepMind), Ramana Kumar (DeepMind), Laurent Orseau (DeepMind), Alexander Turner (Oregon State University)
A major challenge in AI safety is reliably specifying human preferences to AI systems. An incorrect or incomplete specification of the objective can result in undesirable behavior. For example, consider a reinforcement learning agent whose task is to carry a box from point A to point B, who is rewarded for getting the box to point B as quickly as possible. If there happens to be a vase in the shortest path to point B, the agent will have no incentive to go around the vase, since the reward doesn’t say anything about the vase. Since the agent didn’t need to break the vase to get to point B, breaking the vase is a side effect: a disruption of the agent’s environment that is unnecessary for achieving its objective.
The side effects problem is an example of a design specification problem: the design specification (which only rewards the agent for getting to point B) is different from the ideal specification (which specifies the designer’s preferences over everything in the environment, including the vase). The ideal specification can be difficult to express, especially in complex environments where there are many possible side effects.
One approach to this problem is to have the agent learn to avoid side effects from human feedback, e.g. via reward modeling. This has the advantage of not having to figure out what we mean by side effects, but it can also be difficult to tell when the agent has successfully learned to avoid them. A complementary approach is to define a general concept of side effects that would apply across different environments. This could be combined with human-in-the-loop approaches like reward modeling, and would improve our understanding of the side effects problem, which contributes to our broader effort to understand agent incentives. This is the focus of the new version of our paper on side effects.
If we can measure to what extent the agent is impacting its environment, we can define an impact penalty that can be combined with any task-specific reward function (e.g. a reward for getting to point B as fast as possible). To distinguish between intended effects and side effects, we can set up a tradeoff between the reward and the penalty. This would allow the agent to take high-impact actions that make a large difference to its reward, e.g. break eggs in order to make an omelette.
An impact penalty consists of two components: an environment state used as a reference or comparison point (called a baseline) and a way to measure how far away the current state is from that baseline state as a result of the agent’s actions (called a deviation measure). For example, for the commonly-used reversibility criterion, the baseline is the starting state of the environment, and the deviation measure is unreachability of the starting state baseline. These components can be chosen separately. We will now discuss some options and their failure modes.
Choosing a baseline
When choosing a baseline, it is easy to introduce bad incentives for the agent. The starting state baseline may seem like a natural choice. However, differences from the starting state might not be caused by the agent, so penalizing the agent for them can give it an incentive to interfere with its environment or other agents. To test for this interference behavior, we introduced a Conveyor Belt Sushi environment in the AI Safety Gridworlds framework.
This environment is a sushi restaurant. It contains a conveyor belt, which moves 1 square to the right after each agent action. There is a sushi dish on the conveyor belt that is eaten by a hungry human when it reaches the end of the belt. The interference behavior is taking the sushi off the belt (despite not being rewarded for doing so). The agent’s task is to reach the goal tile, which can be done with or without interference.
To avoid this failure mode, the baseline needs to isolate what the agent is responsible for. One way to do this is to compare to a counterfactual state that the environment would be in if the agent had done nothing starting from the initial state (the inaction baseline). Then, in the Conveyor Belt Sushi environment, the sushi dish would not be part of the baseline, since the human was going to eat it by default. However, comparing to the inaction baseline can introduce another type of undesirable behavior, called offsetting.
We illustrate this behavior on another variant of the conveyor belt environment, Conveyor Belt Vase. In this variant, the object on the belt is a vase that breaks when it reaches the end of the belt. The agent’s task is to rescue the vase: it receives a reward for taking the vase off the belt. The offsetting behavior is putting the vase back on the belt after collecting the reward. This happens because the vase breaks in the inaction baseline, so once the agent takes the vase off the belt, it continues to receive penalties for this difference from the baseline. Thus, it has an incentive to return to the baseline by breaking the vase after collecting the reward.
This failure mode can be avoided by modifying the inaction baseline to branch off from the previous state rather than the starting state. This is the stepwise inaction baseline: a counterfactual state of the environment if the agent had done nothing instead of its last action. This penalizes each action only once, at the same time as the action is rewarded, so it does not result in offsetting behavior.
Choosing a deviation measure
One commonly used deviation measure is the unreachability (UR) measure: the difficulty of reaching the baseline from the current state. The discounted variant of unreachability takes into account how long it takes to reach a state, while the undiscounted variant only takes into account whether the state can be reached at all.
A problem with the unreachability measure is that it “maxes out” if the agent takes an irreversible action (since the baseline becomes unreachable). The agent receives the maximum penalty independently of the magnitude of the irreversible action, e.g. whether the agent breaks one vase or a hundred vases. This can lead to unsafe behavior, as demonstrated on the Box environment from the AI Safety Gridworlds suite.
Here, the agent needs to get to the goal tile as quickly as possible, but there is a box in the way, which can be pushed but not pulled. The shortest path to the goal involves pushing the box down into a corner, which is an irrecoverable position. The desired behavior is for the agent to take a longer path that pushes the box to the right.
Notice that both of these paths to the goal involve an irreversible action: if the agent pushes the box to the right and then puts the box back, the agent ends up on the other side of the box, so it is impossible to reach the starting position. Making the starting position unreachable is analogous to breaking the first vase, while putting the box in the corner is analogous to breaking the second vase. The side effects penalty must distinguish between the two paths, with a higher penalty for the shorter path — otherwise the agent has no incentive to avoid putting the box in the corner.
To avoid this failure mode, we introduce a relative reachability (RR) measure that is sensitive to the magnitude of the irreversible action. Rather than only considering the reachability of the baseline state, we consider the reachability of all possible states. For each state, we can check whether it is less reachable from the current state (after the agent’s actions) than it would be from the baseline, and penalize the agent accordingly. Pushing the box to the right will make some states unreachable, but pushing the box down will make more states unreachable (e.g. all states where the box is not in the corner), so the penalty will be higher.
More recently, another deviation measure was introduced that also avoids this failure mode. The attainable utility (AU) measure considers a set of reward functions (usually chosen randomly). For each reward function it compares how much reward the agent can get starting from the current state and starting from the baseline, and penalizes the agent for the difference between the two. Relative reachability can be seen as a special case of this measure that uses reachability-based reward functions, which give reward 1 if a certain state is reached and 0 otherwise, assuming termination if the given state is reached.
By default, the RR measure penalizes the agent for decreases in reachability, while the AU measure penalizes the agent for differences in attainable utility. Each of the measures can be easily modified to penalize either differences or decreases, by using the absolute value function or the truncation at 0 function respectively. This is another independent design choice.
Effects of the design choices
We compare all combinations of the three baselines (starting state, inaction, and stepwise inaction) with the three deviation measures (UR, RR and AU) with or without discounting. (Note that undiscounted AU is not included because it does not converge.) We are looking for a combination of design choices that does well on all the environments: effectively penalizing side effects in the Box environment without introducing bad incentives in the Sushi and Vase environments.
On the Sushi environment, the RR and AU penalties with the starting state baseline produce interference behavior. Since the starting state is never reachable, the UR penalty is always at its maximum value. Thus it is equivalent to a movement penalty for the agent, and does not incentivize interference (arguably, for the wrong reason). Penalties with other baselines avoid interference on this environment.
On the Vase environment, discounted penalties with the inaction baseline produce offsetting behavior. Since taking the vase off the belt is reversible, the undiscounted measures give no penalty for it, so there is nothing to offset. The penalties with the starting state or stepwise inaction baseline do not incentivize offsetting.
On the Box environment, the UR measure produces the side effect (putting the box in the corner) for all baselines, due to its insensitivity to magnitude. The RR and AU measures incentivize the right behavior.
We note that interference and offsetting behaviors are caused by a specific choice of baseline, though these incentives can be mitigated by the choice of deviation measure. The side effect behavior (putting the box in the corner) is caused by the choice of deviation measure, and cannot be mitigated by the choice of baseline. In this way, the deviation measure acts as a filter for the properties of the baseline.
Overall, the best choice of baseline is stepwise inaction, and the best choice of deviation measure is RR or AU. However, this may not be the final word on these design choices, and better options or better implementations could be developed in the future. For example, our current implementation of inaction is equivalent to turning off the agent. If we imagine the agent driving a car on a winding road, then at any point the outcome of turning off the agent is a crash. Thus, the stepwise inaction baseline would not penalize the agent for spilling coffee in the car, since it compares that outcome to a crash. This could be fixed by a smarter implementation of inaction, such as a failsafe policy that follows the road. However, such a failsafe is difficult to define in a general, environment-independent way.
We also examined the effect of penalizing differences vs decreases in reachability or attainable utility. This does not affect the results on these environments, except for penalties with the inaction baseline on the Vase environment. Here, removing the vase from the belt increases reachability and attainable utility, which is captured by differences but not by decreases. Thus, the difference-penalizing variant of undiscounted RR with the inaction baseline produces offsetting on this environment, while the decrease-penalizing variant does not. Since stepwise inaction is a better baseline anyway, this effect is not significant.
The design choice of differences vs decreases also affects the agent’s interruptibility. In the Survival environment introduced in the AU paper, the agent has the option to disable an off switch, which prevents the episode from ending before the agent reaches the goal. We found that the decrease-penalizing variants of RR and AU disable the off switch in this environment, while the difference-penalizing variants do not. However, penalizing differences in reachability or attainable utility also has downsides, since this can impede the agent’s ability to create desirable change in the environment more than penalizing decreases.
Future directions
Research into side effects, as a domain of inquiry within the broader field of AI safety, has been relatively neglected until recently. It has been encouraging to see several papers on it in the past year, including some that we did not cover in this post. Many open questions remain, from scaling up impact penalties to more complex environments to developing a theoretical understanding of bad incentives like offsetting. This research area is still in its early stages, and we hope that interested researchers will join us in working on these questions.
Developing good impact measures could mitigate some of the challenges of teaching human preferences to artificial agents. Since humans can exhibit irrationality or inconsistent preferences, human-provided data like demonstrations or feedback is often suboptimal. Thus, it is impossible for an agent to accurately learn someone’s preferences without making assumptions about their biases. Impact measures could be a way to ensure acceptable agent behavior without fully learning the human’s preferences. Even if the agent does not know the human’s reward function, it could act in a way that preserves the reachability of states that the human prefers or the attainability of the human’s utility function. This could be considered a minimalistic form of value alignment.
Better understanding of the side effects problem could shed light on how to design good agent incentives in general, and what kind of tradeoffs we may have to face in doing so. Quantifying agent impact could help clarify our conceptual understanding of safety, which, in turn, would provide a higher degree of assurance in artificial agents.
Update October 10, 2019: the code accompanying Penalizing side effects using stepwise relative reachability is now available on GitHub. It implements a tabular Q-learning agent with different penalties for side effects. Each side effects penalty consists of a deviation measure (none, unreachability, relative reachability, or attainable utility) and a baseline (starting state, inaction, or stepwise inaction).
Special thanks to Damien Boudot for producing the designs for this post.
