Policy Filtering for Planning in Partially Observable Stochastic Domains y

y UAI'95 submission.
Abstract
Partially observable Markov decision processes (POMDP) can be used as a model for planning in stochastic domains. This paper considers the problem of computing optimal policies for nite horizon POMDPs. In deciding on an action to take, an agent is not only concerned with how the action would a ect the current time point, but also its impacts on the rest of the planning horizon. In a POMDP, the future e ects of an action are not separable from the e ects of the agent's future behavior. Consequently, one needs to consider the agent's future behavior in order to properly evaluate the future impacts of an action. One reason that makes POMDPs di cult to solve is that the agent can behave in an exponential number (in the length of the remaining of the planning horizon) of ways. This paper represents the agent's future behavior in terms of sub-policy-trees and gives a method that reduces the number of possible sub-policy-trees by collapsing similar sub-policy-trees and by pruning inferior sub-policy-trees.
Policy Filtering for Planning in Partially Observable Stochastic Domainsy
Nevin Lianwen Zhang
Technical Report HKUST-CS95-05 February 21, 1995
Department of Computer Science The Hong Kong University of Science and Technology Clear Water Bay, Kowloon, Hong Kong
past information increases exponentially with time. This is usually referred to as the \curse of dimensionality". Zhang and Boerlage (1995) have proposed a method for reducing the number of possible information states by pruning inconsistent information states and by collapsing similar consistent information states. Second, in deciding on an action to take the agent is not only concerned with how the action would a ect the current time point, but also its impacts on the rest of the planning horizon. In a fully observable MDP, the e ects of an action is fully observed at the next time point. Hence one is able to separate the e ects of an action performed at one time point from the e ects of actions performed at other time points. In a POMDP, on the other hand, the e ects of an action is not fully observed at the next time point, and one is unable to separate the e ects of the current action from the e ects of the agent's future behavior. Hence one needs to consider the agent's future behavior in order to properly evaluate the future impacts of the current action. The problem is that the agent can behave in an exponential number (in the length of the remaining of the planning horizon) of ways. This paper represents the agent's future behavior in terms of sub-policy-trees and describes a method that reduces the number of possible sub-policy-trees by collapsing similar sub-policy-trees and by pruning inferior sub-policy-trees. In a sense, this paper is a dual to Zhang and Boerlage (1995). The organization of the rest of the paper is as follows. Section 2 reviews POMDPs and introduces the concepts of policy trees sub-policy-trees. Section 3 shows how to inductively prune pointwise inferior sub-policy-trees and collapse similar sub-policy-trees. A bound on the sacri ce of optimality is given in Section 4. Section 5 de nes inferior sub2
s1
s2
...
st-1
st
st+1
...
sN
o1
o2
. . . . . . ... . .
ot-1
1 Introduction
There is a growing interest in using Markov decision processes (MDP) as a model for planning in stochastic domains (Dean and Wellman 1991, Provan and Clarke 1993, Dean et al 1993, Cassandra et al 1994). In this model, there is a state variable that represents the state of a world. The world evolves stochastically over time. At each time point, the agent obtains some observations about the world and takes an action. The agent receives a reward (or penalty) at each time point depending whether the planning goal is achieved and on the costs of actions. A policy speci es, for each time point, the appropriate action to take in response to each possible contingency. An optimal policy is one that maximizes the expected total reward, i.e. one that leads the agent to achieve the goal with minimum cost. The world can be either fully observable or only partially observable by the agent. The fully observable case has been studied extensively in the dynamic programming e (e.g. Bertsekas 1987, White 1993). Dean et al (1993) have proposed a search algorithm to deal with the applications where the world can be in a large number of states. This paper is concerned with the partially observable case. This case is considerably more di cult than the fully observable case for two related reasons. First, when the agent knows exactly which state the world is currently in, information from the past | past observations and actions | is irrelevant to the current decision. This is the so-called Markov property. On the other hand, when the agent does not have full observation of the world, past information can be used to better estimate the true current state of the world, and hence should be taken into account. The problem is that the number of possible states of 1