Partial Local FriendQ Multiagent Learning Application to Team Automobile Coordination Probl

Partial Local FriendQ Multiagent Learning:Application to Team Automobile Coordination Problem1Julien Laumonier and Brahim Chaib-draa21IntroductionIn real world cooperative multiagent problems,each agent has of-ten a partial view of the environment.If communication has a cost,the multiagent system designer has tofind a compromise between in-creasing the observability and total cost of the multiagent system.Tochoose a compromise,we propose,to take into account the degreeof observability,defined as the agent’s vision distance,for a coop-erative multiagent system by measuring the performance of the as-sociated learned policy.Obviously,decreasing observability reducesthe number of accessible states for agents and therefore decreases theperformance of the policy.We restrict our application to team game,a subclass of coordination problems where all agents have the sameutility function.We consider problems where agents’designer doesnot know the model of the world.Thus,we can use learning algo-rithms which have been proven to converge to Pareto-optimal equi-librium such as Friend Q-learning[4].One can take an optimal al-gorithm tofind the policy for the observable problem.The followingassumptions are defined:(1)Mutually exclusive observations,eachagent sees a partial view of the real state but all agents together seethe real state.(2)Possible communication between agents but notconsidered as an explicit part of the decision making.(3)Only nega-tive interactions between agents.One problem which meets these as-sumptions is the choosing lane decision problem related to IntelligentTransportation Systems which aims to reduce congestion,pollution,stress and increase safety of the traffic.2Formal Model and AlgorithmsReinforcement learning allows an agent to learn by interactingwith its environment.For a mono agent system,the basic formalmodel for reinforcement learning is a Markov decision -ing this model,Q-Learning algorithm calculates the optimal valuesof the expected reward for the agent in a state s if the action a is ex-ecuted.On the other hand,game theory studies formally the interac-tions of rational agents.In a one-stage game,each agent has to choosean action to maximize its own utility which depends on the others’actions.In game theory,the main solution concept is the Nash equi-librium which is the best response for all agents.A solution is Paretooptimal if there does not exist any other solution such that one agentcan improve its reward without decreasing the reward of another.Themodel which combines reinforcement learning and game theory,isthe number of joint actions which has to be tested during the learning.This partial local observability allow us to consider a variable number of agents in the multiagent system.Formally,we define a function g iwhich transforms the joint action a into a partial joint action g i d ( a ,s ).This partial joint action contains all actions of agent which are in the distance d of agent i .The PJA Q-value update function is :Q (f i d (s ),g i d ( a ,s ))=(1−α)Q (f i d (s ),g i d ( a ,s ))+α[r +γmax a d ∈G i d ( A,S )Q (f i d (s ′), a d )]5ResultsWe compare empirically the performance of the totally observ-able problem (FriendQ)and the performance of approximated policy (PJA)on a small problem defined by size X =3,Y =7with 3agents.Figure 1shows that for d =0...2,PJA converges to a local maximum,which increases with d .In these cases,the approximated values are respectively about 76%,86%and 97%from the optimal value given by Friend Q-Learning.When d =3,that is,when the lo-cal view is equivalent to the totally observable view,the average sum rewards converges to the total sum rewards of Friend Q-learning.0 10000 20000 30000 40000 50000S u m R e w a r d EpisodesFigure 1.Rewards for Partial Joint Action Q-learningThe generalization of these results can be done applying PJA on larger problems.Calculating the near optimal policy,using an ap-proximated optimal distance d app ,can be intractable if we need to compare with the results of Friend Q-Learning.We calculate the ra-tio DS =XY /N which represents the degree of space for each agent.As we study a problem where the team of agent has to handle only negative interactions,the higher the ratio,the more space agents have.We compare the performance of PJA algorithm for different ratios.To discover a relation between the ratio DS and the value of d app ,we compare the link between DS and d app。