chinese-chess中国象棋课件

– return the current player’s piece advantage on a scale from -1 to 1
• evaluation function: Y
Y = ∑k=1 to 7 wk * fk
TD(λ) and Updating the Weights
wi, t+1 = wi, t + a (Yt+1 – Yt)S k=1 to t l t-k∆ wiYk
evaluation function while playing on the Free Internet Chess Server (FICS, ), improved from a 1650 rating to a 2100 rating (the level of US Master, world champion are rating around 2900) in just 308 games and 3 days of play.
– property of the game
• feature evaluators
– Rook, Knight, Cannon , Minister, Guard, and Pawn
• weight:
– the value of a specific piece type
• feature function: f
Features Table
t
f1 f2 f3 f4 f5 f6
...
5
0
0 -0.5 0
0 0.4
6
0
00
0
0 -0.4
7
0
00
0 -0.5 0.4
8
0
0 -0.5 0 0.5 -0.4
...
Array of Weights
1.0000 1.0000 0.9987 1.0000 1.0000 1.0101
Why Temporal Difference Learning
• the average branching factor for the game tree is usually around 30
• the avቤተ መጻሕፍቲ ባይዱrage game lasts around 100 ply • the size of a game tree is 30100
is -2 times the board evaluation
• winner
– if is a draw, the final reward is 0 – if the board evaluation is negative, then the final reward
is -2 times the board evaluation – if the board evaluation is positive, then the final reward
Searching
• alpha-beta search • 3 ply search vs 4 ply search • horizon effect • quiescence cutoff search
Horizon Effect
t
t+1
t+2
t+3
Evaluation Function
• feature
is twice the board evaluation
Final Reward
• the weights are normalized by dividing by the greatest weight
• any negative weights are set to zero • the most valuable piece has weight 1
Summary of Main Events
1. Red’s turn 2. Update weights for Red using TD(λ) 3. Red does alpha-beta search. 4. Red executes the best move found 5. Blue’s turn 6. Update weights for Blue using TD(λ) 7. Blue does alpha-beta search 8. Blue executes the best move found (go to 1)
Example
t=5
t=6
t=7
t-8
Final Reward
• loser
– if is a draw, the final reward is 0 – if the board evaluation is negative, then the final reward
is twice the board – if the board evaluation is positive, then the final reward
How to Win a Chinese Chess Game
Reinforcement Learning Cheng, Wen Ju
Set Up
RIVER
General
Guard
Minister
Rook
Knight
Cannon
Pawn
Training
• how long does it to take for a human? • how long does it to take for a computer? • Chess program, “KnightCap”, used TD to learn its
Training
• to play a series of games in a self-play learning mode using temporal difference learning
• The goal is to learn some simple strategies
– piece values or weights
= wi, t + a (Yt+1 – Yt)(fi, t + l fi, t-1 + l 2fi, t-2 + … + l t-1fi, 1)
a = 0.01
learning rate –how quickly the weights can change
l = 0.01
feedback coefficient -how much to discount past values