强化学习算法：Actor-Critic方法

强化学习算法：Actor-Critic方法

1. 技术分析

1.1 Actor-Critic概述

Actor-Critic结合策略和价值函数：

Actor-Critic架构 Actor: 策略网络，输出动作概率 Critic: 价值网络，评估状态价值 优势: 降低方差 提高样本效率 在线学习

1.2 Actor-Critic组成

1.3 Actor-Critic变体

Actor-Critic变体 A2C: 同步优势Actor-Critic A3C: 异步优势Actor-Critic DDPG: 深度确定性策略梯度 TD3: 双延迟深度确定性策略梯度

2. 核心功能实现

2.1 Actor-Critic算法

import numpy as np class ActorCritic: def __init__(self, actor, critic, actor_optimizer, critic_optimizer, gamma=0.99): self.actor = actor self.critic = critic self.actor_optimizer = actor_optimizer self.critic_optimizer = critic_optimizer self.gamma = gamma def compute_advantage(self, state, reward, next_state, done): value = self.critic(state) if done: target = reward else: target = reward + self.gamma * self.critic(next_state) advantage = target - value return advantage, target def train_step(self, state, action, reward, next_state, done): advantage, target = self.compute_advantage(state, reward, next_state, done) actor_loss = -np.log(self.actor(state)[action]) * advantage critic_loss = (target - self.critic(state)) ** 2 self.actor_optimizer.step(actor_loss) self.critic_optimizer.step(critic_loss) return actor_loss, critic_loss def train(self, env, episodes=1000): for episode in range(episodes): state = env.reset() done = False while not done: action_probs = self.actor(state) action = np.random.choice(len(action_probs), p=action_probs) next_state, reward, done = env.step(action) self.train_step(state, action, reward, next_state, done) state = next_state

2.2 A2C算法

class A2C(ActorCritic): def __init__(self, actor, critic, actor_optimizer, critic_optimizer, gamma=0.99, num_workers=4): super().__init__(actor, critic, actor_optimizer, critic_optimizer, gamma) self.num_workers = num_workers def train(self, env, episodes=1000): for episode in range(episodes): states = [] actions = [] rewards = [] next_states = [] dones = [] for _ in range(self.num_workers): state = env.reset() done = False while not done: action_probs = self.actor(state) action = np.random.choice(len(action_probs), p=action_probs) next_state, reward, done = env.step(action) states.append(state) actions.append(action) rewards.append(reward) next_states.append(next_state) dones.append(done) state = next_state total_actor_loss = 0 total_critic_loss = 0 for i in range(len(states)): advantage, target = self.compute_advantage( states[i], rewards[i], next_states[i], dones[i] ) actor_loss = -np.log(self.actor(states[i])[actions[i]]) * advantage critic_loss = (target - self.critic(states[i])) ** 2 total_actor_loss += actor_loss total_critic_loss += critic_loss self.actor_optimizer.step(total_actor_loss / len(states)) self.critic_optimizer.step(total_critic_loss / len(states))

2.3 Actor和Critic网络

class ActorNetwork: def __init__(self, state_dim, action_dim, hidden_dim=64): self.W1 = np.random.randn(state_dim, hidden_dim) * 0.01 self.b1 = np.zeros(hidden_dim) self.W2 = np.random.randn(hidden_dim, action_dim) * 0.01 self.b2 = np.zeros(action_dim) def forward(self, state): h = np.maximum(0, state @ self.W1 + self.b1) logits = h @ self.W2 + self.b2 exp_logits = np.exp(logits - np.max(logits)) probs = exp_logits / np.sum(exp_logits) return probs class CriticNetwork: def __init__(self, state_dim, hidden_dim=64): self.W1 = np.random.randn(state_dim, hidden_dim) * 0.01 self.b1 = np.zeros(hidden_dim) self.W2 = np.random.randn(hidden_dim, 1) * 0.01 self.b2 = np.zeros(1) def forward(self, state): h = np.maximum(0, state @ self.W1 + self.b1) value = h @ self.W2 + self.b2 return value[0]

3. 性能对比

3.1 Actor-Critic变体对比

3.2 Actor-Critic vs 策略梯度

3.3 网络规模影响

4. 最佳实践

4.1 Actor-Critic选择

def choose_actor_critic_method(environment_type): if environment_type == 'continuous': return 'DDPG' elif environment_type == 'discrete': return 'A2C' else: return 'A2C' class ActorCriticSelector: @staticmethod def select(config): methods = { 'a2c': A2C, 'a3c': A3C, 'ddpg': DDPG } return methods[config['method']](**config.get('params', {}))

4.2 训练技巧

class ActorCriticTrainingTips: @staticmethod def separate_learning_rates(actor_lr=0.001, critic_lr=0.005): return {'actor_lr': actor_lr, 'critic_lr': critic_lr} @staticmethod def target_networks(): return {'use_target': True, 'tau': 0.001} @staticmethod def gradient_clipping(max_norm=1.0): return {'max_norm': max_norm}

5. 总结

Actor-Critic是强化学习的主流方法：

1. Actor：策略网络，选择动作
2. Critic：价值网络，评估价值
3. A2C：同步训练，稳定可靠
4. DDPG：连续动作的标准方法

对比数据如下：

• A2C比REINFORCE样本效率更高
• DDPG是连续控制的首选
• 推荐使用分离的学习率
• 目标网络可以提高稳定性