RL From Scratch

Implementations

• REINFORCEPolicy Gradient

  Monte Carlo policy gradient — collect full episodes, compute returns, reinforce actions that led to high reward. The original policy gradient method (Williams, 1992).

  DiscreteOn-policyMLP · CNN
  →
• DQNValue-Based

  Deep Q-Network — neural network approximates Q(s,a), stabilised with experience replay and a frozen target network (Mnih et al., 2015).

  DiscreteOff-policyMLP
  →
• Dueling DQNValue-Based

  Splits Q into state value V(s) and advantage A(s,a). Learns which states are inherently good independently of which action matters (Wang et al., 2016).

  DiscreteOff-policyMLP
  →
• A2CActor-Critic

  Advantage Actor-Critic — actor outputs action probs, critic estimates V(s). Advantage A = r + γV(s') − V(s) replaces high-variance Monte Carlo returns (Mnih et al., 2016).

  DiscreteOn-policyMLP · CNN
  →
• PPOActor-Critic

  Proximal Policy Optimization — clips the probability ratio r = π_new/π_old so updates stay inside a safe region. Multiple gradient epochs per batch without instability (Schulman et al., 2017).

  Discrete + Cont.On-policyMLP · CNN
  →
• DDPGActor-Critic

  Deep Deterministic Policy Gradient — deterministic actor μ(s) outputs actions directly, backpropagating through the critic to improve. Off-policy with Polyak target networks (Lillicrap et al., 2016).

  ContinuousOff-policyMLP · CNN
  →
• TD3Actor-Critic

  Twin Delayed DDPG — two critics (take the min), delayed actor updates, and clipped noise on target actions to fix DDPG's overestimation bias (Fujimoto et al., 2018).

  ContinuousOff-policyMLP · CNN
  →
• SACActor-Critic

  Soft Actor-Critic — maximises reward and policy entropy jointly. Auto-tuned temperature. Robust, exploratory, and the benchmark for off-policy continuous control (Haarnoja et al., 2018).

  ContinuousOff-policyMLP · CNN
  →
• RNDExploration

  Random Network Distillation — prediction error of a fixed random network acts as an intrinsic curiosity reward. High for novel states, low for familiar ones (Burda et al., 2018).

  Intrinsic RewardMLP · CNN
  →

About NeatRL