Chapters 9-10: Prediction and Control Using Approximation
Introduction and Prediction
- Explanation: What is the on-Policy State Distribution
- Example: Semi-gradient TD(0) prediction with linear function approximation using linear features for a simple grid world (no walls).
- Example: Semi-gradient TD(0) prediction with linear function approximation using linear features and Fourier Features for the L-Maze. Note: Linear features don’t work!
Control with Linear Features
How Control with Linear Features Fails
- Semi-gradient Sarsa(0) control with linear function approximation with linear features for the L-Maze.
- Semi-gradient Sarsa(0) control with Linear function approximation with linear features for the 4x3 Grid World.
Control with Feature Construction
- Semi-gradient Sarsa(0) control with linear function approximation with Fourier basis features (L-Maze).
- Semi-gradient Sarsa(0) control with linear function approximation with Fourier basis features (4x3 Gridworld).
- Tile Coding: Richard Sutton’s Tile Coding Software tiles3.py (retrieved from here). Code explanation created with ChatGPT.
Non-Linear Function Approximation
Exercise
License
© 2026 Michael Hahsler. All code and documents in this repository are provided under the Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) License.