Skip to content

Environment API

Base environment

gym_classics2.envs.abstract.base_env.BaseEnv

Bases: Env

Base class for finite Gymnasium environments with explicit model access.

BaseEnv supplies seeded reset and step methods, a discrete action space, optional integer encoding of states, and the complete transition model returned by model. Subclasses define the environment dynamics through _next_state, _reward, _done, and _generate_transitions. Each generated transition has the form (next_state, reward, terminated, probability).

Parameters:

Name Type Description Default
starts

Raw states from which an episode may start.

 required
action_labels

Labels whose positions define the integer action IDs.

 None
tabular

If true, observations are consecutive integer state IDs. If false, subclasses return raw states and must define a suitable observation space.

 True
reachable_states

All reachable raw states. If omitted, they are discovered from starts using the transition model.

 None

Attributes:

Name Type Description
action_space

Discrete Gymnasium action space.
observation_space

Discrete state space in tabular mode; otherwise defined by the subclass.
state

Current raw environment state, or None before the first reset.
tabular

Whether observations use integer state IDs.
Source code in gym_classics2/envs/abstract/base_env.py 
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
class BaseEnv(Env, metaclass=ABCMeta):
    """Base class for finite Gymnasium environments with explicit model access.

    ``BaseEnv`` supplies seeded ``reset`` and ``step`` methods, a discrete action
    space, optional integer encoding of states, and the complete transition model
    returned by ``model``. Subclasses define the environment dynamics through
    ``_next_state``, ``_reward``, ``_done``, and ``_generate_transitions``. Each
    generated transition has the form ``(next_state, reward, terminated,
    probability)``.

    Args:
        starts: Raw states from which an episode may start.
        action_labels: Labels whose positions define the integer action IDs.
        tabular: If true, observations are consecutive integer state IDs. If
            false, subclasses return raw states and must define a suitable
            observation space.
        reachable_states: All reachable raw states. If omitted, they are
            discovered from ``starts`` using the transition model.

    Attributes:
        action_space: Discrete Gymnasium action space.
        observation_space: Discrete state space in tabular mode; otherwise
            defined by the subclass.
        state: Current raw environment state, or ``None`` before the first reset.
        tabular: Whether observations use integer state IDs.
    """

    # needs to provide the following Gymnasium Env functions:
    # - reset
    # - step
    # - render
    # - close

    # added for gym_classics
    # - action_space
    # - observation_space
    # - start states
    # - transitions
    # - rewards
    # - done conditions
    # - encode
    # - decode
    # - is_reachable
    # - actions
    # - model
    # - generate_transitions

    def __init__(self, starts, action_labels = None, tabular = True, reachable_states=None):   
        assert action_labels is not None, "action_labels must be provided"

        self.action_labels = action_labels

        n_actions = len(action_labels)
        self.action_space = Discrete(n_actions)

        # observation space is defined by the subclass
        #self.observation_space = Discrete(len(self._reachable_states))

        # Normalize sets and other iterables so reset() can sample by index and
        # callers see a read-only public representation.
        self._starts = tuple(starts)
        self.state = None

        self._transition_cache = {}

        if reachable_states is None:
            # Get reachable states by searching through the state space
            reachable_states = set()
            for s in self._starts:
                self._search(s, reachable_states)

        # organize the reachable states in a consistent order
        self._reachable_states = reachable_states

        # build encoder and decoder for tabular access
        self._decoder = [s for s in sorted(self._reachable_states)]
        self._encoder = {s: i for i, s in enumerate(self._decoder)}

        self.tabular = tabular
        if self.tabular:
            self.observation_space = Discrete(len(self._reachable_states))


    def reset(self, seed=None, options=None):
        """Start a new episode and return ``(observation, info)``."""
        super().reset(seed=seed)

        self.state = self._starts[self.np_random.integers(len(self._starts))]
        observation = self.state
        info = {}

        #if self.render_mode == "human":
        #    self._render_frame()

        if self.tabular:
            observation = self.state2id(observation)

        return observation, info

    def step(self, action):
        """Sample one transition and return the five-item Gymnasium result."""
        assert self.action_space.contains(action)

        # after the random elements are sampled, the environment transition is deterministic
        elements = self._sample_random_elements(self.state, action)

        next_state, reward, done, _ = self._deterministic_step(self.state, action, *elements)
        self.state = next_state
        info = {}

        if self.tabular:
            next_state = self.state2id(next_state)

        return next_state, reward, done, False, info


    ## render needs to be implemented in each environment, so we don't provide a default implementation here
    ## we use the default close, can be overwritten.

    ## Additional Interface

    @property
    def start_states(self):
        """Tuple of raw states from which an episode may start.

        These are raw environment states even when :attr:`tabular` is true. Use
        ``state2id`` to convert them to integer observations.
        """
        return tuple(self._starts)

    ## Tabular access
    def states(self):
        """Returns a generator over all possible environment states."""
        return range(len(self._encoder))

    def state2id(self, state):
        """Converts a raw state into a unique integer."""

        if isinstance(state, list):
            return [self._encoder[s] for s in state]

        return self._encoder[state]

    def id2state(self, i):
        """Reverts an encoded integer back to its raw state."""
        if isinstance(i, list):
            return [self._decoder[j] for j in i]

        return self._decoder[i]

    def is_reachable(self, state):
        """Returns True if the state can be reached from at least one start location,
        False otherwise."""
        return state in self._reachable_states

    def actions(self):
        """Returns a generator over all possible agent actions."""
        return range(self.action_space.n)

    def action2id(self, action_label):
        """Converts a action label into a numeric action ID."""
        action_ids = dict(zip(self.action_labels, range(len(self.action_labels))))
        id = action_ids.get(action_label, -1)
        return id

    def id2action(self, action, type="text"):
        """Converts a numeric action ID into a label. Choices for type are 'text' and 'arrow'."""
        action_labels = self.action_labels + [""]  # Add empty label for hidden actions

        return action_labels[int(action)]

    def model(self, state, action):
        """Return the complete transition model for a state-action pair.

        Args:
            state: Integer state ID when :attr:`tabular` is true, otherwise a raw
                state.
            action: Integer action ID.

        Returns:
            A four-item list ``[next_states, rewards, terminals, probabilities]``.
            ``next_states`` is a list of IDs in tabular mode and raw states
            otherwise. The remaining items are one-dimensional NumPy arrays in the
            same order.
        """

        if self.tabular:
            state = self.id2state(state)

        # caching the model for efficiency    
        sa_pair = (state, action)
        if sa_pair in self._transition_cache:
            return self._transition_cache[sa_pair]

        tr = list(self._generate_transitions(state, action))
        tr = [[t[0] for t in tr], np.array([t[1] for t in tr]), np.array([t[2] for t in tr]), np.array([t[3] for t in tr])]

        assert (tr[3] >= 0.0).all(), "transition probabilities must be nonnegative"
        assert np.sum(tr[3]) == 1.0, "transition probabilities must sum to 1"

        if self.tabular:
            tr[0] = self.state2id(tr[0])

        self._transition_cache[sa_pair] = tr
        return tr

    def _search(self, state, visited):
        """A recursive depth-first search that adds all reachable states to the visited set."""
        visited.add(state)
        for a in self.actions():
            for transition in self._generate_transitions(state, a):
                next_state, _, done, prob = transition
                if prob > 0.0:
                    if not done and next_state not in visited:
                        self._search(next_state, visited)
                    # MFH add the final state!
                    if done and next_state not in visited:
                        visited.add(next_state)

    # Do not overwrite!
    def _deterministic_step(self, state, action, *random_elements):
        """An environment step that is deterministic conditioned on the given values
        of the random variables (if there are any).

        Do not override.
        """
        next_state, prob = self._next_state(state, action, *random_elements)
        reward = self._reward(state, action, next_state)
        done = self._done(state, action, next_state)
        #if done:
        #    next_state = state
        return next_state, reward, done, prob

    # these need to be implemented in the subclass
    def _sample_random_elements(self, state, action):
        """Samples values for random elements (if any) that influence the environment
        transition from the current state-action pair (S, A).

        If the environment is deterministic, no need to override this method.
        """
        return ()

    @abstractmethod
    def _next_state(self, state, action, *random_elements):
        """Returns the next state S' induced by the state-action pair (S, A), which must
        be deterministic conditioned on the values of any random_elements. Also returns
        the probability that this particular transition occurred."""
        raise NotImplementedError

    @abstractmethod
    def _reward(self, state, action, next_state):
        """Returns the reward yielded by this (S,A,S') outcome."""
        raise NotImplementedError

    @abstractmethod
    def _done(self, state, action, next_state):
        """Returns True if this (S,A,S') outcome should terminate, False otherwise."""
        raise NotImplementedError

    @abstractmethod
    def _generate_transitions(self, state, action):
        """Returns a generator over all transitions from this state-action pair.

        Should be overridden in the subclass.
        """
        raise NotImplementedError

start_states property 

start_states

Tuple of raw states from which an episode may start.

These are raw environment states even when :attr:tabular is true. Use state2id to convert them to integer observations.

states 

states()

Returns a generator over all possible environment states.

Source code in gym_classics2/envs/abstract/base_env.py 
140
141
142
def states(self):
    """Returns a generator over all possible environment states."""
    return range(len(self._encoder))

state2id 

state2id(state)

Converts a raw state into a unique integer.

Source code in gym_classics2/envs/abstract/base_env.py 
144
145
146
147
148
149
150
def state2id(self, state):
    """Converts a raw state into a unique integer."""

    if isinstance(state, list):
        return [self._encoder[s] for s in state]

    return self._encoder[state]

id2state 

id2state(i)

Reverts an encoded integer back to its raw state.

Source code in gym_classics2/envs/abstract/base_env.py 
152
153
154
155
156
157
def id2state(self, i):
    """Reverts an encoded integer back to its raw state."""
    if isinstance(i, list):
        return [self._decoder[j] for j in i]

    return self._decoder[i]

is_reachable 

is_reachable(state)

Returns True if the state can be reached from at least one start location, False otherwise.

Source code in gym_classics2/envs/abstract/base_env.py 
159
160
161
162
def is_reachable(self, state):
    """Returns True if the state can be reached from at least one start location,
    False otherwise."""
    return state in self._reachable_states

actions 

actions()

Returns a generator over all possible agent actions.

Source code in gym_classics2/envs/abstract/base_env.py 
164
165
166
def actions(self):
    """Returns a generator over all possible agent actions."""
    return range(self.action_space.n)

action2id 

action2id(action_label)

Converts a action label into a numeric action ID.

Source code in gym_classics2/envs/abstract/base_env.py 
168
169
170
171
172
def action2id(self, action_label):
    """Converts a action label into a numeric action ID."""
    action_ids = dict(zip(self.action_labels, range(len(self.action_labels))))
    id = action_ids.get(action_label, -1)
    return id

id2action 

id2action(action, type='text')

Converts a numeric action ID into a label. Choices for type are 'text' and 'arrow'.

Source code in gym_classics2/envs/abstract/base_env.py 
174
175
176
177
178
def id2action(self, action, type="text"):
    """Converts a numeric action ID into a label. Choices for type are 'text' and 'arrow'."""
    action_labels = self.action_labels + [""]  # Add empty label for hidden actions

    return action_labels[int(action)]

model 

model(state, action)

Return the complete transition model for a state-action pair.

Parameters:

Name Type Description Default
state

Integer state ID when :attr:tabular is true, otherwise a raw state.

 required
action

Integer action ID.

 required

Returns:

Type Description

A four-item list [next_states, rewards, terminals, probabilities].

next_states is a list of IDs in tabular mode and raw states

otherwise. The remaining items are one-dimensional NumPy arrays in the

same order.
Source code in gym_classics2/envs/abstract/base_env.py 
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
def model(self, state, action):
    """Return the complete transition model for a state-action pair.

    Args:
        state: Integer state ID when :attr:`tabular` is true, otherwise a raw
            state.
        action: Integer action ID.

    Returns:
        A four-item list ``[next_states, rewards, terminals, probabilities]``.
        ``next_states`` is a list of IDs in tabular mode and raw states
        otherwise. The remaining items are one-dimensional NumPy arrays in the
        same order.
    """

    if self.tabular:
        state = self.id2state(state)

    # caching the model for efficiency    
    sa_pair = (state, action)
    if sa_pair in self._transition_cache:
        return self._transition_cache[sa_pair]

    tr = list(self._generate_transitions(state, action))
    tr = [[t[0] for t in tr], np.array([t[1] for t in tr]), np.array([t[2] for t in tr]), np.array([t[3] for t in tr])]

    assert (tr[3] >= 0.0).all(), "transition probabilities must be nonnegative"
    assert np.sum(tr[3]) == 1.0, "transition probabilities must sum to 1"

    if self.tabular:
        tr[0] = self.state2id(tr[0])

    self._transition_cache[sa_pair] = tr
    return tr

Linear walk

gym_classics2.envs.abstract.linear_walk.LinearWalk

Bases: BaseEnv

Finite one-dimensional walk with terminal outcomes at both ends.

Raw states are integer positions from 0 through length - 1. Every episode starts at the center position. Action 0 moves left and action 1 moves right; movement is deterministic. Attempting to move left from position 0 or right from position length - 1 terminates the episode and yields the corresponding boundary reward. All other transitions have reward zero.

Observations are consecutive integer state IDs. For this environment, each ID is equal to its raw position.

Parameters:

Name Type Description Default
length

Odd number of nonterminal positions in the walk.

 required
left_reward

Reward for terminating beyond the left boundary.

 required
right_reward

Reward for terminating beyond the right boundary.

 required
Source code in gym_classics2/envs/abstract/linear_walk.py 
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
class LinearWalk(BaseEnv):
    """Finite one-dimensional walk with terminal outcomes at both ends.

    Raw states are integer positions from ``0`` through ``length - 1``. Every
    episode starts at the center position. Action ``0`` moves left and action ``1``
    moves right; movement is deterministic. Attempting to move left from position
    ``0`` or right from position ``length - 1`` terminates the episode and yields
    the corresponding boundary reward. All other transitions have reward zero.

    Observations are consecutive integer state IDs. For this environment, each ID
    is equal to its raw position.

    Args:
        length: Odd number of nonterminal positions in the walk.
        left_reward: Reward for terminating beyond the left boundary.
        right_reward: Reward for terminating beyond the right boundary.
    """

    def __init__(self, length, left_reward, right_reward):
        self._length = length
        self._left_reward = left_reward
        self._right_reward = right_reward

        assert length % 2 == 1
        super().__init__(starts={length // 2}, action_labels=["left", "right"])

    def _next_state(self, state, action):
        state += [-1, 1][action]
        next_state = min(max(state, 0), self._length - 1)
        return next_state, 1.0

    def _reward(self, state, action, next_state):
        sa_pair = (state, action)
        return {
            (0, 0):                self._left_reward,
            (self._length - 1, 1): self._right_reward,
        }.get(sa_pair, 0.0)

    def _done(self, state, action, next_state):
        sa_pair = (state, action)
        return sa_pair in {(0, 0), (self._length - 1, 1)}

    def _generate_transitions(self, state, action):
        yield self._deterministic_step(state, action)

Gridworld

gym_classics2.envs.abstract.gridworld.Gridworld

Bases: BaseEnv

Finite rectangular gridworld constructed from an ASCII layout.

Layout rows are written from top to bottom; raw states are (x, y) coordinates with (0, 0) in the lower-left corner. S marks a start, G a terminal goal, X a blocked cell, and a space a traversable cell. Other characters are traversable labels retained for plotting. Optional | characters are ignored when the layout is parsed.

Actions and transitions:

The default actions move up, right, down, and left and are represented in that order by the integers 0 to 3. The default transition model is deterministic. An action that would leave the gridworld or enter a blocked cell leaves the agent in place. Additional actions can be added in the constructor and will use integer IDs starting at 4. The transition model can be changed by subclassing and overwriting _next_state.

For an example of a stochastic transition model, see ClassicGridworld.

Reward model:

Entering a goal yields goal_reward and terminates the episode; other transitions yield step_reward. Since goal states are terminal (absorbing), the reward for a transition from a goal state is always zero.

The default setting is for a world without step cost and reaching the goal is rewarded with 1.0. If you want to use a step cost, set step_reward to a negative value. Note that the goal_reward is for the transition to the goal state and thus needs to be adjusted to reflect the positive reward for reaching the goal minus the cost of getting there.

You can also overwrite the _reward method to implement a custom reward model.

Parameters:

Name Type Description Default
layout_string

Rectangular ASCII representation of the grid.

 required
action_labels

Labels for the four actions, ordered as up, right, down, and left.

 ['up', 'right', 'down', 'left']
goal_reward

Reward for a transition into a goal cell.

 1.0
step_reward

Reward for any other transition.

 0.0
tabular

If true, observations are integer state IDs. If false, they are raw (x, y) coordinates.

 True
render_mode

None to disable rendering, "human" for a window, or "rgb_array" for an RGB image returned by render.

 None
Note

Rendering requires the optional render dependency extra.

Source code in gym_classics2/envs/abstract/gridworld.py 
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
class Gridworld(BaseEnv):
    """Finite rectangular gridworld constructed from an ASCII layout.

    Layout rows are written from top to bottom; raw states are ``(x, y)``
    coordinates with ``(0, 0)`` in the lower-left corner. ``S`` marks a start,
    ``G`` a terminal goal, ``X`` a blocked cell, and a space a traversable cell.
    Other characters are traversable labels retained for plotting. Optional ``|``
    characters are ignored when the layout is parsed.

    Actions and transitions:

    The default actions move up, right, down, and left and are represented in that 
    order by the integers 0 to 3. The default transition model is deterministic.
    An action that would leave the gridworld or enter 
    a blocked cell leaves the agent in place. Additional actions can be added in the constructor and will use 
    integer IDs starting at 4. The transition model can be changed
    by subclassing and overwriting ``_next_state``. 

    For an example of a stochastic transition model, see
    [ClassicGridworld][gym_classics2.envs.gym_classics2.classic_gridworld_v1.ClassicGridworld].

    Reward model:

    Entering a goal yields ``goal_reward`` and terminates the episode; other transitions yield
    ``step_reward``. Since goal states are terminal (absorbing), the reward for a transition from 
    a goal state is always zero.

    The default setting is for a world without step cost and reaching the goal is rewarded with 1.0. If you 
    want to use a step cost, set ``step_reward`` to a negative value. Note that the ``goal_reward`` 
    is for the transition to the goal state and thus needs to be 
    adjusted to reflect the positive reward for reaching the goal minus the cost of getting there.

    You can also overwrite the ``_reward`` method to implement a custom reward model.

    Args:
        layout_string: Rectangular ASCII representation of the grid.
        action_labels: Labels for the four actions, ordered as up, right, down,
            and left.
        goal_reward: Reward for a transition into a goal cell.
        step_reward: Reward for any other transition.
        tabular: If true, observations are integer state IDs. If false, they are
            raw ``(x, y)`` coordinates.
        render_mode: ``None`` to disable rendering, ``"human"`` for a window, or
            ``"rgb_array"`` for an RGB image returned by ``render``.

    Note:
        Rendering requires the optional ``render`` dependency extra.
    """
    metadata = {"render_modes": ["human", "rgb_array"], "render_fps": 4}

    def __init__(self, layout_string, action_labels = ["up", "right", "down", "left"], 
                 goal_reward = 1.0, step_reward = 0.0, 
                 tabular = True, render_mode=None):
        """Initialize the gridworld from ``layout_string``."""

        self._goal_reward = goal_reward
        self._step_reward = step_reward

        self.dims, starts, self._goals, self._blocks, self._extra_labels = parse_gridworld(layout_string)

        assert render_mode is None or render_mode in self.metadata["render_modes"]
        self.render_mode = render_mode
        self.PyGame_window_size = 512  # The size of the PyGame window in pixels
        self.PyGame_window = None
        self.PyGame_clock = None

        super().__init__(starts, action_labels = action_labels, tabular = tabular, reachable_states = None)

        if not tabular:
            self.observation_space = MultiDiscrete(self.dims)

    @property
    def goal_states(self):
        """Tuple containing the raw terminal goal coordinates."""
        return tuple(sorted(self._goals))

    def _next_state(self, state, action, *random_elements):
        next_state = self._move(state, action)
        if self._is_blocked(next_state):
            next_state = state
        return self._clamp(next_state), 1.0

    def _generate_transitions(self, state, action):
        yield self._deterministic_step(state, action)

    def _reward(self, state, action, next_state):      
        if state in self._goals:
            return 0.0
        if next_state in self._goals: 
            return self._goal_reward 
        return self._step_reward

    def _done(self, state, action, next_state):
        return next_state in self._goals

    def step(self, action):
        """Advance the environment and render a frame in human mode."""
        next_state, reward, done, x, info = super().step(action)

        if self.render_mode == "human":
            self._render_frame()

        return next_state, reward, done, x, info

    def reset(self, seed=None, options=None):
        """Reset to a start cell and render a frame in human mode."""
        observation, info = super().reset(seed=seed, options=options)

        if self.render_mode == "human":
            self._render_frame()

        return observation, info

    def _move(self, state, action):
        x, y = state
        return {
            0: (x,   y+1),  # Up
            1: (x+1, y),    # Right
            2: (x,   y-1),  # Down
            3: (x-1, y)     # Left
        }[action]

    def _clamp(self, state):
        """Clamps the state within the grid dimensions."""
        x, y = state
        x = max(0, min(x, self.dims[0] - 1))
        y = max(0, min(y, self.dims[1] - 1))
        return (x, y)

    def _is_blocked(self, state):
        """Returns True if this state cannot be occupied, False otherwise."""
        return state in self._blocks

    def close(self):
        """Release any PyGame display resources."""
        if self.PyGame_window is not None:
            pygame = _load_pygame()
            pygame.display.quit()
            pygame.quit()
            self.PyGame_window = None
            self.PyGame_clock = None

        super().close()

    def render(self):
        """Return an RGB frame when ``render_mode='rgb_array'``."""
        if self.render_mode == "rgb_array":
            return self._render_frame()

    def _render_frame(self):
        pygame = _load_pygame()
        pix_square_size = self.PyGame_window_size / max(self.dims)
        display_size = np.array(pix_square_size) * self.dims

        if self.PyGame_window is None and self.render_mode == "human":
            pygame.init()
            pygame.display.init()
            self.PyGame_window = pygame.display.set_mode(display_size)
        if self.PyGame_clock is None and self.render_mode == "human":
            self.PyGame_clock = pygame.time.Clock()

        canvas = pygame.Surface(display_size)
        canvas.fill((255, 255, 255))

        # draw the goal
        for g in self._goals:
            pos = np.array(g)
            pos[1] = self.dims[1] - pos[1] - 1
            pygame.draw.rect(
                canvas,
                (0, 128, 0),
                pygame.Rect(
                    pix_square_size * pos,
                    (pix_square_size, pix_square_size),
                ),
            )

        # unreachable squares
        for g in np.argwhere(self.to_matrix() == -1):
            pos = np.array(g)
            pos = np.flip(pos)
            pos[1] = self.dims[1] - pos[1] - 1
            pygame.draw.rect(
                canvas,
                (72, 72, 72),
                pygame.Rect(
                    pix_square_size * pos,
                    (pix_square_size, pix_square_size),
                ),
            )

        # draw the agent
        pos = np.array(self.state)
        pos[1] = self.dims[1] - pos[1] - 1
        pygame.draw.circle(
            canvas,
            (0, 0, 255),
            (pos + .5) * pix_square_size,
            pix_square_size / 3,
        )

        # Finally, add some gridlines
        for x in range(self.dims[1] + 1):
            pygame.draw.line(
                canvas,
                0,
                (0, pix_square_size * x),
                (display_size[0], pix_square_size * x),
                width=3,
            )

        for x in range(self.dims[0] + 1):
            pygame.draw.line(
                canvas,
                0,
                (pix_square_size * x, 0),
                (pix_square_size * x, display_size[1]),
                width=3,
            )

        if self.render_mode == "human":
            # The following line copies our drawings from `canvas` to the visible window
            self.PyGame_window.blit(canvas, canvas.get_rect())
            pygame.event.pump()
            pygame.display.update()

            # We need to ensure that human-rendering occurs at the predefined framerate.
            # The following line will automatically add a delay to
            # keep the framerate stable.
            self.PyGame_clock.tick(self.metadata["render_fps"])
        else:  # rgb_array
            return np.transpose(
                np.array(pygame.surfarray.pixels3d(canvas)), axes=(1, 0, 2)
            )

    ### Addition to the interface

    ## overwrite so we have arrows
    def id2action(self, action, type="text"):
        """Converts a numeric action ID into a label. Choices for type are 'text' and 'arrow'."""
        action = int(action)

        action_labels = self.action_labels + [""]  # Add empty label for hidden actions

        # empty has index 4 and is used to hide actions
        if (type == "arrow"):
             action_labels[0:4] = ['↑', '→', '↓', '←']

        return action_labels[action]

    def to_matrix(self, value = None):
        """Arrange state-indexed values as a grid-shaped matrix.

        Args:
            value: One value per reachable state. If omitted, use state IDs.

        Returns:
            Matrix with grid rows ordered for display. Unreachable cells contain
            ``-1``, an empty string, ``NaN``, or zero according to the input type.
        """

        if value is None:
            value = list(self.states())

        value = np.array(value)

        if np.issubdtype(value.dtype, np.integer):
            m = np.full(self.dims, -1, dtype=value.dtype)
        elif np.issubdtype(value.dtype, np.str_):
            m = np.full(self.dims, "", dtype=value.dtype)
        elif np.issubdtype(value.dtype, np.floating):
            m = np.full(self.dims, np.nan, dtype=value.dtype)
        else:
            m = np.zeros(self.dims, dtype=value.dtype)

        for y in range(self.dims[1]):
            for x in range(self.dims[0]):
                state = (x, y)
                if self.is_reachable(state):
                    m[x,y] = value[self.state2id(state)]
                else:
                    pass

        return m.transpose() 

    def print(self, array, decimals=2, separator=' ' * 2, signed=True, transpose=False):
        """Prints a gridworld array in a human-readable format. The array should be a vector with values for states in the gridworld, 
        such as a value function or policy."""

        def formatter(x):
            string = '{:' + ('+' if signed else '') + '.' + str(decimals) + 'f}'
            return string.format(x)
        maxlen = max([len(formatter(x)) for x in array])

        # Now we can actually print the values
        for y in reversed(range(self.dims[1])):
            for x in range(self.dims[0]):
                state = (x, y) if not transpose else (y, x)
                if self.is_reachable(state):
                    s = self.state2id(state)
                    print(formatter(array[s]).rjust(maxlen), end=separator)
                else:
                    print(' ' * maxlen, end=separator)
            print()

    def image(self, V=None, policy=None, episode = None, labels=None, title=None, cmap = 'auto', origin='lower', clim = None):
        """Display the gridworld, values, and optional actions as an image.

        Args:
            V: One value per state, such as a value function. If omitted, cells
                display their state IDs and the color bar is hidden.
            policy: One action ID per state. Actions are drawn as arrows and
                replace ``labels`` when provided.
            episode: Sequence of transitions whose first two entries are the state
                and action IDs. Actions taken in visited states are drawn as arrows
                and replace ``policy``.
            labels: One label per state. If ``True``, use values from ``V`` rounded
                to two decimal places.
            title: Plot title.
            cmap: Matplotlib colormap or colormap name. ``"auto"`` selects a
                sequential or diverging colormap from the displayed values.
            origin: ``"lower"`` places coordinate ``(0, 0)`` at the lower-left;
                ``"upper"`` places it at the upper-left.
            clim: Optional ``(minimum, maximum)`` limits for the color scale.
        """

        colorbar = True

        if not V is None:
            m = self.to_matrix(V)
        else:
            m = np.zeros(self.dims).transpose()
            # missing positions have -1
            m[self.to_matrix(labels) == -1] = np.nan
            labels = self.states()
            colorbar = False

        if not episode is None:
            # start with policy that hides all actions with an index past the last action.
            policy = np.full(len(self.states()), len(self.actions()))
            for step in episode:
                policy[step[0]] = step[1]

        if not policy is None:
            labels = [self.id2action(a, type = "arrow") for a in policy]

        if isinstance(labels, bool) and labels:
                labels = np.round(V, 2)

        if not labels is None:
            labels = self.to_matrix(labels)

        extra = [""] * self.dims[0] * self.dims[1]
        for s in self._starts:
            extra[self.state2id(s)] = "S"
        for s in self._goals:
            extra[self.state2id(s)] = "G"
        for s, label in self._extra_labels:
            extra[self.state2id(s)] = label
        extra = self.to_matrix(extra)

        _image(m, title=title, labels=labels, extra=extra, cmap=cmap, clim = clim, origin=origin, colorbar=colorbar)  


    def image_list(self, Vs = None, policies = None, episodes = None, cmap = 'auto', clim = None, origin='lower'):
        """
        Creates a sequence of images, one for each episode.
        """
        n_states = len(self.states())
        if Vs is not None:
            iterations = len(Vs)
        elif policies is not None:
            iterations = len(policies)
        else:
            iterations = len(episodes)

        V = None
        policy = None
        episode = None

        for i in range(iterations):
            if not Vs is None:
                V = Vs[i]
            if not policies is None:
                policy = policies[i]
            if not episodes is None:
                episode = episodes[i]    

            self.image(V, policy=policy, episode=episode, title=f'After Iteration {i}', cmap=cmap, clim = clim, origin=origin)

goal_states property 

goal_states

Tuple containing the raw terminal goal coordinates.

reset 

reset(seed=None, options=None)

Reset to a start cell and render a frame in human mode.

Source code in gym_classics2/envs/abstract/gridworld.py 
132
133
134
135
136
137
138
139
def reset(self, seed=None, options=None):
    """Reset to a start cell and render a frame in human mode."""
    observation, info = super().reset(seed=seed, options=options)

    if self.render_mode == "human":
        self._render_frame()

    return observation, info

step 

step(action)

Advance the environment and render a frame in human mode.

Source code in gym_classics2/envs/abstract/gridworld.py 
123
124
125
126
127
128
129
130
def step(self, action):
    """Advance the environment and render a frame in human mode."""
    next_state, reward, done, x, info = super().step(action)

    if self.render_mode == "human":
        self._render_frame()

    return next_state, reward, done, x, info

render 

render()

Return an RGB frame when render_mode='rgb_array'.

Source code in gym_classics2/envs/abstract/gridworld.py 
172
173
174
175
def render(self):
    """Return an RGB frame when ``render_mode='rgb_array'``."""
    if self.render_mode == "rgb_array":
        return self._render_frame()

print 

print(array, decimals=2, separator=' ' * 2, signed=True, transpose=False)

Prints a gridworld array in a human-readable format. The array should be a vector with values for states in the gridworld, such as a value function or policy.

Source code in gym_classics2/envs/abstract/gridworld.py 
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
def print(self, array, decimals=2, separator=' ' * 2, signed=True, transpose=False):
    """Prints a gridworld array in a human-readable format. The array should be a vector with values for states in the gridworld, 
    such as a value function or policy."""

    def formatter(x):
        string = '{:' + ('+' if signed else '') + '.' + str(decimals) + 'f}'
        return string.format(x)
    maxlen = max([len(formatter(x)) for x in array])

    # Now we can actually print the values
    for y in reversed(range(self.dims[1])):
        for x in range(self.dims[0]):
            state = (x, y) if not transpose else (y, x)
            if self.is_reachable(state):
                s = self.state2id(state)
                print(formatter(array[s]).rjust(maxlen), end=separator)
            else:
                print(' ' * maxlen, end=separator)
        print()

image 

image(V=None, policy=None, episode=None, labels=None, title=None, cmap='auto', origin='lower', clim=None)

Display the gridworld, values, and optional actions as an image.

Parameters:

Name Type Description Default
V

One value per state, such as a value function. If omitted, cells display their state IDs and the color bar is hidden.

 None
policy

One action ID per state. Actions are drawn as arrows and replace labels when provided.

 None
episode

Sequence of transitions whose first two entries are the state and action IDs. Actions taken in visited states are drawn as arrows and replace policy.

 None
labels

One label per state. If True, use values from V rounded to two decimal places.

 None
title

Plot title.

 None
cmap

Matplotlib colormap or colormap name. "auto" selects a sequential or diverging colormap from the displayed values.

 'auto'
origin

"lower" places coordinate (0, 0) at the lower-left; "upper" places it at the upper-left.

 'lower'
clim

Optional (minimum, maximum) limits for the color scale.

 None
Source code in gym_classics2/envs/abstract/gridworld.py 
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
def image(self, V=None, policy=None, episode = None, labels=None, title=None, cmap = 'auto', origin='lower', clim = None):
    """Display the gridworld, values, and optional actions as an image.

    Args:
        V: One value per state, such as a value function. If omitted, cells
            display their state IDs and the color bar is hidden.
        policy: One action ID per state. Actions are drawn as arrows and
            replace ``labels`` when provided.
        episode: Sequence of transitions whose first two entries are the state
            and action IDs. Actions taken in visited states are drawn as arrows
            and replace ``policy``.
        labels: One label per state. If ``True``, use values from ``V`` rounded
            to two decimal places.
        title: Plot title.
        cmap: Matplotlib colormap or colormap name. ``"auto"`` selects a
            sequential or diverging colormap from the displayed values.
        origin: ``"lower"`` places coordinate ``(0, 0)`` at the lower-left;
            ``"upper"`` places it at the upper-left.
        clim: Optional ``(minimum, maximum)`` limits for the color scale.
    """

    colorbar = True

    if not V is None:
        m = self.to_matrix(V)
    else:
        m = np.zeros(self.dims).transpose()
        # missing positions have -1
        m[self.to_matrix(labels) == -1] = np.nan
        labels = self.states()
        colorbar = False

    if not episode is None:
        # start with policy that hides all actions with an index past the last action.
        policy = np.full(len(self.states()), len(self.actions()))
        for step in episode:
            policy[step[0]] = step[1]

    if not policy is None:
        labels = [self.id2action(a, type = "arrow") for a in policy]

    if isinstance(labels, bool) and labels:
            labels = np.round(V, 2)

    if not labels is None:
        labels = self.to_matrix(labels)

    extra = [""] * self.dims[0] * self.dims[1]
    for s in self._starts:
        extra[self.state2id(s)] = "S"
    for s in self._goals:
        extra[self.state2id(s)] = "G"
    for s, label in self._extra_labels:
        extra[self.state2id(s)] = label
    extra = self.to_matrix(extra)

    _image(m, title=title, labels=labels, extra=extra, cmap=cmap, clim = clim, origin=origin, colorbar=colorbar)

image_list 

image_list(Vs=None, policies=None, episodes=None, cmap='auto', clim=None, origin='lower')

Creates a sequence of images, one for each episode.

Source code in gym_classics2/envs/abstract/gridworld.py 
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
def image_list(self, Vs = None, policies = None, episodes = None, cmap = 'auto', clim = None, origin='lower'):
    """
    Creates a sequence of images, one for each episode.
    """
    n_states = len(self.states())
    if Vs is not None:
        iterations = len(Vs)
    elif policies is not None:
        iterations = len(policies)
    else:
        iterations = len(episodes)

    V = None
    policy = None
    episode = None

    for i in range(iterations):
        if not Vs is None:
            V = Vs[i]
        if not policies is None:
            policy = policies[i]
        if not episodes is None:
            episode = episodes[i]    

        self.image(V, policy=policy, episode=episode, title=f'After Iteration {i}', cmap=cmap, clim = clim, origin=origin)

Noisy gridworld

gym_classics2.envs.abstract.noisy_gridworld.NoisyGridworld

Bases: Gridworld

Gridworld with classic 80-10-10 stochastic action outcomes.

The requested action is executed with probability 0.8. With probability 0.1 each, it is instead rotated 90 degrees clockwise or counterclockwise. The resulting move follows the boundary, blocking, reward, and termination rules defined by Gridworld. The model method enumerates all three possible outcomes, including duplicate next states when movement is blocked.

Parameters:

Name Type Description Default
layout_string

Rectangular ASCII representation of the grid.

 required
action_labels

Labels for the four actions, ordered as up, right, down, and left.

 ['up', 'right', 'down', 'left']
goal_reward

Reward for a transition into a goal cell.

 1.0
step_reward

Reward for any other transition.

 0.0
tabular

If true, observations are integer state IDs. If false, they are raw (x, y) coordinates.

 True
render_mode

None to disable rendering, "human" for a window, or "rgb_array" for an RGB image returned by render.

 None
Note

Rendering requires the optional render dependency extra.

Source code in gym_classics2/envs/abstract/noisy_gridworld.py 
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
class NoisyGridworld(Gridworld):
    """Gridworld with classic 80-10-10 stochastic action outcomes.

    The requested action is executed with probability 0.8. With probability 0.1
    each, it is instead rotated 90 degrees clockwise or counterclockwise. The
    resulting move follows the boundary, blocking, reward, and termination rules
    defined by ``Gridworld``. The ``model`` method enumerates all three
    possible outcomes, including duplicate next states when movement is blocked.

    Args:
        layout_string: Rectangular ASCII representation of the grid.
        action_labels: Labels for the four actions, ordered as up, right, down,
            and left.
        goal_reward: Reward for a transition into a goal cell.
        step_reward: Reward for any other transition.
        tabular: If true, observations are integer state IDs. If false, they are
            raw ``(x, y)`` coordinates.
        render_mode: ``None`` to disable rendering, ``"human"`` for a window, or
            ``"rgb_array"`` for an RGB image returned by ``render``.

    Note:
        Rendering requires the optional ``render`` dependency extra.
    """

    def _sample_random_elements(self, state, action):
        return [self._noisy_action(action)]

    def _next_state(self, state, action, noisy_action):
        next_state, _ = super()._next_state(state, noisy_action)
        if action == noisy_action:
            return next_state, 0.8
        return next_state, 0.1

    def _noisy_action(self, action):
        p = self.np_random.random()
        # 10% chance: rotate the action clockwise
        if 0.8 <= p < 0.9:
            action += 1
        # 10% chance: rotate the action counter-clockwise
        elif 0.9 <= p:
            action -= 1
        return action % self.action_space.n

    def _generate_transitions(self, state, action):
        for i in [-1, 0, 1]:
            noisy_action = (action + i) % self.action_space.n
            yield self._deterministic_step(state, action, noisy_action)

Concrete environments

gym_classics2.envs.gym_classics2.classic_gridworld_v1.ClassicGridworld

Bases: Gridworld

A 4x3 pedagogical gridworld. The agent starts in the bottom-left cell. Actions are noisy; with a 10% chance each, a move action may be rotated by 90 degrees clockwise or counter-clockwise (the "80-10-10 rule"). Cell (1, 1) is blocked and cannot be occupied by the agent.

Reference: Russell and Norvig, Artificial Intelligence: A Modern Approach (3rd ed., 2010), p. 646.

state: Grid location.

actions: Move up/right/down/left.

rewards: +1 for taking any action in cell (3, 2). -1 for taking any action in cell (3, 1). NOTE: v1 uses the original -0.04 penalty for each state.

termination: Earning a nonzero reward.

Source code in gym_classics2/envs/gym_classics2/classic_gridworld_v1.py 
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
class ClassicGridworld(Gridworld):
    """A 4x3 pedagogical gridworld. The agent starts in the bottom-left cell. Actions
    are noisy; with a 10% chance each, a move action may be rotated by 90 degrees
    clockwise or counter-clockwise (the "80-10-10 rule"). Cell (1, 1) is blocked and
    cannot be occupied by the agent.

    **Reference:** Russell and Norvig,
    [*Artificial Intelligence: A Modern Approach* (3rd ed., 2010)](https://aima.cs.berkeley.edu/3rd-ed/),
    p. 646.

    **state**: Grid location.

    **actions**: Move up/right/down/left.

    **rewards**: +1 for taking any action in cell (3, 2). -1 for taking any
    action in cell (3, 1). *NOTE:*  v1 uses the original -0.04 penalty for each state.

    **termination**: Earning a nonzero reward.
    """

    layout = """
|   G|
| X G|
|S   |
"""

    def __init__(self, goal_reward = 1.0, trap_reward = -1.0, step_reward = -0.04, **args):
        self._trap_reward = trap_reward
        super().__init__(ClassicGridworld.layout, goal_reward = goal_reward, step_reward = step_reward, **args)

    def _reward(self, state, action, next_state):
        if state in self._goals:
            return 0.0
        return {(3, 1): self._trap_reward, (3, 2): self._goal_reward}.get(next_state, self._step_reward)

    def _done(self, state, action, next_state):
        return next_state in self._goals

    # Implement the non-deterministic actions

    # This method is called in the step function to create random events.
    # Here, the random event is that the environment executes a different
    # noisy action instead of the action the agent asked for.  
    # We return actually executed action as a list of random elements.
    def _sample_random_elements(self, state, action):
        noisy_action = (action + self.np_random.choice([-1, 0, 1], p=[.1,.8,.1])) % self.action_space.n
        return [noisy_action]

    # Returns an iterator for all possible outcomes. The random element is that
    # we have a noisy action, that may not be the intended action. 
    def _generate_transitions(self, state, action):
        # goal state is absorbing
        if state in self._goals:
            yield state, 0, True, 1.0

        else:
            for i in [-1, 0, 1]:
                noisy_action = (action + i) % self.action_space.n
                yield self._deterministic_step(state, action, noisy_action)

    # execute the noisy action and the probability
    def _next_state(self, state, action, noisy_action):
        next_state, _ = super()._next_state(state, noisy_action)
        p = 0.8 if action == noisy_action else 0.1
        return next_state, p

gym_classics2.envs.gym_classics2.cliff_walk_v1.CliffWalk

Bases: Gridworld

The Cliff Walking task, a 12x4 gridworld often used to contrast Sarsa with Q-Learning. The agent begins in the bottom-left cell and must navigate to the goal (bottom-right cell) without entering the region along the bottom ("The Cliff").

v1 follows the textbook and does not end episodes when the cliff is reached. Also, the goal is a real state.

Reference: Sutton and Barto, Reinforcement Learning: An Introduction (2nd ed., 2018), p. 132, Example 6.6.

state: Grid location.

actions: Move up/right/down/left.

rewards: -100 for entering The Cliff. -1 for all other transitions.

termination: reaching the goal.

Source code in gym_classics2/envs/gym_classics2/cliff_walk_v1.py 
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
class CliffWalk(Gridworld):
    """The Cliff Walking task, a 12x4 gridworld often used to contrast Sarsa with
    Q-Learning. The agent begins in the bottom-left cell and must navigate to the goal
    (bottom-right cell) without entering the region along the bottom ("The Cliff").

    v1 follows the textbook and does not end episodes when the cliff is reached. Also, the goal is 
    a real state.

    **Reference:** Sutton and Barto,
    [*Reinforcement Learning: An Introduction* (2nd ed., 2018)](https://incompleteideas.net/book/the-book-2nd.html),
    p. 132, Example 6.6.

    **state**: Grid location.

    **actions**: Move up/right/down/left.

    **rewards**: -100 for entering The Cliff. -1 for all other transitions.

    **termination**: reaching the goal.
    """

    layout = """
|            |
|            |
|            |
|S          G|
"""

    def __init__(self, goal_reward=0.0, step_reward=-1.0, cliff_reward=-100.0, **args):
        self._cliff = frozenset((x, 0) for x in range(1, 11))
        self._cliff_reward = cliff_reward
        super().__init__(CliffWalk.layout, goal_reward=goal_reward, step_reward=step_reward, **args)

    # cliff is unreachable. Leads to the start state    
    def _next_state(self, state, action):
        state, _ = super()._next_state(state, action)
        if (state in self._cliff):
            state = self._starts[0]
        return state, 1.0

    def _reward(self, state, action, next_state):
        if next_state in self._goals: 
            return self._goal_reward

        n_state, _ = super()._next_state(state, action)
        if n_state in self._cliff: 
            return self._cliff_reward

        return self._step_reward

    def _done(self, state, action, next_state):
        return next_state in self._goals

gym_classics2.envs.gym_classics2.dyna_maze.DynaMaze

Bases: Gridworld

A 9x6 deterministic gridworld with barriers to make navigation more challenging. The agent starts in cell (0, 3); the goal is the top-right cell.

Reference: Sutton and Barto, Reinforcement Learning: An Introduction (2nd ed., 2018), p. 164, Example 8.1.

state: Grid location.

actions: Move up/right/down/left.

rewards: +1 for episode termination.

termination: Reaching the goal.

Source code in gym_classics2/envs/gym_classics2/dyna_maze.py 
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
class DynaMaze(Gridworld):
    """A 9x6 deterministic gridworld with barriers to make navigation more challenging.
    The agent starts in cell (0, 3); the goal is the top-right cell.

    **Reference:** Sutton and Barto,
    [*Reinforcement Learning: An Introduction* (2nd ed., 2018)](https://incompleteideas.net/book/the-book-2nd.html),
    p. 164, Example 8.1.

    **state**: Grid location.

    **actions**: Move up/right/down/left.

    **rewards**: +1 for episode termination.

    **termination**: Reaching the goal.
    """

    layout = """
|       XG|
|  X    X |
|S X    X |
|  X      |
|     X   |
|         |
"""

    def __init__(self, **args):
        super().__init__(DynaMaze.layout, **args)

gym_classics2.envs.gym_classics2.four_rooms.FourRooms

Bases: Gridworld

An 11x11 gridworld segmented into four rooms. The agent begins in the bottom-left cell; the goal is in the top-right cell.

Reference: Sutton, Precup, and Singh, Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning (1999), p. 192.

state: Grid location.

actions: Move up/right/down/left.

rewards: +1 for episode termination.

termination: Taking any action in the goal.

Source code in gym_classics2/envs/gym_classics2/four_rooms.py 
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
class FourRooms(Gridworld):
    """An 11x11 gridworld segmented into four rooms. The agent begins in the bottom-left
    cell; the goal is in the top-right cell.

    **Reference:** Sutton, Precup, and Singh,
    [*Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in
    Reinforcement Learning* (1999)](https://doi.org/10.1016/S0004-3702%2899%2900052-1),
    p. 192.

    **state**: Grid location.

    **actions**: Move up/right/down/left.

    **rewards**: +1 for episode termination.

    **termination**: Taking any action in the goal.
    """

    layout = """
|     X     |
|     X   G |
|           |
|     X     |
|     X     |
|X XXXX     |
|     XXX XX|
|     X     |
|     X     |
|           |
|S    X     |
"""

    def __init__(self, **args):
        super().__init__(FourRooms.layout, **args)

gym_classics2.envs.gym_classics2.L_maze.LMazeGridworld

Bases: Gridworld

A deterministic 10x10 maze separated by an L-shaped barrier.

The agent begins below the horizontal barrier and must travel around it to reach the goal near the upper-right corner.

Reference: Dijkstra's algorithm, Wikipedia.

Source code in gym_classics2/envs/gym_classics2/L_maze.py 
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
class LMazeGridworld(Gridworld):
    """A deterministic 10x10 maze separated by an L-shaped barrier.

    The agent begins below the horizontal barrier and must travel around it to
    reach the goal near the upper-right corner.

    **Reference:** [Dijkstra's algorithm](https://en.wikipedia.org/wiki/Dijkstra%27s_algorithm),
    Wikipedia.
    """
    layout = """
|          |
|        G |
|          |
| XXXXXX   |
|      X   |
|      X   |
|      X   |
|   S  X   |
|          |
|          |
"""

    def __init__(self, **args):
        super().__init__(self.layout, **args)

gym_classics2.envs.gym_classics2.sparse_gridworld.SparseGridworld

Bases: NoisyGridworld

A 10x8 featureless gridworld. The agent starts in cell (1, 3) and the goal is at cell (6, 3). To make it more challenging, the same 80-10-10 transition probabilities from ClassicGridworld are used. Great for testing various forms of credit assignment in the presence of noise.

Reference: Sutton and Barto, Reinforcement Learning: An Introduction (2nd ed., 2018), p. 147, Figure 7.4.

states: Grid location.

actions: Move up/right/down/left.

rewards: +1 for episode termination.

termination: Reaching the goal.

Source code in gym_classics2/envs/gym_classics2/sparse_gridworld.py 
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
class SparseGridworld(NoisyGridworld):
    """A 10x8 featureless gridworld. The agent starts in cell (1, 3) and the goal is at
    cell (6, 3). To make it more challenging, the same 80-10-10 transition probabilities
    from `ClassicGridworld` are used. Great for testing various forms of credit
    assignment in the presence of noise.

    **Reference:** Sutton and Barto,
    [*Reinforcement Learning: An Introduction* (2nd ed., 2018)](https://incompleteideas.net/book/the-book-2nd.html),
    p. 147, Figure 7.4.

    **states:** Grid location.

    **actions:** Move up/right/down/left.

    **rewards:** +1 for episode termination.

    **termination:** Reaching the goal.
    """

    layout = """
|          |
|          |
|          |
|          |
| S    G   |
|          |
|          |
|          |
"""

    def __init__(self, **args):
        super().__init__(SparseGridworld.layout, **args)

gym_classics2.envs.gym_classics2.windy_gridworld.WindyGridworld

Bases: Gridworld

A 10x7 deterministic gridworld where some columns are affected by an upward wind. The agent starts in cell (0, 3) and the goal is at cell (7, 3). If an agent executes an action from a cell with wind, the resulting position is given by the vector sum of the action's effect and the wind.

Reference: Sutton and Barto, Reinforcement Learning: An Introduction (2nd ed., 2018), p. 130, Example 6.5.

state: Grid location.

actions: Move up/right/down/left.

rewards: -1 for all transitions unless the episode terminates.

termination: Reaching the goal.

Source code in gym_classics2/envs/gym_classics2/windy_gridworld.py 
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
class WindyGridworld(Gridworld):
    """A 10x7 deterministic gridworld where some columns are affected by an upward wind.
    The agent starts in cell (0, 3) and the goal is at cell (7, 3). If an agent executes
    an action from a cell with wind, the resulting position is given by the vector sum
    of the action's effect and the wind.

    **Reference:** Sutton and Barto,
    [*Reinforcement Learning: An Introduction* (2nd ed., 2018)](https://incompleteideas.net/book/the-book-2nd.html),
    p. 130, Example 6.5.

    **state:** Grid location.

    **actions:** Move up/right/down/left.

    **rewards:** -1 for all transitions unless the episode terminates.

    **termination:** Reaching the goal.
    """

    layout = """
|          |
|          |
|          |
|S      G  |
|          |
|          |
|          |
"""

    def __init__(self, **args):
        super().__init__(WindyGridworld.layout, **args)

    def _next_state(self, state, action):
        wind_strength = self._wind_strength(state)
        state, _ = super()._next_state(state, action)
        state = self._apply_wind(state, wind_strength)
        return self._clamp(state), 1.0

    def _apply_wind(self, state, strength):
        x, y = state
        return (x, y + strength)

    def _wind_strength(self, state):
        """Returns wind strength in the given state."""
        x, _ = state
        if x in {3, 4, 5, 8}:
            return 1
        elif x in {6, 7}:
            return 2
        else:
            return 0

gym_classics2.envs.gym_classics2.linear_walks.Walk5

Bases: LinearWalk

A 5-state deterministic linear walk. Ideal for implementing random walk experiments.

Reference: Sutton and Barto, Reinforcement Learning: An Introduction (2nd ed., 2018), p. 125, Example 6.2.

state: Discrete position {0, ..., 4} on the number line.

actions: Move left/right.

rewards: +1 for moving right in the extreme right state.

termination: Moving right in the extreme right state or moving left in the extreme left state.

Source code in gym_classics2/envs/gym_classics2/linear_walks.py 
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
class Walk5(LinearWalk):
    """A 5-state deterministic linear walk. Ideal for implementing random walk
    experiments.

    **Reference:** Sutton and Barto,
    [*Reinforcement Learning: An Introduction* (2nd ed., 2018)](https://incompleteideas.net/book/the-book-2nd.html),
    p. 125, Example 6.2.

    **state:** Discrete position {0, ..., 4} on the number line.

    **actions:** Move left/right.

    **rewards:** +1 for moving right in the extreme right state.

    **termination:** Moving right in the extreme right state or moving left in the
    extreme left state.
    """

    def __init__(self):
        super().__init__(length=5, left_reward=0.0, right_reward=1.0)

gym_classics2.envs.gym_classics2.linear_walks.Walk19

Bases: LinearWalk

Same as 5Walk but with 19 states and an additional -1 reward for moving left in the extreme left state.

Reference: Sutton and Barto, Reinforcement Learning: An Introduction (2nd ed., 2018), p. 145, Example 7.1.

Source code in gym_classics2/envs/gym_classics2/linear_walks.py 
26
27
28
29
30
31
32
33
34
35
36
class Walk19(LinearWalk):
    """Same as `5Walk` but with 19 states and an additional -1 reward for moving left
    in the extreme left state.

    **Reference:** Sutton and Barto,
    [*Reinforcement Learning: An Introduction* (2nd ed., 2018)](https://incompleteideas.net/book/the-book-2nd.html),
    p. 145, Example 7.1.
    """

    def __init__(self):
        super().__init__(length=19, left_reward=-1.0, right_reward=1.0)