Reinforcement learning with a two-armed bandit (cpm.models, cpm.generators, cpm.optimisation)¶

This is an intermediate level tutorial, where we assume that you are familiar with the basics of reinforcement learning and the two-armed bandit problem. In this example, we will apply and fit a reinforcement learning model to a two-armed bandit problem. The model will be consist of a $\epsilon$-greedy policy and a prediction error term.

In the following tutorial, we will use prettyformatter to print some nice and organised output in the Jupyter Notebook. In your python script, you do not need to use prettyformatter.

Two-armed bandits¶

Two-armed bandit is a two-alternative forced-choice reinforcement learning problem, part of the bigger set of multi-armed bandit problems. In these problems, the person is faced with multiple choices, each with a different degree of reward. The goal of the person is to learn which choice is the best and to maximize the reward over time. In this example, we will consider a two-armed bandit problem, where the person is faced with two choices (select an item on the left or the item on the right), each with a different reward. There are 4 different items that can appear in combinations of two, and the reward for each item varies. For example, if item 1 has a chance of 0.7 of giving a reward, then you can expect to receive a reward 70% of the time when you select item 1. The problem is that the underlying structure of the item-reward mapping is unknown.

Import the data¶

First, we will import the data and get it ready for the toolbox.

Let us look at what each column represents:

• index: variable to identify each row - this variable is clutter.
• left: the stimulus presented on the left side.
• right: the stimulus presented on the right side.
• reward_left: the reward received when the left stimulus is selected.
• reward_right: the reward received when the right stimulus is selected.
• ppt: the participant number.
• responses: the response of the participant (1 for right, 0 for left).

Here, we will quickly add a column observed to specify for the toolbox, what is the dependent variable we actually want to predict.

The model description¶

Let each stimulus have an associated value, which is the expected reward that can be obtained from selecting that stimulus. Let also $Q(a)$ be the estimated value of action $a$. We set the starting value for all $Q(a)$ to be nonzero and equally distributed between all stimuli.

In each trial, $t$, there are two stimuli present, so $Q(a)$ could be $Q(\text{left})$ or $Q(\text{right})$, where the corresponding Q values are derived from the associated value of the stimuli present on the left or right. More formally, we can say that the expected value of the action $a$ selected at time $t$ is given by:

\begin{equation} Q_t(a) = \mathbb{E}[R_t | A_t = a] \end{equation}

where $R_t$ is the reward received at time $t$, and $A_t$ is the action selected at time $t$. In each trial $t$, the Softmax choice rule (Bridle, 1990) will select an action (left or right) based on the following policy:

\begin{equation} P(a_t) = \frac{e^{Q_{a,t} \beta}}{\sum_{i = 1}^{k}{e^{Q_{i,t} \beta}}} \end{equation}

where $\beta$ is the inverse temperature parameter, also referred to as choice stochasticity, and $Q_{a,t}$ is the estimated value of the action $a$ at time $t$. $k$ is the number of actions available, and in our case, $k = 2$. The model uses the variant of the delta rule (Rescorla & Wagner, 1972; Rumelhart, Hinton & Williams, 1986) adapted for multi-armed bandit problems where each option has a single dimension (Barto & Sutton, 2018), reducing Rescorla-Wagner's summed error-term to the following equation, similar to single linear operators (Bush and Mosteller, 1955):

\begin{equation} \Delta Q_t(A_t) = \alpha \times \Big[ R_t - Q_t(A_t) \Big] \end{equation}

where $\alpha$ is the learning rate and $R_t$ is the reward received at time $t$, also called a teaching signal and sometimes annotated as $\lambda$. $A_t$ is the action chosen for the trial $t$. Then we update the Q-values, such as:

\begin{equation} Q_{t+1}(A_t) = Q_t(A_t) + \Delta Q_t(A_t) \end{equation}

Building the model¶

In order to use the toolbox, you will have to specify the computations for one single trial. Rest assured, you do not have to build the model from scratch. We have fully-fledged models in cpm.applications that you can use, but we also have all the building blocks in cpm.components that you can use to build your own model.

Parameters¶

Now, before we build the model, we also have to talk about the cpm.Parameter class. This class is used to specify the parameters of the model, including various bounds and priors. Let us specify the parameters for the model.

You can immediately see that we have two types of variables in the Parameters class: one with priors and one without. The ones with defined priors are the ones we will estimate later on, they are freely varying parameters (alpha and temperature). All freely varying parameters have to be defined with priors.

Parameters without priors are part of the initial state of the model, such as starting Q-values for each stimulus. Here we initialized them at 0.25.

That was enough preparation, let us build a model. The model will take in the parameters class and the data, and it will output all variables of interest we specify.

Note: The model will take in a variable (as its second argument) that is in our case a row in the pandas.DataFrame, behaving as a pandas.Series.

The important bit here is that everything we want to store must be specified in the model output. There are also variables we want to update, such as the values, which again, we have to include in the model output. They will be used by other methods in the toolbox to identify key variables.

The important part is the dependent key, which outputs the main thing we fit the model to later.

Fitting to data¶

First we wrap our model specification into a Wrapper, that loops it through the data

Now we have ran the model on a single session, let's investagete the output. cpm is great for this because everything that you might need to look at is stored inside the Wrapper object.

Maybe a better way to do this is to export the simulation details as a pandas.DataFrame:

Fitting the model¶

Now that we have the model, we can fit it to the data.

Simulation¶

We can instantly simulate model output from these results by using the cpm.generator.Simulator class.