RLRW#

class RLRW(data=None, dimensions=2, parameters_settings=None, generate=False)[source]#

Bases: Application

The class implements a simple reinforcement learning model for a multi-armed bandit tasks using a standard update rule calculating prediction error and a Softmax decision rule. The model is an n-dimensional and k-armed implementation of model 3 from Wilson and Collins (2019), which largely corresponds to the model presented by Sutton & Barto (2020) in Chapter 14.

Parameters:

data: pandas.DataFrame: The data to be fit by the model. The data must contain columns for the choice and reward for each dimension. See Notes for more information on what columns should you include.
dimensions: int: The number of distinct stimuli present in the data.
parameters_settings: list-like: The parameters to be fit by the model. The parameters must be specified as a list of lists, with each list containing the value, lower, and upper bounds of the parameter. See Notes for more information on how to specify parameters and for the default settings.
generate: bool: If True, the model samples a choice on each trial from its policy instead of using the observed one, and learns from the reward of the sampled arm. Default is False.

Returns:

cpm.generators.Wrapper: A cpm.generators.Wrapper object. Its outputs include response, the arm chosen on each trial: the sampled one if generate is True, otherwise the observed one. Simulated data can therefore be fitted directly.

Notes

The model implementation uses two parameters:

alpha: the learning rate, which determines how much the model updates its values based on the prediction error.
temperature: the inverse temperature, which determines the choice stochasticity – how sensitive is the model to value differences.

Data must contain the following columns:

response: the choice of the participant from the available options, starting from 0.
arm_n: the stimulus identifier for each option (arms in the bandit task), where n is the option available on a given trial. If there are more than one options, the stimulus identifier should be specified as separate columns of arm_1, arm_2, arm_3, etc. or arm_left, arm_middle, arm_right, etc.
reward_n: the reward given after each options, where n is the corresponding arm of the bandit available on a given trial. If there are more than one options, the reward should be specified as separate columns of reward_1, reward_2, reward_3, etc.

parameters_settings must be a 2D array, like [[0.5, 0, 1], [5, 0, 10]], where the first list specifies the alpha parameter and the second list specifies the temperature parameter. The first element of each list is the initial value of the parameter, the second element is the lower bound, and the third element is the upper bound. The default settings are 0.5 for alpha with a lower bound of 0 and an upper bound of 1, and 5 for temperature with a lower bound of 0 and an upper bound of 10.

The model is defined as follows:

Let each stimulus have an associated value, which is the expected reward that can be obtained from selecting that stimulus. Let also \(Q(a)\) be the estimated value of action \(a\). In each trial, \(t\), there are two stimuli present, so \(Q(a)\) could be \(Q(\text{left})\) or \(Q(\text{right})\), where the corresponding \(Q\) values are derived from the associated value of the stimulus present on the left or right. In each trial \(t\) , the Softmax choice rule (Bridle, 1990) will convert the estimated value of each action into a probability according to the following policy:

\[P(a) = \frac{e^{Q(a)/\tau}}{\sum_{a'} e^{Q(a')/\tau}}\]

where \(\tau\) is the temperature parameter. After the choice is made and feedback is received, the value of the chosen stimulus is updated. The current implementation uses the variant of the delta rule (Rescorla & Wagner, 1972; Rumelhart, Hinton, & Williams, 1986) adapted for multi-armed bandit problems where each option has a single stimulus dimension (Sutton & Barto, 2020), reducing Rescorla-Wagner’s summed error-term to the following equation, similar to (Bush & Mosteller, 1951):

\[\Delta Q(a_t) = \alpha \times [R - Q(a_t)]\]

where \(\alpha\) is the learning rate, \(R\) is the reward received for the chosen action, and \(Q(a_t)\) is the estimated value of the chosen action before updating. The values of unchosen stimuli remain unchanged.

Computation

The model computes all trials of a participant in one call, compiled with numba if numba is installed (pip install cpm-toolbox[numba]) and as plain Python otherwise, with the same results. model is the model function for a single trial, model(parameters, trial), which computes one trial from the states in parameters and returns its outputs.

References

Bridle, J. S. (1990). Probabilistic Interpretation of Feedforward Classification Network Outputs, with Relationships to Statistical Pattern Recognition. In F. F. Soulié & J. Hérault (Eds.), Neurocomputing (pp. 227–236). Springer. https://doi.org/10.1007/978-3-642-76153-9_28

Bush, R. R., & Mosteller, F. (1951). A mathematical model for simple learning. Psychological Review, 58(5), 313–323. https://doi.org/10.1037/h0054388

Rescorla, R. A. (1972). A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and non-reinforcement. Classical Conditioning, Current Research and Theory, 2, 64–69.

Robert C Wilson & Anne GE Collins (2019) Ten simple rules for the computational modeling of behavioral data eLife 8:e49547.

Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323(6088), 533–536. https://doi.org/10.1038/323533a0

Sutton, R. S., & Barto, A. (2020). Reinforcement learning: An introduction (Second edition). The MIT Press.

Examples

>>> from cpm.applications.reinforcement_learning import RLRW
>>> from cpm.datasets import load_bandit_data
>>> data = load_bandit_data()
>>> model = RLRW(data=data[data.ppt == 1], dimensions=4)
>>> model.run()
>>> model.export().head()