Example 1: An associative learning model and blocking¶
This is a beginner's walkthrough of an associative learning model and the blocking effect. The model is an instance of the Rescorla-Wagner learning rule (Rescorla and Wagner, 1972), which is a simple and widely used model of associative learning. The blocking effect (Kamin, 1961) is a well-known phenomenon in the psychology of learning, and it was the original target phenomena for the Rescorla-Wagner model.
The Blocking Effect¶
The blocking effect (Kamin, 1961) is a type of cue competition when learning about one cue is restricted in the presence of another cue that was trained separately. In a simple example, consider a rat that has been trained to associate light (A) with food (+). and subsequently encounters the compound of light and sound (AB) with food (+). Under these conditions, learning about B is restricted. In humans, learning (in associative or contingency learning experiments) is often measured by asking them to rate the likelihood of an outcome (e.g., food) given a cue (e.g., light). Participants rate blocked cues (B) as less likely to result in an outcome than a control cue (e.g. X following Y- and XY+ training). This is the blocking effect.
In this tutorial, we will use the toolbox to fit the Bush and Mosteller (1951) separable error term and the Rescorla-Wagner (1972) summed-error term to data from a blocking experiment. It is a partial recreation of Spicer et al. (2021).
Importing the data and getting it ready for the toolbox¶
import pandas as pd
import numpy as np
import copy
from prettyformatter import pprint as pp
Defining our two models¶
Here we will look at one bad model and one good model of blocking.
In both models, the error is scaled by a global and stimulus-specific learning rates (sometimes thought to represent cue salience). For simplicity, we will assume that the learning rate is the same for all cues and all participants, so we will use a single global learning rate, $\alpha$ and assume that all stimuli have the same salience, therefore we can omit that parameter.
The Bush and Mosteller (1951) rule¶
$$ \Delta V_i = \alpha (R - V_i) $$
The Rescorla-Wagner (1972) rule¶
$$ \Delta V_i = \alpha (R - \sum_{i=0} ^{k} V) $$
Building the models¶
import numpy as np
import pandas as pd
from cpm.models import learning, utils
from cpm.generators import Parameters, Wrapper, Simulator
parameters = Parameters(alpha = 0.2, values = np.array([0.25, 0.25, 0.25, 0.25]))
parameters.values
[0.25 0.25 0.25 0.25]
input = pd.Series({
"trials": np.array([3, 4]),
"feedback": np.array([1]),
})
In the cpm toolbox, you will need to write a function that takes in the parameters and the dataand outputs all the things you want to compute on each trial.
The parameters must be a Parameter object and the input must be a dictionary with the mandatory keys trials and feedback.
The output must be a dictionary with the mandatory keys values and dependent.
Both the input and output dictionaries can have any other keys you want or need.
Bush and Mosteller (1951)'s single linear operator¶
def bush_and_mosteller(parameters, input):
# get parameters
alpha = parameters.alpha
values = np.array(parameters.values)
# get trial-specific inputs
stimulus = input.get("trials")
stimulus = utils.Nominal(target = stimulus, bits = 4)
feedback = input.get("feedback")
# mute all non-presented stimuli values
activations = stimulus * values
# compute what we believe the policy will be
policy = np.sum(activations)
error = learning.SeparableRule(weights=values, feedback=feedback, alpha=alpha, input = stimulus)
error.compute()
values += error.weights[0]
output = {
"activations": activations,
"values": values,
"policy": policy,
"error": error.weights[0],
"dependent": policy,
}
return output
bush_and_mosteller(parameters, input)
/var/folders/w1/7prlf4xs12536zfppmztb1_m0000gn/T/ipykernel_15841/4170433186.py:4: DeprecationWarning: __array__ implementation doesn't accept a copy keyword, so passing copy=False failed. __array__ must implement 'dtype' and 'copy' keyword arguments. values = np.array(parameters.values)
{'activations': array([0. , 0. , 0.25, 0.25]),
'values': array([0.25, 0.25, 0.4 , 0.4 ]),
'policy': np.float64(0.5),
'error': array([0. , 0. , 0.15, 0.15]),
'dependent': np.float64(0.5)}
Rescorla-Wagner (1972)'s summed error term¶
def rescorla_wagner(parameters, input):
# get parameters
alpha = parameters.alpha
values = np.array(parameters.values) # copy to avoid changing the original
# get trial-specific inputs
stimulus = input.get("trials")
stimulus = utils.Nominal(target = stimulus, bits = 4)
feedback = input.get("feedback")
# mute all non-presented stimuli values
activations = stimulus * values
# compute what we believe the policy will be
policy = np.sum(activations)
error = learning.DeltaRule(weights=values, feedback=feedback, alpha=alpha, input = activations)
error.compute()
values += error.weights[0]
output = {
"activations": activations,
"values": values,
"policy": policy,
"error": error.weights[0],
"dependent": policy,
}
return output
rescorla_wagner(parameters, input)
/var/folders/w1/7prlf4xs12536zfppmztb1_m0000gn/T/ipykernel_15841/2133887932.py:4: DeprecationWarning: __array__ implementation doesn't accept a copy keyword, so passing copy=False failed. __array__ must implement 'dtype' and 'copy' keyword arguments. values = np.array(parameters.values) # copy to avoid changing the original
{'activations': array([0. , 0. , 0.25, 0.25]),
'values': array([0.25 , 0.25 , 0.29375, 0.29375]),
'policy': np.float64(0.5),
'error': array([0. , 0. , 0.04375, 0.04375]),
'dependent': np.float64(0.5)}