DeltaRule#

class DeltaRule(alpha=None, zeta=None, weights=None, feedback=None, input=None, **kwargs)[source]#

Bases: object

DeltaRule class computes the prediction error for a given input and target value.

Parameters:

alphafloat: The learning rate.
zetafloat: The constant fraction of the magnitude of the prediction error.
weightsarray_like: The value matrix, where rows are outcomes and columns are stimuli or features. The values can be anything; for example belief values, association weights, connection weights, Q-values.
feedbackarray_like: The target values or feedback, sometimes referred to as teaching signals. These are the values that the algorithm should learn to predict.
inputarray_like: The input value. The stimulus representation in the form of a 1D array, where each element can take a value of 0 and 1.
**kwargsdict, optional: Additional keyword arguments.

See also

cpm.models.learning.SeparableRule: A class representing a learning rule based on the separable error-term of Bush and Mosteller (1951).

Notes

The delta-rule is a summed error term, which means that the error is defined as the difference between the target value and the summed activation of all values for a given output units target value available on the current trial/state. For separable error term, see the Bush and Mosteller (1951) rule.

The current implementation is based on the Gluck and Bower’s (1988) delta rule, an extension of the Rescorla and Wagner (1972) learning rule to multi-outcome learning. Such that

\[\Delta w_{ij} = \alpha \cdot (\lambda_i - \sum_j w_{ij}) \cdot x_j\]

where \(\Delta w_{ij}\) is the change in weight for the \(j\)-th stimulus for the \(i\)-th outcome, \(\lambda_i\) is the target (feedback) value for the i-th outcome, \(w_{ij}\) is the weights of stimulus \(j\) for the \(i\)-th outcome, \(x_j\) is the j-th stimulus input, and \(\alpha\) is the learning rate. This is consistent with the Rescorla and Wagner (1972)’s learning rule incorporating the summed error term.

References

Gluck, M. A., & Bower, G. H. (1988). From conditioning to category learning: An adaptive network model. Journal of Experimental Psychology: General, 117(3), 227–247.

Rescorla, R. A., & Wagner, A. R. (1972). A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In A. H. Black & W. F. Prokasy (Eds.), Classical conditioning II: Current research and theory (pp. 64-99). New York:Appleton-Century-Crofts.

Widrow, B., & Hoff, M. E. (1960, August). Adaptive switching circuits. In IRE WESCON convention record (Vol. 4, No. 1, pp. 96-104).

Examples

>>> import numpy as np
>>> from cpm.models.learning import DeltaRule
>>> weights = np.array([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]])
>>> teacher = np.array([1, 0])
>>> input = np.array([1, 1, 0])
>>> delta_rule = DeltaRule(alpha=0.1, zeta=0.1, weights=weights, feedback=teacher, input=input)
>>> delta_rule.compute()
array([[ 0.07,  0.07,  0.  ],
       [-0.09, -0.09, -0.  ]])
>>> delta_rule.noisy_learning_rule()
array([[ 0.05755793,  0.09214091,  0.],
       [-0.08837513, -0.1304325 ,  0.]])

This implementation generalises to n-dimensional matrices, which means that it can be applied to both single- and multi-outcome learning paradigms.

>>> weights = np.array([0.1, 0.6, 0., 0.3])
>>> teacher = np.array([1])
>>> input = np.array([1, 1, 0, 0])
>>> delta_rule = DeltaRule(alpha=0.1, weights=weights, feedback=teacher, input=input)
>>> delta_rule.compute()
array([[0.03, 0.03, 0.  , 0.  ]])

compute()[source]#

Compute the prediction error using the delta learning rule. It is based on the Gluck and Bower’s (1988) delta rule, an extension to Rescorla and Wagner (1972), which was identical to that of Widrow and Hoff (1960).

Returns:

ndarray: The prediction error for each stimuli-outcome mapping with learning noise. It has the same shape as the weights input argument.

noisy_learning_rule()[source]#

Add random noise to the prediction error computed from the delta learning rule as specified Findling et al. (2019). It is inspired by Weber’s law of intensity sensation.

Returns:

ndarray: The prediction error for each stimuli-outcome mapping with learning noise. It has the same shape as the weights input argument.

References

Findling, C., Skvortsova, V., Dromnelle, R., Palminteri, S., and Wyart, V. (2019). Computational noise in reward-guided learning drives behavioral variability in volatile environments. Nature Neuroscience 22, 2066–2077

reset()[source]#: Reset the weights to zero.