In various everyday situations humans make judgments about which of several possible causes is responsible for a particular effect. Is thunder the sound you hear when almighty Zeus calls the lesser Gods to order by banging his fist on the table, or is it caused by two electrically charged regions in the atmosphere equalizing? Is the wine sweet because the grapes grew in clay soil or because Bacchus was merciful?

In a series of three blog posts we will introduce a simple learning principle that can simulate human causal judgements rather well. This example is an adapted and simplified version of the learning problem in causal judgement presented in Spellman (1996). She used a situation with two fertilising liquids (of two different colours: blue and red) and counts of growing vegetables (TOMATO and NO_TOMATO, i.e., no plant whatsoever) if they were given no liquid, one of the two liquids, or both liquids, in a growing pot that is considered to be a constant background (i.e., an ambient or context). The participants’ task was to draw causal inferences from what is, in essence, a \(2 \times 2\) table of possibilities (red liquid: yes/no; blue liquid: yes/no; and two outcomes – tomatoes or no plant at all).

This information can be expanded with counts of how frequently each situation occurred (Table 2).

If we expand Table 2 to present each learning trial separately (given the number of repetitions, as specified in the column Frequency), we get Table 3 below.

In Table 3 identical trials are grouped (i.e., repeated as many times as Frequency indicates). We may want to shuffle them to make each trial appear at random (i.e., in an unsystematic order). If we do this, the information would look similar to Table 4.

We now have all the ingredients to simulate the process of iterative learning. That is, we have learning events, which consist of input cues and outcomes. The goal of such learning would be to (gradually) estimate how valuable cues are in discriminating (or predicting) each outcome. We could use pencil and paper to calculate this, and we will actually do this for the first three trials from Table 4.

But before we can get started, we need to have our learning principle handy. Our learning principle is the Rescorla-Wagner (1972) rule which learns about the relationship between a cue and an outcome. (An essentially identical learning rule was proposed by Widrow and Hoff in 1960, later known as the Widrow-Hoff rule or the Least Mean Square method in the field of Machine Learning. The Rescorla-Wagner/Widrow-Hoff rule also corresponds closely to the Delta rule which serves as the basis in many connectionist networks.)

The rule defines the state of learning in terms of a learning weight \(w\) (i.e., strength of connection) between cue \(i\) and the outcome \(j\) at any given time \(t\) (thus, we write: \(w_{ij}^{t}\)). Here, time is not continuous as it appears in reality, but it consists of discrete steps – these steps are the learning Trials from the previous tables. The Rescorla-Wagner rule states that the cue-outcome weight at the next time step (\(w_{ij}^{t+1}\)) will be the sum of weights at the current step (\(w_{ij}^{t}\)) plus some change that is due to the current experience, i.e., learning (\(\Delta w^{t}\)):

\[ w_{ij}^{t+1} = w_{ij}^{t} + \Delta w^{t} \]

Learning differentiates only three possible situations and, consequently, three possible changes. These are:

Although the equations above might look a bit intimidating, especially (2) and (3), most of the terms are shared: the free parameter \(\gamma\) – the learning rate (or speed of learning), and the sum of weights for a given outcome \(j\) (\(\sum{w_{.j}}\)) present at the current time step are part of both (2) and (3). The learning rate, typically, takes on a very small value (e.g., 0.01 or even 0.001). This ensures that adjustments are subtle and learning is gradual. Large values would trigger large adjustments. Somewhat simplified, a large value would imply that a single event can dramatically change predictions regardless of the amount of prior experience. Animals and humans do not do make drastic changes to their predictions but (more or less) trust their experience: if it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck.

The difference between (2) and (3) is that if both cue and outcome are present and the evidence is positive then you subtract the sum of weights (\(\sum{w_{.j}}\)) from 1. Otherwise, when the cue is present, but the outcome is not and you obtain negative evidence, you need to correct for the error in predicting a given outcome (recall that our learning is error-driven) and, thus, you subtract the sum of weights (\(\sum{w_{.j}}\)) from 0.

We are now ready to use the Rescorla-Wagner learning rule to calculate how we might learn to predict which fluid (or maybe a combination of both fluids) is most likely to yield tomatoes.

Petar/Dagmar

References:

Spellman, B. A. (1996). Conditionalizing causality. In D. R. Shanks, K. Holyoak, & D. L. Medin (Eds.), Causal learning (pp. 167-206). San Diego, CA: Academic Press.

Rescorla, R. A., & Wagner, R. A. (1972). A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and non-reinforcement. In H. Black & W. F. Proksay (Eds.), Classical conditioning II (pp. 64-99). New York, NY: Appleton-Century-Crofts.

Widrow, B., & Hoff, M. E. (1960). Adaptive switching circuits. In WESCON Convention Record Part IV (pp. 96-104). New York, NY: Institute of Radio Engineers. [Reprinted in 1988: J. A. Anderson & E. Rosenfeld (Eds.), Neurocomputing: Foundations of Research (pp. 126-134). Cambridge, MA: MIT Press.]