Reward Learning

Reward Learning

1. Recap

   • Chemical brain
     • https://www.youtube.com/watch?v=p5zFgT4aofA
   • PET scans
     • Positron Emission Tomography - Measures neurotransmitters - Radiation from radioactively labelled compounds - More activity = increased blood supply = more compounds releasing radiation - Assesses many aspects of function - Not great spatial resolution (but better than EEG, MEG) - Poor temporal resolution (relies on blood response) - Requires injection of radioactive substance - Bulky and costly to make radioisotopes
     • - Radiotracer passed into body = radioactive chemical that releases tiny particles called positrons (injected into vein, or can be inhaled as a gas) - FDG (fluorodeoxyglucose) - similar to glucose but radioactive - body treats in similar way to actual glucose - How parts of body respond to glucose
   • fMRI
     • - Measured MRI signal changes - affected by different magnetic properties of oxygenated and deoxygenated blood - related to neural activity - Good spatial resolution, non-invasive - Better temporal resolution than PET, but worse than EEG/MEG - Expensive etc.

2. Learning

   • Innate instinctual responses are selected (survival of the fittest) - If environment changes then animals need to adapt - need to learn
   • Classical Conditioning
     • Pavlov's
     • CS + UCS = UCR
       • Conditioned stimulus (bell) + unconditioned stimulus (bone) = unconditioned response (salivation)
       • CS = CSR
         • Conditioned stimulus = Conditioned stimulus response Produce an adaptive automatic response to a signla that predicts and event
     • Blocking
       • No new information comes from additional CS (e.g. bell + LIGHT = food) - learns only about bell - learning could only occur if something changes/was omitted (omit what is expected)
       • Karmin
         • Rats - noise and shock already paired - therefore light did not add any new information - more intense shock with introduction to light - better predictor of fearful stimuli than noise - more effective conditioning
   • Operant/instumental conditioning
     • Animals learnt to respond/operate environment to get a reward or avoid punishment - reinforcer = something that is more likely to make a behaviour occur
     • Skinner box
       • Behaviour as consequence of stimulus (but no explanation as to why - behaviourism)

3. Prediction errors

   • -Learning driven by deviations or errors between predicted time and amount of rewards and actual experienced times and magnitudes
   • Drive learning about expected reward to guide decision making
   • Rescorla-Wagner model
     • Assumes learning only occurs if US is surprising - first time get a reward (US) = surprising (you learn something) - Next time you get a US, paired with CS (bell)- learning that CS predicts the US - if CS always indicates US - wight of association increases  - US becomes less surprising = associate strength (V)- Once predicts perfectly (1) - learning will no longer occur- Large US = large asymptote (lambda) - e.g. larger shock
     • Error driven learning
       • Minimise discrepancy between received reward (r) and predicted reward (v)
       • R-W rule
         • Estimates the expected reward using an exponentially weighted average of recently received reward - Influence of past rewards on choice behaviour also shows positive increase in weight
     • What you get MINUS what you expected to get
       • Change in associative strength (weight/predictive value) = what you get (reward) MINUS what you expect to get (expected reward) - can be stronger or weaker depending on salience of stimuli (e.g. magnitude of the reward) --> whether you get more than you expected or less
   • Dopamine
     • Neuromodulator Dopamine and most prominent target- the Striatum (part of basal ganglia) - implicated in motivation and action - DA neurons recorded in primates receiving reward - display error signals
     • Learning driven by deviations or errors 1) Neurons mediate predictive expectation of reward triggered by a CS 2) DA neurons play role in 'teaching' new predictions by signalling occurrences of outcomes that deviate from expected
     • Activity of DA neurons during classical/operant conditioning = similar to theoretical models (Rescorla-Wagner) --> DA neurons learn associations between unexpected events (rewards) and cues (e.g. bell) - Tonic DA levels = sustained release - Phasic release = related to learning associations
     • Dopamine prediction errors
       • DA responses to reward as wighted sun of current and past rewards r - V (current minus wighted average of past) - DA responses to reward as function of prediction error r - V (linear, negative error cut off due to low baseline responses)
     • Dopamine responses
       • - Burst to unexpected reward (before learning has occurred) - Response transfers to reward predictors (on presentation of CS) - Pauses at time of omitted reward (burst at CS, dip when no expected reward)
       • Predictive responses
         • - Reward following 0% predictive cue = burst at reward presentation - 50% - burst of activity at CS and reward presentation - 100% - burst of activity at CS (learning has occurred) - no reward following 100% predictive cue - pause/stop of activity when expected reward is omitted
         • - after repeat pairings, DA neurons change time of activation from reward delivery to cue onset - resembling behavioural reaction from the US to CS - When expected reward omitted, DA neurons depressed below basal firing rate at exact time reward should have occurred - Spike on DA activity = better than predcited - no spike = as predicted - negative spike = worse than predicted (omitted)
     • Mesolimbic DA neurons
       • Activated by range of hedonic rewards (e.g. palatable food, sex, drugs) - related to incentive value of reward - Also activated by pavlovian CSs for reward (predictive cue - neurons activate before hedonic reward occurs)
     • Contingencies/probabilities
       • DA firing can also track probabilities that cue is related to reward
       • Fiorillo (2013)
         • Showed that as the probability that the cue is related to the reward changes, so does the DA response - Support DA can code the discrepancy between predicted and actual rewards - DA neurons had sustained firing (not phasic) especially for the uncertain trials (probability 0.5 - 50%) 
         • 3 important characteristics
           • DA neurons signal: 1) The prediction of future rewards 2) It's unexpected occurrence (increased activity on reward presentation) 3) It's unexpected absence (decreased activity at time reward was expected)
   • Human Studies
     • Ramnania et al
       • Prediction error for free monetary reward in PFC - Examine prediction error related activity for classical conditioning (contingencies between CS and monetary reward  - fMRI study - so cannot measure DA neurons - but showed what areas of the brain responded in similar way to R-W model- examined: 1) the failure of expected rewards 2) the occurence of unexpected rewards (control - predicted event occurred as expected)
       • Results Brain regions track what you expect to get and if you don't get what's expected - brain regions responded if get a surprise reward and if don't get the reward
       • Processes and encodes
         • - Results support view that changes in these circuits do not only represent the processing of predictive stimuli and rewards but actively encode the associations between them
       • Frontotemporal circuits
         • - Distinct changes in separate frontotemproal circuits in relation to prediction error for monetary rewards delivered independently of behaviour as well as unexpected rewards and unexpected omission on rewards
         • medial orbitofrontal cortex
           • Activity changes that are specific to unexpected reward
         • Anterior PFC
           • Changes relate to both types of prediction error (unexpected reward and unexpected omission of reward)
   • Temporal difference errors
     • TD algorithms often used in reinforcement learning to predict a measure of the total amount of reward expected over time (the future) - R-W does not account for timing within the trial - TD learning - time within a trial is taken into account --> provide prediction for each time (t) in the trial a CS is presented and of the total future reward that will be gained in the trial from time to the end of the trial (always updating what you expect to happen - allowing for change)
     • O'Doherty et al (2006)
       • TD learning provides specific predictions - Before learning: positive PE response should occur to reward presentation (UCS) - During learning: should shift to CS - After learning - unexpected reward should lead to a positive PE response at time of reward delivery, unexpected reward omission- negative PE response at expected time of reward delivery - Wanted to see if ventral striatum, orbitofrontal cortex or others were consistent with a temporal difference prediction error signal during appetitive learning
       • Methods
         • Calculated a TD prediction error from real behaviour at the times when the CS was presented and the time of reward, then used this info as a regressor in the fRI analysis - looking for correlations between the change in the TD error (behaviour) and change in brain activity --> show what parts of the brain are tracking the change in learning (TD errors)
       • Findings
         • - Backward shift in the time to peak of the hemodynamic response in the ventral striatum over the course of learning - Reflects a transfer in evoked neuronal activity from time of actual receipt of a reward to the time at which a predictive cue was presented - Output of a PE response from a TD model of learning predicts neuroimagery data - in ventral striatum, orbitofrontal cortex + others
     • Rolls et al (2008)
       • Brain has increasing activity to increasing amounts of reward (shown in O'Doherty) - Wanted to know how a decision is influence if the reward magnitude (RM) is high, but there is a small probability of obtaining the reward - What brain areas can track changes in probability
       • Methods
         • - Expected value (EV) of a choice is calculated, after choice has been made - this can be updated based on a TD prediction error  between the EV and the RM (reward magnitude) obtained - EV = probability of obtaining a reward x RM - Changes in EV, RM and TD = regressors in the fMRI model - Reveals parts of the brian that track these behaviours
       • Results
         • TD error correlated with nucleus accumbens, IFG and midbrain activity - Some areas of the brain (e.g. medial orbitofrontal cortex) respond to reward outcomes (RM) and a prediction that reflects the probability of how much reward will be obtained for a particular choice (EV) - Both signals are crucial fro decision making and could contribute to the calculation of reward prediction (TD) error
         • Also show that the ventral striatum and midbrain are correlated in a probabilistic decision-making task with TD error and how activations in the insula may also be related to decision making by reflecting low EV

4. Implications

   • Understanding the neural correlates of decision making can help us understand psychiatric disorders like depression or schizophrenia, where there is known cognitive dysfunction   http://brain.oxfordjournals.org/content/131/8/2084.long
   • Kumar et al
     • Abnormal temporal difference reward-learning signals in major depression (P. Kumar, G. Waiter, T. Ahearn, M. Milders, I. Reid, J. D. Steele)                   
     • - Adults with MDD receiving long-term antidepressant meds - citalopram (and comparison controls - unmedicated and acutely medicated) - scanned using fMRI during a reward-learning task
     • Findings
       • Patients with MDD - significantly reduced reward-learning signals in ventral striatum (VS), rostral + dorsal anterior cingulate, retrosplenial cortex (RC), midbrain and hippocampus - TD signal increased in brainstem of pateints - Acute antidepressant administration to control - associated with blunted TD signal and brainstem TD not increased
       • Beginning to look at reward-prediction errors (TD) in depression - implications for cognitive dysfunction in depression - teaches us about how current antidepressants interact with the reward signal - help understand why some people might not respond well to antidepressants