PSYC 318-Lecture#4(II)

learning
1. Stimulus-stimulus learning
  1. learn associations between diff. stimuli
    1. not rely on dopamine reinforcement signals.
    2. only requires one trial
    3. observations + seeing patterns
  2. could theoretically be used to control movements
    1. but this is not how motor learning works: b/c the brain doesn't know how it produced a certain type of neural activity.
2. Procedural Learning
  1. reinforcement mechanisms.
    1. required for motor learning
      1. Most of the neural activity that is produced by our brain trying to influence behaviour is thought to be blocked or b/c the neural activity is not strong enough to get through all the filters in the brain.
        Anytime there is movement, some neural activity was successful in getting through the action selection filter.
    2. every moment, the brain is assessing if things are getting better or worse for the animal.
      1. this decision will result in increases or decreases in a reinforcement signal in the brain, which acts to strengthen or weaken recent neural activity, such that it becomes more or less likely that that neural activity will make it past the action selection filter next time a similar context arises
        Most movement-related neural activity is probably randomly generated
  2. Predicted Error Signal:
    1. signifies: unexpected change in your current
      1. signals brain that credit or blame should be appointed.
        credit or blame should be used to adjust your value estimates of things (cost-benefit)
        your probability of repeating that behaviour in the future.
  3. Predicted Error Signal Formula:
    1. Predicted Error Signal = actual – expected value of current situation
    2. Fluctuations in the reinforcement signal
      1. make it more or less likely that your brain will generate the same neural activity in similar situation
  4. we learn through multiple trial and error
    1. i.e. how to move our muscles
    2. i.e. CPP
      1. unclear if this is SS or PL
  5. Process
    1. Step 1: animals envision what movement they want to make
    2. Step 2: animals reinforce any neural activity that gets them closer to achieving that goal
    3. you don’t need to have a goal for reinforcement learning to improve your life Any randomly generated neural activity that seems to improve your situation in life in any manner, that neural activity would be reinforced
Where in the brain do actions get reinforced?
1. Dopaminergic Projections of the Rat Brain
2. The Dopamine System
  1. Dopamine as a Prediction Error Signal
    1. D. released
      1. estimation of the current moment is better than you anticipated it to be
    2. D. withheld
      1. estimation of the current moment is worse than you anticipated it to be
    3. Dopamine signalling will strengthen recently active glutamatergic synapses
      1. motor commands G.S. encode become more likely to win
    4. As your expectations grow, the dopamine system becomes more and more selective
  2. very few dopamine neurons
    1. large
    2. sends many projections
    3. homogenous group of neurons; all fire at the same time
      1. excluding PFC projecting neurones
    4. unmyelinated & can’t fire v. fast (0-40 Hz)
    5. dopamine
      1. Cleared from extracellular space 100 times slower ( compared to glutamate or GABA )
3. All motor commands sent to the NAc (striatum)
  1. this is the input nucleus of the basal ganglia
    1. information is encoded in excitatory, glutamatergic inputs from the cortex
      1. striatum also receives dopamine
        Glutamate inputs
        what gets reinforced
        (actual information, movements, decision = motor commands)
        Dopamine inputs
        reinforcement signals
Three-Factor Rule
1. DEF
  1. neurons that fire together become eligible for dopamine-induced synaptic plasticity.
  2. changes in dopamine levels
    1. if dopamine is at baseline (there is no learning that occurred)
    2. strengthen or weaken recently active glutamatergic synapses in the striatum.
    3. strength of glutamatergic inputs in the striatum can change when the synapses experience:
      1. Postsynaptic activity
      2. Pre-synaptic activity
      3. Abrupt changes in local dopamine receptor activity
Dopamine Neurons Encode a Prediction Error
1. Full Experiment
  1. guaranteed reward: 100% percent predictive by stimulus, dopamine neurones will fire when stimulus is present, but won’t fire when the reward is presente
  2. unexpected reward: produces the same amount of dopamine neurone activation, as the amount of firing that occurs when they receive a stimulus that tells them they are guaranteed to get a reward.
  3. dopamine neurone activity is reduced (response is muted)
    1. dopamine response occurs when the animal first realizes the session will start
2. Test: Electrophysiological recordings of dopamine neurons in a monkey when it unexpectedly receives food.
  1. raster plot graph
    1. each row is a 2 second long interval
      1. Halfway into each trial, the monkey unexpectedly gets a juice reward.
        Each dopamine neuron fires about 5-10 times across each 1 second long interval
      2. first trial = top row, last trial = bottom row
    2. Black dot = when a midbrain dopamine neuron fired an action potential
  2. summary histogram
    1. summary of how many dopamine neurons fire at any point in time, but the data is summed across 50 trials
    2. 'collapsed raster plots'
  3. 50% prediction = you get half amount of dopamine activity that you would if it was a 100% unexpected reward
    1. will fire to stimulus and again when they receive reward: these two events sums up the amount of dopamine activity that occurs when dopamine is guaranteed or unexpected
  4. guaranteed reward
    1. firing of the dopamine neurones won’t change
  5. The amount that dopamine neurones fire upon stimulus presentation depends on how well the stimulus predicts reward
  6. 25% prediction
    1. if the animal can sense that there is a 25% chance it will receive a reward, the value to the animal is 25% of the kool aid, and then the remaining 75% value of the kool aid is delivery once it receives the reward.
Expectation of Reward Affect Learning
Demonstration from Blocking
1. i.e. tone, light sugar water
Expectation of Reward Affect Dopamine neurone Activity
1. when a reward-predictive stimulus is only presented in conjunction with another stimulus, that has already been learned to be fully predictive of reward, then learning to the second stimulus is blocked
  1. 2nd stimulus is a redundant cue
How does dopamine discriminate between cues when it reinforces neural activity?
Second Order Conditioning
1. 15th dopamine
  1. only fire to the cue, not the reward
2. 5th dopamine
  1. only fire a little to the reward
3. you can see the dopamine signalling moves back further and further to earliest predictor, to the first time the animal has changes in expectations, the first time the animal can predict presentation of a reward
4. people think different types of learning have diff. time window involved
Pleasure Versus Prediction Error
Dopamine has Two Distinct Functions
Phasic versus tonic dopamine signalling
1. phasic dopamine doesn’t correlate well with perceived pleasure
  1. b/c pleasure occurs in a situation dependent manner
  2. unexpected dopamine events always increase phasic dopamine signals
2. dopamine, is a teaching signal that notifies the brain as soon as your anticipated value changes.
3. timing and amount of pleasure of rewarding events was largely anticipated, phasic dopamine signals will not change during the event itself and little to no learning will occur.
4. test: if we opto-genetically stimulate dopamine neurones while they are drinking the reward (sugar water) this should cause animals to learn about the light stimulus
5. stimulated dopamine neurones to make the animal think that something better than expected happened
  1. similar to drugs
Messing with Tonic Dopamine Signalling
1. artificially inc. dopamine receptor
  1. seem to be more engaged with their environment
  2. are more willing to take risks and do hard things to get rewards
  3. seem more motivated
2. if you artificially dec. dopamine receptor
  1. seem less motivated
  2. seem to be less engaged with their environment
  3. seem less motivated
3. if you lose all dopamine receptor activity (severe Parkinson’s) you can’t initiate purposeful movement.
How does dopamine discriminate between cues when it reinforces neural activity?
1. dopamine neurones are needed to learn about antecedent cues
  1. when the stimulus is presented, dopamine neurones fire, that is the moment when the value of the world changes
    1. However, after this moment, there was no change in expectation, therefore, no change in learning needs to occur
Negative Prediction Error
1. current situation is worse than anticipated.
2. Dopamine neurons abruptly stop firing when an expected reward is not received
  1. dopamine firing b/c they withhold the same amount of dopamine that was released previously
    1. amount released previously to predictor was the same amount as what they believe the value of the reward be
Dopamine has 2 distinct forms
1. phasic dopamine signals regulate reinforcement learning by encoding a feedback signal ( a prediction error signal)
  1. Abrupt changes in Phasic dopamine signals drive learning depends on how much base line activity is there
    1. a) per neurone
    2. c) ‘amount of phasic activity’
    3. b) across all neurones
      1. all three can causes changes in resting rate of dopamine
  2. “high gain setting”
    1. high background levels of dopamine
      1. in extracellularl striatum
    2. easy to get actions through the filers, willingness to exert effort
    3. gain setting determines " how excitable neurones are to glutamate inputs"
      1. a) how excitable neurons are within the striatum
      2. b) how responsive they are to glutamate inputs
  3. low gain setting
    1. unmotivated to do anything
  4. learning= phasic
  5. tonic= motivational
    1. general dopamine neurons fire at ~4Hz
    2. slight changes in a) b) c) can cause large changes in the resting (tonic) amount of extracellular dopamine levels.
    3. speed of this tonic activity
    4. number of dopamine neurones that are participating (firing)
    5. amount of phasic activity
    6. influence the ‘gain’ settings in the system
    7. willingness to exert effort: correspond to the animals estimate of overall value of current situation
2. tonic, baseline dopamine levels regulate motivational state
3. pre synaptic activity” glutamate input into the basal ganglia
  1. when that occurs there is a potential to change the strength of that synaptic reaction, strength of synapse dec. when dopamine signals dec. (and vice versa)
4. dopamine neurones usually fire around 4 Hz,
  1. firing rate corresponds to animals motivation and effort)
    1. dopamine is cleared from the extracellular space slowly
      1. there will be more dopamine in the system if a neurones ifs firing at 6Hz than if neurone is firing at 2 Hz

Next up

PSYC 318-Lecture#4(II)

Description

Resource summary

Similar

	Created by Pascale Bockelmann over 7 years ago