Basal ganglia reinforcement

Reinforcement learning
1. Reinforcement = selection bias
2. Thorndikes law of effect
  1. “Any act which in a given situation produces satisfaction becomes associated with that situation so that when the situation recurs the act is more likely than before to recur also"
3. In the basal ganglia
  1. Selective disinhibition in the parallel looped architecture component of basal ganglia = selection mechanism
  2. Reinforcement learning are processes which bias future selections
  3. Processes of reinforcement likely to operate within a selection machine
  4. Phasic dopamine widely acknowledged can provide a reinforcement signal
    1. Short latency (70-100ms)
    2. Short duration (~ 100ms) burst of impulses
    3. Elicited by biologically salient stimuli
    4. Defining characteristics of phasic dopamine signals
      1. Fast and short
      2. Mono-phasic
      3. Bi-phasic
      4. Post-gaze shift
      5. Insight
        Sensory-evoked phasic DA responses seem to operate like a time-stamp
        What are the signals in DA target regions at the time of the DA time-stamp ?
        …. these are the signals the timed dopamine input will be interacting with
Reward prediction errors
1. Phasic DA signals similar to reward prediction error term in the temporal difference (TD) reinforcement learning algorithm (Barto, Montague, Dayan)
2. Reward prediction errors = unexpected sensory events that are ‘better’ or ‘worse’ than predicted
3. Reward prediction errors reinforce the selection of actions that will maximise the future acquisition of reward
Action discovery problem
1. Actions are multi-dimensional
  1. Where must the action take place?
  2. When must the action take place?
  3. What exactly must be done to what?
  4. How fast and with what force?
  5. How are critical parameters of different dimensions discovered?
    1. Development of novel actions
      1. Trial and error repetition
        DA makes agent "want" to repeat/reselect preceding movements in preceding contexts
      2. Variation/exploration
        not all contextual/behavioural components in each iteration
      3. Mechanism
        LTP in the prescence of phasic DA
        LTD in the abscence of phasic DA
        provides reinforcement required for system to converge on critical causative components
      4. How do we test if its true?
        A behavioural paradigm to investigate different aspects of action discovery
        1) Mechanisms of reinforcement
        2) Convergence on critical parameters of the critical 3WH dimensions
        Ideal task requirements
        1) Must be able to discriminate learning of WHERE, WHEN, WHAT and HOW dimensions
        2) Difficulty should be continuously variable
        3) Repeated measures
        4) Same task used to investigate comparative competences of a range of subjects – rodent, monkey, man and robot
        5) Should be simple, practical and efficient – different versions to suit experimental context
      5. SEE PPT FOR EXAMPLES - IMPORTANT

Next up

Basal ganglia reinforcement

Description

Resource summary

Media attachments

Similar

	Created by Kristi Brogden almost 10 years ago