PrincetonUniversity / PsyNeuLink / 7235829825

# Princeton University licenses this file to You under the Apache License, Version 2.0 (the "License");

# you may not use this file except in compliance with the License.  You may obtain a copy of the License at:

#     http://www.apache.org/licenses/LICENSE-2.0

# Unless required by applicable law or agreed to in writing, software distributed under the License is distributed

# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and limitations under the License.

# *******************************************  LearningMechanism *******************************************************

Contents

--------

  * `LearningMechanism_Overview`

  * `LearningMechanism_Creation`

      - `LearningMechanism_Automatic_Creation`

      - `LearningMechanism_Explicit_Creation`

  * `LearningMechanism_Structure`

      - `LearningMechanism_InputPorts`

      - `LearningMechanism_Function`

      - `LearningMechanism_OutputPorts`

      - `LearningMechanism_Additional_Attributes`

      - `LearningMechanism_Learning_Configurations`

  * `LearningMechanism_Execution`

  * `LearningMechanism_Class_Reference`

.. _LearningMechanism_Overview:

Overview

--------

A LearningMechanism is a `ModulatoryMechanism <ModulatoryMechanism>` that modifies the `matrix <MappingProjection.matrix>`

parameter of one or more `MappingProjections <MappingProjection>`.  Its function takes one or more `error_signals

<LearningMechanism_Input_Error_Signal>` (usually the output of a `ComparatorMechanism` or one or more other

`LearningMechanisms <LearningMechanism>`), as well as information about the `MappingProjection(s) and activity

<LearningMechanism_Additional_Attributes>` that generated the error(s), and calculates a `learning_signal

<LearningMechanism.learning_signal>` that is used to modify the MappingProjection(s) by way of its

`LearningProjection(s) <LearningProjection>`.  Typically, a LearningMechanism is used to "train" a single

MappingProjection (its `primary_learned_projection`), using the output of the Mechanism to which that

MappingProjection projects (its `output_source`) as the source of the error it attempts to reduce.  A

LearningMechanism can be used to train multiple MappingProjections, by assigning it `additional LearningProjections

<LearningMechanism_Multiple_LearningSignals>`; however, these will all use the same `learning_signal

<LearningMechanism.learning_signal>`, generated by the `primary_learned_projection` and its associated `output_source`.

All of the MappingProjection(s) modified by a LearningMechanism must project from one `ProcessingMechanism

<ProcessingMechanism>` to another in the same `Composition`. The learning components of a Composition can be

displayed using the Composition's `show_graph`show_graph <ShowGraph.show_graph>` method with its

**show_learning** argument assigned `True` or *ALL*.

.. _LearningMechanism_Note

*A Note about the Implementation of Learning*

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The implementation of learning in PsyNeuLink was designed for exposition rather than efficiency. Unlike its

implementation in most other environments -- where the learning algorithm is tightly integrated with the

elements of processing that it modifies --  PsyNeuLink separates it into three constituent components, often including:

an `ObjectiveMechanism` used to evaluate the most proximal source of error; a `LearningMechanism` that uses that error

(or one derived from it by another LearningMechanism) to calculate a learning signal;  and one or more

`LearningProjections <LearningProjection>` that use that learning signal to modify the weight `matrix

<MappingProjection.matrix>` of the `MappingProjection(s) <MappingProjection>` being learned.  This has the advantage

of isolating and exposing the constituent computations, making it clearer what these are and how they operate, and

also making each individually accessible for reconfiguration. However, it comes at the cost of efficiency.  For

efficient execution of supervised forms of learning (e.g., reinforcement learning and backpropagation),

an `AutodiffComposition` can be used, which allows the model to be specified using PsyNeuLink, but actually executes

learning either in a compiled form or using `PyTorch <https://pytorch.org>` (see `Compilation` for additional details).

.. _LearningMechanism_Creation:

Creating a LearningMechanism

----------------------------

A LearningMechanism can be created in any of the ways used to `create Mechanisms <Mechanism_Creation>`.

More commonly, however, LearningMechanisms are created automatically.

.. LearningMechanism_Automatic_Creation:

*Automatic Creation*

~~~~~~~~~~~~~~~~~~~~

A LearningMechanism is created automatically when a Composition's `learning method <Composition_Learning_Methods>` is

called.  In that case, a `LearningSignal`, `LearningProjection`, a `ComparatorMechanism` (in the case of `supervised

learning <>`), and any additional Projections required to implement learning that do not already exist are also

instantiated.  This is described below, under `Learning Configurations <LearningMechanism_Learning_Configurations>`.

A LearningMechanism is also created automatically when either the `tuple specification

<MappingProjection_Learning_Tuple_Specification>` is used to specify learning for a MappingProjection, or a

`LearningProjection` is created without specifying its `sender <LearningProjection.sender>` attribute. However, this

is not advised, and should only used in special circumstances, as properly configuring learning generally requires

the instantiation of several other closely related Components, as described below.

.. _LearningMechanism_Explicit_Creation:

*Explicit Creation*

~~~~~~~~~~~~~~~~~~~

If a LearningMechanism is created explicitly (using its constructor), then its **variable** and **error_sources**

arguments must be specified.  The **variable** must have at leaset three items that are compatible (in number and type)

with the `value <InputPort.value>` of the LearningMechanism's `InputPorts <LearningMechanism_InputPorts>`.  Each

item in **error_sources** must be one of the following: a `ComparatorMechanism`, for `single layer learning

<LearningMechanism_Single_Layer_Learning>` or for the last `MappingProjection` in a `learning Pathway

<Composition_Learning_Pathway>` for `multilayer learning <LearningMechanism_Multilayer_Learning>`; or a

`LearningMechanism`.

.. _LearningMechanism_Learning_Signals:

When a LearningMechanism is created explicitly, it can also be assigned existing LearningSignals and/or specified to

create these, as well as `LearningProjections <LearningProjection>` from these to specified MappingProjections.  These

are specified in the **learning_signals** argument of the LearningMechanism's constructor, using any of the forms

allowed for `specifying a LearningSignal <LearningSignal_Specification>`.

.. _LearningMechanism_Structure:

Structure

---------

A LearningMechanism has three types of `InputPorts <InputPort>`, a learning `function <LearningMechanism.function>`,

and two types of `OutputPorts <OutputPort>`. These are used, respectively, to receive, compute, and transmit the

information needed to modify the MappingProjection(s) for which the LearningMechanism is responsible.  In addition,

it has several attributes that govern and provide access to its operation.  These are described below.

.. _LearningMechanism_InputPorts:

*InputPorts*

~~~~~~~~~~~~~

These receive the information required by the LearningMechanism's `function <LearningMechanism.function>`.  They are

listed in the LearningMechanism's `input_ports <LearningMechanism.input_ports>` attribute.  They have the following

names and roles (shown in the `figure <LearningMechanism_Single_Layer_Learning_Figure>` below):

.. _LearningMechanism_Activation_Input:

* *ACTIVATION_INPUT* - receives the value of the input to the `primary_learned_projection`; that is, the `value

  <Projection_Base.value>` of that MappingProjection's `sender <MappingProjection.sender>`. The value is assigned

  as the first item of the LearningMechanism's `variable <LearningMechanism.variable>` attribute.

.. _LearningMechanism_Activation_Output:

* *ACTIVATION_OUTPUT* - receives the value of the LearningMechanism's `output_source <LearningMechanism.output_source>`;

  that is, the `value <OutputPort.value>` of the `OutputPort` of the *ProcessingMechanism* to which the

  `primary_learned_projection` projects.  By default, the `output_source <LearningMechanism.output_source>`'s

  `primary OutputPort <OutputPort_Primary>` is used.  However, a different OutputPort can be designated in

  the constructor for the `output_source <LearningMechanism.output_source>`, by assigning a `parameter specification

  dictionary <ParameterPort_Specification>` to the **params** argument of its constructor, with an entry that uses

  *MONITOR_FOR_LEARNING* as its key and a list containing the desired OutputPort(s) as its value. The `value

  <InputPort.value>` of the *ACTIVATION_OUTPUT* InputPort is assigned as the second item of the LearningMechanism's

  `variable <LearningMechanism.variable>` attribute.

.. _LearningMechanism_Input_Error_Signal:

* *ERROR_SIGNAL* - this receives the `value <OutputPort.value>` from the *OUTCOME* `OutputPort

  <ComparatorMechanism_Structure>` of a `ComparatorMechanism`, or of the *ERROR_SIGNAL* OutputPort of another

  `LearningMechanisms <LearningMechanism_Output_Error_Signal>`. If the `primary_learned_projection` projects

  to the `TERMINAL` Mechanism of the `Composition` to which it belongs, or is not part of a `multilayer learning

  sequence <LearningMechanism_Multilayer_Learning>`, then the LearningMechanism has a single *ERROR_SIGNAL* InputPort,

  that receives its input from a ComparatorMechanism. If the `primary_learned_projection` is part of a `multilayer

  learning pathway <LearningMechanism_Multilayer_Learning>`, then the LearningMechanism will have one or more

  *ERROR_SIGNAL* InputPorts, that receive their input from the next LearningMechanism(s) in the sequence;  that is,

  the one(s) associated with the `efferents <Port_Base.efferents>` (outgoing Projections) of its `output_source`,

  with one *ERROR_SIGNAL* InputPort for each of those Projections.  The `value <InputPort.value>`\\s of the

  *ERROR_SIGNAL* InputPorts are summed by the LearningMechanism's `function <LearningMechanism.function>` to

  calculate the `learning_signal <LearningMechanism.learning_signal>` (see `below <LearningMechanism_Function>`);

  note that the value of the *ERROR_SIGNAL* InputPort may not be the same as that of the LearningMechanism's

  `error_signal <LearningMechanism.error_signal>` attribute or *ERROR_SIGNAL* `OutputPort

  <LearningMechanism_Output_Error_Signal>` (see `note <LearningMechanism_Error_Signal>` below).  If a LearningMechanism

  has more than one *ERROR_SIGNAL* InputPort, their names are suffixed with a hyphenated index, that is incremented for

  each additional InputPort (e.g., ``error_signal-1``, ``error_signal-2``, etc.).  These are listed in the

  LearningMechanism's `error_signal_input_ports` attribute, and the `value <InputPort.value>` of each is assigned

  as an item of the LearningMechanism's `variable <LearningMechanism.variable>` attribute, beginning with its third

  item (i.e., following the `value <InputPort.value>` of the *ACTIVATION_INPUT* and *ACTIVATION_VALUE* InputPorts).

.. _LearningMechanism_Covariates:

* *COVARIATES* - one or more InputPorts that receive the value of the inputs to `output_source (i.e., the

  *ProcessingMechanism* to which the `primary_learned_projection` projects) other than `primary_learned_projection

  <LearningMechanism_Primary_Learned_Projection>`, if the `output_source <LearningMechanism.output_source>`'s

  `function <Mechanism_Base.function>` takes more than one argument and they affect its derivative. These are

  assigned as a list to the fourth item in the LearningMechanism's `variable <LearningMechanism.variable>`

  attribute, and are used to calculate the `learning_signal <LearningMechanism.learning_signal>` (see `below

  <LearningMechanism_Function>`).

  .. note::

     The LearningMechanism has a separate InputPort for each covariate, however their

     values are combined into a single item of the LearningMechanism's `variable <LearningMechanism.variable>`

     that is provided to its `function <LearningMechanism.function>` at the fourth position.

  .. technical_note::

     Although the *COVARIATES* InputPorts receive Projections from the same senders as the corresponding

     Projections to the `output_source <LearningMechanism.output_source>`, these are for display only,

     and are not to actually compute the LearningSignal, since they are not subject to learning and therefore

     do not provide the same value as the actual Projections they shadow.  Rather, those values are accessed

     directly by the LearningMechanism from the InputPorts of the `output_source <LearningMechanism.output_source>`,

     in a manner similar to its access of the `error_matrices <LearningMechanism.error_matrices>` from the

     `ParameterPort`s of the `error_sources <LearningMechanism.error_sources>`.

The Mechanisms from the which the `value <InputPort.values>`\\s above are received are listed in the

LearningMechanism's `input_source <LearningMechanism.input_source>`, `output_source <LearningMechanism.output_source>`,

and `error_sources <LearningMechanism.error_sources>` attributes, respectively (see

`LearningMechanism_Additional_Attributes` for additional details).

.. _LearningMechanism_Function:

*Learning Function*

~~~~~~~~~~~~~~~~~~~

The `function <LearningMechanism.function>` of a LearningMechanism uses the values received by the Mechanism's

InputPorts (described `above <LearningMechanism_InputPorts>`) to calculate the value of its `learning_signal

<LearningMechanism.learning_signal>` and `error_signal <LearningMechanism.error_signal>` attributes.

.. _LearningMechanism_Learning_Signal:

* `learning_signal` - the set of changes to the `matrix <MappingProjection.matrix>` parameter of the

  `MappingProjections <MappingProjection>` being learned, calculated to reduce the summed value of the

  LearningMechanism's *ERROR_SIGNAL* `InputPort(s) <LearningMechanism_Input_Error_Signal>`.

.. _LearningMechanism_Error_Signal:

* `error_signal <LearningMechanism.error_signal>` - the contribution made by the `primary_learned_projection` to the

  error_signal(s) received by the LearningMechanism's *ERROR_SIGNAL* `InputPort(s)

  <LearningMechanism_Input_Error_Signal>`. It is used by the LearningMechanism's `function <LearningMechanism.function>`

  to calculate the `learning_signal <LearningMechanism.learning_signal>`. Depending upon the context and specific

  `LearningFunction <LearningFunctions>` used, it may also take account of the `value <Mechanism_Base.value>` of its

  `output_source`, as well as the `matrix <MappingProjection.matrix>` parameter of any of the `output_source`'s

  outgoing Projections that are also being learned (these are listed in the LearningMechanism's `error_matrices

  <LearningMechanism.error_matrices>` attribute).  The value of the `error_signal <LearningMechanism.error_signal>`

  is assigned as the value of the LearningMechanism's *ERROR_SIGNAL* `OutputPort

  <LearningMechanism_Output_Error_Signal>`.

  .. _LearningMechanism_Error_Signal_Note:

  .. note::

     A LearningMechanism's *ERROR_SIGNAL* `InputPort(s) <LearningMechanism_Input_Error_Signal>` and its

     *ERROR_SIGNAL* `OutputPort <LearningMechanism_Output_Error_Signal>` may not have the same value.

     The former are the error signal(s) received from a `ComparatorMechanism` or one or more `LearningMechanisms

     <LearningMechanism>`, while the latter is the contribution made to those errors by the `primary_learned_projection`

     and the `output_source`, as calculated by the LearningMechanism's `function <LearningMechanism.function>`

     (see `error_signal <LearningMechanism_Error_Signal>` above).

The default `function <LearningMechanism.function>` of a LearningMechanism is `BackPropagation` (also known as the

*Generalized Delta Rule*; see

`Rumelhart et al., 1986 <http://www.nature.com/nature/journal/v323/n6088/abs/323533a0.html>`_).  However, it can be

assigned any other PsyNeuLink `LearningFunction <LearningFunctions>`, or any other Python function that takes as its

input a list or np.array containing three lists or 1d np.arrays of numeric values, and returns two lists or 1d

np.arrays.  The two values it returns are assigned to the LearningMechanism's `learning_signal

<LearningMechanism.learning_signal>` and `error_signal <LearningSignal.error_signal>` attributes, respectively,

as well as to its two OutputPorts, as described below.

.. _LearningMechanism_OutputPorts:

*OutputPorts*

~~~~~~~~~~~~~~

By default, a LearningMechanism has two `OutputPorts <OutputPort>`, the first of which is named *ERROR_SIGNAL* and

is assigned the value of the `error_signal <LearningMechanism.error_signal>` returned by the LearningMechanism's

`function <LearningMechanism.function>`, and the second of which is a `LearningSignal` and is assigned the value of the

`learning_signal <LearningMechanism.learning_signal>` returned by the `function <LearningMechanism.function>`.

They are each described below:

.. _LearningMechanism_Output_Error_Signal:

* *ERROR_SIGNAL* - this is the `primary OutputPort <OutputPort_Primary>` of a LearningMechanism, and  receives the

  value of the `error_signal <LearningMechanism.error_signal>` used to calculate the `learning_signal

  <LearningMechanism.learning_signal>`.  Its value is assigned as the first item of the LearningMechanism's

  `output_values <LearningMechanism.output_values>` attribute.  If the LearningMechanism is part of a `multilayer

  learning pathway <LearningMechanism_Multilayer_Learning>`, the *ERROR_SIGNAL* OutputPort is assigned a Projection

  to the LearningMechanism for the preceding MappingProjection in the sequence being learned - see `figure

  <LearningMechanism_Multilayer_Learning_Figure>` below).  Note that the `value <OutputPort.value>` of the

  *ERROR_SIGNAL* OutputPort may not be the same as that of the LearningMechanism's *ERROR_SIGNAL* `InputPorts

  <LearningMechanism_Input_Error_Signal>` (see `error_signal <LearningMechanism_Error_Signal>`).

.. _LearningMechanism_LearningSignal:

* `LearningSignal(s) <LearningSignal>` - by default, a LearningMechanism has a single LearningSignal, which is a

  special type of OutputPort that receives the `learning_signal <LearningMechanism.learning_signal>` generated by the

  LearningMechanism's `function <LearningMechanism.function>`, and used to modify the `matrix

  <MappingProjection.matrix>` parameter of the `primary_learned_projection`.  The LearningSignal is assigned as the

  second item in the list of the LearningMechanism's OutputPorts (i.e., of its `output_ports

  <LearningMechanism.output_ports>` attribute), and its `value <LearningSignal.value>` is assigned as the second

  item of the LearningMechanism's `output_values <LearningMechanism.output_values>` attribute.

.. _LearningMechanism_Multiple_LearningSignals:

  **Multiple LearningSignals and LearningProjections.** Though not common, it is possible for a LearningMechanism to

  be assigned more than one `LearningSignal`, and/or more than one `LearningProjection` to its LearningSignal(s). This

  allows multiple MappingProjections to be trained by a single LearningMechanism. Note, however, that all of the

  LearningSignals of a LearningMechanism (and therefore its LearningProjections) use the same

  `learning_signal <LearningMechanism.learning_signal>`, that is calculated based on the LearningMechanism's

  `primary_learned_projection` and its associated `output_source`.  This can be useful in some settings, such as for

  certain forms of `convolutional neural networks <https://en.wikipedia.org/wiki/Convolutional_neural_network>`_.

  If all of the LearningProjections are used to implement the same form of `modulation <ModulatorySignal_Modulation>`,

  (determined by their LearningSignals' `modulation <LearningSignal.modulation>` attribute), then they should be

  assigned to a single LearningSignal. Multiple LearningProjections can be assigned to a LearningSignal by specifying

  them in the **projections** argument of its constructor, or the *PROJECTIONS* entry of a dictionary assigned to its

  **params** argument); however, the `matrix <MappingProjection.matrix>` parameter for the MappingProjection to which

  they project must have the same shape for all of them. If different forms of modulation are required to train

  different MappingProjections, then multiple LearningSignals should be assigned to the LearningMechanism, each

  specified for one of the required types of modulation, and then LearningProjections assigned to them for the

  corresponding MappingProjections. Multiple LearningSignals can be specified for a LearningMechanism by including

  them in a list assigned to the **learning_signals** argument of the LearningMechanism's constructor.

  COMMENT:

  The `learning_rate <LearningSignal.learning_rate>` for each LearningSignal, and the `learning_rate

  <LearningProjection.learning_rate>` for each of its `LearningProjections <LearningProjection>`, can all be assigned

  different values (with the latter taking precedence over the former).  If none of these are specified, the

  `learning_rate <LearningMechanism.learning_rate>` of the LearningMechanism is used (see `below

  <LearningMechanism_Learning_Rate>`).

  COMMENT

  All of the LearningSignals of a LearningMechanism are listed in its `learning_signals` attribute.  Because these

  are `OutputPorts <OutputPort>`, they are also listed in the `output_ports <LearningMechanism.output_ports>`

  attribute, after the *ERROR_SIGNAL* OutputPort.  All of the LearningMechanism's LearningProjections (that is, those

  belonging to all of its LearningSignals) are listed in its `learning_projections` attribute.

.. _LearningMechanism_Additional_Attributes:

*Additional Attributes*

~~~~~~~~~~~~~~~~~~~~~~~

In addition to its `InputPorts <LearningMechanism_InputPorts>`, `function <LearningMechanism_Function>` and

`OutputPorts <LearningMechanism_OutputPorts>`, a LearningMechanism has the following attributes that

refer to the Components being learned and/or its operation:

.. _LearningMechanism_Primary_Learned_Projection:

* `primary_learned_projection` - the `MappingProjection` with the `matrix <MappingProjection.matrix>` parameter for

  which the `learning_signal <LearningMechanism.learning_signal>` is computed.  This is always the first Projection

  listed first in LearningMechanism's `learned_projections` attribute.

..

* `learned_projections` - a list of all of the MappingProjections to which the LearningMechanism sends a

  `LearningProjection`, listed in the order of the `LearningSignal(s) <LearningSignal>` to which they belong,

  as those are listed in the LearningMechanism's `learning_signals <LearningMechanism.learning_signals>` attribute.

..

* `learning_enabled <LearningMechanism.learning_enabled>` - determines whether and when the LearningMechanism's

  `learning_projections <LearningMechanism.learning_priojections>` are executed (see `learning_enabled

  <LearningMechanism.learning_enabled>` for additional details).

..

* `input_source` - the `Mechanism <Mechanism>` that sends the `primary_learned_projection`, and projects to the

  LearningMechanism's *ACTIVATION_INPUT* `InputPort <LearningMechanism_Activation_Input>`.

..

* `output_source` - the `Mechanism <Mechanism>` that receives the `primary_learned_projection`, and  provides the

  input to the LearningMechanism's *ACTIVATION_OUTPUT* `InputPort <LearningMechanism_Activation_Output>`.

..

* `error_sources` - a `ComparatorMechanism`, `LearningMechanism`, or list of them that calculate the error signal(s)

  provided to the LearningMechanism's *ERROR_SIGNAL(s)* `InputPort(s) <LearningMechanism_Input_Error_Signal>`.

..

* `error_matrices` - the `matrix <MappingProjection.matrix>` parameters of the Projections associated with the

  `error_sources <LearningMechanism.error_sources>`;  that is, of any of the `output_source

  <LearningMechanism.output_source>`'s `efferents <Port_Base.efferents>` that are also being learned.

..

* `covariates_sources` - the `InputPort`s of `Mechanism`(s) that provide covariates used in calculating the derivative

  of the `output_source <LearningMechanism.output_source>`'s `function <Mechanism_Base.function>` (see `above

  `LearningMechanism_Covariates`), and project to its *COVARIATES* `InputPort <LearningMechanism_Covariates>`.

..

* `modulation` - the default value used for the `modulation <LearningSignal.modulation>` attribute of

  LearningMechanism's `LearningSignals <LearningSignal>` (i.e. those for which it is not explicitly specified).

  This determines the way in which the `learning_signal <LearningMechanism.learning_signal>` is used to modify the

  `matrix <MappingProjection.matrix>` parameter of the `learned_projections`.  By default its value is

  Modulation.ADD, which causes the weight changes in the `learning_signal` to be added to the current value of the

  `matrix <MappingProjection.matrix>` parameter (see `LearningMechanism_Execution` for a description of how the

  modifications are executed).

.. _LearningMechanism_Learning_Rate:

* `learning_rate <LearningMechanism.learning_rate>` - specifies the `learning_rate <LearningFunction.learning_rate>`

  parameter used by the LearningMechanism's `function <LearningMechanism.function>` which, for most forms of learning,

  uses it to multiply the weight change matrix before returning it as the `learning_signal

  <LearningMechanism.learning_signal>`.  The value can be specified in the **learning_rate** argument of the

  LearningMechanism's constructor, or in the constructor for its `function <LearningMechanism.function>`.  If both

  are specified, the specification for the function takes precedence; in either case, the value of the learning_rate

  parameter is always the same for the LearningMechanism and its function.  If neither is specified, then it inherits

  its value from any specifications made for the `Composition` or a `learning Pathway <Composition_Learning_Pathway>`

  of the Composition to which it belongs (see `Composition_Learning_Rate` for additional details). However, after the

  Composition and/or its `learning pathways <Composition_Learning_Pathway>` have been constructed, specifying the

  `learning_rate <LearningMechanism.learning_rate>` for a LearningMechanism (or its function) overrides any

  specifications made for the Composition or its learning pathways, including in calls the Composition's `learn

  <Composition.learn>` method.  In this way, individual LearningMechanisms can be assigned specific learning rates

  that apply whenever those are executed. If the learning_rate is not explicitly specified anywhere, the `default

  value <Parameter_Defaults>` for the LearningMechanism's `function <LearningMechanism.function>` is used.

  COMMENT: TBI

  A `learning_rate` parameter can also be specified for individual `LearningSignals <LearningSignal>` and/or their

  associated `LearningProjections <LearningProjection>`. Those have a direct multiplicative effect on the

  `learning_signal <LearningProjection.learning_signal>` of the LearningSignal and/or it LearningProjections (see

  `LearningSignal learning_rate <LearningSignal_Learning_Rate>` for additional details).

  A `learning_rate` parameter can also be specified for individual `LearningProjections <LearningProjection>`.

  Those have a direct multiplicative effect on the `learning_signal <LearningProjection.learning_signal>` of the

  LearningSignal (see `LearningProjection_Function_and_Learning_Rate>` for additional details).

  COMMENT

.. _LearningMechanism_Learning_Configurations:

COMMENT:

@@@ THE FOLLOWING SECTIONS SHOULD BE MOVED TO THE "USER'S MANUAL" WHEN THAT IS WRITTEN

COMMENT

*Learning Configurations*

~~~~~~~~~~~~~~~~~~~~~~~~~

When learning is enabled for a `RecurrentTransferMechanism <RecurrentTransferMechanism_Learning>` (for `unsupervised

learning <Composition_Learning_Unsupervised>`) or using the `learning method <Composition_Learning_Methods>` of a

Composition, all of the Components required for learning are created automatically. The types of Components that are

generated depend on the type of learning specified and the configuration of the `Composition <Composition>`, as

described below. All of the learning Components of a Composition can be displayed using its `show_graph

<ShowGraph.show_graph` method with the **show_learning** argument assigned `True` or *ALL*.

.. _LearningMechanism_Single_Layer_Learning:

Single layer learning

^^^^^^^^^^^^^^^^^^^^^

This configuration occurs when `unsupervised learning <Composition_Learning_Unsupervised>` is used, or `supervised

learning <Composition_Learning_Supervised>` is used for a pathway in a Composition with only two Mechanisms (i.e.,

for the Projection between them).  In this case, a single `ComparatorMechanism` and LearningMechanism are created

(if they do not already exist) as well as the following MappingProjections:

* from an `OutputPort` of the LearningMechanism's `output_source` to the ComparatorMechanism's *SAMPLE* `InputPort

  <ComparatorMechanism_Structure>`.  By default, the `primary OutputPort <OutputPort_Primary>` of the

  `output_source` is used; however, this can be modified by specifying its *MONITOR_FOR_LEARNING* parameter

  (see `above <LearningMechanism_Activation_Output>`).

..

* from the `TARGET_MECHANISM <Composition_Learning_Components>` in the Composition to the ComparatorMechanism's

  *TARGET* `InputPort <ComparatorMechanism_Structure>`;

..

* from the ComparatorMechanism's *OUTCOME* `OutputPort <ComparatorMechanism_Structure>` to the

  LearningMechanism's *ERROR_SIGNAL* `InputPort <LearningMechanism_Activation_Input>`.

In addition, a `LearningProjection` is created from the `LearningSignal<LearningMechanism_LearningSignal>` for the

`primary_learned_projection` to the `ParameterPort` for the `matrix <MappingProjection.matrix>` of the

`primary_learned_projection`.  Because this configuration involves only a single layer of learning, *no* Projection

is created or assigned to the LearningMechanism's *ERROR_SIGNAL* `OutputPort <LearningMechanism_Output_Error_Signal>`.

.. _LearningMechanism_Single_Layer_Learning_Figure.svg:

    **Components for Single Layer Learning**

    .. figure:: _static/LearningMechanism_Single_Layer_Learning_fig.svg

       :alt: Schematic of Mechanisms and Projections involved in learning for a single MappingProjection

       Components generated by a call to a `learning method <Composition_Learning_Methods>` of a Composition

       (e.g., ``add_backpropagaption_learning_pathway(pathway=[X, Y]``), labeled by type of Component (in **bold**,

       camelCase), NodeRole(s) assigned (*italics*), learning component type used as key in dictionary returned by

       the learning method (*UPPER_CASE* italics, outside of object) and, where relevant, the name of the attribute

       of the LearningMechanism with which it is associated (*italicized* lower case, outside of object). The

       ComparatorMechanism and LearningMechanism are shown with their InputPorts, OutputPorts and functions

       diagrammed.

.. _LearningMechanism_Multilayer_Learning:

Multilayer learning

^^^^^^^^^^^^^^^^^^^

This refers to learning in a pathway that has three or more Mechanisms in a sequence, and therefore two or more

MappingProjections that are being learned.  The learning components for such a configuration are created

automatically when a `learning method <Composition_Learning_Methods>` of a Composition is used that supports

multilayer learning (at present, this is only `add_backpropagation_learning_pathway`).  If the learning components

are `constructed explicitly <LearningMechanism_Explicit_Creation>` for a multilayer learning configuration,

then each LearningMechanism must use a `LearningFunction <LearningFunctions>` that can calculate the influence

that each MappingProjection and its output have on the error that the LearningMechanism receives from the previous

one(s) in the sequence (e.g., `BackPropagation`).  Furthermore, the construction of the other `learning-related

Components <Composition_Learning_Components>` associated with each LearningMechanism depend on the position of its

`primary_learned_projection` and `output_source <LearningMechanism.output_source>` in the sequence.  If it is the

last one in the sequence, it is treated in the same way the LearningMechanism in a `single layer learning configuration

configuration <LearningMechanism_Single_Layer_Learning>`.  This is the  case if the `output_source` is a standalone

Mechanism or the `TERMINAL` Mechanism of a Composition. In these cases, as for single layer learning, a

`ComparatorMechanism` is created that receives the output of the `output_source` as well as the target for learning

(see `TARGET Mechanisms <LearningMechanism_Targets>` below), and projects to a LearningMechanism that is created for

the `primary_learned_projection`.  For all other MappingProjections being learned in the

sequence, the following additional MappingProjections are created for learning (shown in the `figure

<LearningMechanism_Multilayer_Learning_Figure>` below):

* from the `input_source <LearningMechanism.input_source>` to the LearningMechanism's *ACTIVATION_INPUT* `InputPort

  <LearningMechanism_Activation_Input>`;

..

* from the `output_source <LearningMechanism.output_source>` to the LearningMechanism's *ACTIVATION_OUTPUT* `InputPort

  <LearningMechanism_Activation_Output>`;

..

* from the *ERROR_SIGNAL* `OutputPort <LearningMechanism_Output_Error_Signal>` of each of the LearningMechanism's

  `error_sources <LearningMechanisms.error_sources>` to each of its corresponding *ERROR_SIGNAL* `InputPort(s)

  <LearningMechanism_Input_Error_Signal>`;

* from the `covariates_source <LearningMechanism.covariates_source>` to the LearningMechanism's *COVARIATES*

  `InputPort`(s)  if the `function <Mechaism_Base.function>` of the `output_source <LearningMechanism.output_source>`

  takes more than one argument and they affect its derivative (see `above <LearningMechanism_Covariates>`).

In addition, a `LearningProjection` is created from the `LearningSignal <LearningMechanism_LearningSignal>` for the

`primary_learned_projection` of each LearningMechanism in the sequence, to the `ParameterPort` for the `matrix

<MappingProjection.matrix>` of the `primary_learned_projection`.  If the `primary_learned_projection` is the first in

the sequence, then *no* Projection is created or assigned to its LearningMechanism's *ERROR_SIGNAL* `OutputPort

<LearningMechanism_Output_Error_Signal>`.

.. _LearningMechanism_Multilayer_Learning_Figure:

    **Components for Multilayer Learning**

    .. figure:: _static/LearningMechanism_Multilayer_Learning_fig.svg

       :alt: Schematic of Mechanisms and Projections involved in learning for a sequence of MappingProjections

       Components generated by a call to a `learning method <Composition_Learning_Methods>` of a Composition for a

       sequence with three ProcessingMechanisms (and therefore two MappingProjections to be learned; e.g.,

       ``add_backpropagaption_learning_pathway(pathway=[X, Y, Z]``).  Components are labeled by: type of Component (in

       **bold**, camelCase); NodeRole(s) assigned (*italics*) to it; name of learning component type (used as key in

       dictionary returned by the learning method; *UPPER_CASE* italics, outside of object); and, where relevant,

       the name of the attribute of the LearningMechanism with which it is associated (*italicized* lower case,

       outside of object). The ComparatorMechanism and LearningMechanism are shown with their InputPorts, OutputPorts

       and functions diagrammed.

.. _LearningMechanism_Targets:

`TARGET` and `OBJECTIVE` Mechanisms**.  When `supervised learning <Composition_Learning_Supervised>` is

implemented using one of a Composition's `learning methods <Composition_Learning_Methods>`, it automatically creates a

number of  `learning-related Components <Composition_Learning_Components>`.  This includes a `TARGET_MECHANISM`, that

receives the target stimulus specifed in the **inputs** argument of the Composition's `learn <Composition.learn>`

method; and a `OBJECTIVE_MECHANISM`, that computes the error_signal for the sequence.  The output of the

`OBJECTIVE_MECHANISM` is the `error_source <LearningMechanism.error_sources>` for the last MappingProjection in the

learning pathway.  If a multilayer learning pathway is implemented explicitly, it must include these Components.

.. _LearningMechanism_Execution:

Execution

---------

LearningMechanisms that implement `unsupervised learning <Composition_Learning_Unsupervised>`

(`AutoassociativeLearningMechanisms <AutoAssociativeLearningMechanism>`) execute when the `RecurrentTransferMechanism`

with which they are associated executes.  LearningMechanisms that are part of a `supervised learning pathway

<Composition_Learning_Supervised>` execute after all of the ProcessingMechanisms in the `Composition` to which they

belong have executed.  When a LearningMechanism is executed, it uses the `value <InputPort.value>`(s) of its

*ERROR_SIGNAL* `InputPort(s) <LearningMechanism_Input_Error_Signal>` to calculate changes to the weights of the

`matrix <MappingProjection.MappingProjection.matrix>` parameter of its `primary_learned_projection

<LearningMechanism.primary_learned_projection>` and any of its other `learned_projections

<LearningMechanis.learned_projections>`. Those weight changes are assigned as the LearningMechanism's

`learning_signal <LearningMechanism.learning_signal>` attribute, the `value <LearningSignal.value>` of each of its

`LearningSignals <LearningMechanism_LearningSignal>`, and as the `value <LearningProjection.value>` of each of their

LearningProjections.  That value is used, in turn, to modify the `value <ParameterPort.value>` of the *MATRIX*

`ParameterPort` of each of the MappingProjections being learned (listed in the LearningMechanism's

`learned_projections <LearningMechanism.learned_projections>` attribute).

Each ParameterPort uses the value it receives from the `LearningProjection` that projects to it to modify the

parameter of its `function <ParameterPort.function>`, in a manner specified by the `modulation

<LearningSignal.modulation>` attribute of the `LearningSignal` from which it receives the LearningProjection (see

`Modulation <ModulatorySignal_Modulation>` for a description of how modulation operates). By default, the

`modulation <LearningSignal.modulation>` attribute of a LearningSignal is `ADDITIVE`, the `function

<ParameterPort.function>` of a *MATRIX* ParameterPort for a MappingProjection is `AccumulatorIntegrator`,

and the parameter it uses for additive modulation is its `increment <AccumulatorIntegrator.increment>` parameter.

These assignments cause the value of the LearningProjection to be added to the previous value of the *MATRIX*

ParameterPort, thus incrementing the weights by an amount specified by the LearningMechanism's `learning_signal

<LearningMechanism.learning_signal>`. Note, that the changes to the `matrix

<MappingProjection.MappingProjection.matrix>` parameter itself do not take effect until the next time the

`learned_projection` is executed (see `Lazy Evaluation <Component_Lazy_Updating>` for an explanation of

"lazy" updating).

A LearningMechanism's `function <LearningMechanism.function>` also computes an `error_signal

<LearningMechanism.error_signal>` that is assigned as the `value <OutputPort.value>` of its *ERROR_SIGNAL*

`OutputPort <LearningMechanism_Output_Error_Signal>`;  in a `multilayer learning configuration

<LearningMechanism_Multilayer_Learning>`, that value is provided to the *ERROR_SIGNAL* `InputPort(s)

<LearningMechanism_Input_Error_Signal>` of the LearningMechanism(s) for the preceding MappingProjection(s)

being learned in the sequence.

.. _LearningMechanism_Class_Reference:

Class Reference

---------------

"""

    ADDITIVE, ASSERT, ENABLED, INPUT_PORTS, \

    LEARNING, LEARNING_MECHANISM, LEARNING_PROJECTION, LEARNING_SIGNAL, LEARNING_SIGNALS, MATRIX, \

    MODULATION, NAME, OUTPUT_PORT, OWNER_VALUE, PARAMS, PROJECTIONS, REFERENCE_VALUE, SAMPLE, PORT_TYPE, VARIABLE

    convert_to_list

    'ACTIVATION_INPUT', 'ACTIVATION_INPUT_INDEX', 'ACTIVATION_OUTPUT', 'ACTIVATION_OUTPUT_INDEX',

    'COVARIATES', 'COVARIATES_SOURCES', 'DefaultTrainingMechanism',

    'ERROR_SIGNAL', 'ERROR_SIGNAL_INDEX', 'ERROR_SOURCES',

    'LearningMechanism', 'LearningMechanismError', 'input_port_names', 'output_port_names'

]

    """Evaluate whether spec is a valid learning specification

    Return `True` if spec is LEARNING or a valid projection_spec (see Projection_Base._is_projection_spec)

    Otherwise, return `False`

    """

    # MODIFIED 11/28/17 OLD:

        else:

                                       type=LEARNING_PROJECTION,

                                       include_matrix_spec=include_matrix_spec)

    """

        Denotes whether LearningMechanism requires a target input.

    Attributes

    ----------

    UNSUPERVISED

        implements (and requires a Projection to) a *ERROR_SIGNAL* InputPort.

    SUPERVISED

        does not implement a *ERROR_SIGNAL* InputPort.

    """

    """

        Denotes

    Attributes

    ----------

    EXECUTION_PHASE

        LearningMechanism (and associated `LearningProjections(s) <LearningProjection>`) executed during the

        `execution phase <Composition_Execution>` of the Composition to which they belong, usually immediately after

        execution of of the `Mechanism <Mechanism>` that receives the `primary_learned_projection`

    LEARNING_PHASE

        LearningMechanism (and associated `LearningProjections(s) <LearningProjection>`) executed during the

        `learning phase <Composition_Execution>` of the Composition to which they belong.

    """

# Parameters:

# Used to index variable:

# Used to name input_ports and output_ports:

    """

    LearningMechanism(                    \

        variable,                         \

        covariates_sources,               \

        error_sources,                    \

        function=BackPropagation,         \

        learning_rate=None,               \

        learning_signals=LEARNING_SIGNAL, \

        modulation=ADDITIVE,              \

        learning_enabled=True)

    Subclass of ModulatoryMechanism that modifies the `matrix <MappingProjection.matrix>` parameter of a

    `MappingProjection`. See `Mechanism <Mechanism_Class_Reference>` for additional arguments and attributes.

    COMMENT:

        Description:

            LearningMechanism is a subtype of the ModulatoryMechanism Type of the Mechanism Category of Component

            It implements a Mechanism that calculates changes to a Projection's parameters.

            Its function takes the output of an ObjectiveMechanism and generates a

            learning_signal (ndarray of parameter changes) to be used by the recipient of a LearningProjection

            that projects from the LearningMechanism to a MappingProjection.

        # DOCUMENT: ??NOT SURE WHETHER THIS IS STILL RELEVANT

        #    IF objective_mechanism IS None, IT IS LEFT UNSPECIFIED (FOR FURTHER IMPLEMENTATION BY COMPOSITION)

        #    THESE ARE HANDLED BY A MODULE METHOD _instantiate_objective_mechanism (AS PER OBJECTIVE MECHANISM):

        #        IF objective_mechanism IS SPECIFIED AS ObjectiveMechanism, AN OBJECTIVE MECHANISM IS CREATED FOR IT

        #        IF objective_mechanism IS SPECIFIED AS A MECHANISM OR OutputPort,

        #               a MappingProjection WITH AN IDENTITY MATRIX IS IMPLEMENTED FROM IT TO THE LearningMechanism

        Learning function:

            Generalized delta rule:

            dE/dW  =          learning_rate   *    dE/dA          *       dA/dW             *    I

            weight = weight + (learning_rate  * error_derivative  *  activation_derivative  *  input)

            for sumSquared error fct =        (target - output)

            for logistic activation fct =                           output * (1-output)

            where:

                output = activity of output (target) units (higher layer)

                input = activity of sending units (lower layer)

            Needs:

            - activation_derivative:  get from FUNCTION of sample_activation_mechanism/receiver_mech

                                      assumes derivative of Logistic unless otherwise specified

            - error_derivative:  get from FUNCTION of error_sources/next_level_mech;  but handled in ObjectiveMechanism

    COMMENT

    Arguments

    ---------

    variable : List or 2d np.array

        it must have three items that correspond to the three values required by the LearningMechanism's `function

        <LearningMechanism.function>`;  they must each be compatible (in number and type) with the `value

        <InputPort.value>` of the corresponding `InputPort <LearningMechanism_InputPorts>` (see `variable

        <LearningMechanism.variable>` for additional details).

    covariates_sources : InputPort or list of them : default None

        specifies the `InputPort`(s) of the LearningMechanism's `output_source <LearningMechanism.output_source>`

        other than the one to which the `primary_learned_projection` projects. (see `LearningMechanism_Covariates`

        for additional details).

    error_sources : ComparatorMechanism, LearningMechanism, OutputPort or list of them

        specifies the source(s) of the error signal(s) used by the LearningMechanism's `function

        <LearningMechanism.function>`.  Each must be a `ComparatorMechanism` for `single layer learning

        <LearningMechanism_Single_Layer_Learning>`, or for the last `MappingProjection` in a learning pathway in

        `multilayer learning <LearningMechanism_Multilayer_Learning>`;  otherwise they must be a `LearningMechanism`

        or the *ERROR_SIGNAL* OutputPort of one.

    function : LearningFunction or function : default BackPropagation

        specifies the function used to calculate the LearningMechanism's `learning_signal

        <LearningMechanism.learning_signal>` and `error_signal <LearningMechanism.error_signal>` attributes.  It's

        `variable <Function_Base.variable>` must have three items, each of which must be a list or 1d array of

        numeric values, corresponding to values provided by the LearningMechanism's *ACTIVATION_INPUT*,

        *ACTIVATION_OUTPUT*, and *ERROR_SOURCES* InputPorts, respectively (see `LearningMechanism_InputPorts

        `LearningMechanism_Function` and `LearningMechanism_InputPorts` for additional details). If supports an

        activation function that takes more than one argument that impact its derivative, then it must also accept a

        potential fourth keyword argument (``covariates``), that is a list or 1d array of numeric values provided by

        the LearningMechanism's *COVARIATES* InputPort(s) (see `LearningMechanism_Covariates` for additional details).

    learning_rate : float : default None

        specifies the learning rate for the LearningMechanism (see `learning_rate <LearningMechanism.learning_rate>`

        for details).

    learning_signals : List[parameter of Projection, ParameterPort, Projection, tuple[str, Projection] or dict] :

        default *LEARNING_SIGNAL* specifies the parameter(s) to be learned (see `learning_signals

        <LearningMechanism.learning_signals>` for details).

    modulation : str : default ADDITIVE

        specifies the default form of modulation used by the LearningMechanism's LearningSignals,

        unless they are `individually specified <LearningSignal_Specification>`.

    learning_enabled : bool or Enum[ONLINE|AFTER] : True

        specifies whether and when the LearningMechanism's `LearningProjections <LearningProjection>` are executed

        (see `learning_enabled <LearningMechanism.learning_enabled>` for additional details).

    Attributes

    ----------

    COMMENT:

        componentType : LEARNING_MECHANISM

    COMMENT

    variable : 2d np.array

        has three items that serve as the template for the three inputs required by the LearningMechanism's `function

        <LearningMechanism.function>` (corresponding to its three `InputPorts <LearningMechanism_InputPorts>`:

        the input to the `primary_learned_projection` (from `input_source`), the output of the Mechanism to which

        that projects (i.e., of `output_source`); and the error signal (from `LearningMechanism.error_sources`).

    input_ports : ContentAddressableList[OutputPort]

        list containing the LearningMechanism's three `InputPorts <LearningMechanism_InputPorts>`:

        *ACTIVATION_INPUT*, *ACTIVATION_OUTPUT*, *ERROR_SIGNAL*, and possible its *COVARIATES*.

    error_signal_input_ports : list[InputPorts]

        list of InputPorts that receive error_signals from the LearningMechanism's `error_sources

        <LearningMechanism.error_sources>`.

    input_source : ProcessingMechanism

        the Mechanism that sends the `primary_learned_projection`, and projects to the

        LearningMechanism's *ACTIVATION_INPUT* `InputPort <LearningMechanism_Activation_Input>`.

    output_source : ProcessingMechanism

        the Mechanism that receives the `primary_learned_projection`, and  projects to the

        LearningMechanism's *ACTIVATION_OUTPUT* `InputPort <LearningMechanism_Activation_Output>`.

    covariates_sources : List[InputPort]

        the `InputPort`(s) of the LearningMechanism's `output_source <LearningMechanism.output_source>`

        other than the one to which the `primary_learned_projection` projects;  these are used as

        covariates in the calculation of the derivative of `output_source <LearningMechanism.output_source>`\\'s

        `function <LearningMechanism.function>` with respect to `value <MappingProjection.value>` of the

        `primary_learned_projection` (see `LearningMechanism_Covariates` for additional details).

    covariates_values : List[1d np.array]

        the values of the InputPorts to which the `covariates_sources <LearningMechanism.covariates_sources>` project;

        passed to the LearningMechanism's `function <LearningMechanism.function>` as the *COVARIATES* item of its

        `variable <LearningMechanism.variable>`, and assigned as the `value <InputPort.value>` of the LearningMechanism's

        *COVARIATES* `InputPort <LearningMechanism_Covariates>`s.

    error_sources : list[ComparatorMechanism or LearningMechanism]

        the Mechanism(s) that calculate the error signal(s) provided to the

        LearningMechanism's *ERROR_SIGNAL(s)* `InputPort(s) <LearningMechanism_Input_Error_Signal>`.

    error_matrices : list[ParameterPort]

        the matrices of the Projections associated with the `error_sources <LearningMechanism.error_sources>`,

        (i.e., for the next Projection(s) in the learning_sequence, or to the `ComparatorMechanism`);

        note: these are *not* for the LearningMechanism's `learned_projections <LearningMechanism.learned_projections>`.

    primary_learned_projection : MappingProjection

        the Projection with the `matrix <MappingProjection.matrix>` parameter used to generate the

        LearningMechanism's `error_signal <LearningMechanism.error_signal>` and `learning_signal

        <LearningMechanism.learning_signal>` attributes.  It is always the first Projection listed in the

        LearningMechanism's `learned_projections <LearningMechanism.learned_projections>` attribute.

    learned_projections : List[MappingProjection]

        all of the MappingProjections modified by the LearningMechanism;  the first item in the list is always the

        `primary_learned_projection <LearningMechanism.primary_learned_projection>`.

    function : LearningFunction or function : default BackPropagation

        specifies the function used to calculate the `learning_signal <LearningMechanism.learning_signal>` (assigned

        to the LearningMechanism's `LearningSignal(s) <LearningMechanism_LearningSignal>`), and the `error_signal

        <LearningMechanism.error_signal>` (passed to the LearningMechanism for the preceding `MappingProjection` in a

        `multilayer learning pathway <LearningMechanism_Multilayer_Learning>`).  It takes the following

        three arguments, each of which must be a list or 1d array: **input**,  **output**, and **error** (see

        `LearningMechanism_Function` for additional details).

    learning_rate : float : None

        determines the learning rate for the LearningMechanism.  It is used to specify the :keyword:`learning_rate`

        parameter for the LearningMechanism's `learning function <LearningMechanism.function>`

        (see description of `learning_rate <LearningMechanism_Learning_Rate>` for additional details).

    error_signal : 1d np.array

        one of two values returned by the LearningMechanism's `function <LearningMechanism.function>`.  For

        `single layer learning <LearningMechanism_Single_Layer_Learning>`, this is the same as the value received in

        the LearningMechanism's *ERROR_SIGNAL* `InputPort <LearningMechanism_Input_Error_Signal>`;  for `multilayer