PrincetonUniversity / PsyNeuLink / 15917088825

# Princeton University licenses this file to You under the Apache License, Version 2.0 (the "License");

# you may not use this file except in compliance with the License.  You may obtain a copy of the License at:

#     http://www.apache.org/licenses/LICENSE-2.0

# Unless required by applicable law or agreed to in writing, software distributed under the License is distributed

# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and limitations under the License.

# **************************************  OptimizationControlMechanism *************************************************

# FIX: REWORK WITH REFERENCES TO `outcome <OptimizationControlMechanism.outcome>`

#      INTRODUCE SIMULATION INTO DISCUSSION OF COMPOSITION-BASED

"""

Contents

--------

  * `OptimizationControlMechanism_Overview`

      - `Expected Value of Control <OptimizationControlMechanism_EVC>`

      - `Agent Representation and Types of Optimization <OptimizationControlMechanism_Agent_Representation_Types>`

          - `"Model-Free" Optimization <OptimizationControlMechanism_Model_Free>`

          - `Model-Based Optimization <OptimizationControlMechanism_Model_Based>`

  * `OptimizationControlMechanism_Creation`

      - `Agent Rep <OptimizationControlMechanism_Agent_Rep_Arg>`

      - `State Features <OptimizationControlMechanism_State_Features_Arg>`

          - `agent_rep Composition <OptimizationControlMechanism_Agent_Rep_Composition>`

          - `agent_rep CompositionFunctionApproximator <OptimizationControlMechanism_Agent_Rep_CFA>`

      - `State Feature Functions <OptimizationControlMechanism_State_Feature_Function_Arg>`

      - `Outcome  <OptimizationControlMechanism_Outcome_Args>`

  * `OptimizationControlMechanism_Structure`

      - `Agent Representation <OptimizationControlMechanism_Agent_Rep>`

          - `State <OptimizationControlMechanism_State>`

      - `Input <OptimizationControlMechanism_Input>`

          - `state_input_ports <OptimizationControlMechanism_State_Input_Ports>`

          - `outcome_input_ports <OptimizationControlMechanism_Outcome>`

              - `objective_mechanism <OptimizationControlMechanism_ObjectiveMechanism>`

              - `monitor_for_control <OptimizationControlMechanism_Monitor_for_Control>`

      - `Function <OptimizationControlMechanism_Function>`

          - `OptimizationControlMechanism_Custom_Function`

          - `OptimizationControlMechanism_Search_Functions`

          - `OptimizationControlMechanism_Default_Function`

      - `Output <OptimizationControlMechanism_Output>`

          - `Randomization ControlSignal <OptimizationControlMechanism_Randomization_Control_Signal>`

  * `OptimizationControlMechanism_Execution`

      - `OptimizationControlMechanism_Execution_Timing`

      - `OptimizationControlMechanism_Optimization_Procedure`

      - `OptimizationControlMechanism_Estimation_Randomization`

  * `OptimizationControlMechanism_Class_Reference`

.. _OptimizationControlMechanism_Overview:

Overview

--------

An OptimizationControlMechanism is a `ControlMechanism <ControlMechanism>` that uses an `OptimizationFunction` to

optimize the performance of the `Composition` for which it is a `controller <Composition_Controller>`.  It does so

by using the `OptimizationFunction` (assigned as its `function <OptimizationControlMechanism.function>`) to execute

its `agent_rep <OptimizationControlMechanism.agent_rep>` -- a representation of the Composition to be optimized --

under different `control_allocations <ControlMechanism.control_allocation>`, and selecting the one that optimizes

its `net_outcome <ControlMechanism.net_outcome>`.  An OptimizationControlMechanism can be configured to implement

various forms of optimization, ranging from fully `model-based optimization <OptimizationControlMechanism_Model_Based>`

that uses the Composition itself as the  `agent_rep <OptimizationControlMechanism.agent_rep>` to simulate the

outcome for a given `state <OptimizationControlMechanism_State>` (i.e., a combination of the current input and a

particular `control_allocation <ControlMechanism.control_allocation>`), to fully `model-free optimization

<OptimizationControlMechanism_Model_Free>` by using a `CompositionFunctionApproximator` as the `agent_rep

<OptimizationControlMechanism.agent_rep>` that learns to  predict the outcomes for a state. Intermediate forms of

optimization can also be implemented, that use simpler Compositions to approximate the dynamics of the full

Composition. The outcome of executing the `agent_rep <OptimizationControlMechanism.agent_rep>` is used to compute a

`net_outcome <ControlMechanism.net_outcome>` for a given `state <OptimizationControlMechanism_State>`, that takes

into account the `costs <ControlMechanism_Costs_NetOutcome>` associated with the `control_allocation

<ControlMechanism.control_allocation>`, and is used to determine the optimal `control_allocations

<ControlMechanism.control_allocation>`.

.. _OptimizationControlMechanism_EVC:

**Expected Value of Control**

The `net_outcome <ControlMechanism.net_outcome>` of an OptimizationControlMechanism's `agent_rep

<OptimizationControlMechanism.agent_rep>` is computed -- for a given `state <OptimizationControlMechanism_State>`

(i.e., set of `state_feature_values <OptimizationControlMechanism.state_feature_values>` and a `control_allocation

<ControlMechanism.control_allocation>`) -- as the difference between the `outcome <ControlMechanism.outcome>` computed

by its `objective_mechanism <ControlMechanism.objective_mechanism>` and the aggregated `costs <ControlMechanism.costs>`

of its `control_signals <OptimizationControlMechanism.control_signals>` computed by its `combine_costs

<ControlMechanism.combine_costs>` function.  If the `outcome <ControlMechanism.outcome>` computed by the

`objective_mechanism <ControlMechanism.objective_mechanism>` is configured to measure the value of processing (e.g.,

reward received, time taken to respond, or a combination of these, etc.), and the `OptimizationFunction` assigned as

the OptimizationControlMechanism's `function <OptimizationControlMechanism.function>` is configured to find the

`control_allocation <ControlMechanism.control_allocation>` that maximizes its `net_outcome

<ControlMechanism.net_outcome>` (that is, the `outcome <ControlMechanism.outcome>` discounted by the

result of the `combine_costs <ControlMechanism.combine_costs>` function), then the OptimizationControlMechanism is

said to be maximizing the `Expected Value of Control (EVC) <https://www.ncbi.nlm.nih.gov/pubmed/23889930>`_.  That

is, it implements a cost-benefit analysis that weighs the `costs <ControlMechanism.costs>` of the ControlSignal

`values <ControlSignal.value>` associated with a `control_allocation <ControlMechanism.control_allocation>` against

the `outcome <ControlMechanism.outcome>` expected to result from it.  The costs are computed based on the

`cost_options <ControlSignal.cost_options>` specified for each of the OptimizationControlMechanism's `control_signals

<OptimizationControlMechanism.control_signals>` and its `combine_costs <ControlMechanism.combine_costs>` function.

The EVC is determined by its `compute_net_outcome <ControlMechanism.compute_net_outcome>` function (assigned to its

`net_outcome <ControlMechanism.net_outcome>` attribute), which is computed for a given `state

<OptimizationControlMechanism_State>` by the OptimizationControlMechanism's `evaluate_agent_rep

<OptimizationControlMechanism.evaluate_agent_rep>` method. In these respects, optimization of a Composition's

performance by its OptimizationControlMechanism -- as indexed by its `net_outcome <ControlMechanism.net_outcome>`

attribute -- implement a form of `Bounded Rationality <https://psycnet.apa.org/buy/2009-18254-003>`_,

also referred to as `Resource Rationality <https://www.cambridge.org/core/journals/behavioral-and-brain-sciences/article/abs/resourcerational-analysis-understanding-human-cognition-as-the-optimal-use-of-limited-computational-resources/586866D9AD1D1EA7A1EECE217D392F4A>`_,

in which the constraints imposed by the "bounds" or resources are reflected in the `costs` of the ControlSignals

(also see `Computational Rationality <https://onlinelibrary.wiley.com/doi/full/10.1111/tops.12086>`_ and `Toward a

Rational and Mechanistic Account of Mental Effort

<https://www.annualreviews.org/doi/abs/10.1146/annurev-neuro-072116-031526?casa_token=O2pFelbmqvsAAAAA:YKjdIbygP5cj_O7vAj4KjIvfHehHSh82xm44I5VS6TdTtTELtTypcBeET4BGdAy0U33BnDXBasfqcQ>`_).

COMMENT:

The table `below <OptimizationControlMechanism_Examples>` lists different

parameterizations of OptimizationControlMechanism that implement various models of EVC Optimization.

COMMENT

.. _OptimizationControlMechanism_Agent_Representation_Types:

**Agent Representation and Types of Optimization**

Much of the functionality described above is supported by a `ControlMechanism` (the parent class of an

OptimizationControlMechanism). The defining  characteristic of an OptimizationControlMechanism is its `agent

representation <OptimizationControlMechanism_Agent_Rep>`, that is used to determine the `net_outcome

<ControlMechanism.net_outcome>` for a given `state <OptimizationControlMechanism_State>`, and find the

`control_allocation <ControlMechanism.control_allocation>` that optimizes this.  The `agent_rep

<OptimizationControlMechanism.agent_rep>` can be the `Composition` to which the OptimizationControlMechanism

belongs (and controls), another (presumably simpler) one, or a `CompositionFunctionApproximator` that is used to

estimate the `net_outcome <ControlMechanism.net_outcome>` of the Composition of which the OptimizationControlMechanism

is the `controller <Composition.controller>`.  These different types of `agent representation

<OptimizationControlMechanism_Agent_Rep>` correspond closely to the distinction between *model-based* and

*model-free* optimization in the `machine learning

<https://www.google.com/books/edition/Reinforcement_Learning_second_edition/uWV0DwAAQBAJ?hl=en&gbpv=1&dq=Sutton,+R.+S.,+%26+Barto,+A.+G.+(2018).+Reinforcement+learning:+An+introduction.+MIT+press.&pg=PR7&printsec=frontcover>`_

and `cognitive neuroscience <https://www.nature.com/articles/nn1560>`_ literatures, as described below.

    .. figure:: _static/Optimization_fig.svg

       :scale: 50%

       :alt: OptimizationControlMechanism

       **Functional Anatomy of an OptimizationControlMechanism.** *Panel A:* Examples of use in fully model-based

       and model-free optimization.  Note that in the example of `model-based optimization

       <OptimizationControlMechanism_Model_Based>` (left), the OptimizationControlMechanism uses the entire

       `Composition` that it controls as its `agent_rep <OptimizationControlMechanism.agent_rep>`, whereas in

       the example of `model-free optimization <OptimizationControlMechanism_Model_Free>` (right) the

       the `agent_rep <OptimizationControlMechanism.agent_rep>` is a `CompositionFunctionApproximator`. The `agent_rep

       <OptimizationControlMechanism.agent_rep>` can also be another (presumably simpler) Composition that can be used

       to implement forms of optimization intermediate between fully model-based and model-free. *Panel B:* Flow of

       execution during optimization.  In both panels, faded items show process of adaptation when using a

       `CompositionFunctionApproximator` as the `agent_rep <OptimizationControlMechanism.agent_rep>`.

|

.. _OptimizationControlMechanism_Model_Based:

*Model-Based Optimization*

The fullest form of this is implemented by assigning the Composition for which the OptimizationControlMechanism is the

`controller <Composition.controller>`) as its a`agent_rep  <OptimizationControlMechanism.agent_rep>`

On each `TRIAL <TimeScale.TRIAL>`, that Composition *itself* is provided with either the most recent inputs

to the Composition, or ones predicted for the upcoming trial (as determined by the `state_feature_values

<OptimizationControlMechanism.state_feature_values>` of the OptimizationControlMechanism), and then used to simulate

processing on that trial in order to find the `control_allocation <ControlMechanism.control_allocation>` that yields

the best `net_outcome <ControlMechanism.net_outcome>` for that trial.  A different Composition can also be assigned as

the `agent_rep  <OptimizationControlMechanism.agent_rep>`, that approximates in simpler form the dynamics of processing

in the Composition for which the OptimizationControlMechanism is the `controller <Composition.controller>`,

implementing a more restricted form of model-based optimization.

.. _OptimizationControlMechanism_Model_Free:

*"Model-Free" Optimization*

    .. note::

       The term *model-free* is placed in apology quotes to reflect the fact that, while this term is

       used widely (e.g., in machine learning and cognitive science) to distinguish it from *model-based* forms of

       processing, model-free processing nevertheless relies on *some* form of model -- albeit usually a much simpler

       one -- for learning, planning and decision making.  In the context of a OptimizationControlMechanism, this is

       addressed by use of the term "agent_rep", and how it is implemented, as described below.

This clearest form of this uses a `CompositionFunctionApproximator`, that learns to predict the `net_outcome

<ControlMechanism.net_outcome>` for a given state (e.g., using reinforcement learning or other forms of

function approximation, such as a `RegressionCFA`).  In each `TRIAL <TimeScale.TRIAL>` the  `agent_rep

<OptimizationControlMechanism.agent_rep>` is used to search over `control_allocation

<ControlMechanism.control_allocation>`\\s, to find the one that yields the best predicted `net_outcome

<ControlMechanism.net_outcome>` of processing on the upcoming trial, based on the current or (expected)

`state_feature_values <OptimizationControlMechanism.state_feature_values>` for that trial.  The `agent_rep

<OptimizationControlMechanism.agent_rep>` is also given the chance to adapt in order to improve its prediction of

its `net_outcome <ControlMechanism.net_outcome>` based on the `state <OptimizationControlMechanism_State>` and

`net_outcome <ControlMechanism.net_outcome>` of the prior `TRIAL <TimeScale.TRIAL>`.  A Composition can also be

used to generate such predictions, permitting forms of optimization that are intermediate between the extreme

examples of model-based and model-free, as noted above.

.. _OptimizationControlMechanism_Creation:

Creating an OptimizationControlMechanism

----------------------------------------

The constructor has the same arguments as a `ControlMechanism <ControlMechanism>`, with the following

exceptions/additions, which are specific to the OptimizationControlMechanism:

.. _OptimizationControlMechanism_Agent_Rep_Arg:

* **agent_rep** -- specifies the `Composition` used by the OptimizationControlMechanism's `evaluate_agent_rep

  <OptimizationControlMechanism.evaluate_agent_rep>` method to calculate the predicted `net_outcome

  <ControlMechanism.net_outcome>` for a given `state <OptimizationControlMechanism_State>` (see `below

  <OptimizationControlMechanism_Agent_Rep>` for additional details). If it is not specified, then the

  `Composition` to which the OptimizationControlMechanism is assigned becomes its `agent_rep

  <OptimizationControlMechanism.agent_rep>`, and the OptimizationControlMechanism is assigned as that Composition's

  `controller <Composition.controller>`, implementing fully `model-based <OptimizationControlMechanism_Model_Based>`

  optimization.  If that Composition already has a `controller <Composition.controller>` specified,

  the OptimizationControlMechanism is disabled. If another Composition is specified, it must conform to the

  specifications for an `agent_rep <OptimizationControlMechanism.agent_rep>` as described `below

  <OptimizationControlMechanism_Agent_Rep>`.  The  `agent_rep <OptimizationControlMechanism.agent_rep>` can also be

  a `CompositionFunctionApproximator` for `model-free <OptimizationControlMechanism_Model_Free>` forms of

  optimization.  The type of Component assigned as the `agent_rep <OptimizationControlMechanism.agent_rep>` is

  identified in the OptimizationControlMechanism's `agent_rep_type <OptimizationControlMechanism.agent_rep_type>`

  attribute.

.. _OptimizationControlMechanism_State_Features_Arg:

* **state_features** -- specifies the sources of input to the OptimizationControlMechanism's `agent_rep

  <OptimizationControlMechanism.agent_rep>` which, together with a selected `control_allocation

  <ControlMechanism.control_allocation>`, are provided as input to it's `evaluate <Composition.evaluate>` method

  when that is executed to estimate or predict the Composition's `net_outcome <ControlMechanism.net_outcome>`.

  Those sources of input are used to construct the OptimizationControlMechanism's `state_input_ports

  <OptimizationControlMechanism.state_input_ports>`, one for each `external InputPort

  <Composition_Input_External_InputPorts>` of the `agent_rep <OptimizationControlMechanism.agent_rep>`. The input to

  each `state_input_port <OptimizationControlMechanism.state_input_ports>`, after being processed by it `function

  <InputPort.function>`, is assigned as the corresponding value of `state_feature_values

  <OptimizationControlMechanism.state_feature_values>`, the values of which provided as the input to the corresponding

  InputPorts of the `INPUT <NodeRole.INPUT>` `Nodes <Composition_Nodes>` of the agent_rep each time it is `evaluated

  <Composition.evaluate>`.  Accordingly, the specification requirements for **state_features** depend on whether the

  `agent_rep<OptimizationControlMechanism.agent_rep>` is a `Composition` or a `CompositionFunctionApproximator`,

  as described in each of the two sections below.

  |

  .. _OptimizationControlMechanism_Agent_Rep_Composition:

  **state_features** *for an agent_rep that is a* **Composition**

  |

  .. _OptimizationControlMechanism_State_Features_Automatic_Assignment:

  *Automatic assignment.*  By default, if **state_features**, **state_feature_default** and **state_feature_function**

  are not specified, the `state_input_ports <OptimizationControlMechanism.state_input_ports>` are configured to

  `shadow the inputs <InputPort_Shadow_Inputs>` of every `external InputPort <Composition_Input_External_InputPorts>`

  of the `agent_rep <OptimizationControlMechanism.agent_rep>` Composition;  as a result, each time `agent_rep

  <OptimizationControlMechanism.agent_rep>` is `evaluated <Composition.evaluate>`, it receives the same `external

  input <Composition_Execution_Inputs>`) it received during its last `TRIAL<TimeScale.TRIAL>` of execution.

  |

  .. _OptimizationControlMechanism_State_Features_Explicit_Specification:

  *Explicit specification.* Specifying the **state_features**, **state_feature_default** and/or

  **state_feature_function** arguments explicitly can be useful if: values need to be provided as input to the

  `agent_rep <OptimizationControlMechanism.agent_rep>` when it is evaluated other than its `external inputs

  <Composition_Execution_Inputs>`; to restrict evaluation to a subset of its inputs (while others are held constant);

  and/or to assign specific functions to one or more `state_input_ports

  <OptimizationControlMechanism.state_input_ports>` (see `below

  <OptimizationControlMechanism_State_Feature_Function_Arg>`) that allow them to process the inputs

  (e.g., modulate and/or integrate them) before they are assigned to `state_feature_values

  <OptimizationControlMechanism.state_feature_values>` and passed to the `agent_rep

  <OptimizationControlMechanism.agent_rep>`. Assignments can be made to **state_features** corresponding

  to any or all InputPorts of the `agent_rep <OptimizationControlMechanism.agent_rep>`\\'s `INPUT <NodeRole.INPUT>`

  `Nodes <Composition_Nodes>`, as described `below <OptimizationControlMechanism_State_Features_Specification>`.

  Any that are not specified are assigned the value specified for **state_feature_default** (*SHADOW_INPUTS* by

  default; see `state_feature_default <OptimizationControlMechanism.state_feature_default>` for additional details).

  A single assignment can be made for all **state_features**, or they can be specified individually for each `INPUT

  <NodeRole.INPUT>` `Nodes <Composition_Nodes>` InputPort, as descdribed below.

  .. _OptimizationControlMechanism_State_Features_Shapes:

      .. note::

         If **state_features** are specified explicitly, the `value <Component.value>`\\s of the specified Components

         must match the `input_shape <InputPort.input_shape>` of the corresponding InputPorts of the `agent_rep

         <OptimizationControlMechanism.agent_rep>`\\'s `INPUT <NodeRole.INPUT>` `Nodes <Composition_Nodes>`. Those

         InputPorts are listed in the `agent_rep <OptimizationControlMechanism.agent_rep>`\\'s

         `external_input_ports_of_all_input_nodes <Composition.external_input_ports_of_all_input_nodes>` attribute

         and, together with examples of their values, in the OptimizationControlMechanism's `state_feature_values

         <OptimizationControlMechanism.state_feature_values>` attribute. A failure to properly meet these requirements

         produces an error.

  .. _OptimizationControlMechanism_State_Features_Specification:

  The **state_features** argument can be specified using any of the following formats:

  .. _OptimizationControlMechanism_State_Feature_Single_Spec:

  * *Single specification* -- any of the indivdiual specifications described `below

    <OptimizationControlMechanism_State_Feature_Individual_Specs>` can be directly to **state_features**, that is

    then used to construct *all* of the `state_input_ports <OptimizationControlMechanism.state_input_ports>`, one

    for each `external InputPort <Composition_Input_External_InputPorts>` of the `agent_rep

    <OptimizationControlMechanism.agent_rep>`.

  .. _OptimizationControlMechanism_State_Feature_Input_Dict:

  * *Inputs dictionary* -- specifies state_features (entry values) for individual `InputPorts <InputPort>` and/or

    `INPUT <NodeRole.INPUT>` `Nodes <Composition_Nodes>` of the `agent_rep <OptimizationControlMechanism.agent_rep>`

    (entry keys). It must conform to the format used to `specify external inputs <Composition_Input_Dictionary>`

    to the `agent_rep <OptimizationControlMechanism.agent_rep>`, in which entries consist of a key specifying either

    an `INPUT <NodeRole.INPUT>` `Node <Composition_Nodes>` of the `agent_rep <OptimizationControlMechanism.agent_rep>`

    or one of their `external InputPorts <Composition_Input_External_InputPorts>`, and a value that is the source of

    the input that can be any of the forms of individual input specifications listed `below

    <OptimizationControlMechanism_State_Feature_Individual_Specs>`. The format required for the entries can be seen

    using either the `agent_rep <OptimizationControlMechanism.agent_rep>`

    `get_input_format <Composition.get_input_format>` method (for inputs to its `INPUT <NodeRole.INPUT>` <Nodes

    <Composition_Nodes>`) or its `external_input_ports_of_all_input_nodes

    <Composition.external_input_ports_of_all_input_nodes>` (for all of their `external InputPorts

    <Composition_Input_External_InputPorts>`). If a nested Composition is specified (that is, one that is an `INPUT

    <NodeRole.INPUT>` Node of `agent_rep <OptimizationControlMechanism.agent_rep>`), the state_feature assigned to it

    is used to construct the `state_input_ports <OptimizationControlMechanism.state_input_ports>` for *all* of the

    `external InputPorts <Composition_Input_External_InputPorts>` for that nested Composition, and any nested within

    it at all levels of nesting. If any `INPUT <NodeRole.INPUT>` Nodes or their InputPorts are not specified in the

    dictionary, `state_feature_default <OptimizationControlMechanism.state_feature_default>` is assigned as their

    state_feature specification (this includes cases in which some but not all `INPUT <NodeRole.INPUT>` Nodes of a

    nested Composition, or their InputPorts, are specified; any unspecified INPUT Nodes of the corresponding

    Compositions are assigned `state_feature_default <OptimizationControlMechanism.state_feature_default>` as their

    state_feature specification).

  .. _OptimizationControlMechanism_State_Feature_List_Inputs:

  * *List* -- a list of individual state_feature specifications, that can be any of the forms of individual

    input specifications listed `below <OptimizationControlMechanism_State_Feature_Individual_Specs>`. The items

    correspond to all of the `external InputPorts <Composition_Input_External_InputPorts>` of the `agent_rep

    <OptimizationControlMechanism.agent_rep>`, and must be specified in the order they are listed in the

    `agent_rep <OptimizationControlMechanism.agent_rep>`\\'s `external_input_ports_of_all_input_nodes

    <Composition.external_input_ports_of_all_input_nodes>` attribute. If the list is incomplete, the remaining

    InputPorts are assigned `state_feature_default <OptimizationControlMechanism.state_feature_default>`

    as their state_feature specification, which by default is *SHADOW_INPUTS* (see `below

    <OptimizationControlMechanism_SHADOW_INPUTS_State_Feature>`. Items can be included in the list that

    have not yet been added to the OptimizationControlMechanism's Composition or its `agent_rep

    <OptimizationControlMechanism.agent_rep>`. However, these must be added before the Composition is executed,

    and must appear in the list in the same position that the InputPorts to which they pertain are listed in

    the `agent_rep <OptimizationControlMechanism.agent_rep>`\\'s `external_input_ports_of_all_input_nodes

    <Composition.external_input_ports_of_all_input_nodes>` attribute, once construction of the `agent_rep

    <OptimizationControlMechanism.agent_rep>` is complete.

  .. _OptimizationControlMechanism_State_Feature_Set_Inputs:

  * *Set* -- a set of `INPUT <NodeRole.INPUT>` `Nodes <Composition_Nodes>` of the `agent_rep

    <OptimizationControlMechanism.agent_rep>` that are assigned *SHADOW_INPUTS* as their state_feature

    -- that is, that should receive the same inputs during evaluation as when the Composition of which

    the OptimizationControlMechanism is the `controller <Composition_Controller>` is fully executed

    (see `below <_OptimizationControlMechanism_SHADOW_INPUTS_State_Feature>`). The order of their specification

    does not matter;  however, any of the `agent_rep <OptimizationControlMechanism.agent_rep>`\\'s `INPUT

    <NodeRole.INPUT>` Nodes that are *not* included in the set are assigned `state_feature_default

    <OptimizationControlMechanism.state_feature_default>` as their state_feature specification.  Note that,

    since the default for `state_feature_default <OptimizationControlMechanism.state_feature_default>` is

    *SHADOW_INPUTS*, unless this is specified otherwise omitting items from a set has no effect (i.e., they

    too are assigned *SHADOW_INPUTS*);  for omitted items to be treated differently, `state_feature_default

    <OptimizationControlMechanism.state_feature_default>` must be specified; for example by assigning it

    ``None`` so that items omitted from the set are assigned their default input value (see `below

    <OptimizationControlMechanism_None_State_Feature>`.

  .. _OptimizationControlMechanism_State_Feature_Individual_Specs:

  * *Individual state_feature specifications* -- any of the specifications listed below can be used singly,

    or in a dict, list or set as described `above <OptimizationControlMechanism_State_Feature_Input_Dict>`,

    to configure `state_input_ports <OptimizationControlMechanism.state_input_ports>`.

    .. _OptimizationControlMechanism_None_State_Feature:

    * *None* -- no `state_input_port <OptimizationControlMechanism.state_input_ports>` is constructed for

      the corresponding `INPUT <NodeRole.INPUT>` `Node <Composition_Nodes>` InputPort, and its the value

      of its `default variable <Component.defaults>` is used as the input to that InputPort whenever the

      <OptimizationControlMechanism.agent_rep>` is `evaluated <Composition.evaluate>`, irrespective of its input when

      the `agent_rep <OptimizationControlMechanism.agent_rep>` was last executed.

    .. _OptimizationControlMechanism_Numeric_State_Feature:

    * *numeric value* -- create a `state_input_port <OptimizationControlMechanism.state_input_ports>` has

      no `afferent Projections <Mechanism_Base.afferents>`, and uses the specified value as the input to its

      `function <InputPort.function>`, the result of which is assigned to the corresponding value of

      `state_feature_values <OptimizationControlMechanism.state_feature_values>` and provided as the input to

      the corresponding `INPUT <NodeRole.INPUT>` `Node <Composition_Nodes>` InputPort each time the `agent_rep

      <OptimizationControlMechanism.agent_rep>`  is `evaluated <Composition.evaluate>`. The specified value must

      be compatible with the shape of all of the `external InputPorts <Composition_Input_External_InputPorts>`

      of the `agent_rep <OptimizationControlMechanism.agent_rep>` (see `note

      <OptimizationControlMechanism_State_Features_Shapes>` above).

    .. _OptimizationControlMechanism_SHADOW_INPUTS_State_Feature:

    * *SHADOW_INPUTS* -- create a `state_input_port <OptimizationControlMechanism.state_input_ports>` that `shadows

      the input <InputPort_Shadow_Inputs>` of the InputPort to which the specification is assigned; that is, each time

      `agent_rep <OptimizationControlMechanism.agent_rep>` is  `evaluated <Composition.evaluate>`, the state_input_port

      receives the same input that the corresponding `INPUT <NodeRole.INPUT>` `Node <Composition_Nodes>` InputPort

      received during the last `TRIAL <TimeScale.TRIAL>` of execution.

    .. _OptimizationControlMechanism_Input_Port_State_Feature:

    * *InputPort specification* -- create a `state_input_port <OptimizationControlMechanism.state_input_ports>` that

      `shadows <InputPort_Shadow_Inputs>` the input to the specified `InputPort`;  that is, each time `agent_rep

      <OptimizationControlMechanism.agent_rep>` is  `evaluated <Composition.evaluate>`, the state_input_port receives

      the same input that the specified InputPort received during the last `TRIAL <TimeScale.TRIAL>` in which the

      Composition for which the OptimizationControlMechanism is the `controller <Composition.controller>` was executed.

      The specification can be any form of `InputPort specification <InputPort_Specification>` for the `InpuPort`

      of any `Mechanism <Mechanism>` that is an `INPUT <NodeRole.INPUT>` `Node <Composition_Nodes>` in the Composition

      (not limited to the `agent_rep <OptimizationControlMechanism.agent_rep>`).  This includes an

      `InputPort specification dictionary <InputPort_Specification_Dictionary>`, that can be used to configure the

      corresponding `state_input_port <OptimizationControlMechanism.state_input_ports>`, if `Parameters <Parameter>`

      other than its `function <InputPort.function>` need to be specified (which can be done directly using a

      `2-item tuple <OptimizationControlMechanism_Tuple_State_Feature>` specification or the **state_feature_function**

      arg as described `below <OptimizationControlMechanism_State_Feature_Function_Arg>`), such as the InputPort's

      `name <InputPort.name>` or more than a single `afferent Projection <Mechanism_Base.afferents>`.

      .. _OptimizationControlMechanism_INPUT_Node_Specification:

      .. note::

         Only the `INPUT <NodeRole.INPUT>` `Nodes <Composition_Nodes>` of a `nested Composition <Composition_Nested>`

         can be shadowed.  Therefore, if the Composition that an OptimizationControlMechanism controls contains any

         nested Compositions, only its `INPUT <NodeRole.INPUT>` `Nodes <Composition_Nodes>` can be specified for

         shadowing in the **state_features** argument of the OptimizationControlMechanism's constructor.

      .. hint::

         Shadowing the input to a Node of a `nested Composition <Composition_Nested>` that is not an `INPUT

         <NodeRole.INPUT>` Node of that Composition can be accomplished in one or of two ways, by: a) assigning it

         `INPUT <NodeRole.INPUT>` as a `required NodeRole <Composition_Node_Role_Assignment>` where it is added to

         the nested Composition; and/or b) adding an additional Node to that Composition that shadows the desired one

         (this is allowed *within* the *same* Composition), and is assigned as an `OUTPUT <NodeRole.OUTPUT>` Node of

         that Composition, the `OutputPort` of which which can then be specified in the **state_features** argument of

         the OptimizationControlMechanism's constructor (see below).

      .. technical_note::

        The InputPorts specified as state_features are designated as `internal_only <InputPort.internal_only>` = `True`.

    .. _OptimizationControlMechanism_Output_Port_State_Feature:

    * *OutputPort specification* -- create a `state_input_port <OptimizationControlMechanism.state_input_ports>` that

      receives a `MappingProjection` from the specified `OutputPort`;  that is, each time `agent_rep

      <OptimizationControlMechanism.agent_rep>` is  `evaluated <Composition.evaluate>`, the state_input_port receives

      the `value <OutputPort.value>` of the specified OutputPort after the last `TRIAL <TimeScale.TRIAL>` in which the

      Composition for which the OptimizationControlMechanism is the `controller <Composition.controller>` was executed.

      The specification can be any form of `OutputPort specification <OutputPort_Specification>` for any `OutputPort`

      of a `Mechanism <Mechanism>` in the Composition (not limited to the `agent_rep

      <OptimizationControlMechanism.agent_rep>`.

    .. _OptimizationControlMechanism_Mechanism_State_Feature:

    * *Mechanism* -- create a `state_input_port <OptimizationControlMechanism.state_input_ports>` that `shadows

      <InputPort_Shadow_Inputs>` the input to the `primary InputPort <InputPort_Primary>` of the specified Mechanism

      (this is the same as explicitly specifying the Mechanism's  input_port, as described `above

      <OptimizationControlMechanism_Input_Port_State_Feature>`). If the Mechanism is in a `nested Composition

      <Composition_Nested>`, it must be an `INPUT <NodeRole.INPUT>` `Node <Composition_Nodes>` of that Composition

      (see `note <OptimizationControlMechanism_INPUT_Node_Specification>` above).  If the Mechanism's `OutputPort`

      needs to be used, it must be specified explicitly (as described `above

      <OptimizationControlMechanism_Output_Port_State_Feature>`).

      .. note::

         The use of a Mechanism to specify the shadowing of its `primary InputPort <InputPort_Primary>` is unique to

         its specification in the **state_features** argument of an OptimizationControlMechanism, and differs from the

         ordinary usage where it specifies a Projection from its `primary OutputPort <OutputPort_Primary>` (see

         `InputPort specification <InputPort_Projection_Source_Specification>`).  This difference extends to the use

         of a Mechanism in the *PROJECTIONS* entry of an `InputPort specification dictionary

         <InputPort_Specification_Dictionary>` used in the **state_features** argument, where there too

         it designates shadowing of its `primary InputPort <InputPort_Primary>` rather than a `Projection` from its

         `primary OutputPort <OutputPort_Primary>`.

    .. _OptimizationControlMechanism_Tuple_State_Feature:

    * *2-item tuple* -- the first item must be any of the forms of individual state_feature specifications

      described `above <OptimizationControlMechanism_State_Feature_Individual_Specs>`, and the second

      item must be a `Function`, that is assigned as the `function <InputPort.function>` of the corresponding

      `state_input_port <OptimizationControlMechanism.state_input_ports>`; this takes precedence over

      any other state_feature_function specifications (e.g., in an `InputPort specification dictionary

      <InputPort_Specification_Dictionary>` or the **state_feature_function** argument of the

      OptimizationControlMechanism's constructor; see `state_feature_function

      <OptimizationControlMechanism_State_Feature_Function_Arg>` for additional details).

  |

  .. _OptimizationControlMechanism_Agent_Rep_CFA:

  **state_features** *for an agent_rep that is a* **CompositionFunctionApproximator**

  |

  The **state_features** specify the **feature_values** argument to the `CompositionFunctionApproximator`\\'s

  `evaluate <CompositionFunctionApproximator.evaluate>` method. These cannot be determined automatically and so

  they *must be specified explicitly*, in a list, with the correct number of items in the same order and with

  the same shapes they are expected have in the array passed to the **feature_values** argument of the

  `evaluate <CompositionFunctionApproximator.evaluate>` method (see warning below).

      .. warning::

         The **state_features** for an `agent_rep <OptimizationControlMechanism.agent_rep>` that is a

         `CompositionFunctionApproximator` cannot be created automatically nor can they be validated;

         thus specifying the wrong number or invalid **state_features**, or specifying them in an incorrect

         order may produce errors that are unexpected or difficult to interpret.

  The list of specifications can contain any of the forms of specification used for an `agent_rep

  <OptimizationControlMechanism.agent_rep>` that is a Composition as described `above

  <OptimizationControlMechanism_State_Feature_Input_Dict>`, with the following exception: if a

  `Mechanism` is specified, its `primary OutputPort <OutputPort_Primary>` is used (rather than

  shadowing its primary InputPort), since that is more typical usage, and there are no assumptions

  made about the state features of a `CompositionFunctionApproximator` (as there are about a Composition

  as `agent_rep <OptimizationControlMechanism.agent_rep>`); if the input to the Mechanism *is* to be

  `shadowed <InputPort_Shadow_Inputs>`, then its InputPort must be specified explicitly (as described

  `above <OptimizationControlMechanism_Input_Port_State_Feature>`).

|

.. _OptimizationControlMechanism_State_Feature_Function_Arg:

* **state_feature_function** -- specifies a `function <InputPort.function>` to be used as the default

  function for `state_input_ports <OptimizationControlMechanism.state_input_ports>`. This is assigned as

  the `function <InputPort.function>` to any state_input_ports for which *no other* `Function` is specified --

  that is, in either an `InputPort specification dictionary <InputPort_Specification_Dictionary>` or a `2-item tuple

  <OptimizationControlMechanism_Tuple_State_Feature>` in the **state_features** argument (see `state_features

  <OptimizationControlMechanism_State_Features_Arg>`).  If either of the latter is specified, they override

  the specification in **state_feature_function**.  If **state_feature_function** is *not* specified, then

  `LinearCombination` (the standard default `Function` for an `InputPort`) is assigned to any `state_input_ports

  <OptimizationControlMechanism.state_input_ports>` that are not otherwise assigned a `Function`.

  Specifying functions for `state_input_ports <OptimizationControlMechanism.state_input_ports>` can be useful,

  for example to provide an average or integrated value of prior inputs to the `agent_rep

  <OptimizationControlMechanism.agent_rep>`\\'s `evaluate <Composition.evaluate>` method during the optimization

  process, or to use a generative model of the environment to provide those inputs.

    .. note::

       The value returned by a function assigned to the **state_feature_function** argument must preserve the

       shape of its input, and must also accommodate the shape of the inputs to all of the `state_input_ports

       <OptimizationControlMechanism.state_input_ports>` to which it is assigned (see `note

       <OptimizationControlMechanism_State_Features_Shapes>` above).

.. _OptimizationControlMechanism_Outcome_Args:

* **Outcome arguments** -- these specify the Components, the values of which are assigned to the `outcome

  <ControlMechanism.outcome>` attribute, and used to compute the `net_outcome <ControlMechanism.net_outcome>` for a

  given `control_allocation <ControlMechanism.control_allocation>` (see `OptimizationControlMechanism_Execution`).

  As with a ControlMechanism, these can be sepcified directly in the **monitor_for_control** argument, or through the

  use of `ObjectiveMechanism` specified in the  **objecctive_mechanism** argument (see

  ControlMechanism_Monitor_for_Control for additional details).  However, an OptimizationControlMechanism places some

  restrictions on the specification of these arguments that, as with specification of `state_features

  <OptimizationControlMechanism_State_Features_Arg>`, depend on the nature of the `agent_rep

  <OptimizationControlMechanism.agent_rep>`, as described below.

  * *agent_rep is a Composition* -- the items specified to be monitored for control must belong to the `agent_rep

    <OptimizationControlMechanism.agent_rep>`, since those are the only ones that will be executed when the

    `evaluate_agent_rep <OptimizationControlMechanism.evaluate_agent_rep>` is called; an error will be generated

    identifying any Components that do not belong to the `agent_rep <OptimizationControlMechanism.agent_rep>`.

  * *agent_rep is a CompositionFunctionApproximator* -- the items specified to be monitored for control can be any

    within the Composition for which the OptimizationControlMechanism is the `controller <Composition_Controller>`;

    this is because their values during the last execution of the Composition are used to determine the `net_outcome

    <ControlMechanism.net_outcome>` that the `agent_rep <OptimizationControlMechanism.agent_rep>`\\'s

    `adapt <CompositionFunctionApproximator.adapt>` method -- if it has one -- seeks to predict.  Accordingly,

    the values of the items specified to be monitored control must match, in shape and order, the

    **net_outcome** of that `adapt <CompositionFunctionApproximator.adapt>` method.

* **Optimization arguments** -- these specify parameters that determine how the OptimizationControlMechanism's

  `function <OptimizationControlMechanism.function>` searches for and determines the optimal `control_allocation

  <ControlMechanism.control_allocation>` (see `OptimizationControlMechanism_Execution`); this includes specification

  of the `num_estimates <OptimizationControlMechanism.num_estimates>` and `num_trials_per_estimate

  <OptimizationControlMechanism.num_trials_per_estimate>` parameters, as well as the `random_variables

  <OptimizationControlMechanism.random_variables>`, `initial_seed <OptimizationControlMechanism.initial_seed>` and

  `same_seed_for_all_allocations <OptimizationControlMechanism.same_seed_for_all_allocations>` Parameters, which

  determine how the `net_outcome <ControlMechanism.net_outcome>` is estimated for a given `control_allocation

  <ControlMechanism.control_allocation>` (see `OptimizationControlMechanism_Estimation_Randomization` for additional

  details).

.. _OptimizationControlMechanism_Structure:

Structure

---------

An OptimizationControlMechanism conforms to the structure of a `ControlMechanism`, with the following exceptions

and additions.

.. _OptimizationControlMechanism_Agent_Rep:

*Agent Representation*

^^^^^^^^^^^^^^^^^^^^^^

The defining feature of an OptimizationControlMechanism is its agent representation, specified in the **agent_rep**

argument of its constructor, and assigned to its `agent_rep <OptimizationControlMechanism.agent_rep>` attribute.  This

designates a representation of the `Composition` (or parts of it) that the OptimizationControlMechanism uses to

evaluate sample `control_allocations <ControlMechanism.control_allocation>` in order to find one that optimizes the

the `net_outcome <ControlMechanism.net_outcome>` of the Composition when it is fully executed. The `agent_rep

<OptimizationControlMechanism.agent_rep>` can be the Composition itself for which the OptimizationControlMechanism is

the `controller <Composition_Controller>` (fully `model-based optimization <OptimizationControlMechanism_Model_Based>`,

or another one `model-free optimization <OptimizationControlMechanism_Model_Free>`), that is usually a simpler

Composition or a `CompositionFunctionApproximator`  used to estimate the `net_outcome <ControlMechanism.net_outcome>`

for the full Composition (see `above <OptimizationControlMechanism_Agent_Representation_Types>`).  The `evaluate

<Composition.evaluate>` method of the `agent_rep <OptimizationControlMechanism.agent_rep>` is assigned as the

`evaluate_agent_rep <OptimizationControlMechanism.evaluate_agent_rep>` method of the OptimizationControlMechanism.

If the `agent_rep <OptimizationControlMechanism.agent_rep>` is not the Composition for which the

OptimizationControlMechanism is the controller, then it must meet the following requirements:

* Its `evaluate <Composition.evaluate>` method must accept as its first four positional arguments:

  - values that correspond in shape to  the `state_feature_values

    <OptimizationControlMechanism.state_feature_values>` (inputs for estimate);

  - `control_allocation <ControlMechanism.control_allocation>` (the set of parameters for which estimates

    of `net_outcome <ControlMechanism.net_outcome>` are made);

  - `num_trials_per_estimate <OptimizationControlMechanism.num_trials_per_estimate>` (number of trials executed by

    agent_rep for each estimate).

..

* If it has an `adapt <CompositionFunctionApproximator.adapt>` method, that must accept as its first three

  arguments, in order:

  - values that correspond to the shape of the  `state_feature_values

    <OptimizationControlMechanism.state_feature_values>` (inputs that led to the net_come);

  - `control_allocation <ControlMechanism.control_allocation>` (set of parameters that led to the net_outcome);

  - `net_outcome <ControlMechanism.net_outcome>` (the net_outcome that resulted from the `state_feature_values

    <OptimizationControlMechanism.state_feature_values>` and `control_allocation

    <ControlMechanism.control_allocation>`) that must match the shape of `outcome <ControlMechanism.outcome>`.

  COMMENT:

  - `num_estimates <OptimizationControlMechanism.num_trials_per_estimate>` (number of estimates of `net_outcome

    <ControlMechanism.net_outcome>` made for each `control_allocation <ControlMechanism.control_allocation>`).

  COMMENT

.. _OptimizationControlMechanism_State:

*State*

~~~~~~~

The current state of the OptimizationControlMechanism -- or, more properly, of its `agent_rep

<OptimizationControlMechanism.agent_rep>` -- is determined by the OptimizationControlMechanism's current

`state_feature_values <OptimizationControlMechanism.state_feature_values>` (see `below

<OptimizationControlMechanism_State_Input_Ports>`) and `control_allocation <ControlMechanism.control_allocation>`.

These are used by the `evaluate_agent_rep <OptimizationControlMechanism.evaluate_agent_rep>` method,

the results of which are combined with the `costs <ControlMechanism_Costs_NetOutcome>` associated with the

`control_allocation <ControlMechanism.control_allocation>`, to evaluate the `net_outcome

<ControlMechanism.net_outcome>` for that state. The current state is listed in the OptimizationControlMechanism's

`state <OptimizationControlMechanism.state>` attribute, and `state_dict <OptimizationControlMechanism.state_dict>`

contains the Components associated with each value of `state <OptimizationControlMechanism.state>`.

COMMENT:

          > Attributes that pertain to the state of the agent_rep for a given evaluation:

              state_feature_specs: a list of the sources for state_feature_values, one for each InputPort of each INPUT Node of the agent_rep

                                   (at all levels of nesting);  None for any sources not specified in **state_features**

                                   (corresponding InputPorts are are assigned their default input when agent_rep.evaluate() is executed).

              state_feature_values: a dict with entries for each item specified in **state_features** arg of constructor,

                                    in which the key of each entry is an InputPort of an INPUT Node of agent_rep (at any level of nesting)

                                    and the value is the current value of the corresponding state_input_port;  the dict is suitable for

                                    use as the **predicted_inputs** or **feature_values** arg of the agent_rep's evaluate() method

                                    (depending on whether the agent_rep is a Composition or a CFA);

                                    note:  there are not entries for InputPorts of INPUT Nodes that are not specified in **state_features**;

                                           those are assigned either their default input values (LINK XXX) or the shadowed input of

                                           the corresponding INPUT Node InputPort (LINK XXX), depending on how **state_features** was formatted;

                                           (see LINK XXX for details of formatting).

              state_features: a dict with entries corresponding to each item of state_feature_specs,

                              the keys of which are InputPorts of the INPUT Nodes of the agent_rep,

                              and values of which are the corresponding state_feature_specs

                              (i.e., sources of input for those InputPorts when evaluate() is called);

              control_allocation: a list of the current values of the OCM's control_signals that are used to modulate the Parameters

                                  specified for control when the agent_rep's evaluate() method is called;

              state: a list of the values of the current state, starting with state_feature_values and ending with

                     control_allocations

              state_dict: a dictionary with entries for each state_feature and ControlSignal, keys?? values??

                                   their source/destination, and their current values

          > Constituents of state specifications:

            - agent_rep_input_port:  an InputPort of an INPUT Node of the agent_rep,

                                     that will receive a value from state_feature_values passed to agent_rep.evaluate()

            - source: the source of the input to an agent_rep_input_port,

                      that sends a Projection to the corresponding state_input_port

          > Relationship of numeric spec to ignoring it (i.e. assigning it None):

               allows specification of value as input *just* for simulations (i.e., agent_rep_evaluate)

               and not normal execution of comp

COMMENT

.. _OptimizationControlMechanism_Input:

*Input*

^^^^^^^

An OptimizationControlMechanism has two types of `input_ports <Mechanism_Base.input_ports>`, corresponding to the two

forms of input it requires: `state_input_ports <OptimizationControlMechanism.state_input_ports>` that provide the values

of the Components specified as its `state_features <OptimizationControlMechanism_State_Features_Arg>`, and that are used

as inputs to the `agent_rep <OptimizationControlMechanism.agent_rep>` when its `evaluate <Composition.evaluate>` method

is used to execute it; and `outcome_input_ports <OptimizationControlMechanism.outcome_input_ports>` that provide the

outcome of executing the `agent_rep <OptimizationControlMechanism.agent_rep>`, that is used to compute the `net_outcome

<ControlMechanism.net_outcome>` for the `control_allocation <ControlMechanism.control_allocation>` under which the

execution occurred. Each of these is described below.

.. _OptimizationControlMechanism_State_Input_Ports:

*state_input_ports*

~~~~~~~~~~~~~~~~~~~

The `state_input_ports <OptimizationControlMechanism.state_input_ports>` receive `Projections <Projection>`

from the Components specified as the OptimizationControlMechanism's `state_features

<OptimizationControlMechanism_State_Features_Arg>`, the values of which are assigned as the `state_feature_values

<OptimizationControlMechanism.state_feature_values>`, and conveyed to the `agent_rep

<OptimizationControlMechanism.agent_rep>`\\'s `evaluate <Composition.evaluate>` method when it is `executed

<OptimizationControlMechanism_Execution>`.  The OptimizationControlMechanism has a `state_input_port

<OptimizationControlMechanism.state_input_ports>` for every specification in the **state_features** arg of its

constructor (see `above <OptimizationControlMechanism_State_Features_Arg>`).

COMMENT:

OLD

If the `agent_rep is a Composition <OptimizationControlMechanism_Agent_Rep_Composition>`, then the

OptimizationControlMechanism has a state_input_port for every specification in the **state_features** arg of its

constructor (see `above <OptimizationControlMechanism_State_Features_Arg>`). If the `agent_rep is a

CompositionFunctionApproximator <OptimizationControlMechanism_Agent_Rep_CFA>`, then the OptimizationControlMechanism

has a state_input_port that receives a Projection from each Component specified in the **state_features** arg of its

constructor.

COMMENT

COMMENT:

In either, case the the `values <InputPort.value>` of the

`state_input_ports <OptimizationControlMechanism.state_input_ports>` are assigned to the `state_feature_values

<OptimizationControlMechanism.state_feature_values>` attribute that is used, in turn, by the

OptimizationControlMechanism's `evaluate_agent_rep <OptimizationControlMechanism.evaluate_agent_rep>` method to

estimate or predict the `net_outcome <ControlMechanism.net_outcome>` for a given `control_allocation

<ControlMechanism.control_allocation>` (see `OptimizationControlMechanism_Execution`).

State features can be of two types:

* *Input Features* -- these are values that either shadow the input received by an `InputPort` of a `Mechanisms

  <Mechanism>` in the `Composition` for which the OptimizationControlMechanism is a `controller

  <Composition.controller>` (irrespective of whether that is the OptimizationControlMechanism`s `agent_rep

  <OptimizationControlMechanism.agent_rep>`). They are implemented as `shadow InputPorts <InputPort_Shadow_Inputs>`

  (see `OptimizationControlMechanism_SHADOW_INPUTS_State_Feature` for specification) that receive a `Projection`

  from the same source as the Mechanism being shadowed.

..

* *Output Features* -- these are the `value <OutputPort.value>` of an `OutputPort` of a `Mechanism <Mechanism>` in

  the `Composition` for which the OptimizationControlMechanism is a `controller <Composition.controller>` (again,

  irrespective of whether it is the OptimizationControlMechanism`s `agent_rep

  <OptimizationControlMechanism.agent_rep>`); and each is assigned a

  `Projection` from the specified OutputPort(s) to the InputPort of the OptimizationControlMechanism for that feature.

The InputPorts assigned to the **state_features** are listed in the OptimizationControlMechanism's `state_input_port

<OptimizationControlMechanism's.state_input_port>` attribute, and their current `values <InputPort.value>` are

listed in its `state_feature_values <OptimizationControlMechanism.state_feature_values>` attribute.

The InputPorts assigned to the **state_features** are listed in the OptimizationControlMechanism's `state_input_port

<OptimizationControlMechanism's.state_input_port>` attribute, and their current `values <InputPort.value>` are

listed in its `state_feature_values <OptimizationControlMechanism.state_feature_values>` attribute.

COMMENT

.. _OptimizationControlMechanism_Outcome:

*outcome_input_ports*

~~~~~~~~~~~~~~~~~~~~~

The `outcome_input_ports <OptimizationControlMechanism.outcome_input_ports>` comprise either a single `OutputPort`

that receives a `Projection` from the OptimizationControlMechanism's `objective_mechanism

<OptimizationControlMechanism_ObjectiveMechanism>` if has one; or, if it does not, then an OutputPort for each

Component it `monitors <OptimizationControlMechanism_Monitor_for_Control>` to determine the `net_outcome

<ControlMechanism.net_outcome>` of executing its `agent_rep <OptimizationControlMechanism.agent_rep>` (see `outcome

arguments <OptimizationControlMechanism_Outcome_Args>` for how these are specified). The value(s) of the

`outcome_input_ports <OptimizationControlMechanism.outcome_input_ports>` are assigned to the

OptimizationControlMechanism's `outcome <ControlMechanism.outcome>` attribute.

.. _OptimizationControlMechanism_ObjectiveMechanism:

*objective_mechanism*

If an OptimizationControlMechanism has an `objective_mechanism <ControlMechanism.objective_mechanism>`, it is

assigned a single outcome_input_port, named *OUTCOME*, that receives a Projection from the objective_mechanism's

`OUTCOME OutputPort <ObjectiveMechanism_Output>`. The OptimizationControlMechanism's `objective_mechanism

<ControlMechanism.objective_mechanism>` is used to evaluate the outcome of executing its `agent_rep

<OptimizationControlMechanism.agent_rep>` for a given `state <OptimizationControlMechanism_State>`. This passes

the result to the OptimizationControlMechanism's *OUTCOME* InputPort, that is placed in its `outcome

<ControlMechanism.outcome>` attribute.

    .. note::

        An OptimizationControlMechanism's `objective_mechanism <ControlMechanism.objective_mechanism>` and the `function

        <ObjectiveMechanism.function>` of that Mechanism, are distinct from and should not be confused with the

        `objective_function <OptimizationFunction.objective_function>` parameter of the OptimizationControlMechanism's

        `function <OptimizationControlMechanism.function>`.  The `objective_mechanism

        <ControlMechanism.objective_mechanism>`\\'s `function <ObjectiveMechanism.function>` evaluates the `outcome

        <ControlMechanism.outcome>` of processing without taking into account the `costs <ControlMechanism.costs>` of

        the OptimizationControlMechanism's `control_signals <OptimizationControlMechanism.control_signals>`.  In

        contrast, its `evaluate_agent_rep <OptimizationControlMechanism.evaluate_agent_rep>` method, which is assigned

        as the `objective_function` parameter of its `function <OptimizationControlMechanism.function>`, takes the

        `costs <ControlMechanism.costs>` of the OptimizationControlMechanism's `control_signals

        <OptimizationControlMechanism.control_signals>` into account when calculating the `net_outcome` that it

        returns as its result.

COMMENT:

ADD HINT HERE RE: USE OF CONCATENATION

the items specified by `monitor_for_control

<ControlMechanism.monitor_for_control>` are all assigned `MappingProjections <MappingProjection>` to a single

*OUTCOME* InputPort.  This is assigned `Concatenate` as it `function <InputPort.function>`, which concatenates the

`values <Projection_Base.value>` of its Projections into a single array (that is, it is automatically configured

to use the *CONCATENATE* option of a ControlMechanism's `outcome_input_ports_option

<ControlMechanism.outcome_input_ports_option>` Parameter). This ensures that the input to the

OptimizationControlMechanism's `function <OptimizationControlMechanism.function>` has the same format as when an

`objective_mechanism <ControlMechanism.objective_mechanism>` has been specified, as described below.

COMMENT

.. _OptimizationControlMechanism_Monitor_for_Control:

*monitor_for_control*

If an OptimizationControlMechanism is not assigned an `objective_mechanism <ControlMechanism.objective_mechanism>`,

then its `outcome_input_ports <OptimizationControlMechanism.outcome_input_ports>` are determined by its

`monitor_for_control <ControlMechanism.monitor_for_control>` and `outcome_input_ports_option

<ControlMechanism.outcome_input_ports_option>` attributes, specified in the corresponding arguments of its

constructor (see `Outcomes arguments <OptimizationControlMechanism_Outcome_Args>`). The value(s) of the specified

Components are assigned as the OptimizationControlMechanism's `outcome <ControlMechanism.outcome>` attribute,

which is used to compute the `net_outcome <ControlMechanism.net_outcome>` of executing its `agent_rep

<OptimizationControlMechanism.agent_rep>`.

    .. note::

       If a `Node <Composition_Nodes>` other than an `OUTPUT <NodeRole.OUTPUT>` of a `nested <Composition_Nested>`

       Composition is `specified to be monitored <ControlMechanism_Monitor_for_Control>`, it is assigned as a `PROBE

       <NodeRole.PROBE>` of that nested Composition. Although `PROBE <NodeRole.PROBE>` Nodes are generally treated

       like `OUTPUT <NodeRole.OUTPUT>` Nodes (since they project out of the Composition to which they belong), their

       `value <Mechanism_Base.value>` is not included in the `output_values <Composition.output_values>` or `results

       <Composition.results>` attributes of the Composition for which the OptimizationControlMechanism is the

       `controller <Composition.controller>`, unless that Composition's `include_probes_in_output

       <Composition.include_probes_in_output>` attribute is set to True (see Probes `Composition_Probes` for additional

       information).

.. _OptimizationControlMechanism_Function:

*Function*

^^^^^^^^^^

The `function <OptimizationControlMechanism.function>` of an OptimizationControlMechanism is used to find the

`control_allocation <ControlMechanism.control_allocation>` that optimizes the `net_outcome

<ControlMechanism.net_outcome>` for the current (or expected) `state <OptimizationControlMechanism_State>`.

It is generally an `OptimizationFunction`, which in turn has `objective_function

<OptimizationFunction.objective_function>`, `search_function <OptimizationFunction.search_function>` and

`search_termination_function <OptimizationFunction.search_termination_function>` methods, as well as a `search_space

<OptimizationFunction.search_space>` attribute.  The `objective_function <OptimizationFunction.objective_function>`

is automatically assigned the OptimizationControlMechanism's `evaluate_agent_rep

<OptimizationControlMechanism.evaluate_agent_rep>` method, that is used to evaluate each `control_allocation

<ControlMechanism.control_allocation>` sampled from the `search_space <OptimizationFunction.search_space>` by the

`search_function <OptimizationFunction.search_function>` until the `search_termination_function

<OptimizationFunction.search_termination_function>` returns `True`.  The `net_outcome <ControlMechanism.net_outcome>`

returned by the call to `evaluate_agent_rep <OptimizationControlMechanism.evaluate_agent_rep>` is assigned to the

OptimizationControlMechanism's `optimal_net_outcome <OptimizationControlMechanism.optimal_net_outcome>` attribute,

and the `control_allocation  <ControlMechanism.control_allocation>` that yielded it is assigned to the

OptimizationControlMechanism's `optimal_control_allocation <OptimizationControlMechanism.optimal_control_allocation>`

attribute (see `OptimizationControlMechanism_Execution` for additional details).

.. _OptimizationControlMechanism_Custom_Function:

*Custom Function*

~~~~~~~~~~~~~~~~~

A custom function can be assigned as the OptimizationControlMechanism's `function

<OptimizationControlMechanism.function>`, however it must meet the following requirements:

  - It must accept as its first argument and return as its result an array with the same shape as the

    OptimizationControlMechanism's `control_allocation <ControlMechanism.control_allocation>`.

  ..

  - It must be able to execute the OptimizationControlMechanism's `evaluate_agent_rep

    <OptimizationControlMechanism.evaluate_agent_rep>` `num_estimates <OptimizationControlMechanism.num_estimates>`

    times, and aggregate the results in computing the `net_outcome <ControlMechanism.net_outcome>` for a given

    `control_allocation <ControlMechanism.control_allocation>` (see

    `OptimizationControlMechanism_Estimation_Randomization` for additional details).

  ..

  - It must implement a `reset` method that can accept as keyword arguments **objective_function**,

    **search_function**, **search_termination_function**, and **search_space**, and implement attributes

    with corresponding names.

.. _OptimizationControlMechanism_Search_Functions:

*Search Function, Search Space and Search Termination Function*

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Subclasses of OptimizationControlMechanism may implement their own `search_function

<OptimizationControlMechanism.search_function>` and `search_termination_function

<OptimizationControlMechanism.search_termination_function>` methods, as well as a

`control_allocation_search_space <OptimizationControlMechanism.control_allocation_search_space>` attribute, that are

passed as parameters to the `OptimizationFunction` when it is constructed.  These can be specified in the

constructor for an `OptimizationFunction` assigned as the **function** argument in the

OptimizationControlMechanism's constructor, as long as they are compatible with the requirements of

the OptimizationFunction and OptimizationControlMechanism.  If they are not specified, then defaults specified

either by the OptimizationControlMechanism or the OptimizationFunction are used.

.. _OptimizationControlMechanism_Default_Function:

*Default Function: GridSearch*

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If the **function** argument is not specified, the `GridSearch` `OptimizationFunction` is assigned as the default,

which evaluates the `net_outcome <ControlMechanism.net_outcome>` using the OptimizationControlMechanism's

`control_allocation_search_space <OptimizationControlMechanism.control_allocation_search_space>` as its

`search_space <OptimizationFunction.search_space>`, and returns the `control_allocation

<ControlMechanism.control_allocation>` that yields the greatest `net_outcome <ControlMechanism.net_outcome>`,

thus implementing a computation of `EVC <OptimizationControlMechanism_EVC>`.

.. _OptimizationControlMechanism_Output:

*Output*

^^^^^^^^

The output of OptimizationControlMechanism are its `control_signals <ControlMechanism.control_signals>` that implement

the `control_allocations <ControlMechanism.control_allocation>` it evaluates and optimizes. These their effects are

estimated over variation in the values of Components with random variables, then the OptimizationControlMechanism's

`control_signals <ControlMechanism.control_signals>` include an additional *RANDOMIZATION_CONTROL_SIGNAL* that

implements that variablity for the relevant Components, as described below.

.. _OptimizationControlMechanism_Randomization_Control_Signal:

*Randomization ControlSignal*

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If `num_estimates <OptimizationControlMechanism.num_estimates>` is specified (that is, it is not None), and

`agent_rep <OptimizationControlMechanism.agent_rep>` has any `Components <Component>` with random variables