• Home
  • Features
  • Pricing
  • Docs
  • Announcements
  • Sign In

PrincetonUniversity / PsyNeuLink / 15917088825

05 Jun 2025 04:18AM UTC coverage: 84.482% (+0.5%) from 84.017%
15917088825

push

github

web-flow
Merge pull request #3271 from PrincetonUniversity/devel

Devel

9909 of 12966 branches covered (76.42%)

Branch coverage included in aggregate %.

1708 of 1908 new or added lines in 54 files covered. (89.52%)

25 existing lines in 14 files now uncovered.

34484 of 39581 relevant lines covered (87.12%)

0.87 hits per line

Source File
Press 'n' to go to next uncovered line, 'b' for previous

80.68
/psyneulink/core/globals/mdf.py
1
"""
2

3
Contents
4
--------
5

6
  * `MDF_Overview`
7
  * `MDF_Examples`
8
  * `MDF_Model_Specification`
9

10
.. _MDF_Overview:
11

12

13
Overview
14
--------
15

16
The developers of PsyNeuLink are collaborating with the scientific community, as part of the `OpenNeuro effort
17
<https://openneuro.org>`_, to create a standard, serialzied format for the description and exchange of computational
18
models of brain and psychological function across different simulation environments. As part of this effort,
19
PsyNeuLink supports the `ModECI Model Description Format <https://github.com/ModECI/MDF/includes>`_ (MDF) by
20
including the ability to produce an MDF-compatible model from a PsyNeuLink model and to construct valid Python
21
scripts that express a PsyNeuLink model from an MDF model.
22

23
Any PsyNeuLink `Composition` or `Component` can be exported to MDF format using its `as_mdf_model` method or
24
to serialized format using its `json_summary` or `yaml_summary` methods. These methods generate strings that, passed into the
25
`generate_script_from_mdf` function, produce a valid Python script replicating the original PsyNeuLink model.
26
`write_mdf_file` can be used to write the serialization for one or more Compositions into a specified file (though
27
see `note <MDF_Write_Multiple_Compositions_Note>`). `generate_script_from_mdf` can accept either the string returned
28
by `get_mdf_serialized` or the name of a file containing one.
29
Calling ``exec(generate_script_from_mdf(<input>))`` will load into the current namespace all of the PsyNeuLink
30
objects specified in the ``input``; and `get_compositions` can be used to retrieve a list of all of the Compositions
31
in that namespace, including any generated by execution of `generate_script_from_mdf`. `generate_script_from_mdf`
32
may similarly be used to create a PsyNeuLink Python script from a ModECI MDF Model object, such as that created
33
by `as_mdf_model <Composition.as_mdf_model>`.
34

35
.. _MDF_Security_Warning:
36

37
.. warning::
38
   Use of `generate_script_from_json` or `generate_script_from_mdf` to generate a Python script from a file without taking proper precautions can
39
   introduce a security risk to the system on which the Python interpreter is running.  This is because it calls
40
   exec, which has the potential to execute non-PsyNeuLink-related code embedded in the file.  Therefore,
41
   `generate_script_from_json` or `generate_script_from_mdf` should be used to read only files of known and secure origin.
42

43
.. _MDF_Examples:
44

45
Model Examples
46
--------------
47

48
Below is an example of a script that implements a PsyNeuLink model of the Stroop model with conflict monitoring,
49
and its output in JSON. Running `generate_script_from_json` on the output will produce another PsyNeuLink script
50
that will give the same results when run on the same input as the original.
51

52
:download:`Download stroop_conflict_monitoring.py
53
<../../tests/mdf/stroop_conflict_monitoring.py>`
54

55
:download:`Download stroop_conflict_monitoring.json
56
<../../docs/source/_static/stroop_conflict_monitoring.json>`
57

58
.. _MDF_Model_Specification:
59

60
MDF Model Specification
61
------------------------
62

63
.. note::
64
    The format is in development, and is subject to change.
65

66
See https://github.com/ModECI/MDF/blob/main/docs/README.md#model
67

68

69
.. _MDF_Simple_Edge_Format:
70

71
MDF Simple Edge Format
72
----------------------
73

74
Models may be output as they are in PsyNeuLink or in "simple edge"
75
format. In simple edge format, PsyNeuLink Projections are written as a
76
combination of two Edges and an intermediate Node, because the generic
77
MDF execution engine does not support using Functions on Edges.
78
PsyNeuLink is capable of re-importing models exported by PsyNeuLink in
79
either form.
80
"""
81

82
import ast
1✔
83
import base64
1✔
84
import binascii
1✔
85
import dill
1✔
86
import enum
1✔
87
import graph_scheduler
1✔
88
import inspect
1✔
89
import json
1✔
90
import math
1✔
91
import numbers
1✔
92
import numpy
1✔
93
import os
1✔
94
import pickle
1✔
95
import pint
1✔
96
import psyneulink
1✔
97
import re
1✔
98
import tempfile
1✔
99
import tokenize
1✔
100
import types
1✔
101
import time
1✔
102
import warnings
1✔
103

104
from psyneulink._typing import Any, Union
1✔
105
from psyneulink.core.globals.keywords import \
1✔
106
    MODEL_SPEC_ID_GENERIC, MODEL_SPEC_ID_PARAMETER_SOURCE, \
107
    MODEL_SPEC_ID_PARAMETER_INITIAL_VALUE, MODEL_SPEC_ID_PARAMETER_VALUE, MODEL_SPEC_ID_PSYNEULINK, \
108
    MODEL_SPEC_ID_TYPE, MODEL_SPEC_ID_MDF_VARIABLE, MODEL_SPEC_ID_SHAPE, MODEL_SPEC_ID_METADATA, MODEL_SPEC_ID_INPUT_PORT_COMBINATION_FUNCTION
109
from psyneulink.core.globals.parameters import ParameterAlias
1✔
110
from psyneulink.core.globals.sampleiterator import SampleIterator
1✔
111
from psyneulink.core.globals.utilities import convert_to_list, gen_friendly_comma_str, get_all_explicit_arguments, \
1✔
112
    parse_string_to_psyneulink_object_string, parse_valid_identifier, safe_equals, convert_to_np_array
113

114
__all__ = [
1✔
115
    'MDFError', 'MDFSerializable', 'PNLJSONEncoder',
116
    'generate_json', 'generate_script_from_json', 'generate_script_from_mdf',
117
    'write_json_file', 'get_mdf_model', 'get_mdf_serialized', 'write_mdf_file'
118
]
119

120

121
# file extension to mdf common name
122
supported_formats = {
1✔
123
    'json': 'json',
124
    'yml': 'yaml',
125
    'yaml': 'yaml',
126
}
127

128

129
class MDFError(Exception):
1✔
130
    pass
1✔
131

132

133
class MDFSerializable:
1✔
134
    @property
1✔
135
    def json_summary(self):
1✔
136
        return self.as_mdf_model().to_json()
×
137

138
    @property
1✔
139
    def yaml_summary(self):
1✔
140
        return self.as_mdf_model().to_yaml()
×
141

142

143
# leaving this due to instructions in test_documentation_models
144
# (useful for exporting Composition results to JSON)
145
class PNLJSONEncoder(json.JSONEncoder):
1✔
146
    """
147
        A `JSONEncoder
148
        <https://docs.python.org/3/library/json.html#json.JSONEncoder>`_
149
        that parses `_dict_summary <Component._dict_summary>` output
150
        into a more JSON-friendly format.
151
    """
152
    def default(self, o):
1✔
153
        import modeci_mdf.mdf as mdf
×
154
        from psyneulink.core.components.component import Component, ComponentsMeta
×
155

156
        if isinstance(o, ComponentsMeta):
×
157
            return o.__name__
×
158
        elif isinstance(o, (type, types.BuiltinFunctionType)):
×
159
            if o.__module__ == 'builtins':
×
160
                # just give standard type, like float or int
161
                return f'{o.__name__}'
×
162
            elif o is numpy.ndarray:
×
163
                return f'{o.__module__}.array'
×
164
            else:
165
                # some builtin modules are internally "_module"
166
                # but are imported with "module"
167
                return f"{o.__module__.lstrip('_')}.{o.__name__}"
×
168
        elif isinstance(o, (enum.Enum, types.FunctionType, types.SimpleNamespace)):
×
169
            return str(o)
×
170
        elif isinstance(o, types.MethodType):
×
171
            return o.__qualname__
×
172
        elif o is NotImplemented:
×
173
            return None
×
174
        elif isinstance(o, Component):
×
175
            return o.name
×
176
        elif isinstance(o, SampleIterator):
×
177
            return f'{o.__class__.__name__}({repr(o.specification)})'
×
178
        elif isinstance(o, numpy.ndarray):
×
179
            try:
×
180
                return list(o)
×
181
            except TypeError:
×
182
                return o.item()
×
183
        elif isinstance(o, numpy.random.RandomState):
×
184
            return f'numpy.random.RandomState({o.seed})'
×
185
        elif isinstance(o, numpy.number):
×
186
            return o.item()
×
187
        elif isinstance(o, mdf.BaseWithId):
×
188
            return json.loads(o.to_json())
×
189

190
        try:
×
191
            return super().default(o)
×
192
        except TypeError:
×
193
            return str(o)
×
194

195

196
def _parse_pint_object(obj: Any) -> Union[pint.Unit, pint.Quantity, None]:
1✔
197
    # varying and don't have a consistent way to identify otherwise
198
    pint_errs = (
1✔
199
        AssertionError,
200
        AttributeError,
201
        TypeError,
202
        ValueError,
203
        pint.errors.DefinitionSyntaxError,
204
        pint.errors.UndefinedUnitError,
205
        tokenize.TokenError,
206
    )
207
    try:
1✔
208
        return psyneulink._unit_registry.Unit(obj)
1✔
209
    except pint_errs:
1✔
210
        pass
1✔
211

212
    try:
1✔
213
        return psyneulink._unit_registry.Quantity(obj)
1✔
214
    except pint_errs:
1✔
215
        pass
1✔
216

217
    return None
1✔
218

219

220
def _get_variable_parameter_name(obj):
1✔
221
    try:
1✔
222
        if obj.parameters.variable.mdf_name is not None:
1✔
223
            return obj.parameters.variable.mdf_name
1✔
224
    except AttributeError:
×
225
        pass
×
226

227
    return MODEL_SPEC_ID_MDF_VARIABLE
1✔
228

229

230
def _mdf_obj_from_dict(d):
1✔
231
    import modeci_mdf.mdf as mdf
1✔
232

233
    def _get_mdf_object(obj, cls_):
1✔
234
        try:
1✔
235
            model_id = obj['id']
1✔
236
        except KeyError:
1✔
237
            try:
1✔
238
                model_id = obj['metadata']['name']
1✔
239
            except KeyError:
1✔
240
                model_id = f'{cls_.__name__}_{time.perf_counter_ns()}'
1✔
241

242
        return cls_.from_dict({model_id: obj})
1✔
243

244
    for cls_name in mdf.__all__:
1✔
245
        cls_ = getattr(mdf, cls_name)
1✔
246
        if all([attr.name in d or attr.name in {'id', 'parameters'} for attr in cls_.__attrs_attrs__]):
1✔
247
            return _get_mdf_object(d, cls_)
1✔
248

249
    if 'function' in d and 'args' in d:
1✔
250
        return _get_mdf_object(d, mdf.Function)
1✔
251

252
    # nothing else seems to fit, try Function (unreliable)
253
    if 'value' in d:
1✔
254
        return _get_mdf_object(d, mdf.Function)
1✔
255

256
    return None
1✔
257

258

259
def _get_parameters_from_mdf_base_object(model, pnl_type):
1✔
260
    model_params = getattr(model, pnl_type._model_spec_id_parameters)
1✔
261

262
    if isinstance(model_params, list):
1✔
263
        parameters = {p.id: p.value for p in model_params}
1✔
264
    elif isinstance(model_params, dict):
1✔
265
        parameters = dict(model_params)
1✔
266
    else:
267
        parameters = {}
1✔
268

269
    return parameters
1✔
270

271

272
def _get_id_for_mdf_port(port, owner=None, afferent=None):
1✔
273
    if owner is None:
1!
274
        try:
1✔
275
            owner = port.owner
1✔
276
        except AttributeError:
1✔
277
            # handle owner as str
278
            pass
1✔
279
    if owner is NotImplemented:
1!
NEW
280
        owner = None
×
281
    owner = getattr(owner, 'name', owner)
1✔
282
    port = getattr(port, 'name', port)
1✔
283
    afferent = getattr(afferent, 'name', afferent)
1✔
284

285
    res = port
1✔
286
    if owner is not None:
1✔
287
        res = f'{owner}_{res}'
1✔
288
    if afferent is not None:
1✔
289
        res = f'{res}_input_port_from_{afferent}'
1✔
290

291
    return parse_valid_identifier(res)
1✔
292

293

294
def _parse_component_type(model_obj):
1✔
295
    def get_pnl_component_type(s):
1✔
296
        from psyneulink.core.components.component import ComponentsMeta
1✔
297

298
        try:
1✔
299
            return getattr(psyneulink, s)
1✔
300
        except AttributeError:
1✔
301
            for o in dir(psyneulink):
1✔
302
                if s.lower() == o.lower():
1!
303
                    o = getattr(psyneulink, o)
×
304
                    if isinstance(o, ComponentsMeta):
×
305
                        return o
×
306
            # if matching component not found, raise original exception
307
            raise
1✔
308

309
    type_str = None
1✔
310
    try:
1✔
311
        try:
1✔
312
            type_dict = model_obj.metadata[MODEL_SPEC_ID_TYPE]
1✔
313
        except AttributeError:
1✔
314
            # could be a dict specification
315
            type_str = model_obj[MODEL_SPEC_ID_METADATA][MODEL_SPEC_ID_TYPE]
1✔
316
    except (KeyError, TypeError):
1✔
317
        # specifically for functions the keyword is not 'type'
318
        type_str = model_obj.function
1✔
319

320
    if type_str is None:
1✔
321
        try:
1✔
322
            type_str = type_dict[MODEL_SPEC_ID_PSYNEULINK]
1✔
323
        except KeyError:
1✔
324
            # catch error outside of this function if necessary
325
            type_str = type_dict[MODEL_SPEC_ID_GENERIC]
×
326
        except TypeError:
1✔
327
            # actually a str
328
            type_str = type_dict
1✔
329
    elif isinstance(type_str, dict):
1!
330
        if len(type_str) != 1:
×
331
            raise MDFError
332
        else:
333
            elem = list(type_str.keys())[0]
×
334
            # not a function_type: args dict
335
            if MODEL_SPEC_ID_METADATA in type_str[elem]:
×
336
                raise MDFError
337
            else:
338
                type_str = elem
×
339

340
    try:
1✔
341
        # gets the actual psyneulink type (Component, etc..) from the module
342
        return get_pnl_component_type(type_str)
1✔
343
    except (AttributeError, TypeError):
1✔
344
        pass
1✔
345

346
    try:
1✔
347
        from modeci_mdf.functions.standard import mdf_functions
1✔
348
        mdf_functions[type_str]['function']
1✔
349
    # remove import/module errors when modeci_mdf is a package
350
    except (ImportError, KeyError, ModuleNotFoundError):
×
351
        pass
×
352
    else:
353
        return f"modeci_mdf.functions.standard.mdf_functions['{type_str}']['function']"
1✔
354

355
    try:
×
356
        getattr(math, type_str)
×
357
    except (AttributeError, TypeError):
×
358
        pass
×
359
    else:
360
        return f'math.{type_str}'
×
361

362
    try:
×
363
        eval(type_str)
×
364
    except (TypeError, SyntaxError):
×
365
        pass
×
366
    except NameError:
×
367
        return type_str
×
368
    else:
369
        return type_str
×
370

371
    raise MDFError(f'Invalid type specified for MDF object: {model_obj}')
372

373

374
def _parse_parameter_value(value, component_identifiers=None, name=None, parent_parameters=None):
1✔
375
    import modeci_mdf.mdf as mdf
1✔
376

377
    if component_identifiers is None:
1✔
378
        component_identifiers = {}
1✔
379

380
    exec('import numpy')
1✔
381
    try:
1✔
382
        pnl_type = _parse_component_type(value)
1✔
383
    except (AttributeError, TypeError, MDFError):
1✔
384
        # ignore parameters that aren't components
385
        pnl_type = None
1✔
386

387
    if isinstance(value, list):
1✔
388
        new_val = [_parse_parameter_value(x, component_identifiers, name, parent_parameters) for x in value]
1✔
389

390
        # check for ParameterPort spec
391
        if (
1✔
392
            len(value) == 2
393
            and isinstance(value[0], (numbers.Number, numpy.ndarray))
394
            and isinstance(value[1], dict)
395
        ):
396
            # make tuple instead of list
397
            value = f"({', '.join([str(x) for x in new_val])})"
1✔
398
        else:
399
            value = f"[{', '.join([str(x) for x in new_val])}]"
1✔
400
    elif isinstance(value, dict):
1✔
401
        if (
1!
402
            MODEL_SPEC_ID_PARAMETER_SOURCE in value
403
            and MODEL_SPEC_ID_PARAMETER_VALUE in value
404
        ):
405
            # handle ParameterPort spec
406
            try:
×
407
                value_type = eval(value[MODEL_SPEC_ID_TYPE])
×
408
            except Exception as e:
×
409
                raise MDFError(
410
                    'Invalid python type specified in MDF object: {0}'.format(
411
                        value[MODEL_SPEC_ID_TYPE]
412
                    )
413
                ) from e
414

415
            value = _parse_parameter_value(
×
416
                value[MODEL_SPEC_ID_PARAMETER_VALUE],
417
                component_identifiers,
418
                name,
419
                parent_parameters,
420
            )
421

422
            # handle tuples and numpy arrays, which both are dumped
423
            # as lists in JSON form
424
            if value_type is tuple:
×
425
                # convert list brackets to tuple brackets
426
                assert value[0] == '[' and value[-1] == ']'
×
427
                value = f'({value[1:-1]})'
×
428
            elif value_type is numpy.ndarray:
×
429
                value = f'{value[MODEL_SPEC_ID_TYPE]}({value})'
×
430
        elif MODEL_SPEC_ID_PARAMETER_INITIAL_VALUE in value:
1!
431
            # is a stateful parameter with initial value
432
            value = _parse_parameter_value(
×
433
                value[MODEL_SPEC_ID_PARAMETER_INITIAL_VALUE],
434
                component_identifiers,
435
                name,
436
                parent_parameters
437
            )
438
        elif MODEL_SPEC_ID_PARAMETER_VALUE in value and pnl_type is None:
1!
439
            # is a standard mdf Parameter class with value
440
            value = _parse_parameter_value(
×
441
                value[MODEL_SPEC_ID_PARAMETER_VALUE],
442
                component_identifiers,
443
                name,
444
                parent_parameters
445
            )
446
        else:
447
            if len(value) == 1:
1✔
448
                try:
1✔
449
                    identifier = list(value.keys())[0]
1✔
450
                except KeyError:
×
451
                    identifier = name
×
452

453
                mdf_object = value[identifier]
1✔
454
            else:
455
                try:
1✔
456
                    identifier = value['id']
1✔
457
                except KeyError:
1✔
458
                    identifier = name
1✔
459

460
                mdf_object = value
1✔
461

462
            # it is either a Component spec or just a plain dict
463
            if (
1✔
464
                identifier in component_identifiers
465
                and component_identifiers[identifier]
466
            ):
467
                # if this spec is already created as a node elsewhere,
468
                # then just use a reference
469
                value = identifier
1✔
470
            else:
471
                if not isinstance(mdf_object, mdf.Base):
1!
472
                    mdf_object = _mdf_obj_from_dict(mdf_object)
1✔
473

474
                try:
1✔
475
                    value = _generate_component_string(
1✔
476
                        mdf_object,
477
                        component_identifiers,
478
                        component_name=identifier,
479
                        parent_parameters=parent_parameters
480
                    )
481
                except (AttributeError, MDFError, KeyError, TypeError):
1✔
482
                    # standard dict handling
483
                    value = '{{{0}}}'.format(
1✔
484
                        ', '.join([
485
                            '{0}: {1}'.format(
486
                                str(_parse_parameter_value(k, component_identifiers, name)),
487
                                str(_parse_parameter_value(v, component_identifiers, name))
488
                            )
489
                            for k, v in value.items()
490
                        ])
491
                    )
492

493
    elif isinstance(value, str):
1✔
494
        # handle pointer to parent's parameter value
495
        try:
1✔
496
            return _parse_parameter_value(parent_parameters[value])
1✔
497
        except (KeyError, TypeError):
1✔
498
            pass
1✔
499

500
        # handle reference to psyneulink object
501
        obj_string = parse_string_to_psyneulink_object_string(value)
1✔
502
        if obj_string is not None:
1✔
503
            return f'psyneulink.{obj_string}'
1✔
504

505
        # handle dill string
506
        try:
1✔
507
            dill_str = base64.decodebytes(bytes(value, 'utf-8'))
1✔
508
            dill.loads(dill_str)
1✔
509
            return f'dill.loads({dill_str})'
1✔
510
        except (binascii.Error, pickle.UnpicklingError, EOFError):
1✔
511
            pass
1✔
512

513
        # handle IO port specification
514
        match = re.match(r'(.+)\.(.+)_ports\.(.+)', value)
1✔
515
        if match is not None:
1✔
516
            comp_name, port_type, name = match.groups()
1✔
517
            comp_identifer = parse_valid_identifier(comp_name)
1✔
518

519
            if comp_identifer in component_identifiers:
1!
520
                name_as_kw = parse_string_to_psyneulink_object_string(name)
1✔
521
                if name_as_kw is not None:
1!
522
                    name = f'psyneulink.{name_as_kw}'
1✔
523
                else:
524
                    name = f"'{name}'"
×
525

526
                return f'{comp_identifer}.{port_type}_ports[{name}]'
1✔
527

528
        # if value is just a non-fixed component name, use the fixed name
529
        identifier = parse_valid_identifier(value)
1✔
530
        if identifier in component_identifiers:
1✔
531
            value = identifier
1✔
532

533
        if _parse_pint_object(value) is not None:
1!
534
            value = f"'{value}'"
×
535

536
        evaluates = False
1✔
537
        try:
1✔
538
            eval(value)
1✔
539
            evaluates = True
1✔
540
        except (TypeError, NameError, SyntaxError):
1✔
541
            pass
1✔
542

543
        # handle generic string
544
        if (
1✔
545
            value not in component_identifiers
546
            # assume a string that contains a dot is a command, not a raw
547
            # string, this is definitely imperfect and can't handle the
548
            # legitimate case, but don't know how to distinguish..
549
            and '.' not in value
550
            and not evaluates
551
        ):
552
            value = f"'{value}'"
1✔
553

554
    elif isinstance(value, mdf.Base):
1✔
555
        value = _generate_component_string(
1✔
556
            value,
557
            component_identifiers,
558
            component_name=value.id,
559
            parent_parameters=parent_parameters
560
        )
561

562
    return value
1✔
563

564

565
def _generate_component_string(
1✔
566
    component_model,
567
    component_identifiers,
568
    component_name=None,
569
    parent_parameters=None,
570
    assignment=False,
571
    default_type=None   # used if no PNL or generic types are specified
572
):
573
    from psyneulink.core.components.functions.function import Function_Base
1✔
574
    from psyneulink.core.components.functions.userdefinedfunction import UserDefinedFunction
1✔
575

576
    try:
1✔
577
        component_type = _parse_component_type(component_model)
1✔
578
    except AttributeError as e:
1✔
579
        # acceptable to exclude type currently
580
        if default_type is not None:
1!
581
            component_type = default_type
×
582
        else:
583
            raise type(e)(
1✔
584
                f'{component_model} has no PNL or generic type and no '
585
                'default_type is specified'
586
            ) from e
587

588
    if component_name is None:
1✔
589
        name = component_model.id
1✔
590
    else:
591
        name = component_name
1✔
592
        try:
1✔
593
            assert component_name == component_model.id
1✔
594
        except KeyError:
×
595
            pass
×
596

597
    is_user_defined_function = False
1✔
598
    try:
1✔
599
        parameters = _get_parameters_from_mdf_base_object(component_model, component_type)
1✔
600
    except AttributeError:
1✔
601
        is_user_defined_function = True
1✔
602

603
    if is_user_defined_function or component_type is UserDefinedFunction:
1✔
604
        custom_func = component_type
1✔
605
        component_type = UserDefinedFunction
1✔
606
        parameters = _get_parameters_from_mdf_base_object(component_model, component_type)
1✔
607
        parameters['custom_function'] = f'{custom_func}'
1✔
608
        try:
1✔
609
            del component_model.metadata['custom_function']
1✔
610
        except KeyError:
×
611
            pass
×
612

613
    try:
1✔
614
        # args in function dict
615
        parameters.update(component_model.function[list(component_model.function.keys())[0]])
1✔
616
    except (AttributeError, KeyError):
1✔
617
        pass
1✔
618

619
    parameter_names = {}
1✔
620

621
    # TODO: remove this?
622
    # If there is a parameter that is the psyneulink identifier string
623
    # (as of this comment, 'pnl'), then expand these parameters as
624
    # normal ones. We don't check and expand for other
625
    # special strings here, because we assume that these are specific
626
    # to other modeling platforms.
627
    try:
1✔
628
        parameters.update(parameters[MODEL_SPEC_ID_PSYNEULINK])
1✔
629
        del parameters[MODEL_SPEC_ID_PSYNEULINK]
×
630
    except KeyError:
1✔
631
        pass
1✔
632

633
    try:
1✔
634
        functions = component_model.functions
1✔
635
    except AttributeError:
1✔
636
        try:
1✔
637
            functions = [_mdf_obj_from_dict(v) for k, v in component_model.metadata['functions'].items()]
1✔
638
        except KeyError:
1✔
639
            functions = None
1✔
640
        except AttributeError:
1✔
641
            functions = component_model.metadata['functions']
1✔
642

643
    # pnl objects only have one function unless specified in another way
644
    # than just "function"
645

646
    if functions is not None:
1✔
647
        dup_function_names = set([f.id for f in functions if f.id in component_identifiers])
1✔
648
        if len(dup_function_names) > 0:
1!
649
            warnings.warn(
×
650
                f'Functions ({gen_friendly_comma_str(dup_function_names)}) of'
651
                f' {name} share names of mechanisms or compositions in this'
652
                ' model. This is likely to cause incorrect script reproduction.'
653
            )
654

655
        function_determined_by_output_port = False
1✔
656

657
        try:
1✔
658
            output_ports = component_model.output_ports
1✔
659
        except AttributeError:
1✔
660
            pass
1✔
661
        else:
662
            if len(output_ports) == 1 or isinstance(output_ports, list):
1!
663
                try:
1✔
664
                    primary_output_port = output_ports[0]
1✔
665
                except KeyError:
1✔
666
                    primary_output_port = output_ports[list(output_ports)[0]]
×
667
                function_determined_by_output_port = True
1✔
668
            else:
669
                try:
×
670
                    # 'out_port' appears to be the general primary output_port term
671
                    # should ideally have a marker in mdf to define it as primary
672
                    primary_output_port = output_ports['out_port']
×
673
                except KeyError:
×
674
                    pass
×
675
                else:
676
                    function_determined_by_output_port = True
×
677

678
        # neuroml-style mdf has MODEL_SPEC_ID_PARAMETER_VALUE in output port definitions
679
        if function_determined_by_output_port and hasattr(primary_output_port, MODEL_SPEC_ID_PARAMETER_VALUE):
1✔
680
            parameter_names['function'] = re.sub(r'(.*)\[\d+\]', '\\1', getattr(primary_output_port, MODEL_SPEC_ID_PARAMETER_VALUE))
1✔
681
        else:
682
            parameter_names['function'] = [
1✔
683
                f.id for f in functions
684
                if not f.id.endswith(MODEL_SPEC_ID_INPUT_PORT_COMBINATION_FUNCTION)
685
            ][0]
686

687
        parameters['function'] = [f for f in functions if f.id == parameter_names['function']][0]
1✔
688

689
    assignment_str = f'{parse_valid_identifier(name)} = ' if assignment else ''
1✔
690

691
    additional_arguments = []
1✔
692
    # get the nonvariable arg and keyword arguments for the component's
693
    # constructor
694
    constructor_arguments = get_all_explicit_arguments(
1✔
695
        component_type,
696
        '__init__'
697
    )
698

699
    # put name as first argument
700
    if 'name' in constructor_arguments:
1✔
701
        additional_arguments.append(f"name='{name}'")
1✔
702

703
    if parent_parameters is None:
1✔
704
        parent_parameters = parameters
1✔
705

706
    parameters = {
1✔
707
        **{k: v for k, v in parent_parameters.items() if isinstance(v, dict) and MODEL_SPEC_ID_PARAMETER_INITIAL_VALUE in v},
708
        **parameters,
709
        **(component_model.metadata if component_model.metadata is not None else {})
710
    }
711

712
    # MDF input ports do not have functions, so their shape is
713
    # equivalent to ours after the InputPort function is run (this
714
    # function may change the shape of the default variable), so ignore
715
    # the input port shape if input_ports parameter is specified
716
    if 'variable' not in parameters and 'input_ports' not in parameters:
1✔
717
        try:
1✔
718
            ip = getattr(parameters['function'], Function_Base._model_spec_id_parameters)[MODEL_SPEC_ID_MDF_VARIABLE]
1✔
719
            var = convert_to_np_array(
×
720
                numpy.zeros(ast.literal_eval(component_model.input_ports[ip][MODEL_SPEC_ID_SHAPE])),
721
                dimension=2
722
            ).tolist()
723
            parameters['variable'] = var
×
724
        except KeyError:
1✔
725
            pass
1✔
726

727
    def parameter_value_matches_default(component_type, param, value):
1✔
728
        default_val = getattr(component_type.defaults, param)
1✔
729
        evaled_val = NotImplemented
1✔
730

731
        # see if val is a psyneulink class instantiation
732
        # if so, do not instantiate it (avoid offsetting rng for
733
        # testing - see if you can bypass another way?)
734
        try:
1✔
735
            eval(re.match(r'(psyneulink\.\w+)\(', value).group(1))
1✔
736
            is_pnl_instance = True
1✔
737
        except (AttributeError, TypeError, NameError, ValueError):
1✔
738
            is_pnl_instance = False
1✔
739

740
        if not is_pnl_instance:
1✔
741
            # val may be a string that evaluates to the default value
742
            # also skip listing in constructor in this case
743
            try:
1✔
744
                evaled_val = eval(value)
1✔
745
            except (TypeError, NameError, ValueError):
1✔
746
                pass
1✔
747
            except Exception:
×
748
                # Assume this occurred in creation of a Component
749
                # that probably needs some hidden/automatic modification.
750
                # Special handling here?
751
                # still relevant after testing for instance above?
752
                pass
×
753

754
        # skip specifying parameters that match the class defaults
755
        if (
1✔
756
            not safe_equals(value, default_val)
757
            and (
758
                evaled_val is NotImplemented
759
                or not safe_equals(evaled_val, default_val)
760
            )
761
        ):
762
            # test for dill use/equivalence
763
            try:
1✔
764
                is_dill_str = value[:5] == 'dill.'
1✔
765
            except TypeError:
1✔
766
                is_dill_str = False
1✔
767

768
            if (
1✔
769
                not is_dill_str
770
                or dill.dumps(eval(value)) != dill.dumps(default_val)
771
            ):
772
                return False
1✔
773

774
        return True
1✔
775

776
    mdf_names_to_pnl = {
1✔
777
        p.mdf_name: p.name for p in component_type.parameters
778
        if p.mdf_name is not None and not isinstance(p, ParameterAlias)
779
    }
780

781
    # sort on arg name
782
    for arg, val in sorted(parameters.items(), key=lambda p: p[0]):
1✔
783
        try:
1✔
784
            arg = mdf_names_to_pnl[arg]
1✔
785
        except KeyError:
1✔
786
            pass
1✔
787

788
        try:
1✔
789
            constructor_parameter_name = getattr(component_type.parameters, arg).constructor_argument
1✔
790
            # Some Parameters may be stored just to be replicated here, and
791
            # they may have different names than are used in the
792
            # constructor of the object.
793
            # Examples:
794
            #   Component.variable / default_variable
795
            #   ControlMechanism.output_ports / control
796
            if constructor_parameter_name is not None:
1✔
797
                constructor_arg = constructor_parameter_name
1✔
798
            else:
799
                constructor_arg = arg
1✔
800
        except AttributeError:
1✔
801
            constructor_arg = arg
1✔
802

803
        if constructor_arg in constructor_arguments:
1✔
804
            try:
1✔
805
                val = _parse_parameter_value(
1✔
806
                    val, component_identifiers,
807
                    name=parameter_names[arg],
808
                    parent_parameters=parent_parameters,
809
                )
810
            except KeyError:
1✔
811
                val = _parse_parameter_value(val, component_identifiers, parent_parameters=parent_parameters)
1✔
812

813
            if (
1✔
814
                (arg in component_type.parameters or constructor_arg in component_type.parameters)
815
                and not parameter_value_matches_default(component_type, arg, val)
816
            ):
817
                additional_arguments.append(f'{constructor_arg}={val}')
1✔
818
        elif component_type is UserDefinedFunction:
1✔
819
            try:
1✔
820
                val[MODEL_SPEC_ID_PARAMETER_INITIAL_VALUE]
1✔
821
            except (KeyError, TypeError):
1✔
822
                pass
1✔
823
            else:
824
                # is a stateful parameter corresponding to this function
825
                if val[MODEL_SPEC_ID_PARAMETER_VALUE] == name:
×
826
                    additional_arguments.append(f"stateful_parameter='{arg}'")
×
827

828
            if arg != MODEL_SPEC_ID_MDF_VARIABLE:
1✔
829
                val = _parse_parameter_value(
1✔
830
                    val, component_identifiers, parent_parameters=parent_parameters
831
                )
832

833
                try:
1✔
834
                    matches = parameter_value_matches_default(component_type, arg, val)
1✔
835
                except AttributeError:
1✔
836
                    matches = False
1✔
837

838
                if not matches:
1✔
839
                    additional_arguments.append(f'{constructor_arg}={val}')
1✔
840

841
    output = '{0}psyneulink.{1}{2}{3}{4}'.format(
1✔
842
        assignment_str,
843
        component_type.__name__,
844
        '(' if len(additional_arguments) > 0 else '',
845
        ', '.join(additional_arguments),
846
        ')' if len(additional_arguments) > 0 else '',
847
    )
848

849
    return output
1✔
850

851

852
def _generate_scheduler_string(
1✔
853
    scheduler_id,
854
    scheduler_model,
855
    component_identifiers,
856
    blacklist=[]
857
):
858
    output = []
1✔
859

860
    for node, condition in scheduler_model.node_specific.items():
1✔
861
        if node not in blacklist:
1✔
862
            output.append(
1✔
863
                '{0}.add_condition({1}, {2})'.format(
864
                    scheduler_id,
865
                    parse_valid_identifier(node),
866
                    _generate_condition_string(
867
                        condition,
868
                        component_identifiers
869
                    )
870
                )
871
            )
872

873
    termination_str = []
1✔
874
    for scale, cond in scheduler_model.termination.items():
1✔
875
        termination_str.insert(
1✔
876
            1,
877
            'psyneulink.{0}: {1}'.format(
878
                f'TimeScale.{str.upper(scale)}',
879
                _generate_condition_string(cond, component_identifiers)
880
            )
881
        )
882

883
    output.append(
1✔
884
        '{0}.termination_conds = {{{1}}}'.format(
885
            scheduler_id,
886
            ', '.join(termination_str)
887
        )
888
    )
889

890
    return '\n'.join(output)
1✔
891

892

893
def _generate_condition_string(condition_model, component_identifiers):
1✔
894
    def _parse_condition_arg_value(value):
1✔
895
        try:
1✔
896
            identifier = parse_valid_identifier(value)
1✔
897
        except TypeError:
1✔
898
            pass
1✔
899
        else:
900
            if identifier in component_identifiers:
1✔
901
                return str(identifier)
1✔
902

903
        try:
1✔
904
            getattr(psyneulink.core.scheduling.condition, value.type)
1✔
905
        except (AttributeError, KeyError, TypeError):
1✔
906
            pass
1✔
907
        else:
908
            return _generate_condition_string(value, component_identifiers)
1✔
909

910
        # handle value/outputport fix for threshold
911
        try:
1✔
912
            if re.match(r'\w+_OutputPort_0', value):
1!
913
                return '"value"'
×
914
        except TypeError:
1✔
915
            pass
1✔
916

917
        return str(_parse_parameter_value(value, component_identifiers))
1✔
918

919
    def _parse_graph_scheduler_type(typ):
1✔
920
        for ts, pnl_ts in graph_scheduler.time._time_scale_aliases.items():
1✔
921
            ts_class_name = graph_scheduler.time._time_scale_to_class_str(ts)
1✔
922
            pnl_ts_class_name = graph_scheduler.time._time_scale_to_class_str(pnl_ts)
1✔
923

924
            if ts_class_name in typ:
1✔
925
                return typ.replace(ts_class_name, pnl_ts_class_name)
1✔
926

927
        return typ
1✔
928

929
    args_str = ''
1✔
930
    cond_type = _parse_graph_scheduler_type(condition_model.type)
1✔
931
    sig = inspect.signature(getattr(psyneulink, cond_type).__init__)
1✔
932

933
    var_positional_arg_name = None
1✔
934

935
    for name, param in sig.parameters.items():
1✔
936
        if param.kind is inspect.Parameter.VAR_POSITIONAL:
1✔
937
            var_positional_arg_name = name
1✔
938
            break
1✔
939

940
    args_dict = condition_model.kwargs
1✔
941

942
    try:
1✔
943
        pos_args = args_dict[var_positional_arg_name]
1✔
944
    except KeyError:
1✔
945
        pass
1✔
946
    else:
947
        if len(pos_args) > 0:
1✔
948
            arg_str_list = []
1✔
949
            for arg in pos_args:
1✔
950
                # handle nested Conditions
951
                try:
1✔
952
                    arg = _generate_condition_string(arg, component_identifiers)
1✔
953
                except TypeError:
×
954
                    pass
×
955

956
                arg_str_list.append(_parse_condition_arg_value(arg))
1✔
957
            args_str = f", {', '.join(arg_str_list)}"
1✔
958

959
    kwargs_str = ''
1✔
960
    kwargs = {k: v for k, v in args_dict.items() if k not in {'function', var_positional_arg_name}}
1✔
961
    if len(kwargs) > 0:
1✔
962
        kwarg_str_list = []
1✔
963
        for key, val in kwargs.items():
1✔
964
            kwarg_str_list.append(f'{key}={_parse_condition_arg_value(val)}')
1✔
965
        kwargs_str = f", {', '.join(kwarg_str_list)}"
1✔
966

967
    if 'function' in args_dict and args_dict['function'] is not None:
1!
968
        func_str = args_dict['function']
×
969
    else:
970
        func_str = ''
1✔
971

972
    arguments_str = '{0}{1}{2}'.format(
1✔
973
        func_str,
974
        args_str,
975
        kwargs_str
976
    )
977
    if len(arguments_str) > 0 and arguments_str[0] == ',':
1✔
978
        arguments_str = arguments_str[2:]
1✔
979

980
    return f'psyneulink.{cond_type}({arguments_str})'
1✔
981

982

983
def _generate_composition_string(graph, component_identifiers):
1✔
984
    import modeci_mdf.mdf as mdf
1✔
985

986
    # used if no generic types are specified
987
    default_composition_type = psyneulink.Composition
1✔
988
    default_node_type = psyneulink.ProcessingMechanism
1✔
989
    default_edge_type = psyneulink.MappingProjection
1✔
990

991
    control_mechanism_types = (psyneulink.ControlMechanism, )
1✔
992
    # these are not actively added to a Composition
993
    implicit_types = (
1✔
994
        psyneulink.ObjectiveMechanism,
995
        psyneulink.ControlProjection,
996
        psyneulink.AutoAssociativeProjection,
997
        psyneulink.LearningMechanism,
998
        psyneulink.LearningProjection,
999
    )
1000
    implicit_roles = (
1✔
1001
        psyneulink.NodeRole.LEARNING,
1002
    )
1003

1004
    output = []
1✔
1005

1006
    comp_identifer = parse_valid_identifier(graph.id)
1✔
1007

1008
    def alphabetical_order(items):
1✔
1009
        alphabetical = enumerate(
1✔
1010
            sorted(items)
1011
        )
1012
        return {
1✔
1013
            parse_valid_identifier(item[1]): item[0]
1014
            for item in alphabetical
1015
        }
1016

1017
    # get order in which nodes were added
1018
    # may be node names or dictionaries
1019
    try:
1✔
1020
        node_order = graph.metadata['node_ordering']
1✔
1021
        node_order = {
1✔
1022
            parse_valid_identifier(list(node.keys())[0]) if isinstance(node, dict)
1023
            else parse_valid_identifier(node): node_order.index(node)
1024
            for node in node_order
1025
        }
1026

1027
        unspecified_node_order = {
1✔
1028
            node: position + len(node_order)
1029
            for node, position in alphabetical_order([
1030
                parse_valid_identifier(n.id) for n in graph.nodes if n.id not in node_order
1031
            ]).items()
1032
        }
1033

1034
        node_order.update(unspecified_node_order)
1✔
1035

1036
        assert all([
1✔
1037
            (parse_valid_identifier(node.id) in node_order)
1038
            for node in graph.nodes
1039
        ])
1040
    except (KeyError, TypeError, AssertionError):
×
1041
        # if no node_ordering attribute exists, fall back to
1042
        # alphabetical order
1043
        node_order = alphabetical_order([parse_valid_identifier(n.id) for n in graph.nodes])
×
1044

1045
    keys_to_delete = []
1✔
1046

1047
    for node in graph.nodes:
1✔
1048
        try:
1✔
1049
            component_type = _parse_component_type(node)
1✔
1050
        except (AttributeError, KeyError):
×
1051
            # will use a default type
1052
            pass
×
1053
        else:
1054
            # projection was written out as a node for simple_edge_format
1055
            if issubclass(component_type, psyneulink.Projection_Base):
1✔
1056
                assert len(node.input_ports) == 1
1✔
1057
                assert len(node.output_ports) == 1
1✔
1058

1059
                extra_projs_to_delete = set()
1✔
1060

1061
                sender = None
1✔
1062
                sender_port = None
1✔
1063
                receiver = None
1✔
1064
                receiver_port = None
1✔
1065

1066
                for proj in graph.edges:
1✔
1067
                    if proj.receiver == node.id:
1✔
1068
                        assert 'dummy' in proj.id
1✔
1069
                        sender = proj.sender
1✔
1070
                        sender_port = proj.sender_port
1✔
1071
                        extra_projs_to_delete.add(proj.id)
1✔
1072

1073
                    if proj.sender == node.id:
1✔
1074
                        assert 'dummy' in proj.id
1✔
1075
                        receiver = proj.receiver
1✔
1076
                        receiver_port = proj.receiver_port
1✔
1077
                        # if for some reason the projection has node as both sender and receiver
1078
                        # this is a bug, let the deletion fail
1079
                        extra_projs_to_delete.add(proj.id)
1✔
1080

1081
                if sender is None:
1✔
1082
                    raise MDFError(f'Dummy node {node.id} for projection has no sender in projections list')
1083

1084
                if receiver is None:
1✔
1085
                    raise MDFError(f'Dummy node {node.id} for projection has no receiver in projections list')
1086

1087
                main_proj = mdf.Edge(
1✔
1088
                    id=node.id.replace('_dummy_node', ''),
1089
                    sender=sender,
1090
                    receiver=receiver,
1091
                    sender_port=sender_port,
1092
                    receiver_port=receiver_port,
1093
                    metadata={
1094
                        # variable isn't specified for projections
1095
                        **{k: v for k, v in node.metadata.items() if k != 'variable'},
1096
                        'functions': node.functions
1097
                    }
1098
                )
1099
                proj.parameters = {p.id: p for p in node.parameters}
1✔
1100
                graph.edges.append(main_proj)
1✔
1101

1102
                keys_to_delete.append(node.id)
1✔
1103
                for p in extra_projs_to_delete:
1✔
1104
                    del graph.edges[graph.edges.index([e for e in graph.edges if e.id == p][0])]
1✔
1105

1106
                for nr_item in ['required_node_roles', 'excluded_node_roles']:
1✔
1107
                    nr_removal_indices = []
1✔
1108

1109
                    for i, (nr_name, nr_role) in enumerate(
1✔
1110
                        graph.metadata[nr_item]
1111
                    ):
1112
                        if nr_name == node.id:
1✔
1113
                            nr_removal_indices.append(i)
1✔
1114

1115
                    for i in nr_removal_indices:
1✔
1116
                        del graph.metadata[nr_item][i]
1✔
1117

1118
    for name_to_delete in keys_to_delete:
1✔
1119
        del graph.nodes[graph.nodes.index([n for n in graph.nodes if n.id == name_to_delete][0])]
1✔
1120

1121
    # generate string for Composition itself
1122
    output.append(
1✔
1123
        "{0} = {1}\n".format(
1124
            comp_identifer,
1125
            _generate_component_string(
1126
                graph,
1127
                component_identifiers,
1128
                component_name=graph.id,
1129
                default_type=default_composition_type
1130
            )
1131
        )
1132
    )
1133
    component_identifiers[comp_identifer] = True
1✔
1134

1135
    mechanisms = []
1✔
1136
    compositions = []
1✔
1137
    control_mechanisms = []
1✔
1138
    implicit_mechanisms = []
1✔
1139

1140
    try:
1✔
1141
        node_roles = {
1✔
1142
            parse_valid_identifier(node): role for (node, role) in
1143
            graph.metadata['required_node_roles']
1144
        }
1145
    except KeyError:
×
1146
        node_roles = []
×
1147

1148
    try:
1✔
1149
        excluded_node_roles = {
1✔
1150
            parse_valid_identifier(node): role for (node, role) in
1151
            graph.metadata['excluded_node_roles']
1152
        }
1153
    except KeyError:
1✔
1154
        excluded_node_roles = []
1✔
1155

1156
    # add nested compositions and mechanisms in order they were added
1157
    # to this composition
1158
    for node in sorted(
1✔
1159
        graph.nodes,
1160
        key=lambda item: node_order[parse_valid_identifier(item.id)]
1161
    ):
1162
        if isinstance(node, mdf.Graph):
1!
1163
            compositions.append(node)
×
1164
        else:
1165
            try:
1✔
1166
                component_type = _parse_component_type(node)
1✔
1167
            except (AttributeError, KeyError):
×
1168
                component_type = default_node_type
×
1169
            identifier = parse_valid_identifier(node.id)
1✔
1170

1171
            try:
1✔
1172
                node_role = eval(_parse_parameter_value(node_roles[identifier]))
1✔
1173
            except (KeyError, TypeError):
1✔
1174
                node_role = None
1✔
1175

1176
            if issubclass(component_type, control_mechanism_types):
1✔
1177
                control_mechanisms.append(node)
1✔
1178
                component_identifiers[identifier] = True
1✔
1179
            elif (
1✔
1180
                issubclass(component_type, implicit_types)
1181
                or node_role in implicit_roles
1182
            ):
1183
                implicit_mechanisms.append(node)
1✔
1184
            else:
1185
                mechanisms.append(node)
1✔
1186
                component_identifiers[identifier] = True
1✔
1187

1188
    implicit_names = [node.id for node in implicit_mechanisms + control_mechanisms]
1✔
1189

1190
    for mech in mechanisms:
1✔
1191
        try:
1✔
1192
            mech_type = _parse_component_type(mech)
1✔
1193
        except (AttributeError, KeyError):
×
1194
            mech_type = None
×
1195

1196
        if (
1✔
1197
            isinstance(mech_type, type)
1198
            and issubclass(mech_type, psyneulink.Function)
1199
        ):
1200
            # removed branch converting functions defined as nodes
1201
            # should no longer happen with recent MDF versions
1202
            assert False
1203

1204
        output.append(
1✔
1205
            _generate_component_string(
1206
                mech,
1207
                component_identifiers,
1208
                component_name=parse_valid_identifier(mech.id),
1209
                assignment=True,
1210
                default_type=default_node_type
1211
            )
1212
        )
1213
    if len(mechanisms) > 0:
1!
1214
        output.append('')
1✔
1215

1216
    for mech in control_mechanisms:
1✔
1217
        output.append(
1✔
1218
            _generate_component_string(
1219
                mech,
1220
                component_identifiers,
1221
                component_name=parse_valid_identifier(mech.id),
1222
                assignment=True,
1223
                default_type=default_node_type
1224
            )
1225
        )
1226

1227
    if len(control_mechanisms) > 0:
1✔
1228
        output.append('')
1✔
1229

1230
    # recursively generate string for inner Compositions
1231
    for comp in compositions:
1!
1232
        output.append(
×
1233
            _generate_composition_string(
1234
                comp,
1235
                component_identifiers
1236
            )
1237
        )
1238
    if len(compositions) > 0:
1!
1239
        output.append('')
×
1240

1241
    # do not add the controller as a normal node
1242
    try:
1✔
1243
        controller_name = graph.metadata['controller']['id']
1✔
1244
    except (AttributeError, KeyError, TypeError):
1✔
1245
        controller_name = None
1✔
1246

1247
    for node in sorted(
1✔
1248
        graph.nodes,
1249
        key=lambda item: node_order[parse_valid_identifier(item.id)]
1250
    ):
1251
        name = node.id
1✔
1252
        if (
1✔
1253
            name not in implicit_names
1254
            and name != controller_name
1255
        ):
1256
            name = parse_valid_identifier(name)
1✔
1257

1258
            output.append(
1✔
1259
                '{0}.add_node({1}{2})'.format(
1260
                    comp_identifer,
1261
                    name,
1262
                    ', {0}'.format(
1263
                        _parse_parameter_value(
1264
                            node_roles[name],
1265
                            component_identifiers
1266
                        )
1267
                    ) if name in node_roles else ''
1268
                )
1269
            )
1270
    if len(graph.nodes) > 0:
1!
1271
        output.append('')
1✔
1272

1273
    if len(excluded_node_roles) > 0:
1!
1274
        for node, roles in excluded_node_roles.items():
×
1275
            if name not in implicit_names and name != controller_name:
×
1276
                output.append(
×
1277
                    f'{comp_identifer}.exclude_node_roles({node}, {_parse_parameter_value(roles, component_identifiers)})'
1278
                )
1279
        output.append('')
×
1280

1281
    # generate string to add the projections
1282
    for proj in graph.edges:
1✔
1283
        try:
1✔
1284
            projection_type = _parse_component_type(proj)
1✔
1285
        except (AttributeError, KeyError):
×
1286
            projection_type = default_edge_type
×
1287

1288
        receiver = parse_valid_identifier(proj.receiver)
1✔
1289
        receiver_port = parse_valid_identifier(proj.receiver_port)
1✔
1290

1291
        # original InputPort has multiple afferents, so maps to multiple
1292
        # MDF input ports with this id form
1293
        multi_afferent_pat = rf'^{receiver}_(.*){_get_id_for_mdf_port("", afferent=proj.id)}$'
1✔
1294

1295
        as_multi_afferent = re.match(multi_afferent_pat, proj.receiver_port)
1✔
1296
        if as_multi_afferent:
1✔
1297
            receiver_port = as_multi_afferent.group(1)
1✔
1298

1299
        receiver_port = re.sub(rf'({receiver}_)?(.*)', r'\2', receiver_port)
1✔
1300
        # undo parse_valid_identifier to PNL default naming scheme if applicable
1301
        receiver_port = re.sub(r'(InputPort)_(.*)', r'\1-\2', receiver_port)
1✔
1302
        # make receiver point to a specific port if not default
1303
        if receiver_port != 'InputPort-0':
1✔
1304
            receiver = f'{receiver}.input_ports["{receiver_port}"]'
1✔
1305

1306
        if (
1✔
1307
            not issubclass(projection_type, implicit_types)
1308
            and proj.sender not in implicit_names
1309
            and proj.receiver not in implicit_names
1310
        ):
1311
            output.append(
1✔
1312
                '{0}.add_projection(projection={1}, sender={2}, receiver={3})'.format(
1313
                    comp_identifer,
1314
                    _generate_component_string(
1315
                        proj,
1316
                        component_identifiers,
1317
                        default_type=default_edge_type
1318
                    ),
1319
                    parse_valid_identifier(proj.sender),
1320
                    receiver,
1321
                )
1322
            )
1323

1324
    # add controller if it exists (must happen after projections)
1325
    if controller_name is not None:
1✔
1326
        output.append(
1✔
1327
            '{0}.add_controller({1})'.format(
1328
                comp_identifer,
1329
                parse_valid_identifier(controller_name)
1330
            )
1331
        )
1332

1333
    # add schedulers
1334
    # blacklist automatically generated nodes because they will
1335
    # not exist in the script namespace
1336
    output.append('')
1✔
1337
    output.append(
1✔
1338
        _generate_scheduler_string(
1339
            f'{comp_identifer}.scheduler',
1340
            graph.conditions,
1341
            component_identifiers,
1342
            blacklist=implicit_names
1343
        )
1344
    )
1345

1346
    return output
1✔
1347

1348

1349
def generate_script_from_json(model_input, outfile=None):
1✔
1350
    """
1351
        Generate a Python script from JSON **model_input** in the
1352
        `general JSON format <JSON_Model_Specification>`
1353

1354
        .. warning::
1355
           Use of `generate_script_from_json` to generate a Python script from a file without taking proper precautions
1356
           can introduce a security risk to the system on which the Python interpreter is running.  This is because it
1357
           calls exec, which has the potential to execute non-PsyNeuLink-related code embedded in the file.  Therefore,
1358
           `generate_script_from_json` should be used to read only files of known and secure origin.
1359

1360
        Arguments
1361
        ---------
1362

1363
            model_input : str
1364
                a JSON string in the proper format, or a filename
1365
                containing such
1366

1367
        Returns
1368
        -------
1369

1370
            Text of Python script : str
1371

1372

1373

1374
    """
1375
    warnings.warn(
×
1376
        'generate_script_from_json is replaced by generate_script_from_mdf and will be removed in a future version',
1377
        FutureWarning
1378
    )
1379
    return generate_script_from_mdf(model_input, outfile)
×
1380

1381

1382
def generate_script_from_mdf(model_input, outfile=None):
1✔
1383
    """
1384
        Generate a Python script from MDF model **model_input**
1385

1386
        .. warning::
1387
           Use of `generate_script_from_mdf` to generate a Python script from a model without taking proper precautions
1388
           can introduce a security risk to the system on which the Python interpreter is running.  This is because it
1389
           calls exec, which has the potential to execute non-PsyNeuLink-related code embedded in the file.  Therefore,
1390
           `generate_script_from_mdf` should be used to read only model of known and secure origin.
1391

1392
        Arguments
1393
        ---------
1394

1395
            model_input : modeci_mdf.Model
1396

1397
        Returns
1398
        -------
1399

1400
            Text of Python script : str
1401
    """
1402
    import modeci_mdf.mdf as mdf
1✔
1403
    from modeci_mdf.utils import load_mdf
1✔
1404

1405
    def get_declared_identifiers(model):
1✔
1406
        names = set()
1✔
1407

1408
        for graph in model.graphs:
1✔
1409
            names.add(parse_valid_identifier(graph.id))
1✔
1410
            for node in graph.nodes:
1✔
1411
                if isinstance(node, mdf.Graph):
1!
1412
                    names.update(get_declared_identifiers(graph))
×
1413

1414
                names.add(parse_valid_identifier(node.id))
1✔
1415

1416
        return names
1✔
1417

1418
    # accept either json string or filename
1419
    try:
1✔
1420
        model = load_mdf(model_input)
1✔
1421
    except (FileNotFoundError, OSError, ValueError):
1✔
1422
        try:
1✔
1423
            model = mdf.Model.from_json(model_input)
1✔
1424
        except json.decoder.JSONDecodeError:
1✔
1425
            # assume yaml
1426
            # delete=False because of problems with reading file on windows
1427
            with tempfile.NamedTemporaryFile(mode='w', suffix='.yml', delete=False) as f:
1✔
1428
                f.write(model_input)
1✔
1429
                model = load_mdf(f.name)
1✔
1430

1431
    imports_str = ''
1✔
1432
    comp_strs = []
1✔
1433
    # maps declared names to whether they are accessible in the script
1434
    # locals. that is, each of these will be names specified in the
1435
    # composition and subcomposition nodes, and their value in this dict
1436
    # will correspond to True if they can be referenced by this name in the
1437
    # script
1438
    component_identifiers = {
1✔
1439
        i: False
1440
        for i in get_declared_identifiers(model)
1441
    }
1442

1443
    for graph in model.graphs:
1✔
1444
        comp_strs.append(_generate_composition_string(graph, component_identifiers))
1✔
1445

1446
    module_friendly_name_mapping = {
1✔
1447
        'psyneulink': 'pnl',
1448
        'dill': 'dill',
1449
        'numpy': 'np'
1450
    }
1451

1452
    potential_module_names = set()
1✔
1453
    module_names = set()
1✔
1454
    model_output = []
1✔
1455

1456
    for i in range(len(comp_strs)):
1✔
1457
        # greedy and non-greedy
1458
        for cs in comp_strs[i]:
1✔
1459
            cs_potential_names = set([
1✔
1460
                *re.findall(r'([A-Za-z_\.]+)\.', cs),
1461
                *re.findall(r'([A-Za-z_\.]+?)\.', cs)
1462
            ])
1463
            potential_module_names.update(cs_potential_names)
1✔
1464

1465
        for module in potential_module_names:
1✔
1466
            if module not in component_identifiers:
1✔
1467
                try:
1✔
1468
                    exec(f'import {module}')
1✔
1469
                    module_names.add(module)
1✔
1470
                except (ImportError, ModuleNotFoundError, SyntaxError):
1✔
1471
                    pass
1✔
1472

1473
        for j in range(len(comp_strs[i])):
1✔
1474
            for module in module_names.copy():
1✔
1475
                try:
1✔
1476
                    friendly_name = module_friendly_name_mapping[module]
1✔
1477
                    # Use '\b' "beginning of word" to avoid mangling references
1478
                    # to psyneulink in dill pickled strings.
1479
                    comp_strs[i][j] = re.sub(f'\\b{module}\\.', f'{friendly_name}.', comp_strs[i][j])
1✔
1480
                except KeyError:
1✔
1481
                    pass
1✔
1482

1483
        for m in module_names.copy():
1✔
1484
            for n in module_names.copy():
1✔
1485
                # remove potential modules that are substrings of another
1486
                if m is not n and m in n:
1✔
1487
                    module_names.remove(m)
1✔
1488

1489
        for module in sorted(module_names):
1✔
1490
            try:
1✔
1491
                friendly_name = module_friendly_name_mapping[module]
1✔
1492
            except KeyError:
1✔
1493
                friendly_name = module
1✔
1494

1495
            imports_str += 'import {0}{1}\n'.format(
1✔
1496
                module,
1497
                f' as {friendly_name}' if friendly_name != module else ''
1498
            )
1499

1500
        comp_strs[i] = '\n'.join(comp_strs[i])
1✔
1501

1502
    model_output = '{0}{1}{2}'.format(
1✔
1503
        imports_str,
1504
        '\n' if len(imports_str) > 0 else '',
1505
        '\n'.join(comp_strs)
1506
    )
1507

1508
    if outfile is not None:
1!
1509
        # pass through any file exceptions
1510
        with open(outfile, 'w') as outfile:
×
1511
            outfile.write(model_output)
×
1512
            print(f'Wrote script to {outfile.name}')
×
1513
    else:
1514
        return model_output
1✔
1515

1516

1517
def generate_json(*compositions, simple_edge_format=True):
1✔
1518
    """
1519
        Generate the `general JSON format <JSON_Model_Specification>`
1520
        for one or more `Compositions <Composition>` and associated
1521
        objects.
1522
        .. _MDF_Write_Multiple_Compositions_Note:
1523

1524
        .. note::
1525
           At present, if more than one Composition is specified, all
1526
           must be fully disjoint;  that is, they must not share any
1527
           `Components <Component>` (e.g., `Mechanism`, `Projections`
1528
           etc.). This limitation will be addressed in a future update.
1529

1530
        Arguments:
1531
            *compositions : Composition
1532
                specifies `Composition` or iterable of ones to be output
1533
                in JSON
1534
    """
1535
    warnings.warn(
×
1536
        'generate_json is replaced by get_mdf_serialized and will be removed in a future version',
1537
        FutureWarning
1538
    )
1539
    return get_mdf_serialized(*compositions, fmt='json', simple_edge_format=simple_edge_format)
×
1540

1541

1542
def get_mdf_serialized(*compositions, fmt='json', simple_edge_format=True):
1✔
1543
    """
1544
        Generate the `general MDF serialized format <JSON_Model_Specification>`
1545
        for one or more `Compositions <Composition>` and associated
1546
        objects.
1547

1548
        .. note::
1549
           At present, if more than one Composition is specified, all
1550
           must be fully disjoint;  that is, they must not share any
1551
           `Components <Component>` (e.g., `Mechanism`, `Projections`
1552
           etc.). This limitation will be addressed in a future update.
1553

1554
        Arguments:
1555
            *compositions : Composition
1556
                specifies `Composition` or iterable of ones to be output
1557
                in **fmt**
1558

1559
            fmt : str
1560
                specifies file format of output. Current options ('json', 'yml'/'yaml')
1561

1562
            simple_edge_format : bool
1563
                specifies use of
1564
                `simple edge format <MDF_Simple_Edge_Format>` or not
1565
    """
1566
    model = get_mdf_model(*compositions, simple_edge_format=simple_edge_format)
1✔
1567

1568
    try:
1✔
1569
        return getattr(model, f'to_{supported_formats[fmt]}')()
1✔
1570
    except AttributeError as e:
×
1571
        raise ValueError(
1572
            f'Unsupported MDF output format "{fmt}". Supported formats: {gen_friendly_comma_str(supported_formats.keys())}'
1573
        ) from e
1574

1575

1576
def write_json_file(compositions, filename:str, path:str=None, simple_edge_format=True):
1✔
1577
    """
1578
        Write one or more `Compositions <Composition>` and associated objects to file in the `general JSON format
1579
        <JSON_Model_Specification>`
1580

1581
        .. _MDF_Write_Multiple_Compositions_Note:
1582

1583
        .. note::
1584
           At present, if more than one Composition is specified, all must be fully disjoint;  that is, they must not
1585
           share  any `Components <Component>` (e.g., `Mechanism`, `Projections` etc.).  This limitation will be
1586
           addressed in a future update.
1587

1588
        Arguments
1589
        ---------
1590
        compositions : Composition or list˚
1591
             specifies `Composition` or list of ones to be written to **filename**
1592

1593
        filename : str
1594
             specifies name of file in which to write JSON specification of `Composition(s) <Composition>`
1595
             and associated objects.
1596

1597
        path : str : default None
1598
             specifies path of file for JSON specification;  if it is not specified then the current directory is used.
1599

1600
    """
1601
    warnings.warn(
×
1602
        'write_json_file is replaced by write_mdf_file and will be removed in a future version',
1603
        FutureWarning
1604
    )
1605
    write_mdf_file(compositions, filename, path, 'json', simple_edge_format)
×
1606

1607

1608
def write_mdf_file(compositions, filename: str, path: str = None, fmt: str = None, simple_edge_format: bool = True):
1✔
1609
    """
1610
        Write the `general MDF serialized format <MDF_Model_Specification>`
1611
        for one or more `Compositions <Composition>` and associated
1612
        objects to file.
1613

1614
        .. note::
1615
           At present, if more than one Composition is specified, all
1616
           must be fully disjoint;  that is, they must not share any
1617
           `Components <Component>` (e.g., `Mechanism`, `Projections`
1618
           etc.). This limitation will be addressed in a future update.
1619

1620
        Arguments:
1621
            compositions : Composition or list
1622
                specifies `Composition` or list of ones to be written to
1623
                **filename**
1624

1625
            filename : str
1626
                specifies name of file in which to write MDF
1627
                specification of `Composition(s) <Composition>` and
1628
                associated objects.
1629

1630
            path : str : default None
1631
                specifies path of file for MDF specification; if it is
1632
                not specified then the current directory is used.
1633

1634
            fmt : str
1635
                specifies file format of output. Auto-detect based on
1636
                **filename** extension if None.
1637
                Current options: 'json', 'yml'/'yaml'
1638

1639
            simple_edge_format : bool
1640
                specifies use of
1641
                `simple edge format <MDF_Simple_Edge_Format>` or not
1642
    """
1643
    compositions = convert_to_list(compositions)
1✔
1644
    model = get_mdf_model(*compositions, simple_edge_format=simple_edge_format)
1✔
1645

1646
    if fmt is None:
1!
1647
        try:
1✔
1648
            fmt = re.match(r'(.*)\.(.*)$', filename).groups()[1]
1✔
1649
        except (AttributeError, IndexError):
×
1650
            fmt = 'json'
×
1651

1652
    if path is not None:
1!
1653
        filename = os.path.join(path, filename)
×
1654

1655
    try:
1✔
1656
        return getattr(model, f'to_{supported_formats[fmt]}_file')(filename)
1✔
1657
    except AttributeError as e:
×
1658
        raise ValueError(
1659
            f'Unsupported MDF output format "{fmt}". Supported formats: {gen_friendly_comma_str(supported_formats.keys())}'
1660
        ) from e
1661

1662

1663
def get_mdf_model(*compositions, simple_edge_format=True):
1✔
1664
    """
1665
        Generate the MDF Model object for one or more
1666
        `Compositions <Composition>` and associated objects.
1667

1668
        .. note::
1669
           At present, if more than one Composition is specified, all
1670
           must be fully disjoint;  that is, they must not share any
1671
           `Components <Component>` (e.g., `Mechanism`, `Projections`
1672
           etc.). This limitation will be addressed in a future update.
1673

1674
        Arguments:
1675
            *compositions : Composition
1676
                specifies `Composition` or iterable of ones to be output
1677
                in the Model
1678

1679
            simple_edge_format : bool
1680
                specifies use of
1681
                `simple edge format <MDF_Simple_Edge_Format>` or not
1682
    """
1683
    import modeci_mdf
1✔
1684
    import modeci_mdf.mdf as mdf
1✔
1685
    from psyneulink.core.compositions.composition import Composition
1✔
1686

1687
    model_name = "_".join([c.name for c in compositions])
1✔
1688

1689
    model = mdf.Model(
1✔
1690
        id=model_name,
1691
        format=f'ModECI MDF v{modeci_mdf.__version__}',
1692
        generating_application=f'PsyNeuLink v{psyneulink.__version__}',
1693
    )
1694

1695
    for c in compositions:
1✔
1696
        if not isinstance(c, Composition):
1✔
1697
            raise MDFError(
1698
                f'Item in compositions arg of {__name__}() is not a Composition: {c}.'
1699
            )
1700
        model.graphs.append(c.as_mdf_model(simple_edge_format=simple_edge_format))
1✔
1701

1702
    return model
1✔
STATUS · Troubleshooting · Open an Issue · Sales · Support · CAREERS · ENTERPRISE · START FREE · SCHEDULE DEMO
ANNOUNCEMENTS · TWITTER · TOS & SLA · Supported CI Services · What's a CI service? · Automated Testing

© 2025 Coveralls, Inc