• Home
  • Features
  • Pricing
  • Docs
  • Announcements
  • Sign In

nasa / trick / 42060905651

05 Nov 2026 04:02PM UTC coverage: 56.973% (-0.02%) from 56.989%
42060905651

Pull #2277

github

web-flow
Merge b6e73d6ba into 846171e41
Pull Request #2277: feat: Auto-create and activate python venv from requirements.txt (#2274)

11 of 30 new or added lines in 1 file covered. (36.67%)

14997 of 26323 relevant lines covered (56.97%)

480702.52 hits per line

Source File
Press 'n' to go to next uncovered line, 'b' for previous

69.42
/trick_source/sim_services/InputProcessor/IPPython.cpp
1
/*
2
   PURPOSE: ( Python input processor )
3
   REFERENCE: ( Trick Simulation Environment )
4
   ASSUMPTIONS AND LIMITATIONS: ( None )
5
   CLASS: ( N/A )
6
   LIBRARY DEPENDENCY: ( None )
7
   PROGRAMMERS: ( Alex Lin NASA 2009 )
8
*/
9

10
#include <Python.h>
11

12
#include "trick/IPPython.hh"
13

14
#include "trick/MemoryManager.hh"
15
#include "trick/exec_proto.h"
16
#include "trick/io_alloc.h"
17
#include "trick/message_proto.h"
18
#include "trick/message_type.h"
19

20
#include <atomic>
21
#include <climits>
22
#include <cstdio>
23
#include <cstdlib>
24
#include <cstring>
25
#include <iostream>
26
#include <pthread.h>
27
#include <sstream>
28
#include <string>
29
#include <sys/stat.h>
30
#include <unistd.h>
31

32
Trick::IPPython * the_pip ;
33

34
//Constructor
35
Trick::IPPython::IPPython() { the_pip = this; }
190 ✔
36

37
// Need to save the state of the main thread to allow child threads to run PyRun variants.
38
static PyThreadState *_save = NULL;
39

40
// Number of threads currently inside parse()/parse_condition(), i.e. threads that hold
41
// the Python GIL or are waiting to acquire it. See GILGuard and shutdown() below.
42
static std::atomic<int> active_parse_count(0);
43

44
namespace
45
{
46
    /**
47
     * Holds the Python GIL for a scope, with three guarantees that the bare
48
     * PyGILState_Ensure()/PyGILState_Release() pair does not provide.
49
     *
50
     * 1. Thread cancellation is disabled for as long as the GIL is held. Without this a
51
     *    pthread_cancel() (ThreadBase::cancel_thread(), used during shutdown) can destroy the
52
     *    thread between Ensure and Release, leaving the GIL owned by a thread that no longer
53
     *    exists. Nobody can ever reacquire it after that. A cancel requested while
54
     *    cancellation is disabled simply stays pending and is delivered once the GIL has been
55
     *    released, which is exactly the behaviour we want.
56
     *
57
     * 2. The GIL is released even if an exception unwinds out of the scope. Trick's
58
     *    exec_terminate() throws Trick::ExecutiveException from inside PyRun_SimpleString
59
     *    (VariableServerSessionThread_loop.cpp catches it), and the previous straight-line
60
     *    code skipped PyGILState_Release() on that path, leaking the GIL.
61
     *
62
     * 3. active_parse_count tracks how many threads are in the interpreter, so that
63
     *    IPPython::shutdown() can tell whether reacquiring the GIL is safe or would block
64
     *    forever.
65
     *
66
     * The GIL is held for the guard's whole lifetime, so put the guard in a nested scope if
67
     * later code in the same function must run without it. To take the GIL twice in one
68
     * function, use two separate blocks; reusing the variable name is fine. Dropping the GIL
69
     * in between lets other threads change what you are reading, so only split when you
70
     * actually want to yield.
71
     *
72
     * Nesting guards is also legal: PyGILState_Ensure() is reentrant, and a nested call
73
     * returns PyGILState_LOCKED, so the inner Release() does not drop the GIL. That only
74
     * holds because each guard keeps its own gstate; the handles must not be shared
75
     * between calls.
76
     */
77
    class GILGuard
78
    {
79
        public:
80
            GILGuard()
5,758 ✔
81
            {
5,758 ✔
82
                pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &old_cancel_state);
5,758 ✔
83
                // Incremented before Ensure() so that a thread blocked *waiting* for the GIL
84
                // is counted too. Erring towards "someone is in the interpreter" is the safe
85
                // direction: the worst case is that we skip Py_Finalize() unnecessarily.
86
                ++active_parse_count;
5,758 ✔
87
                gstate = PyGILState_Ensure();
5,758 ✔
88
            }
5,758 ✔
89

90
            ~GILGuard()
5,758 ✔
91
            {
92
                PyGILState_Release(gstate);
5,758 ✔
93
                --active_parse_count;
5,758 ✔
94
                pthread_setcancelstate(old_cancel_state, NULL);
5,758 ✔
95
            }
5,758 ✔
96

97
            GILGuard(const GILGuard&)            = delete;
98
            GILGuard& operator=(const GILGuard&) = delete;
99

100
        private:
101
            PyGILState_STATE gstate;
102
            int old_cancel_state;
103
    };
104

105
    static bool path_exists(const std::string& path)
190 ✔
106
    {
107
        struct stat sb;
108
        return stat(path.c_str(), &sb) == 0;
190 ✔
109
    }
110

111
    /**
112
     * If a requirements.txt file exists in the simulation's run directory (the process's
113
     * current working directory), make sure a python virtual environment built from it
114
     * exists alongside it, then "activate" that environment for the embedded interpreter
115
     * by setting VIRTUAL_ENV and prepending <venv>/bin to PATH. Must run before
116
     * Py_Initialize(): the sys.path setup later in init() already appends
117
     * $VIRTUAL_ENV/lib/pythonX.Y/site-packages whenever VIRTUAL_ENV is set, which is the
118
     * only hook this needs.
119
     *
120
     * The venv is created once and reused on later runs; delete it to force a rebuild
121
     * after editing requirements.txt.
122
     */
123
    void setup_virtual_env()
190 ✔
124
    {
125
        char cwd_buf[PATH_MAX];
126
        if (getcwd(cwd_buf, sizeof(cwd_buf)) == NULL)
190 ✔
127
        {
128
            return;
190 ✔
129
        }
130
        std::string cwd(cwd_buf);
190 ✔
131
        std::string requirements_file = cwd + "/requirements.txt";
190 ✔
132
        if (!path_exists(requirements_file))
190 ✔
133
        {
134
            return;
190 ✔
135
        }
136

NEW
137
        std::string venv_dir = cwd + "/venv";
×
NEW
138
        if (!path_exists(venv_dir + "/bin/python3"))
×
139
        {
NEW
140
            message_publish(MSG_INFO, "Found %s, creating python virtual environment in %s\n",
×
141
                            requirements_file.c_str(), venv_dir.c_str());
142

143
            // Prefer a python matching the version this interpreter is embedding, so that
144
            // compiled packages installed into the venv are ABI-compatible with it. Fall
145
            // back to plain python3 if that exact version isn't on the PATH.
NEW
146
            std::stringstream venv_cmd;
×
NEW
147
            venv_cmd << "python" << PY_MAJOR_VERSION << "." << PY_MINOR_VERSION << " -m venv " << venv_dir
×
NEW
148
                     << " 2>/dev/null || python3 -m venv " << venv_dir;
×
NEW
149
            if (system(venv_cmd.str().c_str()) != 0)
×
150
            {
NEW
151
                message_publish(MSG_WARNING, "Failed to create python virtual environment in %s\n", venv_dir.c_str());
×
NEW
152
                return;
×
153
            }
154

NEW
155
            std::stringstream pip_cmd;
×
NEW
156
            pip_cmd << venv_dir << "/bin/pip install -r " << requirements_file;
×
NEW
157
            if (system(pip_cmd.str().c_str()) != 0)
×
158
            {
NEW
159
                message_publish(MSG_WARNING, "Failed to install requirements from %s\n", requirements_file.c_str());
×
NEW
160
                return;
×
161
            }
NEW
162
        }
×
163

164
        // Mirror what the venv's activate script does: point VIRTUAL_ENV at it, and put
165
        // its bin/ first on PATH so any subprocess the interpreter spawns sees it too.
NEW
166
        setenv("VIRTUAL_ENV", venv_dir.c_str(), 1);
×
NEW
167
        const char* old_path = getenv("PATH");
×
NEW
168
        std::string new_path = venv_dir + "/bin" + (old_path ? std::string(":") + old_path : "");
×
NEW
169
        setenv("PATH", new_path.c_str(), 1);
×
170
    }
380 ✔
171
}
172

173
/**
174
@details
175
-# Loops through all of the memorymanager allocations testing if a name handle was given.
176
 -# If a name and a user type_name were given to the allocation
177
  -# If the user_type_name is not a Trick core class, prefixed with "Trick::"
178
  -# Create a python statement to assign the python name to an address: <name> = trick.castAsTYPE(int(<address>))
179
  -# Run the statement in the python interpreter
180
*/
181
void Trick::IPPython::get_TMM_named_variables() {
201 ✔
182
    //std::cout << "top level names at initialization" << std::endl ;
183
    Trick::ALLOC_INFO_MAP_ITER aim_it ;
201 ✔
184
    for ( aim_it = trick_MM->alloc_info_map_begin() ; aim_it != trick_MM->alloc_info_map_end() ; ++aim_it ) {
5,632 ✔
185
        ALLOC_INFO * alloc_info = (*aim_it).second ;
5,431 ✔
186
        if ( alloc_info->name != NULL and alloc_info->user_type_name != NULL ) {
5,431 ✔
187
            std::stringstream ss ;
5,180 ✔
188
            std::string user_type_name = alloc_info->user_type_name ;
5,180 ✔
189
            size_t start_colon ;
190
            while ( ( start_colon = user_type_name.find("::") ) != std::string::npos ) {
5,797 ✔
191
                user_type_name.replace( start_colon , 2 , "__" ) ;
617 ✔
192
            }
193
            // The castAs method may not exist if the class was hidden from SWIG (#ifndef SWIG).
194
            // Use a try/except block to test if the method exists or not.  If it doesn't exist
195
            // don't worry about it.  Also only assign python variable if it is pointing to
196
            // something python doesn't owns.  Otherwise we could free the object we're trying to assign.
197
            ss << "try:" << std::endl ;
5,180 ✔
198
            ss << "    if '" << alloc_info->name << "' not in globals() or " ;
5,180 ✔
199
            ss << alloc_info->name << ".thisown == False:" <<  std::endl ;
5,180 ✔
200
            ss << "        " << alloc_info->name << " = " ;
5,180 ✔
201
            ss << "trick.castAs" << user_type_name << "(int(" << alloc_info->start << "))" << std::endl ;
5,180 ✔
202
            ss << "except AttributeError:" << std::endl ;
5,180 ✔
203
            ss << "    pass" << std::endl ;
5,180 ✔
204
            GILGuard gil_guard;
5,180 ✔
205
            PyRun_SimpleString(ss.str().c_str());
5,180 ✔
206
        }
5,180 ✔
207
    }
208
}
201 ✔
209

210
//Initialize and run the Python input processor on the user input file.
211
int Trick::IPPython::init() {
190 ✔
212
    /** @par Detailed Design: */
213

214
    FILE *input_fp ;
215
    int ret ;
216
    std::string error_message ;
190 ✔
217

218
    // Build and activate a virtual environment from requirements.txt, if present, before
219
    // any Python state exists for this process.
220
    setup_virtual_env();
190 ✔
221

222
    // Run Py_Initialze first for python 2.x
223
#if PY_VERSION_HEX < 0x03000000
224
    Py_Initialize();
225
#endif
226

227
    /* Run the Swig generated routine in S_source_wrap.cpp. */
228
    init_swig_modules() ;
190 ✔
229

230
    // Run Py_Initialze after init_swig_modules for python 3.x
231
#if PY_VERSION_HEX >= 0x03000000
232
    Py_Initialize();
190 ✔
233
#endif
234

235
    // The following PyRun_ calls do not require the PyGILState guards because no threads are launched
236
    /* Import simulation specific routines into interpreter. */
237
    PyRun_SimpleString(
190 ✔
238
     "import sys\n"
239
     "import os\n"
240
     "import struct\n"
241
     "import binascii\n"
242
     "if 'VIRTUAL_ENV' in os.environ:\n"
243
     "    sys.path.append(os.path.join(os.environ['VIRTUAL_ENV'], \"lib\", f\"python{sys.version_info.major}.{sys.version_info.minor}\", \"site-packages\"))\n"
244
     "sys.path.append(os.getcwd() + '/trick.zip')\n"
245
     "sys.path.append(os.path.join(os.environ['TRICK_HOME'], 'share/trick/pymods'))\n"
246
     "sys.path += map(str.strip, os.environ['TRICK_PYTHON_PATH'].split(':'))\n"
247
     "import trick\n"
248
     "sys.path.append(os.getcwd() + \"/Modified_data\")\n"
249
    ) ;
250

251
    /* Make shortcut names for all known sim_objects. */
252
    get_TMM_named_variables() ;
190 ✔
253

254
    /* An input file is not required, if the name is empty just return. */
255
    if ( input_file.empty() ) {
190 ✔
256
        return(0) ;
×
257
    }
258

259
    if ((input_fp = fopen(input_file.c_str(), "r")) == NULL) {
190 ✔
260
        error_message = "No input file found named " + input_file ;
×
261
        exec_terminate_with_return(-1 , __FILE__ , __LINE__ , error_message.c_str() ) ;
×
262
    }
263

264
    /* Read and parse the input file. */
265
    if ( verify_input ) {
190 ✔
266
        PyRun_SimpleString("sys.settrace(trick.traceit)") ;
×
267
    }
268

269
    /* Read and parse the input file. */
270
    if ( save_input ) {
190 ✔
271
        PyRun_SimpleString("trick.open_input_file_log()") ;
×
272
        PyRun_SimpleString("sys.settrace(trick.traceittofile)") ;
×
273
    }
274

275
    if ( (ret = PyRun_SimpleFile(input_fp, input_file.c_str())) !=  0 ) {
190 ✔
276
        exec_terminate_with_return(ret , __FILE__ , __LINE__ , "Input Processor error\n" ) ;
×
277
    }
278

279
    if ( verify_input ) {
188 ✔
280
       std::stringstream ss ;
×
281
       ss << "import hashlib" << std::endl ;
×
282
       ss << "input_file = " << "'" << input_file.c_str() << "'" << std::endl;
×
283
       ss << "print('{0} SHA1: {1}'.format(input_file,hashlib.sha1(open(input_file, 'rb').read()).hexdigest()))" << std::endl ;
×
284
       PyRun_SimpleString(ss.str().c_str()) ;
×
285
       exec_terminate_with_return(ret , __FILE__ , __LINE__ , "Input file verification complete\n" ) ;
×
286
    }
×
287

288
    if ( save_input ) {
188 ✔
289
        PyRun_SimpleString("trick.close_input_file_log()") ;
×
290
        PyRun_SimpleString("sys.settrace(None)") ;
×
291
    }
292

293
    fclose(input_fp) ;
188 ✔
294

295
    // Release the GIL from the main thread.
296
    Py_UNBLOCK_THREADS
188 ✔
297

298
    return(0) ;
188 ✔
299
}
190 ✔
300

301
//Command to parse the given string.
302
int Trick::IPPython::parse(std::string in_string) {
522 ✔
303

304
    int ret ;
305
    in_string += "\n" ;
522 ✔
306

307
    GILGuard gil_guard;
522 ✔
308
    ret = PyRun_SimpleString(in_string.c_str());
522 ✔
309

310
    return ret ;
522 ✔
311

312
}
522 ✔
313

314
/**
315
 @details
316
 The incoming statement is assumed to be a conditional fragment, i.e. "a > b".  We need
317
 to get the return value of this fragment by setting the return value of it to a known
318
 variable name in the input processor.  We can then assign that return value to the
319
 incoming return_value reference.
320

321
-# Lock the input processor mutex
322
-# Create a complete statement that assigns the conditional fragment to our return value
323
-# parse the condition
324
-# copy the return value to the incoming cond_return_value
325
-# Unlock the input processor mutex
326
*/
327
int Trick::IPPython::parse_condition(std::string in_string, int & cond_return_val ) {
56 ✔
328

329
    in_string =  std::string("trick_ip.ip.return_val = ") + in_string + "\n" ;
56 ✔
330
    // Running the simple string will set return_val.
331
    int py_ret;
332
    {
333
        GILGuard gil_guard;
56 ✔
334
        py_ret = PyRun_SimpleString(in_string.c_str());
56 ✔
335
    }
56 ✔
336
    cond_return_val = return_val ;
56 ✔
337

338
    return py_ret ;
56 ✔
339

340
}
341

342
//Restart job that reloads event_list from checkpointable structures
343
int Trick::IPPython::restart() {
11 ✔
344
    /* Make shortcut names for all known sim_objects. */
345
    get_TMM_named_variables() ;
11 ✔
346
    return 0 ;
11 ✔
347
}
348

349
int Trick::IPPython::shutdown() {
157 ✔
350

351
    if ( Py_IsInitialized() ) {
157 ✔
352
        // If any thread is still inside parse()/parse_condition() then it owns the GIL,
353
        // or is queued for it, and may never give it back -- the usual cause is a
354
        // variable server command that blocks inside a model call, since SWIG does not
355
        // release the GIL around wrapped calls. Reacquiring the GIL below would then
356
        // block forever.
357
        //
358
        // Skipping Py_Finalize() in that case is safe. The process is about to exit, so
359
        // the OS reclaims the interpreter's memory regardless; all we give up is Python's
360
        // orderly teardown (atexit handlers, __del__ methods). That is a far better
361
        // outcome than wedging the sim, and it lets Executive::shutdown() carry on and
362
        // report the real return code instead of the run appearing to hang.
363
        int in_interpreter = active_parse_count;
157 ✔
364
        if (in_interpreter > 0)
157 ✔
365
        {
366
            message_publish(
×
367
                MSG_WARNING,
368
                "Trick::IPPython::shutdown() skipping Py_Finalize(): %d thread(s) still executing in the Python "
369
                "interpreter and holding the GIL. The Python interpreter will not be finalized.\n",
370
                in_interpreter);
371
            return (0);
×
372
        }
373

374
        // Obtain the GIL so that we can shut down properly
375
        // Check if the thread state is actually saved before trying to restore it
376
        // Python thread state is NULL when using JIT Input
377
        if (_save != NULL)
157 ✔
378
        {
379
            Py_BLOCK_THREADS _save = NULL;
157 ✔
380
        }
381
        Py_Finalize();
157 ✔
382
    }
383
    return(0) ;
157 ✔
384
}
STATUS · Troubleshooting · Open an Issue · Sales · Support · CAREERS · ENTERPRISE · START FREE TRIAL · SCHEDULE DEMO
ANNOUNCEMENTS · TWITTER · TOS & SLA · Supported CI Services · What's a CI service? · Automated Testing

© 2026 Coveralls, Inc