globus-labs / mof-generation-at-scale / 8633835793

46%

Build Type

push

github

Committed by web-flow

Commit Message

Dedicate a whole node to training (#117)

* Incorporate hyun's changes

Not sure why these fix multi-gpu training, but :shrug:

* Dedicate a whole node to training

* Turn off printing wfn files

* Fix a few typos

* Do not retrain if no new data

* Ensure that you do not pin train to 1 GPU

* Remove superfluous .to statements

Fixes #118

* Flake8 fix: don't need sleep

😫

* Force dataset creation before multiprocessing

I think I was getting errors around multiple ranks writing to
a save file at the same time. This way, each sets up their data loader
themselves

The stacktrace on the training error indicated a problem whem making the
loader:

  File "/lus/eagle/projects/ExaMol/mofa/mof-generation-at-scale/mofa/utils/src/lightning.py", line 126, in setup
    self.val_dataset = dataset_type(
  File "/lus/eagle/projects/ExaMol/mofa/mof-generation-at-scale/mofa/utils/src/datasets.py", line 61, in __init__
    self.data = torch.load(dataset_path, map_location=device)
  File "/lus/eagle/projects/ExaMol/mofa/mof-generation-at-scale/env-polaris/lib/python3.10/site-packages/torch/serialization.py", line 795, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/lus/eagle/projects/ExaMol/mofa/mof-generation-at-scale/env-polaris/lib/python3.10/site-packages/torch/serialization.py", line 1002, in _legacy_load
    magic_number = pickle_module.load(f, **pickle_load_args)
EOFError: Ran out of input

* Don't increase iteration number on failures, clean up

* Fix a problem introduced in merging

* Don't use DDP w/o launcher

* Save updated models in special dir

Run Details

25 of 27 new or added lines in 3 files covered. (92.59%)

1 existing line in 1 file now uncovered.

4400 of 9603 relevant lines covered (45.82%)

0.46 hits per line