globus-labs / mof-generation-at-scale / 8633835793 / 1
48%
main: 48%

DEFAULT BRANCH: main
Ran
Files 80
Run time 2s
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 45.819% (+0.05%) from 45.772%
8633835793.1

push

github

web-flow 
Dedicate a whole node to training (#117)

* Incorporate hyun's changes

Not sure why these fix multi-gpu training, but :shrug:

* Dedicate a whole node to training

* Turn off printing wfn files

* Fix a few typos

* Do not retrain if no new data

* Ensure that you do not pin train to 1 GPU

* Remove superfluous .to statements

Fixes #118

* Flake8 fix: don't need sleep

😫

* Force dataset creation before multiprocessing

I think I was getting errors around multiple ranks writing to
a save file at the same time. This way, each sets up their data loader
themselves

The stacktrace on the training error indicated a problem whem making the
loader:

  File "/lus/eagle/projects/ExaMol/mofa/mof-generation-at-scale/mofa/utils/src/lightning.py", line 126, in setup
    self.val_dataset = dataset_type(
  File "/lus/eagle/projects/ExaMol/mofa/mof-generation-at-scale/mofa/utils/src/datasets.py", line 61, in __init__
    self.data = torch.load(dataset_path, map_location=device)
  File "/lus/eagle/projects/ExaMol/mofa/mof-generation-at-scale/env-polaris/lib/python3.10/site-packages/torch/serialization.py", line 795, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/lus/eagle/projects/ExaMol/mofa/mof-generation-at-scale/env-polaris/lib/python3.10/site-packages/torch/serialization.py", line 1002, in _legacy_load
    magic_number = pickle_module.load(f, **pickle_load_args)
EOFError: Ran out of input

* Don't increase iteration number on failures, clean up

* Fix a problem introduced in merging

* Don't use DDP w/o launcher

* Save updated models in special dir

4400 of 9603 relevant lines covered (45.82%)

0.46 hits per line

Source Files on job 8633835793.1