dpgen Workflow Management: machine.json, record.dpgen, and Run Resumption#

Overview#

dpgen manages iterative active-learning workflows (train → explore → label) through two key files:

• param.json — scientific parameters (model architecture, exploration settings, labeling inputs)
• machine.json — HPC submission configuration (machine type, resources, scheduler settings)
• record.dpgen — append-only checkpoint log that tracks which stage the workflow has completed

The entry point is dpgen run param.json machine.json, which calls run_iter().

How machine.json Is Loaded (Read Once, Not Hot-Reloaded)#

machine.json is read exactly once at process startup inside run_iter() 1. The file is parsed via load_file() and then converted with convert_mdata() 2 into a preprocessed dict (mdata) that splits out keys like train_machine, model_devi_machine, fp_machine, and their corresponding resource specs.

This mdata dict is passed through to each stage function — make_train, run_train, run_model_devi, run_fp, etc. — and is never re-read from disk during the session 3.

Implication: Changes made to machine.json while a dpgen process is running have no effect until the process is restarted. There is no hot-reload mechanism.

Note: param.json (jdata) is also loaded at startup 4, but its contents are used to generate stage-specific input files at each iteration. Because jdata is re-used from the in-memory copy, changes to the file on disk mid-run do not take effect without a restart; however, the community practice of stopping between iterations and restarting with an updated param.json works because the new run reads the file fresh. machine.json has no equivalent mechanism even between iterations.

record.dpgen: Checkpoint Format#

record.dpgen uses an append-only format. After each of the 9 stages per iteration completes, record_iter() appends a line <iteration_index> <stage_index> to the file.

Stage indices per iteration 5:

At startup, run_iter() reads the last line of record.dpgen to determine iter_rec 6. The run loop then skips all completed tasks using the formula ii * max_tasks + jj <= iter_rec[0] * max_tasks + iter_rec[1] 7.

Resuming an Interrupted Run#

Simply re-run the original command 8:

dpgen run param.json machine.json

dpgen will read the last checkpoint from record.dpgen and resume from there. No manual cleanup of intermediate files (e.g., 02.fp/ directories) is needed — dpgen handles existing directories automatically. dpgen also tolerates up to ~5% failed fp calculations by default, so partial labeling does not block resumption 8.

Limitations and Caveats#

Avoid Modifying machine.json Before Resuming#

Do not change machine.json before restarting an interrupted run. The warning from the community 8 is that modifications to machine.json can cause dpgen to generate new working directories instead of reusing existing ones, breaking continuity with prior work.

If machine changes are unavoidable, the safest approach is to complete the current iteration cleanly, then restart.

param.json Can Be Modified Between Iterations#

Unlike machine.json, param.json parameters such as stop_batch and init_data_sys entries can be changed between iterations 8. However, machine.json is ingested only once per dpgen run invocation and cannot be changed mid-run.

Manual record.dpgen Edits#

record.dpgen can be manually edited to restart from a specific stage (e.g., to redo an exploration step with different parameters). However, this should be done cautiously and only when you specifically want to re-execute past work.

Key Source Files#