DPDispatcher Task Grouping#

Overview#

DPDispatcher organizes work at three levels: Tasks (a single command), Jobs (a scheduler submission unit containing one or more tasks), and Submissions (a collection of all jobs for a given workload). Two Resources parameters control how tasks map to jobs and how they execute within each job:

• group_size — how many tasks are packed into one scheduler job
• para_deg — how many tasks within a job run in parallel at any moment

Both live in the Resources class in dpdispatcher/submission.py.

group_size: Tasks → Jobs#

Submission.generate_jobs() is where grouping happens. It:

1. Shuffles all tasks with a fixed seed (42) for load-balance reproducibility.
2. Slices the shuffled list into chunks of group_size.
3. Creates one Job per chunk.

Key edge cases 1:

• group_size = 0 → treated as infinity (all tasks in one job).
• group_size < 0 or non-integer → raises RuntimeError.

The docstring for group_size in Resources.arginfo() summarizes this concisely: 2

"How many tasks are packed into one scheduler job. For example, 20 tasks with group_size=5 are typically split into 4 jobs. Use 1 for the simplest one-task workflow. 0 means no explicit upper limit in the grouping logic."

para_deg: Parallelism Within a Job#

All task commands in the generated script run as background processes (&) 3. para_deg controls the cadence of wait barriers between them.

gen_script_wait() implements this:

• Increments an internal counter resources.task_in_para after each task.
• When the counter reaches para_deg, it resets to 0 and emits "wait \n".
• Otherwise it emits nothing (tasks keep accumulating in the background).

A final wait at the end of every script 4 ensures all processes complete before job-completion tags are checked.

The para_deg docstring distinguishes it from group_size clearly 5:

"group_size controls how many tasks are bundled into a job, while para_deg controls concurrency within that job. Keep para_deg=1 for the safest default."

Multi-GPU: if_cuda_multi_devices#

When strategy["if_cuda_multi_devices"] = True, gen_script_wait() also manages GPU assignment 6:

• Before each task, gen_command_env_cuda_devices() prepends export CUDA_VISIBLE_DEVICES=<gpu_index>; based on gpu_in_use % gpu_per_node.
• A wait barrier is only emitted when task_in_para >= para_deg and the gpu_in_use counter wraps around gpu_per_node — i.e., all GPUs on the node have been cycled through.

Constraints enforced at Resources.__init__() 7: if_cuda_multi_devices requires gpu_per_node >= 1 and number_node == 1.

Practical Patterns#

The workstation example is documented at Running multiple MD tasks on a GPU workstation 8.

Key Source References#