How Parallel Pool Recovery Works

R2026b

Parallel pool recovery is when a parallel pool continues to execute code after the MATLAB^® client loses connection to one or more workers. This functionality helps long-running or large-scale computations to complete successfully, even in the presence of hardware faults, network issues, or other unexpected problems.

A pool is recoverable if it has not enabled Single Program Multiple Data (SPMD) communication support. When you first create a pool, the pool is capable of running SPMD computations, but SPMD support is not yet enabled. If SPMD support is enabled, then the pool is no longer recoverable.

How MATLAB Manages Worker Failures

The code that you run on the pool determines how MATLAB manages worker failures.

When you create a parallel pool using the local Processes profile or a remote cluster profile, the pool can initially support both non-communicating and communicating parallel language features. MATLAB then adapts pool recovery behavior based on the features you use.

• Communicating features, such as distributed arrays, tall arrays, or spmd blocks, require all workers to execute the same program simultaneously and exchange data. These features use the SPMD execution model. The first time you run any communicating feature, MATLAB enables SPMD communication support for the pool and keeps it enabled for the remaining lifetime of the pool. After SPMD support is enabled, MATLAB cannot recover from worker loss. If the client loses connection to any worker, MATLAB shuts down the pool.

• Non-communicating features, such as parfor-loops and parfeval computations, do not require communication between workers. When you use only these features, MATLAB does not enable SPMD communication support for the pool. As a result, the pool can recover from worker loss by continuing execution with the remaining workers. However, the pool with the remaining workers can no longer run communicating features.

You do not need to specify or manage SPMD communication support for the pool. MATLAB manages this transition automatically.

Check SpmdEnabled Property of Pool Objects

Parallel pool objects have a read-only SpmdEnabled property that reflects whether the pool can execute SPMD computations. For example, when you start a parallel pool using the Processes profile, the pool is capable of running SPMD computations, so the SpmdEnabled property value of the pool object returns a logical 1.

pool = parpool("Processes",4);
pool.SpmdEnabled

ans =

  logical

   1

• 1 — indicates that the pool can run SPMD computations.

• 0 — indicates that the pool cannot run SPMD computations.

Use this property only for inspection. MATLAB automatically manages when SPMD support is enabled. Do not attempt to control this behavior.

Pool Recovery Behavior by Cluster Type

All remote cluster pools and local Processes pools support recovery from lost workers, as long as the pool has not enabled SPMD support. MATLAB enables SPMD support when needed.

Before R2026b: Only MATLAB Job Scheduler cluster and local Processes pools support recovery from lost workers. Pools created with other schedulers shut down if the client loses connection to any worker.

Tips

• Do not attempt to directly control SPMD communication behavior. MATLAB manages SPMD communication support automatically.

• Where possible, use non-communicating parallel features, such as parfor and parfeval, to run your code in parallel.

• Running communicating features, such as spmd blocks, on a pool permanently disables pool recovery for that pool. Avoid using communicating features unless your algorithm requires all workers to communicate together.

• To avoid disabling pool recovery when you perform additional setup on workers, such as loading variables for Simulink^® models or loading libraries, use the parfevalOnAll function instead of using spmd blocks.

• If a pool recovers from worker loss, the pool can no longer enable SPMD communication support.

• To see if communicating features can run on a pool, check the SpmdEnabled property of the ProcessPool and ClusterPool pool object.