Configure for Third-Party Scheduler Cluster Discovery

You can let MATLAB^® locate your third-party scheduler cluster configured with either the built-in or the Generic scheduler interface. To discover clusters using the Parallel Computing Toolbox™ cluster discovery functionality, you must create a cluster configuration file and store it at a location accessible to MATLAB users.

Prerequisites

To run jobs on your cluster, you must provide the MATLAB client with a set of plugin scripts. The scripts contain instructions specific to your cluster infrastructure, such as how to interface with the job scheduler, and how to transfer job and task data to cluster nodes.

By default, your MATLAB installation comes with plugin scripts for these schedulers:

Slurm
PBS Pro^® and OpenPBS
LSF^®
Torque
Grid Engine
HTCondor
AWS^® Batch

To learn more about how to configure MATLAB clients to submit jobs to clusters with these schedulers, see Configure for Slurm, Torque, LSF, PBS, Grid Engine, HTCondor, or AWS Batch

If your scheduler does not have a set of sample plugin scripts and you want to configure MATLAB clients with the Generic scheduler interface, you can download and modify one of the sample sets of plugin scripts that most closely matches your setup from a GitHub^® repository. You must save the plugin scripts at a location MATLAB clients can access. For more details, see Configure Using the Generic Scheduler Interface.

Cluster Configuration File Format

The cluster configuration file is a plain text file with the extension .conf containing key-value pairs that describe the cluster configuration information. The MATLAB client will use the cluster configuration file to construct a cluster profile for the user who discovers the cluster. Cluster profiles let you define certain properties for your cluster, then have these properties applied when you create cluster, job, and task objects in the MATLAB client.

The cluster configuration file must conform to these specifications:

The configuration file must have the extension .conf
You must specify each cluster property name and value in the Name = Value format.
```
Name = InstallTest
```
You must specify values for the cluster Name, Numworkers, JobStorageLocation, and PluginScriptsLocation properties.
To add comments to the configuration file, you must start the line with the hash (#) symbol.
You can provide properties as structures. The structure field name must be the property name and the field value must specify the property value.
- You can specify structures using dot notation of the form structName.fieldName = fieldValue. For example, you can specify a value for the Username property in the AdditionalProperties structure.
```
AdditionalProperties.Username = myUserName
```
- Alternatively, you can use square brackets to indicate that the subsequent name-value pairs belong to the same structure.
```
[AdditionalProperties]
Username = myUserName
ClusterHost = cluster-host-name
```
- If you add the AdditionalProperties structure to the configuration file, MATLAB converts the structure to the parallel.cluster property parallel.cluster.AdditionalProperties.
You can specify which operating system applies to a property value. Add one or more of the specifiers "Windows", "Mac", "Linux", or "Unix" in parenthesis to the property value. MATLAB will only execute the property value on the specified platform. For example, you can set different job storage locations for the client computer for Windows^® or UNIX^® operating systems.
```
JobStorageLocation (Windows) = C:\Temp\joblocation
JobStorageLocation (Unix) = /home/Temp/joblocation
```
Specify operating system environmental variable names as "$VARIABLE". MATLAB uses getenv("VARIABLE") to find the values of the environmental variable. For example, you can use environmental variable to set different usernames for Windows, macOS, or Linux^® operating systems.
```
[AdditionalProperties]
Username (Windows) = "$USERNAME"
Username (Mac, Linux) = "$USER"
ClusterHost = cluster-host-name
```
Note
The variable "$MATLAB_VERSION_STRING" returns the release number of the MATLAB client, e.g. 2022a. Add an “R” or "r" as appropriate to complete the MATLAB version.
```
ClusterMATLABRoot = /network/installs/MATLAB/R"$MATLAB_VERSION_STRING"
```
MATLAB converts textual representations of numbers to type double and true and false to type logical.
MATLAB supports cell arrays and string arrays, for example: {'hello', 'world'} or ["hello", "world"].

Store Configuration File

Store the cluster configuration file at a location the MATLAB client of users who want to discover the cluster can find. MATLAB will search for the configuration file in these locations:

matlabroot/toolbox/parallel/user/clusterprofiles – You can use this location if users share a MATLAB installation on the cluster.
The folder defined by the environment variable $MATLAB_CLUSTER_PROFILES_LOCATION – This can be useful if the cluster administrator can set environment variables on the organisation's machines when they are imaged.
The user's home folder – %USERPROFILE% on Windows and $HOME on UNIX.
The user's Downloads folder in their home folder – This can be useful if users can download the configuration file from their organisation's intranet.
Any locations defined by Domain Name System (DNS) text records listed under _mdcs._tcp.<domain> – This can be useful if none of the above options are possible, and is a convenient workflow of discovering a cluster because no changes are required to the user's machine.
A DNS text (TXT) record associates a text string with a particular domain. Your system administrator creates DNS TXT records in your organization’s DNS infrastructure.
The DNS TXT record must have the form "discover_folder=<folder>" where <folder> is an accessible location on the filesystem.
MATLAB will discover configuration files if you store it in <folder>. You can store multiple configuration files in <folder> and multiple DNS TXT records can be registered under_mdcs._tcp. For a description of the required record, and validation information, see DNS TXT Record.

Discover Clusters on Users Machine

To locate a cluster from a MATLAB client, use Discover Clusters. For more information, see Discover Clusters (Parallel Computing Toolbox).

Import Configuration File to Create Cluster Profile

Since R2023b

Alternatively, you can import the configuration file to create a cluster profile in your MATLAB client by using the Cluster Profile Manager or programmatically in the MATLAB Command Window.

Note

The imported configuration file appears as a new cluster profile in your Cluster Profile Manager list. The list contains the cluster profile name, which is not necessarily the file name. If you already have a cluster profile with the same name as the cluster described in the configuration file, the new cluster profile gets an extension added to its name so you can distinguish it.

Import with Cluster Profile Manager

To import the configuration file using the Cluster Profile Manager:

On the Home tab in the Environment section, select Parallel > Create and Manage Clusters.
In the Cluster Profile Manager, select Import.
In the Import profiles from file dialog box, browse to find the configuration file with the .conf extension you want to import. Select the file and select Open.

Import Programmatically with `parallel.importProfile`

To import the configuration file programmatically in the MATLAB Command Window, use the parallel.importProfile (Parallel Computing Toolbox) function.

For example, this code imports the configuration file clusterConfigFile.conf for the cluster named Slurm HPC and creates a cluster profile with the cluster name.

clusterProfile = parallel.importProfile("C:\User\Downloads\clusterConfigFile.conf")

clusterProfie =

    'Slurm HPC'