Get Started with MONAI Label in Medical Image Labeler

Medical Open Network for AI (MONAI) Label [1][2] is a deep learning framework for labeling medical images. You can connect to MONAI Label within the Medical Image Labeler app to apply fully automated and interactive deep learning models for segmenting radiology images.

These images show CT scans labeled using automated MONAI Label models in the Medical Image Labeler app.

Open Labeling Session and Load Data

Open the Medical Image Labeler app and start a volume labeling session. You can create a new volume session or reopen a previous session. If you create a new session, import the DICOM, NIfTI, or NRRD volumes to label. For an example that shows how to create a volume session and import data, see Label 3-D Medical Image Using Medical Image Labeler.

Start MONAI Label

To use MONAI Label in Medical Image Labeler, you must connect to a MONAI Label server. The server is an instance of the MONAI Label software that contains the deep learning models. Medical Image Labeler supports MONAI Label version 0.7.0.

To connect to a server from the app toolstrip, first, on the MONAI Label tab, select Start MONAI Label.

You can install and run a MONAI Label server on your local machine or on a remote machine. This table provides details about each option.

Definition	When to Use	Approach
A local MONAI Label server is one that you install on your local machine. In this situation, MONAI Label and MATLAB® run on the same machine.	You are working independently, and your computer has a CUDA® enabled GPU.	In the MONAI Label tab, select Start MONAI Label > On the local machine. The Medical Image Labeler automatically starts and connects to the MONAI Label server on your machine, if installed. This can take several minutes. Note The first time you try to connect to a local server, Medical Image Labeler prompts you to install the Medical Imaging Toolbox™ Interface for MONAI Label Library support package, which includes the MONAI Label software. For more information about installing add-ons, see Get and Manage Add-Ons. The support package also requires Deep Learning Toolbox™ and Computer Vision Toolbox™.
A remote MONAI Label server is one you connect to on a computer other than your local machine. In this situation, MONAI Label and MATLAB run on different machines.	You know the IP address of a remote machine running MONAI Label. For example, you might work as part of a team with a shared GPU computing machine.	First, start the MONAI Label server on the remote machine. For instructions, see the Installation page of the MONAI Label documentation. Then, in Medical Image Labeler, in the MONAI Label tab, select Start MONAI Label > Connect to server. In the dialog box, enter the IP address of a remote MONAI Label server, or select a recently used IP address, and click OK. Connecting to a remote MONAI Label server does not require the Medical Imaging Toolbox Interface for MONAI Label Library support package.

Once you connect to a server, you can view the server details by selecting Server Info in the app toolstrip. To disconnect from the MONAI Label server when you are done labeling, click Disconnect Server.

Select MONAI Label Model

Select a MONAI Label model from the Models gallery in the app toolstrip. The models listed in the Automated section are fully automated, and do not require any foreground or background markers. The models listed in the Interactive section either optionally accept or require manually drawn foreground and background points.

This table lists the MONAI Label models provided by the support package, with details about the model and a link to the relevant MONAI Label documentation page. Choose a model that predicts the label regions you are looking to segment. If multiple models predict similar labels, you can compare the results using trial and error. Optionally, read the MONAI documentation for training details, as models generally work best for input images with similar voxel spacing and preprocessing as their training images.

Name	Description	MONAI Documentation
localization spine	An automated model for CT segmentation of the spine. Predicts one spine label that includes the cervical, thoracic, and lumbar vertebrae.	On the GitHub Repository for the MONAI Label Radiology Sample Application page, under Multistage Vertebra Segmentation, see Spine Localization.
localization vertebra	An automated model for CT segmentation of individual vertebrae. Predicts up to 24 labels for the individual cervical, thoracic, and lumbar vertebrae. Results are generally less refined than the vertebra pipeline model, but has a lower GPU memory requirement.	On the GitHub Repository for the MONAI Label Radiology Sample Application page, under Multistage Vertebra Segmentation, see Vertebra Localization.
vertebra pipeline	An automated model for CT segmentation of individual vertebrae. Predicts up to 24 labels for the individual cervical, thoracic, and lumbar vertebrae. Results are generally more refined than the vertebra pipelline model, but has a higher GPU memory requirement.	On the GitHub Repository for the MONAI Label Radiology Sample Application page, see Multistage Vertebra Segmentation.
segmentation spleen	An automated model for CT segmentation of the spleen. Predicts one label for the spleen.	On the GitHub Repository for the MONAI Label Radiology Sample Application page, see Segmentation Spleen.
spleen ct segmentation	An automated model for CT segmentation of the spleen. Predicts one label for the spleen.	On the MONAI Model Zoo page, browse for "Spleen ct segmentation" and click Model Details.
pancreas ct dints segmentation	An automated model for CT segmentation. Predicts two labels for the pancreas and pancreatic tumor.	On the MONAI Model Zoo page, browse for "Pancreas ct dints segmentation" and click Model Details.
renalStructures UNEST segmentation	An automated model for CT segmentation of the kidneys. Predicts up to three labels for the cortex, medulla, and pelvicalyceal system.	On the MONAI Model Zoo page, browse for "Renalstructures unest segmentation" and click Model Details.
prostate mri anatomy	An automated model for MRI segmentation of the prostate. Predicts two labels for the central gland and peripheral zone.	On the MONAI Model Zoo page, browse for "Prostate mri anatomy" and click Model Details.
wholeBrainSeg Large UNEST segmentation	An automated model for MRI segmentation of the brain. Trained for T1W MRI images. Predicts up to 132 labels for the individual structures of the brain.	On the MONAI Model Zoo page, browse for "Wholebrainseg large unest segmentation" and click Model Details.
segmentation	An automated model for multi-organ CT segmentation. Predicts up to 25 labels, including major organs such as the spleen, liver, and stomach, the individual lobes of each lung, the individual chambers of the heart, and major veins and arteries such as the aorta and inferior vena cava.	On the GitHub Repository for the MONAI Label Radiology Sample Application page, see Segmentation.
swin unetr btcv segmentation	An automated model for multi-organ CT segmentation. Predicts up to 13 labels for organs including the spleen, kidneys, gallbladder, esophagus, liver, stomach, and pancreas, plus the aorta, inferior vena cava, portal and splenic veins, and adrenal glands.	On the MONAI Model Zoo page, browse for "Swin unetr btcv segmentation" and click Model Details.
wholeBody ct segmentation	An automated model for whole body CT segmentation. Predicts up to 104 labels for organs plus individual bones, muscle groups, lobes of the lungs, chambers of the heart, and major veins and arteries.	On the MONAI Model Zoo page, browse for "Wholebody ct segmentation" and click Model Details.
deepedit	A hybrid automated and interactive model for multi-organ CT segmentation. Predicts up to eight labels for the spleen, kidneys, liver, stomach, aorta, inferior vena cava, and background.	On the GitHub Repository for the MONAI Label Radiology Sample Application page, see DeepEdit.
deepgrow 3d	An interactive model for multi-organ segmentation. Predicts up to seven labels for the spleen, kidneys, liver, stomach, aorta, and inferior vena cava.	On the GitHub Repository for the MONAI Label Radiology Sample Application page, see Deepgrow.

Map Labels

Each MONAI Label model has been trained to predict certain label classes, such as spleen, aorta, or rightKidney. Before running a model, you must map the MONAI Label classes to the label definitions within Medical Image Labeler.

Map labels using the Label Mapping dialog box, which open automatically when you select a new model. The dialog box contains a table listing all of the label classes for the selected model. Select the box in the Predict? column for each label you want to predict. You can select all of the labels or a subset of labels. Then, for each label you select, choose an option in the Label Definition in App column. Select <Create New> to create a new label definition with the same name as the MONAI Label name. If you want to assign the output labels to an existing label definition, select the existing label definition instead. You can assign multiple MONAI Label classes to one Medical Image Labeler label definition. For example, you could map both left kidney and right kidney in the image below to one overall kidney label definition. Click OK to save your selections.

This image shows a Label Mapping dialog box in which all labels have been selected for prediction, and the MONAI output label spleen can be mapped to the existing spleen label definition.

Draw Foreground and Background Markers (Interactive Models Only)

The interactive MONAI Label models accept interactively drawn foreground and background markers to guide the label predictions.

• The deepgrow 3d model requires at least one foreground or background marker before you can run the model.

• The deepedit model optionally accepts foreground and background markers, or you can run deepedit as an automated model. For example, you can try running the model automatically without markers, and then add markers in mislabeled regions and rerun to refine the labels.

You can draw markers for only one label definition at a time. Select the label definition you want from the Label Definitions pane. To draw markers, in the app toolstrip, select Mark Foreground or Mark Background and click and drag to draw on an image slice. For example, this image shows foreground markers, in green, and background markers, in red, for the leftKidney label.

Select Clear Markers to clear all markers from the volume. The app automatically clears all markers if you select Disconnect Server, select a new model from the Models gallery, select a new label in the Label Definitions pane, or select a new volume in the Data Browser.

Run Model

From the app toolstrip, click Run to run the model.

• All automated models and the deepedit model simultaneously predict all of the labels you have selected for prediction in the Label Mapping dialog box.

• The deepgrow 3d model predicts labels one at a time, and predicts only the label that is currently selected in the Label Definitions pane. To predict additional labels, select a new label in the Label Definitions pane, draw foreground or background markers, and click Run.

For example, this image shows the predicted left kidney label using the markers drawn in the previous section.

Troubleshoot MONAI Label Errors

This table lists errors you might encounter when using MONAI Label in Medical Image Labeler, and potential solutions to resolve each error.

Problem	Potential Solutions
When you try to run a model, you receive an out of memory error message.	Try running a different MONAI Label model, because different models have different memory requirements. Instead of starting a server on the local machine, connect to a remote server on a machine with more available GPU memory. Upgrade the GPU resources of the local machine to increase available memory.
When you try to start a MONAI Label server, you receive an error message that the server failed to start.	Uninstall and reinstall the Medical Imaging Toolbox Interface for MONAI Label Library support package. Uninstall the support package. From the Home tab of the MATLAB toolstrip, select Add-Ons > Manage Add-Ons. Click the Options button to the right of the support package name and select Uninstall. Delete these files, if they still exist, in the support package folder returned by matlabshared.supportpkg.getSupportPackageRoot. supportpkgroot\3P.instrset\monai.instrset supportpkgroot\3P.instrset\pythonstandalone_3.10.instrset supportpkgroot\3P.instrset\pytorch_2.0.1_python_3.10.instrset supportpkgroot\toolbox\medical\supportpackages\monailabel Reinstall the support package and try starting the server again.
When you try to start a MONAI Label server, you receive an error message about exceeding a GitHub® rate limit.	The first time you start a local server after you install the support package, or if you delete model files from your machine, Medical Image Labeler downloads the MONAI Label models from GitHub. If you make too many download requests from GitHub within a short time, you can exceed rate limits. To learn more about rate limit issues, see the Rate limits for the REST API GitHub documentation page. Follow these steps to resolve the error. Generate a personal access token by following the procedure on the Managing your personal access tokens GitHub documentation page. You can generate either a classic or fine-grained personal access token. After copying the token string generated by GitHub, type this command in the MATLAB Command Window, where tokenstring is the token string generated by GitHub. setenv("MONAI_ZOO_AUTH_TOKEN","tokenstring") Try starting the server again.

References

[1] Diaz-Pinto, Andres, Sachidanand Alle, Vishwesh Nath, Yucheng Tang, Alvin Ihsani, Muhammad Asad, Fernando Pérez-García, et al. “MONAI Label: A Framework for AI-Assisted Interactive Labeling of 3D Medical Images,” 2022. https://doi.org/10.48550/ARXIV.2203.12362.

[2] Diaz-Pinto, Andres, Pritesh Mehta, Sachidanand Alle, Muhammad Asad, Richard Brown, Vishwesh Nath, Alvin Ihsani, et al. “DeepEdit: Deep Editable Learning for Interactive Segmentation of 3D Medical Images.” In Data Augmentation, Labelling, and Imperfections, edited by Hien V. Nguyen, Sharon X. Huang, and Yuan Xue, 11–21. Lecture Notes in Computer Science. Cham: Springer Nature Switzerland, 2022. https://doi.org/10.1007/978-3-031-17027-0_2.