Train DeepLabCut Model¶

DeepLabCut is an open-source library for pose estimation based on deep learning. This tool allows users to train a DeepLabCut model. The output model can be used downstream to perform pose estimation on behavioral movies.

Parameters¶

Inputs¶

The following sections explain in further detail the expected format for these input files.

Labeled Data Zip File¶

This input is a .zip file containing the labeled data from DLC that should be used for training the model. The contents of this folder should correspond with the labeled-data folder inside a DLC project. When a new DLC project is created, the labeled-data folder is automatically created.

The contents of this folder should look something like the following:

• <video 1>
  • CollectedData_<user>.csv
  • CollectedData_<user>.h5
  • image00xxxx.png
  • ...
• <video 2>
• ...
• <video n>

The labeled data consists of a folder for every video selected for training/testing. Within each folder, there should be many .png files containing the frame extracted from the corresponding video. There should also be a csv and h5 file containing the manually labeled pose estimates for each image in the folder. These images need to be labeled using the DLC GUI, prior to training the model in IDEAS.

To label images using DLC, the library must first be installed, following these instructions.

Launch DLC and create a new project, select the videos that should be used for training & testing the network. Once the project is created, the config file should be edited to specify the body parts and skeleton of the animal pose, as well as any additional advanced configurations that should be applied to the model.

Next, frames must be extracted from the selected videos using the Extract Frames panel in the GUI.

Once frames are extracted, each frame can be labeled using the Label Frames panel in the GUI.

After all frames in all videos have been labeled, navigate to the project folder in a file explorer, and compress the labeled-data folder as a zip file. This file can be uploaded to IDEAS and set to the DLC Labeled Data type

Config File¶

Every DLC project has a config.yaml located at the top-level of the project folder. This configures the DLC project. The most important configuration in this file is the body parts and skeleton of the pose which should be used for training.

There are many other important parameters that can be changed from default values for advanced use cases. These parameters include the training fraction, the engine to use for training, the p-cutoff value to use for evaluation. These three values can also be specified in the parameters of this tool due to their importance in model training and evaluation. There are additional properties in the config which can be adjusted as well, read more here.

Train Pose Config File¶

The train pose config file configures the training settings for a model. It is automatically generated by DLC, but it can be edited for advanced use cases. Some parameters that can be adjusted includes the learning rate to use for iteration/epoch ranges, the max number of epochs/iterations, the frequency to save snapshots, the max snapshots to take, etc. If this file is not provided, the default file generated by DLC will be used. In order to generate this file locally using DLC, you will need to create training dataset to auto generate this file.

Pytorch Config File¶

The pytorch config file configures the pytorch training settings for a model. It is automatically generated by DLC, but it can be edited for advanced use cases. Some parameters that can be adjusted includes the maximum number of epochs, batch size, top-level detectors, etc. If this file is not provided, the default file generated by DLC will be used. In order to generate this file locally using DLC, you will need to create training dataset to auto generate this file.

Test Pose Config File¶

The test pose config file configures the settings for evaluating a model. It is automatically generated by DLC, but it can be edited for advanced use cases. If this file is not provided, the default file generated by DLC will be used. In order to generate this file locally using DLC, you will need to create training dataset to auto generate this file.

Description¶

The tool goes through the following steps to train and evaluate a dlc network using input labeled data.

Create New Project¶

First a new project is created. This executes the function deeplabcut.create_new_project. The input labeled data and config file are copied to the new project. Any config values passed in the parameters, i.e., Engine, Training Fraction, and P Cutoff, are then updated in config file of the new project.

Create Training Dataset¶

Using the new project, a training dataset is created from the labeled data. This executes the function deeplabcut.create_training_dataset.

By default, DLC will use the user-specified Training Fraction to automatically, randomly split the labeled data into train and test data. However, it's also possible to specify a custom split of train and test data by specifying the videos names (excluding the file extension) to use for training as a comma-separated list in the Train Files parameter, and the video names to use for testing as a comma-separated list in the Test Files parameter. The videos in Train Files and Test Files are not allowed to overlap.

The Net Type and Augmenter Type are also used to populate the pose_cfg.yaml file created for the new network. This config file controls many settings for training, which can be tweaked for advanced use cases. A pose_cfg.yaml can also be provided as input in the Train Pose Config parameter and used for training the network. See here for more info about this file.

Train Network¶

The next step is training the model. This executes the function deeplabcut.train_network. If Train Pose Config, Test Pose Config, or Pytorch Config files are provided as input, then the files are copied to the dlc-models or dlc-models-pytorch folder before starting training. By default, all snapshots are saved during model training so that there is enough data to accurately assess the training performance over epochs/iterations.

Evaluate Network¶

The final step is evaluating the network. This executes the function deeplabcut.evaluate_network. By default, this evaluates all snapshots that were saved from training. From the evaluation results, an optimal snapshot is computed using the Optimal Snapshot Metric parameter. By default, only the optimal snapshot is saved in the output zipped model in order to reduce the size of the model uploaded to IDEAS. This can be turned off using the Save Optiomal Snapshot parameter

Outputs¶

Trained Model¶

Once training completes the dlc-models or dlc-models-pytorch folder is compressed into an output zip file. This zip file will be compatible with the DeepLabCut Pose Estimation tool, and can be used to analyze new videos. If Save Optiomal Snapshot is true, then only the optimal snapshot is saved in the output zip file in order to reduce the size of the model uploaded to IDEAS. A few summary evaluation results are also copied to the models folder prior to compression, including the CombinedEvaluation-results.csv, and two .h5 files containing the error distribution of the optimal snapshot, with and without a p-cutoff.

Previews¶

This output has several previews to help visualize the training & evaluation results of the model.

Evaluation Results¶

In addition to the output trained model, the evaluation-results or evaluation-result-pytorch is compressed into an output zip file. This is a separate output file since evaluation results can become significantly large in size if Plot Evaluation Results or Generate Maps is set to true.

This output shares the same previews as the trained model output.

Next Steps¶

After training a model, it can be applied on new videos using the DeepLabCut Pose Estimation tool.

FAQ¶

• Which engine should I use?
  • DeepLabCut recommends using pytorch since it is faster. The tensorflow backend will be deprecated by the end of 2024.
• Why is my task getting cancelled after hours of running?
  • There is a 6 hour timeout on tasks in IDEAS. If you suspect the task is timing out, try decreasing Epochs parameter if you are using the pytorch engine, or decrease the Max Iterations parameter if you are using tensorflow engine.
• Can you create multiple shuffles using this tool?
  • This tool is configured to only train a single network, therefore only one shuffle is required for this tool and cannot be changed using a parameter.
  • To train multiple networks simply execute multiple rows of this tool in an analysis table for batch execution.
  • Each output trained model can be compared for performance after all task complete.