DeepLabCut Pose Estimation¶

DeepLabCut is an open-source library for pose estimation based on deep learning. This tool allows users to upload any pre-trained DeepLabCut model and perform pose estimation on behavioral movies.

Parameters¶

Inputs¶

The following table summarizes the valid input files types for this tool:

The following sections explain in further detail the expected format for these input files.

Zipped Model Folder¶

A DeepLabCut model is stored as a folder with specific content and structure. This folder is expected to compressed in a .zip file. At the start of tool execution, the contents of zipped model folder are examined in order to detect whether it can be used with DeepLabCut.

Direct Usage with DeepLabCut¶

First, the tool checks whether the contents of the zipped model folder can be used directly with DeepLabCut. This check examines if the contents of the zipped model folder has a directory structure which matches the directory structure of DeepLabCut projects. A DeepLabCut project is a folder created by DeepLabCut when models are trained. Subsequently, the same folder can be used by DeepLabCut to analyze new behavioral videos. A DeepLabCut project has the following directory structure:

• config.yaml
• dlc-models
• iteration-<iteration>
  • <task>-<date>-trainset<trainingfraction>shuffle1
  • test
    • pose_cfg.yaml
  • train
    • pose_cfg.yaml
    • snapshot-<snapshotnumber>.data-00000-of-00001
    • snapshot-<snapshotnumber>.index
    • snapshot-<snapshotnumber>.meta

The config.yaml file is generated by DeepLabCut, storing the configuration of the DeepLabCut project that was used to train models. This configuration includes the list of body parts in the animal pose, the skeleton of the animal pose, and the type of neural network used for training.

The dlc-models folder contains the DeepLabCut models trained in the project. Iterations of model training are stored in sub-folders. The value <iteration> is read from config.yaml, which determines the iteration to use for analysis. By default this is set to 0 in config.yaml.

Inside the folder iteration-<iteration>, there should be a sub-folder named <task>-<date>-trainset<trainingfraction>shuffle1. The values <task>, <date>, and <trainingfraction> are all read from config.yaml as well.

Within the folder <task>-<date>-trainset<trainingfraction>shuffle1, there should be two sub-folders named train and test. The train folder contains files used & generated during model training. The test folder contains files used & generated during model testing. Both folders should contain a configuration file named pose_cfg.yaml. The train folder should also contain snapshot files of the model captured during training iterations. A snapshot generally consists of three files .data-00000-of-00001, .index, .meta. Ideally, there should only be files for one snapshot number, indicated by <snapshotnumber>, since only one snapshot of the model is actually used for analysis. The snapshot number to use is indicated by snapshotindex in config.yaml (index of unique snapshot numbers in the training directory). If more than one unique snapshot number is found in the train folder, a warning is logged by the tool.

Indirect Usage with DeepLabCut¶

If the contents of the zipped model folder do not correspond to the structure of a DeepLabCut project, the tool will try to construct a new DeepLabCut project and copy the relevant files in the zipped model folder to the new project.

The tool recursively searches for a sub-folder that contains snapshot files (i.e., file name starts with snapshot).

If more than one unique snapshot number is found in the snapshot folder, a warning is logged.

Otherwise, in the snapshot folder, the tools checks if there is a pose_cfg.yaml file. The tool then creates a new project and copies the following values from pose_cfg.yaml in the zipped model folder to config.yaml in the new project:

• default_net_type -> net_type
• dataset_type -> default_augmenter
• all_joints_names -> bodyparts

Moreover, the tool copies the following keys from pose_cfg.yaml in the zipped model folder to the pose_cfg.yaml files in the test and train folders of the new project:

• dataset
• dataset_type
• num_joints
• all_joints
• all_joints_names
• net_type
• init_weights
• global_scale
• location_refinement
• locref_stdev

This mapping of keys from pose_cfg.yaml to the config files in the new DeepLabCut project is based on this logic in the DeepLabCut Model Zoo API.

Behavior Movie¶

The tool will analyze behavioral movies using the zipped model folder. The behavior movies can be in the one the following file formats: .isxb, .mp4, .avi.

Algorithm Description¶

Once the tool has validated that it can use the zipped model folder input with DeepLabCut, the analysis begins. This tool consists of three main analysis steps, summarized in the following diagram:

The first step is running the deeplabcut.analyze_videos API function, documented here. This function takes a path to the DeepLabCut project which contains the trained model to use, and applies that model on the input behavior movies. The output of this step is an h5 file containing the pose estimates of the model.

The second step is running the deeplabcut.filter_predictions API function, documented here. This functions filters the raw pose estimates, removing outliers and noise from the results. The output of this step is also an h5 file containing the filtered pose estimates of the model. This step is recommended in the DeepLabCut documentation, however it can be skipped by setting the Window Length parameter to one.

The third and final step is running the deeplabcut.create_labeled_videos API function, documented here. This function creates a video of the input movie with annotations of the pose estimates results. If filtering is applied on the pose estimates, it will annotate the input movie with the filtered results. Otherwise, the raw pose estimates will be used for annotations. The output of this step is a mp4 file with the annotated movie frames. This result helps to easily visualize and assess the quality of the model results.

Outputs¶

Pose Estimates H5 File¶

An .h5 file containing the model predictions. The predictions are stored as a MultiIndex Pandas Array. The files contains the name of the network, body part name, (x, y) label position in pixels, and the likelihood (i.e., how confident the model was to place the body part label at those coordinates) for each frame per body part. The following figure shows an example of this output.

Previews¶

Each pose estimates file will have the following previews to visualize the data stored in these files. All of these figures are generated using the plot_trajectories DeepLabCut API function.

Histogram:

The first preview is a histogram plot of the consecutive differences in x and y coordinates for each body part. This plot can help determine if there's any outliers in the results based on significant jumps in the coordinates of a body part. Ideally the distribution for each body part should be close to zero.

Likelihood vs. Time Plot:

The second preview is plot of the likelihood of each body part over time. This plot can help identify body parts or time periods where likelihood is low, indicating less reliable results from the model to filter in subsequent analysis.

Body Parts vs. Time Plot:

The third preview is a plot of the x & y coordinates of each body part over time. This plot can help identify how the position of body parts changes over time.

Body Parts Trajectory:

The fourth preview is a plot of the x & y coordinates of each body part plotted spatially in the FOV (field of view). This plot can help identify where the location of body parts in the FOV throughout the duration of the input movies.

Experiment Annotations File¶

The pose estimates in IDEAS experiment annotations format. This can be either a .csv or .parquet file depending on the Experiment Annotations Format parameter. The following table shows an example of this output.

For every body part estimated by the model, there are three columns that are output: x, y, and likelihood. In addition, the following columns are included:

• Frame number: The frame number in the input movie. If there are multiple input movies, this column will refer to the global frame number across the frames in all the input movies.
• Movie number: The movie number that the frame is from, within the multiple input movies.
• Local frame number: The frame number within the individual movie that the frame is from.
• Time since start (s): Timestamp in seconds for every frame of the input movie, relative to the start of the movie.
• Hardware counter (us): Only included for .isxb input movies. This column contains the hardware counter timestamps in microseconds for every frame of the input movie. These are the hardware counter values generated by the nVision system which can be compared to corresponding hardware counter values in .isxd, .gpio, and .imufiles from the same synchronized recording. These timestamps will be used downstream in the Map Annotations to ISXD Data tool in order map frames from annotations to isxd data, so the two datasets can be compared for analysis.

Labeled MP4 File (for visualization)¶

A .mp4 file containing the model predictions annotated on every frame of the input movie. The following figure shows an example of this output.

Next Steps¶

Here are some examples of subsequent analyses that can be executed using the outputs of this tool:

1. Average DeepLabCut Keypoints: Average the keypoint estimates in each frame of the tool output, and use this to represent the mouse center of mass (COM).
2. Compute Locomotion Metrics: Compute instantaneous speed and label states of rest and movement from averaged keypoints.
3. Compare Neural Activity Across States: Compute population activity during states of rest and movement from locomotion metrics.
4. Compare Neural Circuit Correlations Across States: Compute correlations of cell activity during states of rest and movement from locomotion metrics.
5. Combine and Compare Population Activity Data: Combine and compare population activity from multiple recordings and compare across states of rest and movement.
6. Combine and Compare Correlation Data: Combine and compare correlations data from multiple recordings and compare across states of rest and movement.