nVision Animal Tracking Model¶

This tool applies a deep learning model on behavioral movies, extracting bounding box estimates of the animal subject. The deep learning model used by the tool is the same model used in nVision systems for real-time tracking. This tool enables running the nVision tracking model post-acquisition, which is different from the real-time processing that occurs in IDAS during acquisition.

The deep learning model is based on an animal tracking model in the NVIDA Deepstream SDK. The bounding box estimates of the model can be used as a proxy for animal center of mass (COM), which can then be used for further analysis, such as computing speed or labeling states of rest and movement.

Parameters¶

Details¶

Inputs¶

The tool expects inputs to be nVision, behavioral movies, in .isxb file format. The nVision movies do not need to have existing tracking data in order to run the model. The tool is also compatible with series inputs.

Valid Inputs¶

The following table summarizes the valid input files types for this tool:

Time Ref¶

This parameter represents the time reference to use for the timestamps in the output annotations. Time Ref can be one of the following values:

• start: The timestamps represent the seconds since the start of the input movie. If this option is chosen, the timestamps will be in a column named Frame Timestamp (s).
• unix: The timestamps represent the seconds since the Unix epoch. If this option is chosen, the timestamps will be in a column named Frame Timestamp (s).
• tsc: The timestamps represent the hardware counter values generated during acquisition from IDAS. These values are stored in the metadata of the input .isxb files. If this option is chosen, the timestamps will be in a column named Hardware counter (us). This column is always included in the output annotations, because it's necessary in the downstream Map Annotations To ISXD Data tool, in order to map the behavior annotations originating from this tool, with a cell set file from a corresponding, synchronized recording. Read more about synchronization in the Next Steps section.

Tracking ROIs¶

This parameter represents the ROIs (regions of interest) used for tracking. There are two main types of ROIs that the tool accepts as input:

• Tracking Area: A rectangular ROI which defines the area in the FOV (field of view) of the input movies to use for tracking. Any part of the FOV outside this ROI will be ignored for tracking. The tracking area helps to improve model performance by reducing the size of the input movie. It can also improve accuracy of model results by excluding parts of the FOV that might be misidentified. By default, the full FOV is used as the tracking area.
• Zone ROIs: These ROIs define zone areas to use for zone occupancy detection. The ROIs can be of one of the following shapes: rectangle, ellipse, polygon. ROIs must be drawn within the tracking area, are not allowed to overlap, and the tool accepts no more than five zones in total. If no zones are drawn, then zone occupancy detection is not performed.

When this parameter is selected, and there is an input movie selected in the analysis table row, an ROI editor should open. Inside this ROI editor, the input movie preview should be visible. The tracking area and zone ROIs can be drawn on the movie, saved, and submitted to the tool for execution.

Outputs¶

Experiment Annotations¶

The tracking results of the nVision model are output in IDEAS experiment annotations format. This file will contain one row for every frame of the input movie, and the following columns for each row:

• Global Frame Number: The frame number within the movie series
• Movie Number: The movie number that the frame is from, within the movie series
• Local Frame Number: The frame number within the individual movie
• Frame Timestamp: The timestamp of the frame, derived from the hardware frame timestamps stored by IDAS during acquisition. This can be in units of us if the input Time Ref parameter is tsc, otherwise the units are s.
• Bounding Box Left: The left (x) coordinate of the bounding box identified by the model, in units of pixels.
• Bounding Box Top: The top (y) coordinate of the bounding box identified by the model, in units of pixels.
• Bounding Box Right: The right (x) coordinate of the bounding box identified by the model, in units of pixels.
• Bounding Box Bottom: The bottom (y) coordinate of the bounding box identified by the model, in units of pixels.
• Bounding Box Center X: The center (x) coordinate of the bounding box identified by the model, in units of pixels.
• Bounding Box Center Y: The center (y) coordinate of the bounding box identified by the model, in units of pixels.
• Confidence: The confidence of the model, in units of percentage (%).
• Zone ID: If zones are drawn, and the subject is detected in the zone, this column will contain the id of the zone. The id corresponds to the order that the ROI was drawn in the ROI editor.
• Zone Name: If zones are drawn, and the subject is detected in the zone, this column will contain the name of the zone entered in the ROI editor.

If the model is unable to identify the subject, then the bounding box and confidence columns will be empty. The following table shows an example of this output file.

Previews¶

The output annotations file will have the following preview which visualizes the results.

Bounding box center trajectory colored by time:

This preview shows the bounding box center points plotted in the input field of view (FOV). Each point is colored by time, where points occurring earlier in the movie are plotted with cooler colors, and points occurring later in the movie are plotted with warmer colors. This plot can provide insight on which areas of the FOV the animal was in throughout the duration of the recording.

Annotated Movie¶

This output is an mp4 movie containing the frames of the input movie, annotated with the tracking result of the model. The results can be annotated on the movie in any one of the following ways, configured through the user parameters:

• Text label
• Bounding box
• Bounding box center

This output is only generated if Generate MP4 is set to true. Generating this output does not generally affect the performance of the tool, and it can be helpful for quality control of the results.

Tracking Zones¶

If zone ROIs are drawn, then an output csv will be generated containing the following information about every zone:

• ID: The ID of the zone. This value corresponds to the order that the zone was drawn in the ROI editor.
• Name: The name of the zone, entered in the ROI editor.
• Type: The shape type of the zone, this will be rectangle, polygon, or ellipse.
• X # and Y #: A series of columns representing the X coordinates of the zone. If the type is ellipse, then the columns will be X 0, Y 0, for the centerpoint. If the type is rectangle, then the columns will be X 0, Y 0 through X 3, Y 3. If the type is polygon, then the columns will be X 0, Y 0 through X n, Y n.
• Major & Minor Axis: The major and minor axis diameters of the ellipse.
• Angle: The angle of the major axis of the ellipse.

This file should be compatible as an input to Zone Occupancy Analysis, along with the experiment annotations file containing the tracking results of the model.

The following table shows an example of this output file:

Next Steps¶

The outputs of this tool can be used directly in the following tools:

• Compute Locomotion Metrics: Use the experiment annotations file as input. This tool will compute instantaneous speed and label states of rest and movement from the bounding box center points.
• Zone Occupancy Analysis: Use the experiment annotations file and the tracking zones file as inputs. This tool will analyze the zone occupancy detection results of the tracking model.

Synchronized Workflow Use Cases¶

Many use cases are possible when analyzing nVision movies from synchronized calcium-behavior recordings. The following diagram demonstrates the possible workflow use cases for synchronized data using this tool.

All use cases start with a synchronized recording consisting of miniscope (.isxd) and nVision (.isxb) movies. The miniscope movie is analyzed using one of the cell-identification workflows (i.e., End-to-End CNMF-E, End-to-End PCA-ICA). The cell-identification workflow will produce a cell set file containing neural-activity from the miniscope movie.

The nVision movie is analyzed using the nVision Animal Tracking Model, producing an experiment annotations file containing the tracking results of the model. These tracking results are passed as input to Compute Locomotion Metrics. This tool will compute instantaneous speed and label states of rest and movement from the bounding box center points of the tracking results.

When running this tool on the tracking results, set Time Column to be Frame Timestamp (s) if Time Ref is start or unix. If Time Ref is tsc, then set Time Column as Hardware counter (us) and Time Unit as us. Additionally, set X Column Name to Bounding Box Center X, and Y Column Name to Bounding Box Center Y.

After computing locomotion metrics from the tracking results, the locomotion metrics are almost ready to be compared with the cell set. However, first, the locomotion metrics must be processed with the cell set through Map Annotations To ISXD Data. This tool will map every frame in the cell set, with a frame in the locomotion metrics. The mapping will be done by the tool using the hardware counter timestamps from the cell set file and the locomotion metrics (using the Hardware counter (us) column). The hardware counter timestamps from both files are synchronized, so they can be directly compared in order to map miniscope frames to behavior frames. The mapped experiment annotations file will contain one row for every sample in the cell set.

When running this tool on the locomotion metrics, set Time Column to be Frame Timestamp (s), if Time Ref is start or unix. If Time Ref is tsc, then set Time Column as Hardware counter (us) and Time Unit as us. Additionally, set State Column to State.

Once the locomotion metrics have been mapped to the cell set, the two files can be used in one of the following Neural Circuit Analyses: Compare Neural Activity Across States, Compare Neural Circuit Correlations Across States. Both of these tools can be used to compare the cell set across the rest and move behavior states from the locomotion metrics.

Finally, the results of neural circuit analysis can be processed further in one of the following Study Statistics: Combine and Compare Population Activity Data, Combine and Compare Correlation Data.