Organize data

Overview¶

Organizing data is critical for linking multiple files to a recording and keeping track of data across multiple recordings. Use the dataset table to organize and preview data. Remember that you can create different versions of a dataset to track changes to the table.

File browser¶

The file browser is a key feature in organizing data within IDEAS where you can view and manage all files either uploaded to or generate within IDEAS.

The file browser may be accessed by clicking the Files button in the bottom toolbar or by selecting Files from the View menu as shown below.

Searching for files¶

Use the search bar in the file browser to more easily find files. For example, you may use terms from your filename, file date, data type, and recording ID.

Filter files by dataset¶

Data that has not yet been assigned to a specific dataset will be accessible from any dataset in that project using the file browser. Files assigned to a dataset will be viewable only within that dataset. Use the drop-down menu to select whether to view files assigned to the current dataset or those that have not yet been assigned to a dataset.

Filter files by source¶

You may filter your view based on the file source by selecting one of the 3 options in the drop-down menu:

1. Any source: all files including uploaded files and files from analysis results
2. Uploaded files: files specifically uploaded to IDEAS
3. Analysis results: files generated within IDEAS using the analytical tools

Sorting and organizing files¶

You may select what fields the file browser displays by choosing to show or hide columns.

Based on the columns visible, files may then be sorted based on selected column fields. Click either the column name or the sort fields option from the menu to create a sorting preference. If you select more than one field to sort, you can adjust the sorting order by dragging the field up or down in the list.

File browser tools¶

The file browser includes the following additional tools to help you manage your files.

Rename file

Unassign file from their respective recording

Create a time series

Break a time series

Archive file

Unarchive file

Delete data and history for a file

Download file

Keyboard shortcuts

Create data series¶

If you want to group data into a series (e.g., you performed multiple miniscope recordings where the miniscope was not removed), use the following steps:

• From within a dataset, open the File Browser.
• Select the files to group into a series and click the Create Series icon in the menu bar.

• Modify the default name for the series if desired and then click Create Series. The series is now available for use in the dataset table.

Note that you can break the series using the respective icon in the menu bar.

Assign recordings¶

Each row of a dataset table is a recording containing data. Uploaded data needs to be assigned to a recording before performing an analysis workflow.

Insert recording row¶

Create a recording row using the Insert menu and clicking Insert recording row(s). Select the number of rows to insert.

You should now see row(s) inserted in your dataset table. Note that the recording ID is defined by the pre-fix you provided when creating the dataset. The ID is hard-coded so if you have 3 rows (REC-1, REC-2, REC-3) and delete REC-3, the next row you insert will be given the ID: REC-4.

Use the hamburger icon to modify the recording ID format. You can edit the prefix or include values for data columns in the name.

Insert data column¶

Use the Insert menu to insert data columns. Select one of the column types listed below and then follow the prompts to either select data to be displayed and/or provide a name for the column. Click Add Column to complete.

• Uploaded data: stores uploaded data
• Analysis result: displays result files from analysis tables
• Custom metadata: add free text entry
• Linked metadata: displays metadata values from uploaded data or result files

You should now see the columns you added (example here shows adding columns for miniscope movie, gpio, and imu):

Assign data¶

Now that you have recording rows and data columns, you can assign data to a recording by doing one of the following:

• Click an empty cell to upload data directly to the recording.

• Open the File Browser and drag and drop data into the data table.

Note that the File Browser will show whether a file has been assigned to a recording.

Preview data¶

To preview data, simply click on the file of your choice (you can do this from either the dataset table or the File Browser window). This will open a panel on the right where you can preview the data and see metadata information. Shown below is an example for a miniscope movie.

Note that if you have selected a time series, the preview will only show one movie at a time within the series. Use the drop down to select a different segment in the series to preview.

Adding metadata¶

To add metadata, click Insert from the menu, followed by Insert column. You can select to insert either custom metadata or linked metadata.

Adding custom metadata¶

Select Custom metadata to add data in the form of free text fields. Enter a column name of your choice and a metadata key if desired. Click Add Column to finish. Click within the cell for a metadata column to add text.

Adding linked metadata¶

To add metadata extracted from data you have uploaded, select the Linked Metadata option. Select the data column from which metadata will be extracted. Select from the available metadata options corresponding to the data column you chose. A default column name will be added which you may edit. Click Add Column to complete.

Archive data¶

Individual files can be archived by clicking either for the menu for the file (image below, left) or by clicking on the Archive icon (snowflake) in the preview panel (image below, right).