Skip to content

IDEAS Python API Reference

outputs

Module with framework for registering output data in IDEAS tools.

Functions:

Name Description
register Opens a context manager to register output data for an IDEAS tool.
input_paths_to_output_prefix Generate an output prefix from a set of input paths

Classes:

Name Description
OutputData Collection of output files generated by an IDEAS tool.
OutputFile Results generated by a tool, saved to a file, and saved in IDEAS.
Tag Tag is a categorical label that describes an output file.
Metadata Piece of key, value data that describes an output file.
Preview A file which describes an output file.

register

register(output_dir:Optional[str | Path] = None, append:bool = True, raise_missing_file:bool = True)

Opens a context manager to register output data for an IDEAS tool.

Example usage:

from ideas.tools import outputs

with outputs.register() as output_data:
    output_data.register_file(
        "file.txt",
    ).register_preview(
        "figure1.svg",
        caption="A very special figure!"
    ).register_metadata_dict(
        key1="cool"
        key2="beans"
    )

Parameters:

Name Type Description Default
output_dir Optional[str Path] Output directory where output files are generated. If None, then the current working directory is used.
append bool Flag indicating whether to append new output files to existing output data already registered by the tool. True
raise_missing_file bool Raise an error if an output file is missing, otherwise a warning is logged. True

Raises:

Type Description
OutputDataFileOutsideOutputDir If an output file is outside the output directory.
OutputDataFileDoesNotExist If an output file does not exist.

OutputData

OutputData(output_dir:Optional[str | Path] = None, append:Optional[str | Path] = True, raise_missing_file:Optional[str | Path] = True)

Collection of output files generated by an IDEAS tool. Each output file has associated previews, metadata, and tags to display in IDEAS.

Parameters:

Name Type Description Default
output_files List[OutputFile] List of output files registered by a tool. required
output_dir Optional[str Path] Output directory where output files are generated.
append Flag indicating whether to append new output files to existing output data already registered by the tool. True
raise_missing_file bool Raise an error if an output file is missing True

Functions:

Name Description
close Close output data for writing.
open Open output data for writing.
register_file Register an output file as output data.

Attributes:

Name Type Description
append bool }}
file_pathname }}
output_dir Path }}
output_files List[OutputFile] }}
raise_missing_file bool }}

Initialize output data

Parameters:

Name Type Description Default
output_dir Optional[str Path] Output directory where output files are generated. If None, then the current working directory is used.
append Flag indicating whether to append new output files to existing output data already registered by the tool. True
raise_missing_file bool Raise an error if an output file is missing True

OutputData.append

append: bool = append

OutputData.close

close()

Close output data for writing.

OutputData.file_pathname

file_pathname = output_dir / constants.OUTPUT_DATA_JSON_FILENAME

OutputData.open

open()

Open output data for writing.

OutputData.output_dir

output_dir: Path = output_dir

OutputData.output_files

output_files: List[OutputFile] = []

OutputData.raise_missing_file

raise_missing_file: bool = raise_missing_file

OutputData.register_file

register_file(file_path:str | Path, previews:Optional[List[Preview]] = None, metadata:Optional[List[Metadata]] = None, tags:Optional[List[Tag]] = None, output_dir:Optional[str | Path] = None, subdir:Optional[str | Path] = None, name:Optional[str] = None, prefix:Optional[str] = None, suffix:Optional[str] = None)

Register an output file as output data.

Parameters:

Name Type Description Default
file_path str Path The path to the output file
output_dir Optional[str Path] Output directory where output files are generated.
previews Optional[List[Preview]] List of previews to show in IDEAS for the output file None
metadata Optional[List[Metadata]] List of metadata to show in IDEAS for the output file None
tags Optional[List[Tag]] List of tags to show in IDEAS for the output file None
subdir Optional[str Path] A subdirectory in the output directory where output files should be moved. Helpful for organizing result files into subdirectories for output columns
prefix Optional[str] Prefix to optionally prepend to the output filename, and associated preview filenames. None

Returns:

Type Description
OutputFile The output file (for command chaining)

Raises:

Type Description
OutputDataFileOutsideOutputDir If an output file is outside the output directory.
OutputDataFileDoesNotExist If an output file does not exist.

OutputFile

OutputFile(file_path:Path, output_dir:Path, subdir:Optional[str | Path] = None, prefix:Optional[str] = None, suffix:Optional[str] = None, previews:List[Preview] = list(), metadata:List[Metadata] = list(), tags:List[Tag] = list(), raise_missing_file:bool = True) -> None

Results generated by a tool, saved to a file, and saved in IDEAS.

Parameters:

Name Type Description Default
file_path Path The path to the output file required
output_dir Path Output directory where output files are generated. required
subdir Optional[str Path] A subdirectory in the output directory where output files should be moved. Helpful for organizing result files into subdirectories for output columns
prefix Optional[str] Prefix to optionally prepend to the output filename, and associated preview filenames. None
previews List[Preview] List of previews to show in IDEAS for the output file list()
metadata List[Metadata] List of metadata to show in IDEAS for the output file list()
tags List[Tag] List of tags to show in IDEAS for the output file list()
raise_missing_file bool Raise an error if an output file is missing True

Functions:

Name Description
from_dict Deserialize from dict in output data file
register_metadata Register metadata with the output file
register_metadata_dict Register metadata dictionary with the output file
register_preview Register a preview with the output file
register_tags Register tags with the output file
to_dict Serialize to dict to save in output data file

Attributes:

Name Type Description
file_path Path }}
metadata List[Metadata] }}
output_dir Path }}
prefix Optional[str] }}
previews List[Preview] }}
raise_missing_file bool }}
subdir Optional[str Path]
suffix Optional[str] }}
tags List[Tag] }}

OutputFile.file_path

file_path: Path

OutputFile.from_dict

from_dict(instance:Dict, output_dir:str | Path)

Deserialize from dict in output data file

OutputFile.metadata

metadata: List[Metadata] = field(default_factory=list)

OutputFile.output_dir

output_dir: Path

OutputFile.prefix

prefix: Optional[str] = None

OutputFile.previews

previews: List[Preview] = field(default_factory=list)

OutputFile.raise_missing_file

raise_missing_file: bool = True

OutputFile.register_metadata

register_metadata(key:str, value:str, name:Optional[str] = None)

Register metadata with the output file

Parameters:

Name Type Description Default
key str Unique identifier for the metadata across output files required
name Optional[str] Friendly name for the metadata to display in IDEAS None
value str The value of the metadata required

Returns:

Type Description
OutputFile The output file (for command chaining)

OutputFile.register_metadata_dict

register_metadata_dict(**kwargs:Dict[str, Union[str, bool, int, float, List[str]]])

Register metadata dictionary with the output file

Parameters:

Name Type Description Default
kwargs Dict[str, Union[str, bool, int, float, List[str]]] Dictionary with key-value pairs of metadata {}
value Dict[str, Union[str, bool, int, float, List[str]]] The value of the metadata required

Returns:

Type Description
OutputFile The output file (for command chaining)

OutputFile.register_preview

register_preview(file_path:str | Path, caption:str = '', output_dir:Optional[str | Path] = None, subdir:Optional[str | Path] = None, name:Optional[str] = None, prefix:Optional[str] = None, suffix:Optional[str] = None)

Register a preview with the output file

Parameters:

Name Type Description Default
file_path str Path The path to the preview file
caption str A short description of the preview. ''
output_dir Optional[str Path] Output directory where output files are generated. If not specified, default to output directory for output data.
subdir Optional[str Path] A subdirectory in the output directory where output files should be moved. Helpful for organizing result files into subdirectories for output columns
prefix Optional[str] Prefix to optionally prepend to the output filename, and associated preview filenames. None

Returns:

Type Description
OutputFile The output file (for command chaining)

Raises:

Type Description
OutputDataFileOutsideOutputDir If the preview file is outside the output directory.
OutputDataFileDoesNotExist If the preview file does not exist.

OutputFile.register_tags

register_tags(*tags:List[str])

Register tags with the output file

Parameters:

Name Type Description Default
tags List[str] Tags to register ()

Returns:

Type Description
OutputFile The output file (for command chaining)

OutputFile.subdir

subdir: Optional[str | Path] = None

OutputFile.suffix

suffix: Optional[str] = None

OutputFile.tags

tags: List[Tag] = field(default_factory=list)

OutputFile.to_dict

to_dict()

Serialize to dict to save in output data file

Preview

Preview(file_path:Path, caption:str) -> None

A file which describes an output file. Preview files can be figures (i.e., images), movies (i.e., videos), json, or even an html webpage (static assets only). Previews are meant to be small in size, giving a brief glance, or impression at the results store in particular output file.

Parameters:

Name Type Description Default
file_path Path The path to the preview file required
caption str A short description of the preview. required

Functions:

Name Description
from_dict Deserialize from dict in output data file
to_dict Serialize to dict to save in output data file

Attributes:

Name Type Description
caption str }}
file_path Path }}

Preview.caption

caption: str

Preview.file_path

file_path: Path

Preview.from_dict

from_dict(instance)

Deserialize from dict in output data file

Preview.to_dict

to_dict()

Serialize to dict to save in output data file

Tag

Tag = TypeVar('Tag', bound=str)

input_paths_to_output_prefix

input_paths_to_output_prefix(*input_paths:List[List[str | Path | None] | str | Path | None], max_name_len:List[List[str | Path | None] | str | Path | None] = 15, sep:List[List[str | Path | None] | str | Path | None] = '_')

" Generate an output prefix from a set of input paths

Parameters:

Name Type Description Default
input_paths List[List[str Path None]
max_name_len Maximum number of characters to use from each input path 15
sep Seperator to use between input paths '_'

Metadata

Metadata(key:str, name:str, value:Union[str, bool, int, float, List[str]]) -> None

Piece of key, value data that describes an output file.

Parameters:

Name Type Description Default
key str Unique identifier for the metadata across output files required
name str Friendly name for the metadata to display in IDEAS required
value Union[str, bool, int, float, List[str]] The value of the metadata required

Functions:

Name Description
from_dict Deserialize from dict in output data file
to_dict Serialize to dict to save in output data file

Attributes:

Name Type Description
key str }}
name str }}
value Union[str, bool, int, float, List[str]] }}

Metadata.from_dict

from_dict(instance)

Deserialize from dict in output data file

Metadata.key

key: str

Metadata.name

name: str

Metadata.to_dict

to_dict()

Serialize to dict to save in output data file

Metadata.value

value: Union[str, bool, int, float, List[str]]

types

Attributes:

Name Type Description
IdeasFile Type representing an IDEAS input file, as a string file path }}

IdeasFile

IdeasFile: TypeAlias = str

log

Functions:

Name Description
get_logger Set up logger for IDEAS tools

get_logger

get_logger()

Set up logger for IDEAS tools

exceptions

Classes:

Name Description
ApiException Raised for any non-specific API errors; provides the JSON response from the API.
ContainerDownloadAccessException Raised if a container could not be downloaded from IDEAS container registry.
ContainerException Raised if there is an error with containers.
ContainerNotFoundException Raised if a container is not found in IDEAS container registry.
ContainerNotPublishedError Raised if a container is not published to IDEAS container registry when generating code bundle that references the container.
ContainerPublishExistsException Raised if a container is not found in IDEAS container registry.
ContainerPublishPostException Raised if there was an error creating the container image in IDEAS.
ContainerPythonNotFound Raised if python is not an available command inside a chosen container
ContainerUnsupportedArchException Raised if a container arch is not supported in IDEAS.
DockerExecutionError Raised if there is an unexpected error with the Docker API.
DockerUnavailableError Raised if there is an unexpected error with the Docker API.
FileProcessedTimedOut Raised when creating a series, if the specified files have not been successfully processed in
IdeasError Base class for all ideas exceptions.
MismatchedChecksumError Raised if the checksum returned by AWS doesn't match the local checksum on download/upload of a
NvidiaRuntimeMissingError Raised if the user is trying to use gpus for tool execution but docker reports the nvidia
OutputDataException Raised if there is an exception related to output data
OutputDataFileDoesNotExist Raised if an output file does not exist
OutputDataFileOutsideOutputDir Raise if an output file exists outside of the output directory
QuotaExceededError Raised if the API returns a 413 Payload Too Large error.
ToolBundleBuildError Raised if there is an error building a code bundle.
ToolBundleExtractError Raised if we are unable to extract the tool bundle file.
ToolConfigContainerNotFoundError Raised if the user specifies a tool container image not found when configuring a tool.
ToolConfigFileNotFoundError Raised if the user specifies a tool file not found when configuring a tool.
ToolConfigFunctionNotFoundError Raised if the user specifies a tool function not found when configuring a tool.
ToolException Raised if there is an error with tools.
ToolExecutionError Raised if there is an error executing the tool code inside the tool container.
ToolInputsFileNotFoundError Raised if an input file specified in tool inputs is not found in the user's local filesystem.
ToolInputsFormatError Raised if tool inputs are not formatted correctly.
ToolInputsNotFoundError Raised if a given tool inputs file is not found.
ToolIntrospectionError Raised if we are unable to introspect the given tool code file.
ToolMisconfiguredError Raised if a given tool is not configured correctly.
ToolNotFoundError Raised if the no tools are found in a user's codebase.
ToolSchemaValidationError Raised if we are unable to introspect the given tool code file.
UnauthorizedError Raised if the Cognito authorization fails, or the API returns a 401/403 response.
UnhandledError Raise this (hopefully infrequently) if you have no idea how to process an error.
UserNotTenantMemberError Raised if user is trying to perform an action in a tenant they are not a member of.

ApiException

ApiException(message, json)

Bases: IdeasError

Raised for any non-specific API errors; provides the JSON response from the API.

Attributes:

Name Type Description
json }}

ApiException.json

json = json

ContainerDownloadAccessException

Bases: ContainerException

Raised if a container could not be downloaded from IDEAS container registry.

ContainerException

Bases: IdeasError

Raised if there is an error with containers.

ContainerNotFoundException

Bases: ContainerException

Raised if a container is not found in IDEAS container registry.

ContainerNotPublishedError

ContainerNotPublishedError(message = None, container = None)

Bases: ContainerException

Raised if a container is not published to IDEAS container registry when generating code bundle that references the container.

Attributes:

Name Type Description
container }}

ContainerNotPublishedError.container

container = container

ContainerPublishExistsException

ContainerPublishExistsException(message = None, checksum = None)

Bases: ContainerException

Raised if a container is not found in IDEAS container registry.

Attributes:

Name Type Description
checksum }}

ContainerPublishExistsException.checksum

checksum = checksum

ContainerPublishPostException

Bases: ContainerException

Raised if there was an error creating the container image in IDEAS.

ContainerPythonNotFound

Bases: ContainerException

Raised if python is not an available command inside a chosen container

ContainerUnsupportedArchException

ContainerUnsupportedArchException(message = None, arch = None)

Bases: ContainerException

Raised if a container arch is not supported in IDEAS.

Attributes:

Name Type Description
arch }}

ContainerUnsupportedArchException.arch

arch = arch

DockerExecutionError

Bases: IdeasError

Raised if there is an unexpected error with the Docker API.

DockerUnavailableError

Bases: IdeasError

Raised if there is an unexpected error with the Docker API.

Functions:

Name Description
get_instructions Get instructions on how to resolve this error.

DockerUnavailableError.get_instructions

get_instructions()

Get instructions on how to resolve this error.

FileProcessedTimedOut

Bases: IdeasError

Raised when creating a series, if the specified files have not been successfully processed in a reasonable amount of time.

IdeasError

Bases: Exception

Base class for all ideas exceptions.

MismatchedChecksumError

Bases: IdeasError

Raised if the checksum returned by AWS doesn't match the local checksum on download/upload of a file.

NvidiaRuntimeMissingError

Bases: IdeasError

Raised if the user is trying to use gpus for tool execution but docker reports the nvidia runtime is missing.

Functions:

Name Description
get_instructions Get instructions on how to resolve this error.

NvidiaRuntimeMissingError.get_instructions

get_instructions()

Get instructions on how to resolve this error.

OutputDataException

Bases: Exception

Raised if there is an exception related to output data

OutputDataFileDoesNotExist

Bases: OutputDataException

Raised if an output file does not exist

OutputDataFileOutsideOutputDir

Bases: OutputDataException

Raise if an output file exists outside of the output directory

QuotaExceededError

Bases: ApiException

Raised if the API returns a 413 Payload Too Large error.

Attributes:

Name Type Description
json }}

ToolBundleBuildError

Bases: ToolException

Raised if there is an error building a code bundle.

ToolBundleExtractError

Bases: ToolException

Raised if we are unable to extract the tool bundle file.

ToolConfigContainerNotFoundError

ToolConfigContainerNotFoundError(message = None, container = None)

Bases: ToolException

Raised if the user specifies a tool container image not found when configuring a tool.

Attributes:

Name Type Description
container }}

ToolConfigContainerNotFoundError.container

container = container

ToolConfigFileNotFoundError

Bases: ToolException

Raised if the user specifies a tool file not found when configuring a tool.

ToolConfigFunctionNotFoundError

Bases: ToolException

Raised if the user specifies a tool function not found when configuring a tool.

ToolException

Bases: IdeasError

Raised if there is an error with tools.

ToolExecutionError

Bases: ToolException

Raised if there is an error executing the tool code inside the tool container.

ToolInputsFileNotFoundError

ToolInputsFileNotFoundError(message, file)

Bases: ToolException

Raised if an input file specified in tool inputs is not found in the user's local filesystem.

Attributes:

Name Type Description
file }}

ToolInputsFileNotFoundError.file

file = file

ToolInputsFormatError

ToolInputsFormatError(message = None, key = None, value = None)

Bases: ToolException

Raised if tool inputs are not formatted correctly.

Attributes:

Name Type Description
key }}
message }}
value }}

ToolInputsFormatError.key

key = key

ToolInputsFormatError.message

message = message

ToolInputsFormatError.value

value = value

ToolInputsNotFoundError

Bases: ToolException

Raised if a given tool inputs file is not found.

ToolIntrospectionError

Bases: ToolException

Raised if we are unable to introspect the given tool code file.

ToolMisconfiguredError

ToolMisconfiguredError(message = None, key = None)

Bases: ToolException

Raised if a given tool is not configured correctly.

Attributes:

Name Type Description
key }}

ToolMisconfiguredError.key

key = key

ToolNotFoundError

Bases: ToolException

Raised if the no tools are found in a user's codebase.

ToolSchemaValidationError

Bases: ToolException

Raised if we are unable to introspect the given tool code file.

UnauthorizedError

Bases: IdeasError

Raised if the Cognito authorization fails, or the API returns a 401/403 response.

UnhandledError

Bases: IdeasError

Raise this (hopefully infrequently) if you have no idea how to process an error.

UserNotTenantMemberError

Bases: IdeasError

Raised if user is trying to perform an action in a tenant they are not a member of.