IDEAS Python API¶

Python API reference for ideas-python 1.2.0.

Outputs¶

Module with framework for generating output data in IDEAS tools

Classes:

Name	Description
`Metadata`	Piece of key, value data that describes an output file.
`OutputData`	Collection of output files generated by an IDEAS tool.
`OutputFile`	Results generated by a tool, saved to a file, and saved in IDEAS.
`Preview`	A file which describes an output file.

Functions:

Name	Description
`input_paths_to_output_prefix`	"
`register`	Opens a context manager to register output data for an IDEAS tool.

`Metadata` `dataclass` ¶

Piece of key, value data that describes an output file.

Parameters:

Name	Type	Description	Default
`key`	`str`	Unique identifier for the metadata across output files	required
`name`	`str`	Friendly name for the metadata to display in IDEAS	required
`value`	`Union[str, bool, int, float, List[str]]`	The value of the metadata	required

Methods:

Name	Description
`from_dict`	Deserialize from dict in output data file
`to_dict`	Serialize to dict to save in output data file

`from_dict(instance)` `classmethod` ¶

Deserialize from dict in output data file

`to_dict()` ¶

Serialize to dict to save in output data file

`OutputData` ¶

Collection of output files generated by an IDEAS tool. Each output file has associated previews, metadata, and tags to display in IDEAS.

Parameters:

Name	Type	Description	Default
`output_files`	`List[OutputFile]`	List of output files registered by a tool.	required
`output_dir`	`Optional[str \| Path]`	Output directory where output files are generated.	`None`
`append`		Flag indicating whether to append new output files to existing output data already registered by the tool.	`True`
`raise_missing_file`	`bool`	Raise an error if an output file is missing	`True`

Methods:

Name	Description
`close`	Close output data for writing.
`open`	Open output data for writing.
`register_file`	Register an output file as output data.

`close()` ¶

Close output data for writing.

`open()` ¶

Open output data for writing.

`register_file(file_path, previews=None, metadata=None, tags=None, output_dir=None, subdir=None, name=None, prefix=None, suffix=None)` ¶

Register an output file as output data.

Parameters:

Name	Type	Description	Default
`file_path`	`str \| Path`	The path to the output file	required
`output_dir`	`Optional[str \| Path]`	Output directory where output files are generated.	`None`
`previews`	`Optional[List[Preview]]`	List of previews to show in IDEAS for the output file	`None`
`metadata`	`Optional[List[Metadata]]`	List of metadata to show in IDEAS for the output file	`None`
`tags`	`Optional[List[Tag]]`	List of tags to show in IDEAS for the output file	`None`
`subdir`	`Optional[str \| Path]`	A subdirectory in the output directory where output files should be moved. Helpful for organizing result files into subdirectories for output columns	`None`
`prefix`	`Optional[str]`	Prefix to optionally prepend to the output filename, and associated preview filenames.	`None`

Returns:

Type	Description
`OutputFile`	The output file (for command chaining)

Raises:

Type	Description
`OutputDataFileOutsideOutputDir`	If an output file is outside the output directory.
`OutputDataFileDoesNotExist`	If an output file does not exist.

`OutputFile` `dataclass` ¶

Results generated by a tool, saved to a file, and saved in IDEAS.

Parameters:

Name	Type	Description	Default
`file_path`	`Path`	The path to the output file	required
`output_dir`	`Path`	Output directory where output files are generated.	required
`subdir`	`Optional[str \| Path]`	A subdirectory in the output directory where output files should be moved. Helpful for organizing result files into subdirectories for output columns	`None`
`prefix`	`Optional[str]`	Prefix to optionally prepend to the output filename, and associated preview filenames.	`None`
`previews`	`List[Preview]`	List of previews to show in IDEAS for the output file	`list()`
`metadata`	`List[Metadata]`	List of metadata to show in IDEAS for the output file	`list()`
`tags`	`List[Tag]`	List of tags to show in IDEAS for the output file	`list()`
`raise_missing_file`	`bool`	Raise an error if an output file is missing	`True`

Methods:

Name	Description
`from_dict`	Deserialize from dict in output data file
`register_metadata`	Register metadata with the output file
`register_metadata_dict`	Register metadata dictionary with the output file
`register_preview`	Register a preview with the output file
`register_tags`	Register tags with the output file
`to_dict`	Serialize to dict to save in output data file

`from_dict(instance, output_dir)` `classmethod` ¶

Deserialize from dict in output data file

`register_metadata(key, value, name=None)` ¶

Register metadata with the output file

Parameters:

Name	Type	Description	Default
`key`	`str`	Unique identifier for the metadata across output files	required
`name`	`Optional[str]`	Friendly name for the metadata to display in IDEAS	`None`
`value`	`str`	The value of the metadata	required

Returns:

Type	Description
`OutputFile`	The output file (for command chaining)

`register_metadata_dict(**kwargs)` ¶

Register metadata dictionary with the output file

Parameters:

Name	Type	Description	Default
`kwargs`	`Dict[str, Union[str, bool, int, float, List[str]]]`	Dictionary with key-value pairs of metadata	`{}`
`value`	`Dict[str, Union[str, bool, int, float, List[str]]]`	The value of the metadata	required

Returns:

Type	Description
`OutputFile`	The output file (for command chaining)

`register_preview(file_path, caption='', output_dir=None, subdir=None, name=None, prefix=None, suffix=None)` ¶

Register a preview with the output file

Parameters:

Name	Type	Description	Default
`file_path`	`str \| Path`	The path to the preview file	required
`caption`	`str`	A short description of the preview.	`''`
`output_dir`	`Optional[str \| Path]`	Output directory where output files are generated. If not specified, default to output directory for output data.	`None`
`subdir`	`Optional[str \| Path]`	A subdirectory in the output directory where output files should be moved. Helpful for organizing result files into subdirectories for output columns	`None`
`prefix`	`Optional[str]`	Prefix to optionally prepend to the output filename, and associated preview filenames.	`None`

Returns:

Type	Description
`OutputFile`	The output file (for command chaining)

Raises:

Type	Description
`OutputDataFileOutsideOutputDir`	If the preview file is outside the output directory.
`OutputDataFileDoesNotExist`	If the preview file does not exist.

`register_tags(*tags)` ¶

Register tags with the output file

Parameters:

Name	Type	Description	Default
`tags`	`List[str]`	Tags to register	`()`

Returns:

Type	Description
`OutputFile`	The output file (for command chaining)

`to_dict()` ¶

Serialize to dict to save in output data file

`Preview` `dataclass` ¶

A file which describes an output file. Preview files can be figures (i.e., images), movies (i.e., videos), json, or even an html webpage (static assets only). Previews are meant to be small in size, giving a brief glance, or impression at the results store in particular output file.

Parameters:

Name	Type	Description	Default
`file_path`	`Path`	The path to the preview file	required
`caption`	`str`	A short description of the preview.	required

Methods:

Name	Description
`from_dict`	Deserialize from dict in output data file
`to_dict`	Serialize to dict to save in output data file

`from_dict(instance)` `classmethod` ¶

Deserialize from dict in output data file

`to_dict()` ¶

Serialize to dict to save in output data file

`_load_and_remove_output_metadata()` ¶

[internal] Load and remove v2 output metadata to register

`_resolve_output_file_path(output_file, output_dir, subdir=None, name=None, prefix=None, suffix=None)` ¶

Helper function for resolving output file paths relative to an output directory

`input_paths_to_output_prefix(*input_paths, max_name_len=15, sep='_')` ¶

" Generate an output prefix from a set of input paths

Parameters:

Name	Type	Description	Default
`input_paths`	`List[List[str \| Path \| None] \| str \| Path \| None]`	Set of input paths, can be list of strings, paths, or lists of strings or paths	`()`
`max_name_len`		Maximum number of characters to use from each input path	`15`
`sep`		Seperator to use between input paths	`'_'`

`register(output_dir=None, append=True, raise_missing_file=True)` ¶

Opens a context manager to register output data for an IDEAS tool.

Example usage:

from ideas.tools import outputs

with outputs.register() as output_data:
    output_data.register_file(
        "file.txt",
    ).register_preview(
        "figure1.svg",
        caption="A very special figure!"
    ).register_metadata_dict(
        key1="cool"
        key2="beans"
    )

Parameters:

Name	Type	Description	Default
`output_dir`	`Optional[str \| Path]`	Output directory where output files are generated. If None, then the current working directory is used.	`None`
`append`	`bool`	Flag indicating whether to append new output files to existing output data already registered by the tool.	`True`
`raise_missing_file`	`bool`	Raise an error if an output file is missing, otherwise a warning is logged.	`True`

Raises:

Type	Description
`OutputDataFileOutsideOutputDir`	If an output file is outside the output directory.
`OutputDataFileDoesNotExist`	If an output file does not exist.

Types¶

Logging¶

Functions:

Name	Description
`get_logger`	Set up logger for IDEAS tools

`get_logger()` ¶

Set up logger for IDEAS tools

Exceptions¶

Classes:

Name	Description
`ApiException`	Raised for any non-specific API errors; provides the JSON response from the API.
`ContainerDownloadAccessException`	Raised if a container could not be downloaded from IDEAS container registry.
`ContainerException`	Raised if there is an error with containers.
`ContainerNotFoundException`	Raised if a container is not found in IDEAS container registry.
`ContainerNotPublishedError`	Raised if a container is not published to IDEAS container registry when generating code bundle that references the container.
`ContainerPublishExistsException`	Raised if a container is not found in IDEAS container registry.
`ContainerPublishPostException`	Raised if there was an error creating the container image in IDEAS.
`ContainerPythonNotFound`	Raised if `python` is not an available command inside a chosen container
`ContainerUnsupportedArchException`	Raised if a container arch is not supported in IDEAS.
`DockerExecutionError`	Raised if there is an unexpected error with the Docker API.
`DockerUnavailableError`	Raised if there is an unexpected error with the Docker API.
`FileProcessedTimedOut`	Raised when creating a series, if the specified files have not been successfully processed in
`IdeasError`	Base class for all ideas exceptions.
`InvalidProfileError`	Raised if the specified configuration profile doesn't exist.
`MismatchedChecksumError`	Raised if the checksum returned by AWS doesn't match the local checksum on download/upload of a
`NvidiaRuntimeMissingError`	Raised if the user is trying to use gpus for tool execution but docker reports the nvidia
`OutputDataException`	Raised if there is an exception related to output data
`OutputDataFileDoesNotExist`	Raised if an output file does not exist
`OutputDataFileOutsideOutputDir`	Raise if an output file exists outside of the output directory
`QuotaExceededError`	Raised if the API returns a 409 CONFLICT error.
`ToolBundleBuildError`	Raised if there is an error building a code bundle.
`ToolBundleExtractError`	Raised if we are unable to extract the tool bundle file.
`ToolConfigContainerNotFoundError`	Raised if the user specifies a tool container image not found when configuring a tool.
`ToolConfigFileNotFoundError`	Raised if the user specifies a tool file not found when configuring a tool.
`ToolConfigFunctionNotFoundError`	Raised if the user specifies a tool function not found when configuring a tool.
`ToolException`	Raised if there is an error with tools.
`ToolExecutionError`	Raised if there is an error executing the tool code inside the tool container.
`ToolInputsFileNotFoundError`	Raised if an input file specified in tool inputs is not found in the user's local filesystem.
`ToolInputsFormatError`	Raised if tool inputs are not formatted correctly.
`ToolInputsNotFoundError`	Raised if a given tool inputs file is not found.
`ToolIntrospectionError`	Raised if we are unable to introspect the given tool code file.
`ToolMisconfiguredError`	Raised if a given tool is not configured correctly.
`ToolNotFoundError`	Raised if the no tools are found in a user's codebase.
`ToolSchemaValidationError`	Raised if we are unable to introspect the given tool code file.
`UnauthorizedError`	Raised if the Cognito authorization fails, or the API returns a 401/403 response.
`UnhandledError`	Raise this (hopefully infrequently) if you have no idea how to process an error.
`UserNotTenantMemberError`	Raised if user is trying to perform an action in a tenant they are not a member of.

`ApiException` ¶

Bases: IdeasError

Raised for any non-specific API errors; provides the JSON response from the API.

`ContainerDownloadAccessException` ¶

Bases: ContainerException

Raised if a container could not be downloaded from IDEAS container registry.

`ContainerException` ¶

Bases: IdeasError

Raised if there is an error with containers.

`ContainerNotFoundException` ¶

Bases: ContainerException

Raised if a container is not found in IDEAS container registry.

`ContainerNotPublishedError` ¶

Bases: ContainerException

Raised if a container is not published to IDEAS container registry when generating code bundle that references the container.

`ContainerPublishExistsException` ¶

Bases: ContainerException

Raised if a container is not found in IDEAS container registry.

`ContainerPublishPostException` ¶

Bases: ContainerException

Raised if there was an error creating the container image in IDEAS.

`ContainerPythonNotFound` ¶

Bases: ContainerException

Raised if python is not an available command inside a chosen container

`ContainerUnsupportedArchException` ¶

Bases: ContainerException

Raised if a container arch is not supported in IDEAS.

`DockerExecutionError` ¶

Bases: IdeasError

Raised if there is an unexpected error with the Docker API.

`DockerUnavailableError` ¶

Bases: IdeasError

Raised if there is an unexpected error with the Docker API.

Methods:

Name	Description
`get_instructions`	Get instructions on how to resolve this error.

`get_instructions()` `staticmethod` ¶

Get instructions on how to resolve this error.

`FileProcessedTimedOut` ¶

Bases: IdeasError

Raised when creating a series, if the specified files have not been successfully processed in a reasonable amount of time.

`IdeasError` ¶

Bases: Exception

Base class for all ideas exceptions.

`InvalidProfileError` ¶

Bases: IdeasError

Raised if the specified configuration profile doesn't exist.

`MismatchedChecksumError` ¶

Bases: IdeasError

Raised if the checksum returned by AWS doesn't match the local checksum on download/upload of a file.

`NvidiaRuntimeMissingError` ¶

Bases: IdeasError

Raised if the user is trying to use gpus for tool execution but docker reports the nvidia runtime is missing.

Methods:

Name	Description
`get_instructions`	Get instructions on how to resolve this error.

`get_instructions()` `staticmethod` ¶

Get instructions on how to resolve this error.

`OutputDataException` ¶

Bases: Exception

Raised if there is an exception related to output data

`OutputDataFileDoesNotExist` ¶

Bases: OutputDataException

Raised if an output file does not exist

`OutputDataFileOutsideOutputDir` ¶

Bases: OutputDataException

Raise if an output file exists outside of the output directory

`QuotaExceededError` ¶

Bases: ApiException

Raised if the API returns a 409 CONFLICT error.

`ToolBundleBuildError` ¶

Bases: ToolException

Raised if there is an error building a code bundle.

`ToolBundleExtractError` ¶

Bases: ToolException

Raised if we are unable to extract the tool bundle file.

`ToolConfigContainerNotFoundError` ¶

Bases: ToolException

Raised if the user specifies a tool container image not found when configuring a tool.

`ToolConfigFileNotFoundError` ¶

Bases: ToolException

Raised if the user specifies a tool file not found when configuring a tool.

`ToolConfigFunctionNotFoundError` ¶

Bases: ToolException

Raised if the user specifies a tool function not found when configuring a tool.

`ToolException` ¶

Bases: IdeasError

Raised if there is an error with tools.

`ToolExecutionError` ¶

Bases: ToolException

Raised if there is an error executing the tool code inside the tool container.

`ToolInputsFileNotFoundError` ¶

Bases: ToolException

Raised if an input file specified in tool inputs is not found in the user's local filesystem.

`ToolInputsFormatError` ¶

Bases: ToolException

Raised if tool inputs are not formatted correctly.

`ToolInputsNotFoundError` ¶

Bases: ToolException

Raised if a given tool inputs file is not found.

`ToolIntrospectionError` ¶

Bases: ToolException

Raised if we are unable to introspect the given tool code file.

`ToolMisconfiguredError` ¶

Bases: ToolException

Raised if a given tool is not configured correctly.

`ToolNotFoundError` ¶

Bases: ToolException

Raised if the no tools are found in a user's codebase.

`ToolSchemaValidationError` ¶

Bases: ToolException

Raised if we are unable to introspect the given tool code file.

`UnauthorizedError` ¶

Bases: IdeasError

Raised if the Cognito authorization fails, or the API returns a 401/403 response.

`UnhandledError` ¶

Bases: IdeasError

Raise this (hopefully infrequently) if you have no idea how to process an error.

`UserNotTenantMemberError` ¶

Bases: IdeasError

Raised if user is trying to perform an action in a tenant they are not a member of.

IDEAS Python API¶

Outputs¶

Metadata dataclass ¶

from_dict(instance) classmethod ¶

to_dict() ¶

OutputData ¶

close() ¶

open() ¶

register_file(file_path, previews=None, metadata=None, tags=None, output_dir=None, subdir=None, name=None, prefix=None, suffix=None) ¶

OutputFile dataclass ¶

from_dict(instance, output_dir) classmethod ¶

register_metadata(key, value, name=None) ¶

register_metadata_dict(**kwargs) ¶

register_preview(file_path, caption='', output_dir=None, subdir=None, name=None, prefix=None, suffix=None) ¶

register_tags(*tags) ¶

to_dict() ¶

Preview dataclass ¶

from_dict(instance) classmethod ¶

to_dict() ¶

_load_and_remove_output_metadata() ¶

_resolve_output_file_path(output_file, output_dir, subdir=None, name=None, prefix=None, suffix=None) ¶

input_paths_to_output_prefix(*input_paths, max_name_len=15, sep='_') ¶

register(output_dir=None, append=True, raise_missing_file=True) ¶

Types¶

Logging¶

get_logger() ¶

Exceptions¶

ApiException ¶

ContainerDownloadAccessException ¶

ContainerException ¶

ContainerNotFoundException ¶

ContainerNotPublishedError ¶

ContainerPublishExistsException ¶

ContainerPublishPostException ¶

ContainerPythonNotFound ¶

ContainerUnsupportedArchException ¶

DockerExecutionError ¶

DockerUnavailableError ¶

get_instructions() staticmethod ¶

FileProcessedTimedOut ¶

IdeasError ¶

InvalidProfileError ¶

MismatchedChecksumError ¶

NvidiaRuntimeMissingError ¶

get_instructions() staticmethod ¶

OutputDataException ¶

OutputDataFileDoesNotExist ¶

OutputDataFileOutsideOutputDir ¶

QuotaExceededError ¶

ToolBundleBuildError ¶

ToolBundleExtractError ¶

ToolConfigContainerNotFoundError ¶

ToolConfigFileNotFoundError ¶

ToolConfigFunctionNotFoundError ¶

ToolException ¶

ToolExecutionError ¶

ToolInputsFileNotFoundError ¶

ToolInputsFormatError ¶

ToolInputsNotFoundError ¶

ToolIntrospectionError ¶

ToolMisconfiguredError ¶

ToolNotFoundError ¶

ToolSchemaValidationError ¶

UnauthorizedError ¶

UnhandledError ¶

UserNotTenantMemberError ¶

`Metadata` `dataclass` ¶

`from_dict(instance)` `classmethod` ¶

`to_dict()` ¶

`OutputData` ¶

`close()` ¶

`open()` ¶

`register_file(file_path, previews=None, metadata=None, tags=None, output_dir=None, subdir=None, name=None, prefix=None, suffix=None)` ¶

`OutputFile` `dataclass` ¶

`from_dict(instance, output_dir)` `classmethod` ¶

`register_metadata(key, value, name=None)` ¶

`register_metadata_dict(**kwargs)` ¶

`register_preview(file_path, caption='', output_dir=None, subdir=None, name=None, prefix=None, suffix=None)` ¶

`register_tags(*tags)` ¶

`to_dict()` ¶

`Preview` `dataclass` ¶

`from_dict(instance)` `classmethod` ¶

`to_dict()` ¶

`_load_and_remove_output_metadata()` ¶

`_resolve_output_file_path(output_file, output_dir, subdir=None, name=None, prefix=None, suffix=None)` ¶

`input_paths_to_output_prefix(*input_paths, max_name_len=15, sep='_')` ¶

`register(output_dir=None, append=True, raise_missing_file=True)` ¶

`get_logger()` ¶

`ApiException` ¶

`ContainerDownloadAccessException` ¶

`ContainerException` ¶

`ContainerNotFoundException` ¶

`ContainerNotPublishedError` ¶

`ContainerPublishExistsException` ¶

`ContainerPublishPostException` ¶

`ContainerPythonNotFound` ¶

`ContainerUnsupportedArchException` ¶

`DockerExecutionError` ¶

`DockerUnavailableError` ¶

`get_instructions()` `staticmethod` ¶

`FileProcessedTimedOut` ¶

`IdeasError` ¶

`InvalidProfileError` ¶

`MismatchedChecksumError` ¶

`NvidiaRuntimeMissingError` ¶

`get_instructions()` `staticmethod` ¶

`OutputDataException` ¶

`OutputDataFileDoesNotExist` ¶

`OutputDataFileOutsideOutputDir` ¶

`QuotaExceededError` ¶

`ToolBundleBuildError` ¶

`ToolBundleExtractError` ¶

`ToolConfigContainerNotFoundError` ¶

`ToolConfigFileNotFoundError` ¶

`ToolConfigFunctionNotFoundError` ¶

`ToolException` ¶

`ToolExecutionError` ¶

`ToolInputsFileNotFoundError` ¶

`ToolInputsFormatError` ¶

`ToolInputsNotFoundError` ¶

`ToolIntrospectionError` ¶

`ToolMisconfiguredError` ¶

`ToolNotFoundError` ¶

`ToolSchemaValidationError` ¶

`UnauthorizedError` ¶

`UnhandledError` ¶

`UserNotTenantMemberError` ¶