IDEAS Python API Reference¶
outputs¶
Module with framework for registering output data in IDEAS tools.
Functions:
| Name | Description |
|---|---|
register |
Opens a context manager to register output data for an IDEAS tool. |
input_paths_to_output_prefix |
Generate an output prefix from a set of input paths |
Classes:
| Name | Description |
|---|---|
OutputData |
Collection of output files generated by an IDEAS tool. |
OutputFile |
Results generated by a tool, saved to a file, and saved in IDEAS. |
Tag |
Tag is a categorical label that describes an output file. |
Metadata |
Piece of key, value data that describes an output file. |
Preview |
A file which describes an output file. |
register¶
register(output_dir:Optional[str | Path] = None, append:bool = True, raise_missing_file:bool = True)
Opens a context manager to register output data for an IDEAS tool.
Example usage:
from ideas.tools import outputs
with outputs.register() as output_data:
output_data.register_file(
"file.txt",
).register_preview(
"figure1.svg",
caption="A very special figure!"
).register_metadata_dict(
key1="cool"
key2="beans"
)
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output_dir |
Optional[str |
Path] | Output directory where output files are generated. If None, then the current working directory is used. |
append |
bool |
Flag indicating whether to append new output files to existing output data already registered by the tool. | True |
raise_missing_file |
bool |
Raise an error if an output file is missing, otherwise a warning is logged. | True |
Raises:
| Type | Description |
|---|---|
OutputDataFileOutsideOutputDir |
If an output file is outside the output directory. |
OutputDataFileDoesNotExist |
If an output file does not exist. |
OutputData¶
OutputData(output_dir:Optional[str | Path] = None, append:Optional[str | Path] = True, raise_missing_file:Optional[str | Path] = True)
Collection of output files generated by an IDEAS tool. Each output file has associated previews, metadata, and tags to display in IDEAS.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output_files |
List[OutputFile] |
List of output files registered by a tool. | required |
output_dir |
Optional[str |
Path] | Output directory where output files are generated. |
append |
Flag indicating whether to append new output files to existing output data already registered by the tool. | True |
|
raise_missing_file |
bool |
Raise an error if an output file is missing | True |
Functions:
| Name | Description |
|---|---|
close |
Close output data for writing. |
open |
Open output data for writing. |
register_file |
Register an output file as output data. |
Attributes:
| Name | Type | Description |
|---|---|---|
append |
bool |
}} |
file_pathname |
}} | |
output_dir |
Path |
}} |
output_files |
List[OutputFile] |
}} |
raise_missing_file |
bool |
}} |
Initialize output data
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output_dir |
Optional[str |
Path] | Output directory where output files are generated. If None, then the current working directory is used. |
append |
Flag indicating whether to append new output files to existing output data already registered by the tool. | True |
|
raise_missing_file |
bool |
Raise an error if an output file is missing | True |
OutputData.append¶
append: bool = append
OutputData.close¶
close()
Close output data for writing.
OutputData.file_pathname¶
file_pathname = output_dir / constants.OUTPUT_DATA_JSON_FILENAME
OutputData.open¶
open()
Open output data for writing.
OutputData.output_dir¶
output_dir: Path = output_dir
OutputData.output_files¶
output_files: List[OutputFile] = []
OutputData.raise_missing_file¶
raise_missing_file: bool = raise_missing_file
OutputData.register_file¶
register_file(file_path:str | Path, previews:Optional[List[Preview]] = None, metadata:Optional[List[Metadata]] = None, tags:Optional[List[Tag]] = None, output_dir:Optional[str | Path] = None, subdir:Optional[str | Path] = None, name:Optional[str] = None, prefix:Optional[str] = None, suffix:Optional[str] = None)
Register an output file as output data.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
file_path |
str |
Path | The path to the output file |
output_dir |
Optional[str |
Path] | Output directory where output files are generated. |
previews |
Optional[List[Preview]] |
List of previews to show in IDEAS for the output file | None |
metadata |
Optional[List[Metadata]] |
List of metadata to show in IDEAS for the output file | None |
tags |
Optional[List[Tag]] |
List of tags to show in IDEAS for the output file | None |
subdir |
Optional[str |
Path] | A subdirectory in the output directory where output files should be moved. Helpful for organizing result files into subdirectories for output columns |
prefix |
Optional[str] |
Prefix to optionally prepend to the output filename, and associated preview filenames. | None |
Returns:
| Type | Description |
|---|---|
OutputFile |
The output file (for command chaining) |
Raises:
| Type | Description |
|---|---|
OutputDataFileOutsideOutputDir |
If an output file is outside the output directory. |
OutputDataFileDoesNotExist |
If an output file does not exist. |
OutputFile¶
OutputFile(file_path:Path, output_dir:Path, subdir:Optional[str | Path] = None, prefix:Optional[str] = None, suffix:Optional[str] = None, previews:List[Preview] = list(), metadata:List[Metadata] = list(), tags:List[Tag] = list(), raise_missing_file:bool = True) -> None
Results generated by a tool, saved to a file, and saved in IDEAS.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
file_path |
Path |
The path to the output file | required |
output_dir |
Path |
Output directory where output files are generated. | required |
subdir |
Optional[str |
Path] | A subdirectory in the output directory where output files should be moved. Helpful for organizing result files into subdirectories for output columns |
prefix |
Optional[str] |
Prefix to optionally prepend to the output filename, and associated preview filenames. | None |
previews |
List[Preview] |
List of previews to show in IDEAS for the output file | list() |
metadata |
List[Metadata] |
List of metadata to show in IDEAS for the output file | list() |
tags |
List[Tag] |
List of tags to show in IDEAS for the output file | list() |
raise_missing_file |
bool |
Raise an error if an output file is missing | True |
Functions:
| Name | Description |
|---|---|
from_dict |
Deserialize from dict in output data file |
register_metadata |
Register metadata with the output file |
register_metadata_dict |
Register metadata dictionary with the output file |
register_preview |
Register a preview with the output file |
register_tags |
Register tags with the output file |
to_dict |
Serialize to dict to save in output data file |
Attributes:
| Name | Type | Description |
|---|---|---|
file_path |
Path |
}} |
metadata |
List[Metadata] |
}} |
output_dir |
Path |
}} |
prefix |
Optional[str] |
}} |
previews |
List[Preview] |
}} |
raise_missing_file |
bool |
}} |
subdir |
Optional[str |
Path] |
suffix |
Optional[str] |
}} |
tags |
List[Tag] |
}} |
OutputFile.file_path¶
file_path: Path
OutputFile.from_dict¶
from_dict(instance:Dict, output_dir:str | Path)
Deserialize from dict in output data file
OutputFile.metadata¶
metadata: List[Metadata] = field(default_factory=list)
OutputFile.output_dir¶
output_dir: Path
OutputFile.prefix¶
prefix: Optional[str] = None
OutputFile.previews¶
previews: List[Preview] = field(default_factory=list)
OutputFile.raise_missing_file¶
raise_missing_file: bool = True
OutputFile.register_metadata¶
register_metadata(key:str, value:str, name:Optional[str] = None)
Register metadata with the output file
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key |
str |
Unique identifier for the metadata across output files | required |
name |
Optional[str] |
Friendly name for the metadata to display in IDEAS | None |
value |
str |
The value of the metadata | required |
Returns:
| Type | Description |
|---|---|
OutputFile |
The output file (for command chaining) |
OutputFile.register_metadata_dict¶
register_metadata_dict(**kwargs:Dict[str, Union[str, bool, int, float, List[str]]])
Register metadata dictionary with the output file
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
kwargs |
Dict[str, Union[str, bool, int, float, List[str]]] |
Dictionary with key-value pairs of metadata | {} |
value |
Dict[str, Union[str, bool, int, float, List[str]]] |
The value of the metadata | required |
Returns:
| Type | Description |
|---|---|
OutputFile |
The output file (for command chaining) |
OutputFile.register_preview¶
register_preview(file_path:str | Path, caption:str = '', output_dir:Optional[str | Path] = None, subdir:Optional[str | Path] = None, name:Optional[str] = None, prefix:Optional[str] = None, suffix:Optional[str] = None)
Register a preview with the output file
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
file_path |
str |
Path | The path to the preview file |
caption |
str |
A short description of the preview. | '' |
output_dir |
Optional[str |
Path] | Output directory where output files are generated. If not specified, default to output directory for output data. |
subdir |
Optional[str |
Path] | A subdirectory in the output directory where output files should be moved. Helpful for organizing result files into subdirectories for output columns |
prefix |
Optional[str] |
Prefix to optionally prepend to the output filename, and associated preview filenames. | None |
Returns:
| Type | Description |
|---|---|
OutputFile |
The output file (for command chaining) |
Raises:
| Type | Description |
|---|---|
OutputDataFileOutsideOutputDir |
If the preview file is outside the output directory. |
OutputDataFileDoesNotExist |
If the preview file does not exist. |
OutputFile.register_tags¶
register_tags(*tags:List[str])
Register tags with the output file
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
tags |
List[str] |
Tags to register | () |
Returns:
| Type | Description |
|---|---|
OutputFile |
The output file (for command chaining) |
OutputFile.subdir¶
subdir: Optional[str | Path] = None
OutputFile.suffix¶
suffix: Optional[str] = None
OutputFile.tags¶
tags: List[Tag] = field(default_factory=list)
OutputFile.to_dict¶
to_dict()
Serialize to dict to save in output data file
Preview¶
Preview(file_path:Path, caption:str) -> None
A file which describes an output file. Preview files can be figures (i.e., images), movies (i.e., videos), json, or even an html webpage (static assets only). Previews are meant to be small in size, giving a brief glance, or impression at the results store in particular output file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
file_path |
Path |
The path to the preview file | required |
caption |
str |
A short description of the preview. | required |
Functions:
| Name | Description |
|---|---|
from_dict |
Deserialize from dict in output data file |
to_dict |
Serialize to dict to save in output data file |
Attributes:
| Name | Type | Description |
|---|---|---|
caption |
str |
}} |
file_path |
Path |
}} |
Preview.caption¶
caption: str
Preview.file_path¶
file_path: Path
Preview.from_dict¶
from_dict(instance)
Deserialize from dict in output data file
Preview.to_dict¶
to_dict()
Serialize to dict to save in output data file
Tag¶
Tag = TypeVar('Tag', bound=str)
input_paths_to_output_prefix¶
input_paths_to_output_prefix(*input_paths:List[List[str | Path | None] | str | Path | None], max_name_len:List[List[str | Path | None] | str | Path | None] = 15, sep:List[List[str | Path | None] | str | Path | None] = '_')
" Generate an output prefix from a set of input paths
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input_paths |
List[List[str |
Path | None] |
max_name_len |
Maximum number of characters to use from each input path | 15 |
|
sep |
Seperator to use between input paths | '_' |
Metadata¶
Metadata(key:str, name:str, value:Union[str, bool, int, float, List[str]]) -> None
Piece of key, value data that describes an output file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key |
str |
Unique identifier for the metadata across output files | required |
name |
str |
Friendly name for the metadata to display in IDEAS | required |
value |
Union[str, bool, int, float, List[str]] |
The value of the metadata | required |
Functions:
| Name | Description |
|---|---|
from_dict |
Deserialize from dict in output data file |
to_dict |
Serialize to dict to save in output data file |
Attributes:
| Name | Type | Description |
|---|---|---|
key |
str |
}} |
name |
str |
}} |
value |
Union[str, bool, int, float, List[str]] |
}} |
Metadata.from_dict¶
from_dict(instance)
Deserialize from dict in output data file
Metadata.key¶
key: str
Metadata.name¶
name: str
Metadata.to_dict¶
to_dict()
Serialize to dict to save in output data file
Metadata.value¶
value: Union[str, bool, int, float, List[str]]
types¶
Attributes:
| Name | Type | Description |
|---|---|---|
IdeasFile |
Type representing an IDEAS input file, as a string file path | }} |
IdeasFile¶
IdeasFile: TypeAlias = str
log¶
Functions:
| Name | Description |
|---|---|
get_logger |
Set up logger for IDEAS tools |
get_logger¶
get_logger()
Set up logger for IDEAS tools
exceptions¶
Classes:
| Name | Description |
|---|---|
ApiException |
Raised for any non-specific API errors; provides the JSON response from the API. |
ContainerDownloadAccessException |
Raised if a container could not be downloaded from IDEAS container registry. |
ContainerException |
Raised if there is an error with containers. |
ContainerNotFoundException |
Raised if a container is not found in IDEAS container registry. |
ContainerNotPublishedError |
Raised if a container is not published to IDEAS container registry when generating code bundle that references the container. |
ContainerPublishExistsException |
Raised if a container is not found in IDEAS container registry. |
ContainerPublishPostException |
Raised if there was an error creating the container image in IDEAS. |
ContainerPythonNotFound |
Raised if python is not an available command inside a chosen container |
ContainerUnsupportedArchException |
Raised if a container arch is not supported in IDEAS. |
DockerExecutionError |
Raised if there is an unexpected error with the Docker API. |
DockerUnavailableError |
Raised if there is an unexpected error with the Docker API. |
FileProcessedTimedOut |
Raised when creating a series, if the specified files have not been successfully processed in |
IdeasError |
Base class for all ideas exceptions. |
MismatchedChecksumError |
Raised if the checksum returned by AWS doesn't match the local checksum on download/upload of a |
NvidiaRuntimeMissingError |
Raised if the user is trying to use gpus for tool execution but docker reports the nvidia |
OutputDataException |
Raised if there is an exception related to output data |
OutputDataFileDoesNotExist |
Raised if an output file does not exist |
OutputDataFileOutsideOutputDir |
Raise if an output file exists outside of the output directory |
QuotaExceededError |
Raised if the API returns a 413 Payload Too Large error. |
ToolBundleBuildError |
Raised if there is an error building a code bundle. |
ToolBundleExtractError |
Raised if we are unable to extract the tool bundle file. |
ToolConfigContainerNotFoundError |
Raised if the user specifies a tool container image not found when configuring a tool. |
ToolConfigFileNotFoundError |
Raised if the user specifies a tool file not found when configuring a tool. |
ToolConfigFunctionNotFoundError |
Raised if the user specifies a tool function not found when configuring a tool. |
ToolException |
Raised if there is an error with tools. |
ToolExecutionError |
Raised if there is an error executing the tool code inside the tool container. |
ToolInputsFileNotFoundError |
Raised if an input file specified in tool inputs is not found in the user's local filesystem. |
ToolInputsFormatError |
Raised if tool inputs are not formatted correctly. |
ToolInputsNotFoundError |
Raised if a given tool inputs file is not found. |
ToolIntrospectionError |
Raised if we are unable to introspect the given tool code file. |
ToolMisconfiguredError |
Raised if a given tool is not configured correctly. |
ToolNotFoundError |
Raised if the no tools are found in a user's codebase. |
ToolSchemaValidationError |
Raised if we are unable to introspect the given tool code file. |
UnauthorizedError |
Raised if the Cognito authorization fails, or the API returns a 401/403 response. |
UnhandledError |
Raise this (hopefully infrequently) if you have no idea how to process an error. |
UserNotTenantMemberError |
Raised if user is trying to perform an action in a tenant they are not a member of. |
ApiException¶
ApiException(message, json)
Bases: IdeasError
Raised for any non-specific API errors; provides the JSON response from the API.
Attributes:
| Name | Type | Description |
|---|---|---|
json |
}} |
ApiException.json¶
json = json
ContainerDownloadAccessException¶
Bases: ContainerException
Raised if a container could not be downloaded from IDEAS container registry.
ContainerException¶
Bases: IdeasError
Raised if there is an error with containers.
ContainerNotFoundException¶
Bases: ContainerException
Raised if a container is not found in IDEAS container registry.
ContainerNotPublishedError¶
ContainerNotPublishedError(message = None, container = None)
Bases: ContainerException
Raised if a container is not published to IDEAS container registry when generating code bundle that references the container.
Attributes:
| Name | Type | Description |
|---|---|---|
container |
}} |
ContainerNotPublishedError.container¶
container = container
ContainerPublishExistsException¶
ContainerPublishExistsException(message = None, checksum = None)
Bases: ContainerException
Raised if a container is not found in IDEAS container registry.
Attributes:
| Name | Type | Description |
|---|---|---|
checksum |
}} |
ContainerPublishExistsException.checksum¶
checksum = checksum
ContainerPublishPostException¶
Bases: ContainerException
Raised if there was an error creating the container image in IDEAS.
ContainerPythonNotFound¶
Bases: ContainerException
Raised if python is not an available command inside a chosen container
ContainerUnsupportedArchException¶
ContainerUnsupportedArchException(message = None, arch = None)
Bases: ContainerException
Raised if a container arch is not supported in IDEAS.
Attributes:
| Name | Type | Description |
|---|---|---|
arch |
}} |
ContainerUnsupportedArchException.arch¶
arch = arch
DockerExecutionError¶
Bases: IdeasError
Raised if there is an unexpected error with the Docker API.
DockerUnavailableError¶
Bases: IdeasError
Raised if there is an unexpected error with the Docker API.
Functions:
| Name | Description |
|---|---|
get_instructions |
Get instructions on how to resolve this error. |
DockerUnavailableError.get_instructions¶
get_instructions()
Get instructions on how to resolve this error.
FileProcessedTimedOut¶
Bases: IdeasError
Raised when creating a series, if the specified files have not been successfully processed in a reasonable amount of time.
IdeasError¶
Bases: Exception
Base class for all ideas exceptions.
MismatchedChecksumError¶
Bases: IdeasError
Raised if the checksum returned by AWS doesn't match the local checksum on download/upload of a file.
NvidiaRuntimeMissingError¶
Bases: IdeasError
Raised if the user is trying to use gpus for tool execution but docker reports the nvidia runtime is missing.
Functions:
| Name | Description |
|---|---|
get_instructions |
Get instructions on how to resolve this error. |
NvidiaRuntimeMissingError.get_instructions¶
get_instructions()
Get instructions on how to resolve this error.
OutputDataException¶
Bases: Exception
Raised if there is an exception related to output data
OutputDataFileDoesNotExist¶
Bases: OutputDataException
Raised if an output file does not exist
OutputDataFileOutsideOutputDir¶
Bases: OutputDataException
Raise if an output file exists outside of the output directory
QuotaExceededError¶
Bases: ApiException
Raised if the API returns a 413 Payload Too Large error.
Attributes:
| Name | Type | Description |
|---|---|---|
json |
}} |
ToolBundleBuildError¶
Bases: ToolException
Raised if there is an error building a code bundle.
ToolBundleExtractError¶
Bases: ToolException
Raised if we are unable to extract the tool bundle file.
ToolConfigContainerNotFoundError¶
ToolConfigContainerNotFoundError(message = None, container = None)
Bases: ToolException
Raised if the user specifies a tool container image not found when configuring a tool.
Attributes:
| Name | Type | Description |
|---|---|---|
container |
}} |
ToolConfigContainerNotFoundError.container¶
container = container
ToolConfigFileNotFoundError¶
Bases: ToolException
Raised if the user specifies a tool file not found when configuring a tool.
ToolConfigFunctionNotFoundError¶
Bases: ToolException
Raised if the user specifies a tool function not found when configuring a tool.
ToolException¶
Bases: IdeasError
Raised if there is an error with tools.
ToolExecutionError¶
Bases: ToolException
Raised if there is an error executing the tool code inside the tool container.
ToolInputsFileNotFoundError¶
ToolInputsFileNotFoundError(message, file)
Bases: ToolException
Raised if an input file specified in tool inputs is not found in the user's local filesystem.
Attributes:
| Name | Type | Description |
|---|---|---|
file |
}} |
ToolInputsFileNotFoundError.file¶
file = file
ToolInputsFormatError¶
ToolInputsFormatError(message = None, key = None, value = None)
Bases: ToolException
Raised if tool inputs are not formatted correctly.
Attributes:
| Name | Type | Description |
|---|---|---|
key |
}} | |
message |
}} | |
value |
}} |
ToolInputsFormatError.key¶
key = key
ToolInputsFormatError.message¶
message = message
ToolInputsFormatError.value¶
value = value
ToolInputsNotFoundError¶
Bases: ToolException
Raised if a given tool inputs file is not found.
ToolIntrospectionError¶
Bases: ToolException
Raised if we are unable to introspect the given tool code file.
ToolMisconfiguredError¶
ToolMisconfiguredError(message = None, key = None)
Bases: ToolException
Raised if a given tool is not configured correctly.
Attributes:
| Name | Type | Description |
|---|---|---|
key |
}} |
ToolMisconfiguredError.key¶
key = key
ToolNotFoundError¶
Bases: ToolException
Raised if the no tools are found in a user's codebase.
ToolSchemaValidationError¶
Bases: ToolException
Raised if we are unable to introspect the given tool code file.
UnauthorizedError¶
Bases: IdeasError
Raised if the Cognito authorization fails, or the API returns a 401/403 response.
UnhandledError¶
Bases: IdeasError
Raise this (hopefully infrequently) if you have no idea how to process an error.
UserNotTenantMemberError¶
Bases: IdeasError
Raised if user is trying to perform an action in a tenant they are not a member of.