Artifact Storage

Artifact Storage#

The hastegeo.core.artifact_storage package provides an abstraction layer for storing and retrieving model artifacts (weights, predictions, configuration files).

Supported Backends#

Backend

Class

Description

Local filesystem

LocalFileSystemArtifactStorage

Local development storage

Azure Blob Storage

AzureBlobArtifactStorage

Cloud storage with SAS URL generation

Unified Artifact Storage (hastegeo.core.artifact_storage.unified_artifact_storage)#

Factory wrapper that delegates to the correct backend implementation.

class hastegeo.core.artifact_storage.unified_artifact_storage.UnifiedArtifactStorage(storage_type, partition_key=None, **kwargs)[source]#

Bases: object

__init__(storage_type, partition_key=None, **kwargs)[source]#
fetch_artifact(identifier=None, extra_partition_keys=None, src_path=None, dst_path=None)[source]#

Download the artifact from the storage.

get_base_url()[source]#
get_download_url(identifier=None, artifact_path=None, extra_partition_keys=None)[source]#

Get the download URL for the artifact.

get_file_path(identifier, extra_partition_keys=None)[source]#

Get the file path for the artifact.

store_artifact(artifact_name, data=None, src_path=None, namespace=None)[source]#

Store the artifact in storage.

Parameters:

  • artifact_name (str) – The name of the artifact in storage.

  • data (str) – Optional. The data to write as a string.

  • src_path (str) – Optional. The source path of the file to copy/upload. Either data or src_path must be provided.

  • namespace (str | list) – The namespace aka folder structure to use for the artifact.

Returns:

The path where the artifact was stored.

Return type:

str

Raises:

Abstract Base (hastegeo.core.artifact_storage.abstract_artifact_storage)#

Defines the interface and shared utilities for all artifact storage implementations.

Abstract methods: get_file_path(), get_download_url(), fetch_artifact(), store_artifact(), get_base_url()

Utility functions: is_json(), is_bytes(), is_yaml()

class hastegeo.core.artifact_storage.abstract_artifact_storage.AbstractArtifactStorage(partition_key=None)[source]#

Bases: ABC

__init__(partition_key=None)[source]#
abstract fetch_artifact(identifier=None, extra_partition_keys=None, src_path=None, dst_path=None)[source]#
abstract get_base_url()[source]#
abstract get_download_url(identifier=None, artifact_path=None, extra_partition_keys=None)[source]#
abstract get_file_path(identifier, extra_partition_keys=None)[source]#
is_bytes(data)[source]#

Check if data is a bytes object.

Parameters:

data (Any) – Data to check.

Returns:

True if data is bytes, False otherwise.

Return type:

bool

is_json(data)[source]#

Check if data can be serialized as JSON.

Parameters:

data (Any) – Data to check for JSON compatibility.

Returns:

True if data is JSON-serializable, False otherwise.

Return type:

bool

is_yaml(data)[source]#

Check if data can be serialized as YAML.

Parameters:

data (Any) – Data to check for YAML compatibility.

Returns:

True if data is YAML-serializable, False otherwise.

Return type:

bool

abstract store_artifact(artifact_name, data=None, src_path=None, namespace=None)[source]#

Azure Blob Storage (hastegeo.core.artifact_storage.azure_blob_artifact_storage)#

Stores artifacts in Azure Blob Storage with SAS-token-enabled download URLs.

class hastegeo.core.artifact_storage.azure_blob_artifact_storage.AzureBlobArtifactStorage(account_url, container, connection_string, container_read_policy_name='image-r-policy', blob_read_policy_name='imageblob-r-policy', partition_key=None)[source]#

Bases: AbstractArtifactStorage

__init__(account_url, container, connection_string, container_read_policy_name='image-r-policy', blob_read_policy_name='imageblob-r-policy', partition_key=None)[source]#
fetch_artifact(identifier=None, extra_partition_keys=None, src_path=None, dst_path=None)[source]#

Download the artifact from Azure Blob Storage. :param src_path: The source path of the file in Azure Blob Storage. :type src_path: str :param dst_path: The destination path to save the downloaded file. :type dst_path: str

get_base_url()[source]#
get_download_url(identifier=None, artifact_path=None, extra_partition_keys=None, include_sas=True)[source]#

Get the remote path for the file in Azure Blob Storage. :param identifier: The identifier for the file. Mutually exclusive with url. :type identifier: str :param artifact_path: The path to the artifact relative to storage

root. Either artifact_path or identifier must be provided.

Parameters:

  • extra_partition_keys (list) – Extra partition keys to include in the path.

  • include_sas (bool) – Whether to include a SAS token in the URL.

Returns:

The full remote path for the file in Azure Blob Storage, optionally with SAS token.

Return type:

str

get_file_path(identifier, extra_partition_keys=None)[source]#
store_artifact(artifact_name, data=None, src_path=None, namespace=None)[source]#

Upload the artifact to Azure Blob Storage.

Parameters:

  • artifact_name (str) – The name of the artifact in Azure Blob Storage.

  • data (str) – Optional. The data to upload as a string.

  • src_path (str) – Optional. The source path of the file to upload. Either data or src_path must be provided.

  • namespace (str | list) – The namespace aka folder structure to use for the artifact.

Returns:

The blob path where the artifact was stored.

Return type:

str

Raises:

Local Filesystem (hastegeo.core.artifact_storage.local_file_system_artifact_storage)#

Stores artifacts on the local filesystem for development.

class hastegeo.core.artifact_storage.local_file_system_artifact_storage.LocalFileSystemArtifactStorage(partition_key=None, **kwargs)[source]#

Bases: AbstractArtifactStorage

__init__(partition_key=None, **kwargs)[source]#
fetch_artifact(identifier=None, extra_partition_keys=None, src_path=None, dst_path=None)[source]#

Download the artifact from the source path to the destination path.

get_base_url()[source]#
get_download_url(identifier=None, artifact_path=None, extra_partition_keys=None)[source]#
get_file_path(identifier, extra_partition_keys=None)[source]#
store_artifact(artifact_name, data=None, src_path=None, namespace=None)[source]#

Store the artifact in the local file system.

Parameters:

  • artifact_name (str) – The name of the artifact in the local file system.

  • data (str) – Optional. The data to write as a string.

  • src_path (str) – Optional. The source path of the file to copy. Either data or src_path must be provided.

  • namespace (str | list) – The namespace aka folder structure to use for the artifact.

Returns:

The local path where the artifact was stored.

Return type:

str

Raises:
Contents