Data Layer

The hastegeo.core.data_layer package provides a multi-backend storage abstraction for HASTE metadata and file storage. The unified data layer delegates to backend-specific implementations based on configuration.

Supported Backends

Backend	Class	Use Case
Local filesystem	LocalFileSystemDataLayer	Local development
Azure Blob Storage	AzureBlobStorageDataLayer	Cloud file storage
Azure CosmosDB	AzureCosmosDBDataLayer	Cloud document storage
Azure Data Lake	AzureDataLakeDataLayer	Cloud hierarchical storage
PostgreSQL	AzurePostgreSQLDataLayer	Relational database storage

Unified Data Layer (hastegeo.core.data_layer.unified)

Factory wrapper that instantiates the correct backend based on the StorageType configuration.

class hastegeo.core.data_layer.unified.UnifiedDataLayer(storage_type, partition_key=None, **kwargs)

Bases: object

__init__(storage_type, partition_key=None, **kwargs)

delete(identifier, data_type, data_format='json')

delete_all_from_partition()

finalize_save(identifier, data_type, data_format='tif', data_file_path=None, total_chunks=None)

get_base_url()

get_file_path(identifier, data_type=None, data_format='json', extra_partition_keys=None)

get_file_remote_path(identifier=None, data_type=None, data_format='json', extra_partition_keys=None)

load(identifier, data_type, data_format='json')

load_all(data_type, data_format='json')

load_all_from_partition(data_type, data_format='json')

save(identifier, data_type, data=None, data_file_path=None, data_format='json')

save_chunk(identifier, data_type, data=None, data_file_path=None, data_format='tif', chunk_id=None)

update(identifier, data_type, data=None, data_file_path=None, data_format='json')

Abstract Base (hastegeo.core.data_layer.abstract_data_layer)

Defines the interface all data layer implementations must follow.

Abstract methods: save(), save_chunk(), finalize_save(), update(), load(), load_all(), delete()

class hastegeo.core.data_layer.abstract_data_layer.AbstractDataLayer(partition_key=None)

Bases: ABC

Abstract base class for data layer implementations.

The AbstractDataLayer defines the interface for data storage and retrieval operations across different storage backends (local filesystem, Azure Blob, Azure Cosmos DB, etc.). All concrete data layer implementations must inherit from this class and implement its abstract methods.

Parameters:

partition_key (str, optional) – Key used for data partitioning. Defaults to None.

partition_key

The partition key for organizing data.

Type:

str

__init__(partition_key=None)

Initialize the abstract data layer.

Parameters:

partition_key (str, optional) – Key used for data partitioning. Defaults to None.

abstract delete(identifier, data_type, data_format='json')

Delete a specific data record from the storage backend.

Parameters:

• identifier (str) – Unique identifier for the data record to delete.

• data_type (str) – Type/category of the data being deleted.

• data_format (str, optional) – Format of the data. Defaults to “json”.

Raises:

FileNotFoundError – If the specified data record doesn’t exist.

abstract delete_all_from_partition()

Delete all data records from the current partition.

Warning

This operation is irreversible and will remove all data from the current partition.

abstract finalize_save(identifier, data_type, data_format='tif', data_file_path=None, total_chunks=None)

Finalize a chunked upload operation by combining all chunks.

Parameters:

• identifier (str) – Unique identifier for the data record.

• data_type (str) – Type/category of the data being saved.

• data_format (str, optional) – Format of the data (tif, etc.). Defaults to “tif”.

• data_file_path (str, optional) – Final path for the combined file. Defaults to None.

• total_chunks (int, optional) – Expected total number of chunks. Defaults to None.

Note

This method completes a chunked upload operation started with save_chunk() calls.

is_bytes(data)

Check if data is a bytes object.

Parameters:

data (Any) – Data to check.

Returns:

True if data is bytes, False otherwise.

Return type:

bool

is_json(data)

Check if data can be serialized as JSON.

Parameters:

data (Any) – Data to check for JSON compatibility.

Returns:

True if data is JSON-serializable, False otherwise.

Return type:

bool

is_yaml(data)

Check if data can be serialized as YAML.

Parameters:

data (Any) – Data to check for YAML compatibility.

Returns:

True if data is YAML-serializable, False otherwise.

Return type:

bool

abstract load(identifier, data_type, data_format='json')

Load data from the storage backend.

Parameters:

• identifier (str) – Unique identifier for the data record to load.

• data_type (str) – Type/category of the data being loaded.

• data_format (str, optional) – Expected format of the data. Defaults to “json”.

Returns:

The loaded data object or bytes depending on format.

Return type:

Any

Raises:

FileNotFoundError – If the specified data record doesn’t exist.

abstract load_all(data_type, data_format='json')

Load all data records of a specific type.

Parameters:

• data_type (str) – Type/category of the data to load.

• data_format (str, optional) – Expected format of the data. Defaults to “json”.

Returns:

List of all data records of the specified type.

Return type:

List[Any]

abstract load_all_from_partition(data_type, data_format='json')

Load all data records of a specific type from the current partition.

Parameters:

• data_type (str) – Type/category of the data to load.

• data_format (str, optional) – Expected format of the data. Defaults to “json”.

Returns:

List of all data records of the specified type

from the current partition.

Return type:

List[Any]

load_data_from_file(data_file_path)

Load data from a file based on its extension.

Parameters:

data_file_path (str) – Path to the file containing data to load.

Returns:

Parsed data object for JSON files, or raw bytes for

image/binary files.

Return type:

Any

Raises:

ValueError – If the file format is not supported.

Supported formats:

• .json: Returns parsed JSON object

• .tif/.tiff: Returns raw bytes

• .jpeg/.jpg: Returns raw bytes

• .png: Returns raw bytes

abstract save(identifier, data_type, data=None, data_file_path=None, data_format='json')

Save data to the storage backend.

Parameters:

• identifier (str) – Unique identifier for the data record.

• data_type (str) – Type/category of the data being saved.

• data (Any, optional) – Data object to save. Defaults to None.

• data_file_path (str, optional) – Path to file containing data. Defaults to None.

• data_format (str, optional) – Format of the data (json, tif, etc.). Defaults to “json”.

Note

Either data or data_file_path must be provided, but not both.

abstract save_chunk(identifier, data_type, data=None, data_file_path=None, data_format='tif', chunk_id=None)

Save a chunk of data as part of a chunked upload operation.

Parameters:

• identifier (str) – Unique identifier for the data record.

• data_type (str) – Type/category of the data being saved.

• data (Any, optional) – Data chunk to save. Defaults to None.

• data_file_path (str, optional) – Path to file containing data chunk. Defaults to None.

• data_format (str, optional) – Format of the data (tif, etc.). Defaults to “tif”.

• chunk_id (str, optional) – Unique identifier for this chunk. Defaults to None.

Note

This method is used for uploading large files in smaller chunks. Must be followed by finalize_save() to complete the operation.

abstract update(identifier, data_type, data=None, data_file_path=None, data_format='json')

Update existing data in the storage backend.

Parameters:

• identifier (str) – Unique identifier for the data record to update.

• data_type (str) – Type/category of the data being updated.

• data (Any, optional) – New data object. Defaults to None.

• data_file_path (str, optional) – Path to file containing new data. Defaults to None.

• data_format (str, optional) – Format of the data (json, tif, etc.). Defaults to “json”.

Note

Either data or data_file_path must be provided, but not both.

validate_data_input(data, data_file_path)

Validate that exactly one data input method is provided.

Parameters:

• data (Any) – Data object to validate.

• data_file_path (str) – File path to validate.

Returns:

True if validation passes.

Return type:

bool

Raises:

ValueError – If neither or both data inputs are provided.

Local Filesystem (hastegeo.core.data_layer.local_file_system_data_layer)

class hastegeo.core.data_layer.local_file_system_data_layer.LocalFileSystemDataLayer(directory, partition_key=None)

Bases: AbstractDataLayer

Local filesystem implementation of the AbstractDataLayer.

This class provides data storage and retrieval operations using the local filesystem. Data is organized in directories based on partition keys and data types, with files named according to their identifiers and formats.

Parameters:

• directory (str) – Base directory for data storage.

• partition_key (str, optional) – Key for data partitioning. Defaults to None.

directory

The resolved absolute path to the storage directory.

Type:

str

Example

>>> data_layer = LocalFileSystemDataLayer("/data", "project_123")
>>> data_layer.save("model_1", "models", {"accuracy": 0.95})
>>> model_data = data_layer.load("model_1", "models")

__init__(directory, partition_key=None)

Initialize the local filesystem data layer.

Parameters:

• directory (str) – Base directory for data storage.

• partition_key (str, optional) – Key for data partitioning. Creates a subdirectory for organization. Defaults to None.

delete(identifier, data_type, data_format='json')

Delete a specific data record from the storage backend.

Parameters:

• identifier (str) – Unique identifier for the data record to delete.

• data_type (str) – Type/category of the data being deleted.

• data_format (str, optional) – Format of the data. Defaults to “json”.

Raises:

FileNotFoundError – If the specified data record doesn’t exist.

delete_all_from_partition()

Delete all data records from the current partition.

Warning

This operation is irreversible and will remove all data from the current partition.

finalize_save(identifier, data_type, data_format='tif', data_file_path=None, total_chunks=None)

Finalize a chunked upload (not implemented for local filesystem).

Parameters:

• identifier (str) – Unique identifier for the data record.

• data_type (str) – Type/category of the data being saved.

• data_format (str, optional) – Format of the data (tif, etc.). Defaults to “tif”.

• data_file_path (str, optional) – Final path for the combined file. Defaults to None.

• total_chunks (int, optional) – Expected total number of chunks. Defaults to None.

Raises:

NotImplementedError – This method is not implemented for local filesystem storage as chunked uploads are not necessary for local operations.

get_file_path(identifier, data_type=None, data_format='json', extra_partition_keys=None)

Generate the full file path for a data record.

Parameters:

• identifier (str) – Unique identifier for the data record.

• data_type (str, optional) – Type/category of the data. Defaults to None.

• data_format (str, optional) – File format extension. Defaults to “json”.

• extra_partition_keys (Union[str, List[str]], optional) – Additional partition keys for nested directory structure. Defaults to None.

Returns:

Complete file path for the data record.

Return type:

str

Example

>>> layer.get_file_path("model_1", "models", "json", ["v1", "test"])
"/data/project_123/v1/test/models_model_1.json"

get_file_remote_path(identifier=None, data_type=None, data_format='json', extra_partition_keys=None)

Get the remote path for a file (same as local path for filesystem layer).

This method provides compatibility with cloud storage implementations where remote and local paths may differ. For local filesystem storage, it returns the same result as get_file_path().

Parameters:

• identifier (str, optional) – Unique identifier for the data record. Defaults to None.

• data_type (str, optional) – Type/category of the data. Defaults to None.

• data_format (str, optional) – File format extension. Defaults to “json”.

• extra_partition_keys (Union[str, List[str]], optional) – Additional partition keys. Defaults to None.

Returns:

Complete file path (same as local path for filesystem storage).

Return type:

str

load(identifier, data_type, data_format='json')

Load data from the storage backend.

Parameters:

• identifier (str) – Unique identifier for the data record to load.

• data_type (str) – Type/category of the data being loaded.

• data_format (str, optional) – Expected format of the data. Defaults to “json”.

Returns:

The loaded data object or bytes depending on format.

Return type:

Any

Raises:

FileNotFoundError – If the specified data record doesn’t exist.

load_all(data_type, data_format='json')

Load all data records of a specific type.

Parameters:

• data_type (str) – Type/category of the data to load.

• data_format (str, optional) – Expected format of the data. Defaults to “json”.

Returns:

List of all data records of the specified type.

Return type:

List[Any]

load_all_from_partition(data_type, data_format='json')

Load all data records of a specific type from the current partition.

Parameters:

• data_type (str) – Type/category of the data to load.

• data_format (str, optional) – Expected format of the data. Defaults to “json”.

Returns:

List of all data records of the specified type

from the current partition.

Return type:

List[Any]

save(identifier, data_type, data=None, data_file_path=None, data_format='json')

Save data to the local filesystem.

Parameters:

• identifier (str) – Unique identifier for the data record.

• data_type (str) – Type/category of the data being saved.

• data (Any, optional) – Data object to save. Defaults to None.

• data_file_path (str, optional) – Path to source file to copy. Defaults to None.

• data_format (str, optional) – Format of the data (json, tif, etc.). Defaults to “json”.

Raises:

ValueError – If data format is unsupported or if data input validation fails.

Note

If data_file_path is provided, the source file will be copied to the destination and the original file will be removed. For data objects, JSON data is serialized and bytes data is written directly.

save_chunk(identifier, data_type, data=None, data_file_path=None, data_format='tif', chunk_id=None)

Save a chunk of data (not implemented for local filesystem).

Parameters:

• identifier (str) – Unique identifier for the data record.

• data_type (str) – Type/category of the data being saved.

• data (Any, optional) – Data chunk to save. Defaults to None.

• data_file_path (str, optional) – Path to file containing data chunk. Defaults to None.

• data_format (str, optional) – Format of the data (tif, etc.). Defaults to “tif”.

• chunk_id (str, optional) – Unique identifier for this chunk. Defaults to None.

Raises:

NotImplementedError – This method is not implemented for local filesystem storage as chunked uploads are not necessary for local operations.

Note

For local filesystem operations, use the regular save() method instead as chunked uploads are not required.

update(identifier, data_type, data=None, data_file_path=None, data_format='json')

Update existing data in the storage backend.

Parameters:

• identifier (str) – Unique identifier for the data record to update.

• data_type (str) – Type/category of the data being updated.

• data (Any, optional) – New data object. Defaults to None.

• data_file_path (str, optional) – Path to file containing new data. Defaults to None.

• data_format (str, optional) – Format of the data (json, tif, etc.). Defaults to “json”.

Note

Either data or data_file_path must be provided, but not both.

Azure Blob Storage (hastegeo.core.data_layer.azure_blob_storage_data_layer)

class hastegeo.core.data_layer.azure_blob_storage_data_layer.AzureBlobStorageDataLayer(account_url, container, connection_string, container_read_policy_name='image-r-policy', partition_key=None)

Bases: AbstractDataLayer

__init__(account_url, container, connection_string, container_read_policy_name='image-r-policy', partition_key=None)

Initialize the abstract data layer.

Parameters:

partition_key (str, optional) – Key used for data partitioning. Defaults to None.

delete(identifier, data_type, data_format='json')

Delete a specific data record from the storage backend.

Parameters:

• identifier (str) – Unique identifier for the data record to delete.

• data_type (str) – Type/category of the data being deleted.

• data_format (str, optional) – Format of the data. Defaults to “json”.

Raises:

FileNotFoundError – If the specified data record doesn’t exist.

delete_all_from_partition()

Delete all data records from the current partition.

Warning

This operation is irreversible and will remove all data from the current partition.

finalize_save(identifier, data_type, data_format='tif', data_file_path=None, total_chunks=None)

Finalize a chunked upload operation by combining all chunks.

Parameters:

• identifier (str) – Unique identifier for the data record.

• data_type (str) – Type/category of the data being saved.

• data_format (str, optional) – Format of the data (tif, etc.). Defaults to “tif”.

• data_file_path (str, optional) – Final path for the combined file. Defaults to None.

• total_chunks (int, optional) – Expected total number of chunks. Defaults to None.

Note

This method completes a chunked upload operation started with save_chunk() calls.

get_base_url()

get_file_path(identifier, data_type=None, data_format='json', extra_partition_keys=None)

get_file_remote_path(identifier=None, data_type=None, data_format='json', extra_partition_keys=None)

load(identifier, data_type, data_format='json')

Load data from the storage backend.

Parameters:

• identifier (str) – Unique identifier for the data record to load.

• data_type (str) – Type/category of the data being loaded.

• data_format (str, optional) – Expected format of the data. Defaults to “json”.

Returns:

The loaded data object or bytes depending on format.

Return type:

Any

Raises:

FileNotFoundError – If the specified data record doesn’t exist.

load_all(data_type, data_format='json')

Load all data records of a specific type.

Parameters:

• data_type (str) – Type/category of the data to load.

• data_format (str, optional) – Expected format of the data. Defaults to “json”.

Returns:

List of all data records of the specified type.

Return type:

List[Any]

load_all_from_partition(data_type, data_format='json')

Load all data records of a specific type from the current partition.

Parameters:

• data_type (str) – Type/category of the data to load.

• data_format (str, optional) – Expected format of the data. Defaults to “json”.

Returns:

List of all data records of the specified type

from the current partition.

Return type:

List[Any]

save(identifier, data_type, data=None, data_file_path=None, data_format='json', extra_partition_keys=None)

Save data to the storage backend.

Parameters:

• identifier (str) – Unique identifier for the data record.

• data_type (str) – Type/category of the data being saved.

• data (Any, optional) – Data object to save. Defaults to None.

• data_file_path (str, optional) – Path to file containing data. Defaults to None.

• data_format (str, optional) – Format of the data (json, tif, etc.). Defaults to “json”.

Note

Either data or data_file_path must be provided, but not both.

save_chunk(identifier, data_type, data=None, data_file_path=None, data_format='tif', chunk_id=None)

Save a chunk of data as part of a chunked upload operation.

Parameters:

• identifier (str) – Unique identifier for the data record.

• data_type (str) – Type/category of the data being saved.

• data (Any, optional) – Data chunk to save. Defaults to None.

• data_file_path (str, optional) – Path to file containing data chunk. Defaults to None.

• data_format (str, optional) – Format of the data (tif, etc.). Defaults to “tif”.

• chunk_id (str, optional) – Unique identifier for this chunk. Defaults to None.

Note

This method is used for uploading large files in smaller chunks. Must be followed by finalize_save() to complete the operation.

update(identifier, data_type, data=None, data_file_path=None, data_format='json')

Update existing data in the storage backend.

Parameters:

• identifier (str) – Unique identifier for the data record to update.

• data_type (str) – Type/category of the data being updated.

• data (Any, optional) – New data object. Defaults to None.

• data_file_path (str, optional) – Path to file containing new data. Defaults to None.

• data_format (str, optional) – Format of the data (json, tif, etc.). Defaults to “json”.

Note

Either data or data_file_path must be provided, but not both.

Azure CosmosDB (hastegeo.core.data_layer.azure_cosmos_db_data_layer)

class hastegeo.core.data_layer.azure_cosmos_db_data_layer.AzureCosmosDBDataLayer(endpoint, database, container, partition_key=None)

Bases: AbstractDataLayer

__init__(endpoint, database, container, partition_key=None)

Initialize the abstract data layer.

Parameters:

partition_key (str, optional) – Key used for data partitioning. Defaults to None.

delete(identifier, data_type, data_format='json')

Delete a specific data record from the storage backend.

Parameters:

• identifier (str) – Unique identifier for the data record to delete.

• data_type (str) – Type/category of the data being deleted.

• data_format (str, optional) – Format of the data. Defaults to “json”.

Raises:

FileNotFoundError – If the specified data record doesn’t exist.

delete_all_from_partition()

Delete all data records from the current partition.

Warning

This operation is irreversible and will remove all data from the current partition.

finalize_save(identifier, data_type, data_format='tif', data_file_path=None, total_chunks=None)

Finalize a chunked upload operation by combining all chunks.

Parameters:

• identifier (str) – Unique identifier for the data record.

• data_type (str) – Type/category of the data being saved.

• data_format (str, optional) – Format of the data (tif, etc.). Defaults to “tif”.

• data_file_path (str, optional) – Final path for the combined file. Defaults to None.

• total_chunks (int, optional) – Expected total number of chunks. Defaults to None.

Note

This method completes a chunked upload operation started with save_chunk() calls.

load(identifier, data_type)

Load data from the storage backend.

Parameters:

• identifier (str) – Unique identifier for the data record to load.

• data_type (str) – Type/category of the data being loaded.

• data_format (str, optional) – Expected format of the data. Defaults to “json”.

Returns:

The loaded data object or bytes depending on format.

Return type:

Any

Raises:

FileNotFoundError – If the specified data record doesn’t exist.

load_all(data_type)

Load all data records of a specific type.

Parameters:

• data_type (str) – Type/category of the data to load.

• data_format (str, optional) – Expected format of the data. Defaults to “json”.

Returns:

List of all data records of the specified type.

Return type:

List[Any]

load_all_from_partition(data_type)

Load all data records of a specific type from the current partition.

Parameters:

• data_type (str) – Type/category of the data to load.

• data_format (str, optional) – Expected format of the data. Defaults to “json”.

Returns:

List of all data records of the specified type

from the current partition.

Return type:

List[Any]

save(identifier, data_type, data=None, data_file_path=None, data_format='json')

Save data to the storage backend.

Parameters:

• identifier (str) – Unique identifier for the data record.

• data_type (str) – Type/category of the data being saved.

• data (Any, optional) – Data object to save. Defaults to None.

• data_file_path (str, optional) – Path to file containing data. Defaults to None.

• data_format (str, optional) – Format of the data (json, tif, etc.). Defaults to “json”.

Note

Either data or data_file_path must be provided, but not both.

save_chunk(identifier, data_type, data=None, data_file_path=None, data_format='tif', chunk_id=None)

Save a chunk of data as part of a chunked upload operation.

Parameters:

• identifier (str) – Unique identifier for the data record.

• data_type (str) – Type/category of the data being saved.

• data (Any, optional) – Data chunk to save. Defaults to None.

• data_file_path (str, optional) – Path to file containing data chunk. Defaults to None.

• data_format (str, optional) – Format of the data (tif, etc.). Defaults to “tif”.

• chunk_id (str, optional) – Unique identifier for this chunk. Defaults to None.

Note

This method is used for uploading large files in smaller chunks. Must be followed by finalize_save() to complete the operation.

update(data, identifier, data_type, data_format='json')

Update existing data in the storage backend.

Parameters:

• identifier (str) – Unique identifier for the data record to update.

• data_type (str) – Type/category of the data being updated.

• data (Any, optional) – New data object. Defaults to None.

• data_file_path (str, optional) – Path to file containing new data. Defaults to None.

• data_format (str, optional) – Format of the data (json, tif, etc.). Defaults to “json”.

Note

Either data or data_file_path must be provided, but not both.

Azure Data Lake (hastegeo.core.data_layer.azure_data_lake_data_layer)

class hastegeo.core.data_layer.azure_data_lake_data_layer.AzureDataLakeDataLayer(account_url, file_system, partition_key=None)

Bases: AbstractDataLayer

__init__(account_url, file_system, partition_key=None)

Initialize the abstract data layer.

Parameters:

partition_key (str, optional) – Key used for data partitioning. Defaults to None.

delete(identifier, data_type, data_format='json')

Delete a specific data record from the storage backend.

Parameters:

• identifier (str) – Unique identifier for the data record to delete.

• data_type (str) – Type/category of the data being deleted.

• data_format (str, optional) – Format of the data. Defaults to “json”.

Raises:

FileNotFoundError – If the specified data record doesn’t exist.

delete_all_from_partition()

Delete all data records from the current partition.

Warning

This operation is irreversible and will remove all data from the current partition.

finalize_save(identifier, data_type, data_format='tif', data_file_path=None, total_chunks=None)

Finalize a chunked upload operation by combining all chunks.

Parameters:

• identifier (str) – Unique identifier for the data record.

• data_type (str) – Type/category of the data being saved.

• data_format (str, optional) – Format of the data (tif, etc.). Defaults to “tif”.

• data_file_path (str, optional) – Final path for the combined file. Defaults to None.

• total_chunks (int, optional) – Expected total number of chunks. Defaults to None.

Note

This method completes a chunked upload operation started with save_chunk() calls.

get_file_path(identifier, data_type=None, data_format='json', extra_partition_keys=None)

get_file_remote_path(identifier=None, data_type=None, data_format='json', extra_partition_keys=None)

load(identifier, data_type)

Load data from the storage backend.

Parameters:

• identifier (str) – Unique identifier for the data record to load.

• data_type (str) – Type/category of the data being loaded.

• data_format (str, optional) – Expected format of the data. Defaults to “json”.

Returns:

The loaded data object or bytes depending on format.

Return type:

Any

Raises:

FileNotFoundError – If the specified data record doesn’t exist.

load_all(data_type)

Load all data records of a specific type.

Parameters:

• data_type (str) – Type/category of the data to load.

• data_format (str, optional) – Expected format of the data. Defaults to “json”.

Returns:

List of all data records of the specified type.

Return type:

List[Any]

load_all_from_partition(data_type)

Load all data records of a specific type from the current partition.

Parameters:

• data_type (str) – Type/category of the data to load.

• data_format (str, optional) – Expected format of the data. Defaults to “json”.

Returns:

List of all data records of the specified type

from the current partition.

Return type:

List[Any]

save(identifier, data_type, data=None, data_file_path=None, data_format='json')

Save data to the storage backend.

Parameters:

• identifier (str) – Unique identifier for the data record.

• data_type (str) – Type/category of the data being saved.

• data (Any, optional) – Data object to save. Defaults to None.

• data_file_path (str, optional) – Path to file containing data. Defaults to None.

• data_format (str, optional) – Format of the data (json, tif, etc.). Defaults to “json”.

Note

Either data or data_file_path must be provided, but not both.

save_chunk(identifier, data_type, data=None, data_file_path=None, data_format='tif', chunk_id=None)

Save a chunk of data as part of a chunked upload operation.

Parameters:

• identifier (str) – Unique identifier for the data record.

• data_type (str) – Type/category of the data being saved.

• data (Any, optional) – Data chunk to save. Defaults to None.

• data_file_path (str, optional) – Path to file containing data chunk. Defaults to None.

• data_format (str, optional) – Format of the data (tif, etc.). Defaults to “tif”.

• chunk_id (str, optional) – Unique identifier for this chunk. Defaults to None.

Note

This method is used for uploading large files in smaller chunks. Must be followed by finalize_save() to complete the operation.

update(data, identifier, data_type)

Update existing data in the storage backend.

Parameters:

• identifier (str) – Unique identifier for the data record to update.

• data_type (str) – Type/category of the data being updated.

• data (Any, optional) – New data object. Defaults to None.

• data_file_path (str, optional) – Path to file containing new data. Defaults to None.

• data_format (str, optional) – Format of the data (json, tif, etc.). Defaults to “json”.

Note

Either data or data_file_path must be provided, but not both.

PostgreSQL (hastegeo.core.data_layer.azure_postgresql_data_layer)