phidown.downloader_boto3_deprecated

Attributes

config

Functions

load_credentials(→ Tuple[str, str])

Load username and password from a YAML file or create the file if missing.

check_s3_permissions(→ bool)

Check if we have the necessary S3 permissions before starting downloads.

download_s3_file_with_retry(→ bool)

Download a single file from S3 with retry logic and proper error handling.

get_access_token(→ str)

Retrieve an access token from the authentication server.

get_eo_product_details(→ Tuple[str, str])

Retrieve EO product details using the OData API to determine the S3 path.

parse_s3_path(→ Tuple[str, str])

Parse S3 path to extract bucket name and object key.

get_temporary_s3_credentials(→ dict)

Create temporary S3 credentials by calling the S3 keys manager API.

create_s3_client_with_retry(→ boto3.resource)

Create S3 client with retry logic and validation.

test_s3_connectivity(→ bool)

Test S3 connectivity and basic permissions.

test_s3_connectivity_detailed(→ bool)

Test S3 connectivity with detailed debugging information.

format_filename(filename[, length])

Format a filename to a fixed length, truncating if necessary.

download_file_s3(→ None)

Download a file from S3 with improved error handling.

traverse_and_download_s3(→ None)

Traverse S3 bucket and download files with comprehensive error handling.

check_s3_access_from_token(→ bool)

Check if the access token contains S3-related roles.

download_via_odata_api(→ bool)

Download product using the OData API as fallback when S3 is not available.

pull_down(→ None)

Main function to orchestrate the download process.

main(→ None)

Main entry point for the downloader script.

Module Contents

phidown.downloader_boto3_deprecated.load_credentials(file_name: str = 'secret.yml') → Tuple[str, str][source]

Load username and password from a YAML file or create the file if missing.

If the file is not found, the user is prompted to input credentials, which are then saved.

Parameters:

file_name (str, optional) – Name of the YAML file. Defaults to ‘secret.yml’.

Returns:

A tuple containing (username, password).

Return type:

tuple[str, str]

Raises:

  • yaml.YAMLError – If the YAML file is invalid or cannot be written properly.

  • KeyError – If expected keys are missing in the YAML structure.

phidown.downloader_boto3_deprecated.check_s3_permissions(s3_client, bucket_name: str, test_prefix: str) → bool[source]

Check if we have the necessary S3 permissions before starting downloads.

Parameters:

  • s3_client – Boto3 S3 client object

  • bucket_name – Name of the S3 bucket to test

  • test_prefix – A prefix path to test access with

Returns:

True if permissions are sufficient, False otherwise

Return type:

bool

phidown.downloader_boto3_deprecated.download_s3_file_with_retry(s3_client, bucket_name: str, s3_key: str, local_path: str, max_retries: int = 3) → bool[source]

Download a single file from S3 with retry logic and proper error handling.

Parameters:

  • s3_client – Boto3 S3 client object

  • bucket_name – Name of the S3 bucket

  • s3_key – S3 object key to download

  • local_path – Local file path to save the download

  • max_retries – Maximum number of retry attempts

Returns:

True if download successful, False otherwise

Return type:

bool

phidown.downloader_boto3_deprecated.config[source]
phidown.downloader_boto3_deprecated.get_access_token(config: dict, username: str, password: str) → str[source]

Retrieve an access token from the authentication server.

This token is used for subsequent API calls. Save the token to a file on the machine for reuse.

Parameters:

  • config (dict) – Configuration dictionary containing auth server URL

  • username (str) – Username for authentication

  • password (str) – Password for authentication

Returns:

Access token for API authentication

Return type:

str

Raises:

SystemExit – If token retrieval fails

phidown.downloader_boto3_deprecated.get_eo_product_details(config: dict, headers: dict, eo_product_name: str) → Tuple[str, str][source]

Retrieve EO product details using the OData API to determine the S3 path.

Parameters:

  • config (dict) – Configuration dictionary containing API URLs

  • headers (dict) – HTTP headers for authentication

  • eo_product_name (str) – Name of the EO product to retrieve details for

Returns:

A tuple containing (product_id, s3_path)

Return type:

Tuple[str, str]

Raises:

  • requests.exceptions.HTTPError – If the API request fails

  • KeyError – If expected fields are missing from the response

  • ValueError – If no products are found

phidown.downloader_boto3_deprecated.parse_s3_path(s3_path: str) → Tuple[str, str][source]

Parse S3 path to extract bucket name and object key.

Parameters:

s3_path (str) – S3 path in format ‘/bucket/key’ or ‘bucket/key’

Returns:

A tuple containing (bucket_name, object_key)

Return type:

Tuple[str, str]

Raises:

ValueError – If the S3 path format is invalid

phidown.downloader_boto3_deprecated.get_temporary_s3_credentials(headers: dict) → dict[source]

Create temporary S3 credentials by calling the S3 keys manager API.

Parameters:

headers (dict) – HTTP headers containing authorization token

Returns:

Dictionary containing S3 credentials with keys ‘access_id’ and ‘secret’

Return type:

dict

Raises:

  • requests.exceptions.HTTPError – If the API request fails

  • SystemExit – If credential creation fails or max credentials reached

phidown.downloader_boto3_deprecated.create_s3_client_with_retry(config: dict, s3_credentials: dict, max_retries: int = 3) → boto3.resource[source]

Create S3 client with retry logic and validation.

Parameters:

  • config (dict) – Configuration dictionary containing S3 endpoint

  • s3_credentials (dict) – Dictionary containing S3 access credentials

  • max_retries (int) – Maximum number of retry attempts

Returns:

Configured S3 resource

Return type:

boto3.resource

Raises:

ClientError – If S3 resource creation fails after all retries

phidown.downloader_boto3_deprecated.test_s3_connectivity(s3_resource, bucket_name: str) → bool[source]

Test S3 connectivity and basic permissions.

Parameters:

  • s3_resource – Boto3 S3 resource object

  • bucket_name – Name of the S3 bucket to test

Returns:

True if connectivity is successful, False otherwise

Return type:

bool

phidown.downloader_boto3_deprecated.test_s3_connectivity_detailed(s3_resource, bucket_name: str) → bool[source]

Test S3 connectivity with detailed debugging information.

Parameters:

  • s3_resource – Boto3 S3 resource object

  • bucket_name (str) – Name of the S3 bucket to test

Returns:

True if connectivity is successful, False otherwise

Return type:

bool

phidown.downloader_boto3_deprecated.format_filename(filename, length=40)[source]

Format a filename to a fixed length, truncating if necessary.

phidown.downloader_boto3_deprecated.download_file_s3(s3, bucket_name: str, s3_key: str, local_path: str, failed_downloads: List[str]) → None[source]

Download a file from S3 with improved error handling.

Parameters:

  • s3 – S3 client object

  • bucket_name – Name of the S3 bucket

  • s3_key – S3 object key to download

  • local_path – Local file path to save the download

  • failed_downloads – List to track failed downloads

phidown.downloader_boto3_deprecated.traverse_and_download_s3(s3_resource, bucket_name: str, base_s3_path: str, local_path: str, failed_downloads: List[str]) → None[source]

Traverse S3 bucket and download files with comprehensive error handling.

Parameters:

  • s3_resource – Boto3 S3 resource object

  • bucket_name – Name of the S3 bucket

  • base_s3_path – Base path in S3 bucket to traverse

  • local_path – Local folder to download files to

  • failed_downloads – List to track failed downloads

Raises:

  • ClientError – When S3 access is forbidden or other AWS errors occur

  • NoCredentialsError – When AWS credentials are not configured

  • AssertionError – When input parameters are invalid

phidown.downloader_boto3_deprecated.check_s3_access_from_token(access_token: str) → bool[source]

Check if the access token contains S3-related roles.

Parameters:

access_token (str) – JWT access token from Copernicus

Returns:

True if S3 access is likely available, False otherwise

Return type:

bool

phidown.downloader_boto3_deprecated.download_via_odata_api(config: dict, headers: dict, product_name: str, local_folder: str) → bool[source]

Download product using the OData API as fallback when S3 is not available.

Parameters:

  • config (dict) – Configuration dictionary

  • headers (dict) – HTTP headers for authentication

  • product_name (str) – Name of the product to download

  • local_folder (str) – Local folder to save the download

Returns:

True if download successful, False otherwise

Return type:

bool

phidown.downloader_boto3_deprecated.pull_down(product_name: str | None = None, args: argparse.Namespace | None = None) → None[source]

Main function to orchestrate the download process.

Parameters:

  • product_name (Optional[str]) – Name of the Earth Observation product to download

  • args (Optional[argparse.Namespace]) – Command line arguments namespace

Raises:

  • ValueError – When product_name is not provided and args is None

  • RuntimeError – When all download methods fail

phidown.downloader_boto3_deprecated.main() → None[source]

Main entry point for the downloader script.