phidown.s5cmd_utils

Attributes

logger

Functions

run_s5cmd_with_config(→ str)

Run s5cmd command with configuration file.

get_directory_size(→ int)

Calculate total size of all files in a directory recursively.

pull_down(→ bool)

Download Sentinel-1 SAFE directory from Copernicus Data Space.

Module Contents

phidown.s5cmd_utils.logger[source]
phidown.s5cmd_utils.run_s5cmd_with_config(command: str, config_file: str = '.s5cfg', endpoint_url: str | None = None, verbose: bool = False) → str[source]

Run s5cmd command with configuration file.

This function executes s5cmd commands using credentials from a configuration file and handles endpoint URL configuration for the Copernicus Data Space.

Parameters:

  • command – The s5cmd command to execute (without ‘s5cmd’ prefix)

  • config_file – Path to s5cmd configuration file (default: ‘.s5cfg’)

  • endpoint_url – Optional endpoint URL override

  • verbose – Whether to print command being executed

Returns:

Command output as string

Return type:

str

Raises:

Example

>>> output = run_s5cmd_with_config('ls s3://eodata/Sentinel-1/')
phidown.s5cmd_utils.get_directory_size(directory: str) → int[source]

Calculate total size of all files in a directory recursively.

Parameters:

directory – Path to the directory

Returns:

Total size in bytes

Return type:

int

phidown.s5cmd_utils.pull_down(s3_path: str, output_dir: str = '.', config_file: str = '.s5cfg', endpoint_url: str = 'https://eodata.dataspace.copernicus.eu', download_all: bool = True, reset: bool = False, total_size: int | None = None, show_progress: bool = True) → bool[source]

Download Sentinel-1 SAFE directory from Copernicus Data Space.

This function downloads either individual files or entire SAFE directories from the Copernicus Data Space Ecosystem using the optimized s5cmd tool. Optionally displays a progress bar based on downloaded file size.

Parameters:

  • s3_path – S3 path to the Sentinel-1 data (should start with /eodata/)

  • output_dir – Local output directory for downloaded files

  • config_file – Path to s5cmd configuration file

  • endpoint_url – Copernicus Data Space endpoint URL

  • download_all – If True, downloads entire directory with wildcard pattern

  • reset – If True, prompts for new AWS credentials and resets config file

  • total_size – Expected total size in bytes (for progress bar)

  • show_progress – If True and total_size provided, shows tqdm progress bar

Returns:

True if download was successful

Return type:

bool

Raises:

Example

>>> # Download entire SAFE directory with progress bar
>>> output = pull_down(
...     '/eodata/Sentinel-1/SAR/IW_RAW__0S/2024/05/03/'
...     'S1A_IW_RAW__0SDV_20240503T031926_20240503T031942_053701_0685FB_E003.SAFE',
...     output_dir='/path/to/data',
...     total_size=1073741824,  # 1 GB
...     show_progress=True
... )

Notes

  • s5cmd is executed as a subprocess and its stdout/stderr are streamed in real time to the logger. This means when pull_down runs you will see file copy progress and status messages as they happen.

  • Ensure s5cmd is installed and available on PATH. There is no additional environment variable required to enable streaming; it’s handled by this function.

  • Progress bar monitors actual disk usage and updates in real-time.