Utils

Table of contents

utils.visualization_utils.equalize_tensor(raw_granule_tensor, n_std=2)

Equalizes a tensor for a better visualization by clipping outliers of a histogram higher and lower than pixels value mean *- n_std times the standarda deviation.

Parameters:

• raw_granule_tensor (torch.tensor) – tensor to equalize.

• n_std (int, optional) – Number of times the standard deviation. Defaults to 2.

Returns:

equalized tensor.

Return type:

torch.tensor

utils.visualization_utils.plot_event(img, img_name, bbox_list, alert_matrix=None, save_path=None)

Util function to visualize and compare the bands of two different images. It also allows adding an alert matrix.

Parameters:

• img (torch.tensor) – img.

• img_name (string) – image_name.

• bbox_list (skimage properties) – bbox list.

• alert_matrix (torch.tensor, opional) – if not None, the hotmap of normal band is shown. Defaults to None.

• save_path (string, optional) – if not None, the image is saved at save_path. Defaults to None.

utils.visualization_utils.plot_img1_vs_img2_bands(img1_band, img2_band, img_name_list, alert_matrix=None, alert_matrix_unregistered=None, save_path=None)

Util function to visualize and compare the bands of two different images. It also allows adding an alert matrix.

Parameters:

• img1_band (torch.tensor) – first image band.

• img2_band (torch.tensor) – second image band.

• img_name_list (list) – list of names of different images.

• alert_matrix (torch.tensor, opional) – if not None, the hotmap of normal band is shown. Defaults to None.

• alert_matrix_unregistered (torch.tensor, opional) – if not None, the hotmap of unregstered band is shown. Defaults to None.

• save_path (string, optional) – if not None, the image is saved at save_path. Defaults to None.

utils.parse_csv_utils.parse_csv(csv_name)

Parse a CSV file and return a list of rows.

Csv_name:

CSV name.

Returns:

(list of rows. Each row is a dictionary.)

utils.date_utils.get_timestamp()

Return current timestamp.

Returns:

timestamp

Return type:

str

utils.date_utils.parse_string_date(date_string)

Parse a date in string format and returns a datetime.

Parameters:

date_string (string) – date string

Returns:

correspondent datetime.

Return type:

datetime

utils.l1_utils.export_band_to_tif(band, crs, coords, save_path)

Export band to TIF.

Parameters:

• band (torch.tensor) – band to save.

• crs (string) – epcs.

• coords (list) – list of bands coordinates [UL, BL, BR, UR]. Each point is (LON, LAT).

• save_path (_type_) – _description_

utils.l1_utils.get_event_bounding_box(event_hotmap, coords_dict)

Returns the bounding box and the coordinates of the bounding box top-left, bottom-right corners for each clust of 1 in the event_hotmap.

Parameters:

• event_hotmap (torch.tensor) – event hotmap. Pixels = 1 indicate event.

• coords_dict (dict) – {“lat” : lat, lon : “lon”}, containing coordinates for each pixel in the hotmap.

Returns:

bounding box list: list of coordinates [lat, lon] of top-left,

bottom-right coordinates for each cluster of events in the hotmap.

Return type:

skimage.bb

utils.l1_utils.get_l1C_image_default_path(id_event, cfg_file_dict=None, id_raw_l1_dict=None, database='THRAWS', device=device(type='cpu'))

Returns th default path to the L1C cropped image including the “”id_event_name”” tif files from database.

Parameters:

• id_event (str) – event ID.

• cfg_file_dict (dict, optional) – dictionary containing paths to the different pyraws directories. If None, internal CSV database will be parsed.

• id_raw_l1_dict (dict, optional) – id-raw-l1 dictionary. If None, internal CSV database will be parsed.

• database (string, optional) – database name. Defaults to “THRAWS”.

Returns:

default path to the L1C cropped image tif files.

Return type:

string

utils.l1_utils.get_reprojected_bounds(rasterio_file)

Returns the 4 corners coordinates by reprojecting files.

Parameters:

rasterio_file (rasterio) – rasterio file container.

Returns:

list of coordinates. dict: key params

Return type:

list

utils.l1_utils.read_L1C_event_from_database(id_event, bands_list, cfg_file_dict=None, id_raw_l1_dict=None, database='THRAWS', reproject_bounds=True, verbose=True, device=device(type='cpu'))

Read specific bands of an L1C Sentinel 2 event, specified in “bands_list” from img_name. Every tile of the event is represented as TOA reflectance.

Parameters:

• id_event (str) – event ID.

• bands_list (list) – bands list.

• cfg_file_dict (dict, optional) – dictionary containing paths to the different pyraws directories. If None, internal CSV database will be parsed.

• id_raw_l1_dict (dict, optional) – id-raw-l1 dictionary. If None, internal CSV database will be parsed.

• database (string, optional) – database name. Defaults to “THRAWS”.

• reproject_bounds (bool, optional) – if True, bounds are reprojected to EPGS:4326.

• verbose (bool, optional) – if True, if True, verbose mode is used. Defaults to True.

• device (torch.device, optional) – torch device. Defaults to torch.device(“cpu”).

Raises:

ValueError – impossible to find information on the database.

Returns:

dictionary containing [tile_name, tile].

Each tile is list of tensors, each of them is made of TOA values of the requested Sentinel 2A image bands.

dictionary: dictionary containing [tile_name, coordinates], where coordinates are the corners coordinates of each tile

composing the requested image.

dictionary: dictionary containing [tile_name, coordinates], where coordinates are the footprint’s coordinates

of each tile composing the requested image.

dictionary: dictionary containing [tile_name, band_filename_dict], where band_filename_dict is the dictionary

associating to every band the original jp2 file.

string: expected class name.

Return type:

dictionary

utils.l1_utils.read_L1C_event_from_path(l1c_image_path, bands_list, reproject_bounds=True, verbose=True, device=device(type='cpu'))

Read specific bands of an L1C Sentinel 2 event, specified in “bands_list” from img_name. Every tile of the event is represented as TOA reflectance.

Parameters:

• id_event (str) – event ID.

• bands_list (list) – bands list.

• reproject_bounds (bool, optional) – if True, bounds are reprojected to EPGS:4326.

• verbose (bool, optional) – if True, if True, verbose mode is used. Defaults to True.

• device (torch.device, optional) – torch device. Defaults to torch.device(“cpu”).

Raises:

ValueError – impossible to find information on the database.

Returns:

dictionary containing [tile_name, tile]. Each tile is list of tensors,

each of them is made of TOA values of the requested Sentinel 2A image bands.

dictionary: dictionary containing [tile_name, coordinates], where coordinates are the corners

coordinates of each tile composing the requested image.

dictionary: dictionary containing [tile_name, coordinates], where coordinates are the footprint’s

coordinates of each tile composing the requested image.

dictionary: dictionary containing [tile_name, band_filename_dict], where band_filename_dict is the

dictionary associating to every band the original jp2 file.

Return type:

dictionary

utils.l1_utils.read_L1C_image_from_tif(id_event, out_name_ending=None, cfg_file_dict=None, id_raw_l1_dict=None, database='THRAWS', device=device(type='cpu'))

Read an L1C Sentine2 image from a cropped TIF. The image is represented as TOA reflectance. The image is post processed.

Parameters:

• id_event (str) – event ID.

• out_name_ending (src, optional) – optional ending for the output name. Defaults to None.

• cfg_file_dict (dict, optional) – dictionary containing paths to the different pyraws directories. If None, internal CSV database will be parsed.

• id_raw_l1_dict (dict, optional) – id-raw-l1 dictionary. If None, internal CSV database will be parsed.

• database (string, optional) – database name. Defaults to “THRAWS”.

• device (torch.device, optional) – torch device. Defaults to torch.device(“cpu”).

Raises:

ValueError – impossible to find information on the database.

Returns:

dictionary containing every tile composing the requested image.

Each tensor is made of TOA values of the requested Sentinel 2A image bands.

dictionary: dictionary containing lat and lon for every point. string: expected class name.

Return type:

dictionary

utils.l1_utils.read_L1C_tile_from_path(tile_path, bands_list, reproject_bounds=True, verbose=True, device=device(type='cpu'))

Read specific bands of an L1C Sentine2 tile, specified in “bands_list”. The tile, located at “tile_path”, is divided by a factor specified in the auxiliary file “auxiliary_file_path” to transform it from DN to TOA reflectance.

Parameters:

• tile_path (str) – Sentinel 2 image path.

• bands_list (list) – bands list.

• reproject_bounds (bool, optional) – if True, bounds are reprojected to EPGS:4326.

• verbose (bool, optional) – if True, if True, verbose mode is used. Defaults to True.

• device (torch.device, optional) – torch device. Defaults to torch.device(“cpu”).

Raises:

• ValueError – Impossible to open auxiliary file.

• ValueError – Impossible to open the images with the requested bands.

Returns:

list of the requested Sentinel 2A image bands. Each band is a torch.tensor containing TOA values. list: list of tile’s corners’ coordinates extracted from metadata. list: list of tile’s footprint’s coordinates extracted from metadata. dict: bands_filename_dict

Return type:

list

utils.l1_utils.reproject_raster(rasterio_file, output_filename)

Returns the reprojected raster.

Parameters:

• rasterio_file (rasterio) – rasterio file container.

• output_filename (str) – output_filename of the reprojected raster.

Returns:

reprojected raster

Return type:

raster

utils.l1_utils.swap_latlon(poly)

Function that swaps latitude and logitude values :param poly: list of points coordinates :type poly: list

Returns:

list of points coordinates swapped.

Return type:

poly (list)

utils.shape_utils.create_polygon_from_coordinates(lat, lon, center_distance_x=5000, center_distance_y=14000, json_name=None)

Creates a rectangular polygon from coordinates (lat, lon). If json_name is not None, the polygon is saved as JSON file.

Parameters:

• lat (float) – center latitude.

• lon (float) – center longitude.

• center_distance_x (int, optional) – Horizontal distance from the center in m. Defaults to 5000.

• center_distance_y (int, optional) – Vertical distance from the center in m. Defaults to 14000.

• json_name (str, optional) – output JSON file name. If None, no file is created. Defaults to None.

Returns:

polygon dict {“SW” : (lat, lon), “NW” : (lat, lon), “NE” : (lat, lon), “SE” : (lat, lon)}

Return type:

dict

utils.shape_utils.create_rectangular_polygon(center_coordinates, center_distance_x, center_distance_y)

Creates a rectangular polygon from coordinates (lat, lon). :param center_coordinates: center coordinates {“lat” : lat, “lon” : lon} :type center_coordinates: dict :param center_distance_x: Horizontal distance from the center in m. Defaults to 5000. :type center_distance_x: int, optional :param center_distance_y: Vertical distance from the center in m. Defaults to 14000. :type center_distance_y: int, optional

Returns:

polygon dict {“SW” : (lat, lon), “NW” : (lat, lon), “NE” : (lat, lon), “SE” : (lat, lon)}

Return type:

dict

utils.shape_utils.get_granules_polys(requested_bands, event_name, device)

Returns the polygons over bands coordinates and correspondent indices.

Parameters:

• requested_bands (list) – requested bands.

• event_name (st) – event ID.

• device (torch.device) – device.

Returns:

{granule_name : polygon} dict: {granule_name (e.g., granule_0_STACKED_T_granule_1) : index of granules stacked along track}

Return type:

dict

utils.raw_utils.export_band_to_tif(band, coords, save_path)

Export band to TIF.

Parameters:

• band (torch.tensor) – band to save.

• coords (list) – list of bands coordinates [UL, BL, BR, UR]. Each point is (LON, LAT).

• save_path (str) – save path.

utils.raw_utils.find_granules_names(granules_paths)

Extract name of granules from granules path.

Parameters:

granules_paths (list) – list of paths to the granules.

Returns:

list of granule names.

Return type:

list

utils.raw_utils.get_bands_shift(bands_list, satellite, detector_number, downsampling=True, cfg_file_dict=None)

It returns the number of backward shift pixels of the various bands with respect to the first band in the list. Negative shift means the bands shall translate forward.

Parameters:

• bands_list (list) – list of bands

• satellite (str, optional) – “S2A” or “S2B” respectively for “Sentinel-2A” data and “Sentinel-2B” data.

• detector_number (int) – Detectorn number.

• downsampling (boolean, optional) – if True, shift values for downsampled bands of the chosen satellite are used. Otherwise, values for upsampled bands are used. Defaults to True.

• cfg_file_dict (dict, optional) – dictionary containing paths to the different pyraws directories. If None, internal CSV database will be parsed. Defaults to None.

Returns:

list of relative pixel shift compared to the first band.

Return type:

list

utils.raw_utils.get_granule_px_length(n_stacked_granules, satellite, detector_number, cropped_pixels_along=None)

Returns the length of a granule in px :param n_stacked_granules: number of stacked granules. :type n_stacked_granules: int :param satellite: “S2A” or “S2B” respectively for “Sentinel-2A” data and “Sentinel-2B” data. :type satellite: str, optional :param detector_number: Detectorn number. :type detector_number: int :param cropped_pixels_along: Number of cropped pixels along the along-track direction :type cropped_pixels_along: int

Returns:

Length of a granule in px.

Return type:

int

utils.raw_utils.read_Raw_event_from_database(id_event, bands_list, cfg_file_dict=None, id_raw_l1_dict=None, database='THRAWS', verbose=True, device=device(type='cpu'))

Read specific bands of the Raw Sentinel-2 event “”id_event””, specified in “bands_list”.

Parameters:

• id_event (str) – event ID.

• bands_list (list) – bands list.

• cfg_file_dict (dict, optional) – dictionary containing paths to the different pyraws directories. If None, internal CSV database will be parsed.

• id_raw_l1_dict (dict, optional) – id-raw-l1 dictionary. If None, internal CSV database will be parsed.

• database (string, optional) – database name. Defaults to “THRAWS”.

• verbose (bool, optional) – if True, if True, verbose mode is used. Defaults to True.

• device (torch.device, optional) – torch device. Defaults to torch.device(“cpu”).

Raises:

ValueError – impossible to find information on the database.

Returns:

tensor containing TOA values of the requested Sentinel 2A image bands. string: expected class name. list: list of granules names. list: list of raw data useful granules. list: list of complementary raw data granules (to coregister without 0). list: list of polygon coordinates for each granule. list: list of cloud coverage percentage for each granule. dict: {useful granule : bbox}.

Return type:

torch.tensor

utils.raw_utils.read_Raw_event_from_path(dir_path, bands_list, verbose=True, device=device(type='cpu'))

Read specific bands of an Raw Sentine2 event, specified in “bands_list”. The image contains several granules at “dir_path”.

Parameters:

• dir_path (str) – Sentinel 2 Raw image path.

• bands_list (list) – bands list.

• verbose (bool, optional) – if True, if True, verbose mode is used. Defaults to True.

• device (torch.device, optional) – torch device. Defaults to torch.device(“cpu”).

Raises:

ValueError – Impossible to open the events with the requested bands.

Returns:

list of G granules, each of them is a list of the requested Sentinel 2A bands. list: list of granules paths. list: list of granules polygon coordinates. list: list of cloud coverage percentages.

Return type:

list

utils.raw_utils.read_Raw_granule(granule_path, bands_list, verbose=True, device=device(type='cpu'))

Read specific bands of an Raw Sentine2 granule, specified in “bands_list”. The image contains several granules at “dir_path”.

Parameters:

• granule_path (str) – Sentinel 2 Raw granule path.

• bands_list (list) – bands list.

• verbose (bool, optional) – if True, if True, verbose mode is used. Defaults to True.

• device (torch.device, optional) – torch device. Defaults to torch.device(“cpu”).

Raises:

ValueError – Impossible to open the images with the requested bands.

Returns:

it includes G granules, each of them is a list including torch.tensor for each Sentinel 2A image band. list: metadata including original polygon coordinates. list: metadata including polygon cloud cover percentage.

Return type:

list

utils.raw_utils.shift_point_coordinates(point_coordinates, point_distance_y)

Vertical shift of point coordinates.

Parameters:

• point_coordinates (list) – [lan, lon] - point coordinates.

• point_distance_y (int) – vertical shift. Use negative for shift towards north.

Returns:

coordinates of the shifted point.

Return type:

list

utils.raw_utils.swap_latlon(poly)

Function that swaps latitude and logitude values :param poly: list of points coordinates :type poly: list

Returns:

list of points coordinates swapped.

Return type:

poly (list)

utils.band_shape_utils.image_band_resize(img_upsample_band, band_name, upsampled_img_spatial_resolution, interpolate_mode)

Resize an upsampled image band to a the orginal spatial resolution through an interpolate mode.

Parameters:

• img_upsample_band (torch.tensor) – upsampled image to resize

• band_name (string) – band name

• upsampled_img_spatial_resolution (float) – spatial resolution of the input upsampled image.

• interpolate_mode (string) – interpolated mode.

Raises:

• ValueError – unsupported band name.

• ValueError – unsupported interpolated mode.

Returns:

resized image.

Return type:

torch.tensor

utils.band_shape_utils.image_band_upsample(img_band, band_name, target_spatial_resolution, upsample_mode)

Upsample an image band to a target spatial resolution through an upsample mode.

Parameters:

• img_band (torch.tensor) – image band.

• band_name (string) – band name.

• target_spatial_resolution (float) – target resolution (m).

• upsample_mode (string) – band name.

Raises:

• ValueError – unsupported band name.

• ValueError – unsupported upsample mode.

Returns:

upsampled band.

Return type:

torch.tensor

class utils.database_utils.DatabaseHandler(db_name, datapath='/Users/robertodelprete/Desktop/PyRawS/data')

Creates an Raw-L1C csv database for the APIs.

fetcher()

Fetches the products in the the database.

parser()

Parses all the Raw products and creates the database.

single_parser(raw_Event_folderpath)

Parses a single Raw product. :param raw_Event_folderpath: datapath of the Raw event. :type raw_Event_folderpath: str

utils.database_utils.get_cfg_file_dict()

Returns a dictionary containing paths to the different pyraws directories. :returns: cfg file dict. :rtype: dict

utils.database_utils.get_event_granule_bb_dict(event_id, database='THRAWS', cfg_file_dict=None)

Function to extract the dictionary {useful_granule : bounding_box_list} from the bbox_str got from the database.

Parameters:

• event_id (string) – event ID.

• database (string, optional) – database name. Defaults to “”THRAWS””.

• cfg_file_dict (dict, optional) – dictionary containing paths to the different pyraws directories. If None, internal CSV database will be parsed.

Returns:

{useful_granule : bounding_box_list} for the requested event.

Return type:

dict

utils.database_utils.get_event_info(event_id, cfg_file_dict=None, id_raw_l1_dict=None, database='THRAWS')

From the event_ID, it returns raw directory path, the path to the l1 image, to the L1C auxiliary file, and the class of the image. If no `id_raw_l1_dict`is provided, it is done by parsing the internal CSV database file.

Parameters:

• event_id (string) – image name.

• cfg_file_dict (dict, optional) – dictionary containing paths to the different pyraws directories. If None, internal CSV database will be parsed.

• id_raw_l1_dict (dict, optional) – id-raw-l1 dictionary. If None, internal CSV database will be parsed.

• database (string, optional) – database name. Defaults to “THRAWS”.

Raises:

ValueError – If the image is not in the database.

Returns:

path to the Raw image. string: path to the L1 image. string: path to the post-processed L1C tiff file. string: expected class. list: list of raw useful tiles. list: list of complementary raw tiles (to coregister without 0). dict: event coordinates dict {“lat” : lat, “lon” : lon}. list: list of the requested polygon coordinates.

Return type:

string

utils.database_utils.get_events_list(database='THRAWS', cfg_file_dict=None)

Returns the list of events in the database.

Parameters:

• database (string, optional) – database name. Defaults to “”THRAWS””.

• cfg_file_dict (dict, optional) – dictionary containing paths to the different pyraws directories. If None, internal CSV database will be parsed.

Returns:

list of events.

Return type:

str

utils.database_utils.get_id_raw_l1_dict(database='THRAWS', cfg_file_dict=None)

Returns a dictionary containing raw (directory name), l1c (product ID and correspondent granule) and class, raw useful and complementary tiles. :param database: database name. Defaults to “”THRAWS””. :type database: string, optional :param cfg_file_dict: dictionary containing paths to the different pyraws directories.

If None, internal CSV database will be parsed.

Returns:

label_raw-l1_dict

Return type:

dict

utils.database_utils.get_raw_shift_lut(satellite, detector_number, downsampling=True, cfg_file_dict=None)

Get Raw shift LUT.

Parameters:

• satellite (str, optional) – “S2A” or “S2B” respectively for “Sentinel-2A” data and “Sentinel-2B” data.

• detector_number (int) – Detectorn number.

• downsampling (boolean, optional) – if True, shift values for downsampled bands of the chosen satellite are used. Otherwise, values for upsampled bands are used. Defaults to True.

• cfg_file_dict (dict, optional) – cfg_file_dict (dict, optional): dictionary containing paths to the different pyraws directories. If None, internal CSV database will be parsed. Defaults to None.

Returns:

returns the Raw shift LUT.

Return type:

dict