azul retrieve

Overview

azul retrieve finds and downloads Euclid MER data which contain specified targets, such as coordinates or named astronomical objects. A radius can be specified to download only a region of the tiles. The retrieved files are organized in directories we call workdirs (see Workspace, input and output paths).

As depicted below, azul retrieve consists of two main steps:

  1. Querying the indices of the tiles which contain input targets;

  2. Downloading the data files in full or in part.

card target { } object Query { --from --dsr -n } object Download { -q -r -f } folder workdir { } target --> Query Query -> Download Download --> workdir

Inputs

The command takes as input the list of targets, which may be of different types:

Tiles

Integers are parsed as tile indices.

Coordinates

If the target contains a comma, it is considered as RA/Dec coordinates in degrees, using ICRS reference frame.

Named objects

Other types of input are parsed as object names. Their coordinates are looked up in the CDS name resolver.

For example, the following command would retrieve the tiles covering targets of each type:

$ azul retrieve NGC6505 270.93,67.05 102157949

If the target has to contain spaces or special characters, the argument must be given between quotes. In general, spaces can be omitted, though, such that the two following lines are equivalent:

$ azul retrieve "NGC 6505"
$ azul retrieve NGC6505

Each target can be suffixed with a cone angle for cutout retrieval (more details in section Download). The syntax is <target>[r=<radius>], e.g.:

$ azul retrieve NGC6505[r=1m] 270.93,67.05[r=10s] 102157949

Depending on your system, quotes may be required.

Outputs

The files are downloaded into a dedicated workdir for each target (the current directory by default). It can be set with global option --workspace:

$ azul --workspace /tmp retrieve NGC6505 270.93,67.05 102157949 PGC61356

The way the workspace is structured depends on the output template parameter -o. The template is rendered as follows:

Placeholder name

Substitution value

{workspace}

Workspace path

{tile}

Resolved target tile index

{target}

Verbatim target argument in command line, or nothing if the target is a tile index

By default, the workdirs are named after the targets, grouped by tile index. At the time of writing, the above command creates the following workdirs inside workspace /tmp:

@startfiles /tmp/101832848/NGC6505/ /tmp/101832849/NGC6505/ /tmp/101836362/270.93,67.05/ /tmp/101836362/PGC61356/ /tmp/102157949/ /tmp/102158889/NGC6505/ /tmp/102159776/270.93,67.05/ /tmp/102159776/PGC61356/ @endfiles

Several tiles (101832848 and 101832849) cover a single target (NGC6505), such that several workdirs are created for each of them. Conversely, several targets (270.93,67.05 and PGC61356) belong a same tile (101836362), in which case workdirs have a common parent tile folder.

Query

The querying phase consists in finding the tile indices of the input coordinates and resolved objects, as well as the names of the files to be downloaded.

This step relies on what we call a data provider, passed to option --from (or through environment variable AZULRETRIEVE_FROM). and applies to all of the targets of the command line:

$ azul retrieve NGC6505 270.93,67.05 102157949 --from pdr

There are several such providers, which store different sets of data:

pdr (Public Data Releases)

Contains all public data. This is the only provider which does not require authentication.

idr (Internal Data Releases)

Contains data under embargo which will later be released publicly.

otf (on-the-fly)

Contains unreleased data from the Euclid Science Archive. It is updated from time to time between data releases.

dss (Distributed Storage System)

Contains everything but is much slower than other providers.

Tip

In order to bypass the slow spatial queries of provider dss, some MER tiling file can be downloaded and its path given to option --tiling. In this case, you are responsible for providing the relevant and up-to-date tiling file.

Option --dsr is used to restrict the returned tiles according to the value of their DataSetRelease attribute. By default, dataset releases for Q1 and DR1 releases are enabled.

Option -n limits the number of tiles returned by target. This is especially handy when one wants to get exactly one image per target (-n 1).

Download

When retrieving coordinates or named objects, a cone radius can be passed as option -r, which triggers a cutout service and downloads only a square region around the target. All formats handled by Astropy’s Angle are accepted.

The radius applies to all coordinates and named objects without a specific radius given, but not to the tile targets. Therefore, the following line:

$ azul retrieve NGC6505 270.93,67.05 PGC61356[r=20s] 102157949 -r 1m

downloads 2’ x 2’ regions around NGC6505 and (270.93, 67.05), a 40” x 40” region around PGC61356, as well as the whole 102157949 tile.

Warning

Provider dss does not natively support the cutout service.

A full tile will be downloaded and cut locally. Ensure that the workdirs and tiledirs as specified with option -o are distinct (this is the case with the default value).

Flag -q stands for “query only” and interrupts the command before effectively downloading data, such that the file names you look for will be returned but no file download will happen. With optional value -q tiles, the command stops even earlier to return only the tile indices (filenames are not looked for).

By default, if a file to be downloaded already exists in the specified workdir, it is skipped. This behavior can be changed by forcing downloads with flag -f. In this case, files which already exist will be overwritten. This option is particularly useful to force the download of targets with same file names, e.g. same target with different cutout radii.

More on data providers

What we call a data provider is in fact a couple of Euclid components: a database and a data store.

Querying relies on a database for finding the tile indices and MER file names. There are two such databases provided by the Euclid Archive System (EAS):

  • The database of the Euclid Science Archive, formerly known as Science Archive Service (EAS-SAS, which we still name SAS in Azulero because ESA’s EAS-ESA is funny while we are not);

  • The Data Processing System (EAS-DPS or DPS in short).

The SAS offers the spatial query and cutout service but does not know of all of the internal data, while the DPS does reference everything but does not natively offer the spatial query (at least, not in a reasonable amount of time) or the cutout service.

Once the file names have been resolved, the files are downloaded from a data store. The DSS is the storage associated to the DPS. The other providers are bound to the SAS storage. Therefore, selecting provider DSS means accessing metadata from the DPS and data from the DSS. Only the latter is specified to azul retrieve, and the former is deduced (same goes for the other data providers with the SAS).

ESA Datalabs

Azulero works in ESA Datalabs. In this environment, “retrieving” SAS data is a matter of finding them in the filesystem. Option --data labs triggers:

  • Tile retrieval as symlinks: no copy is performed but the resulting workspace has the same structure as in other environments. Instead of getting proper FITS files in the tiledir, links to the Datalabs data store are created.

  • Direct cutout retrieval: the cutouts are extracted directly from the FITS files in the Datalabs data store.

For convenience, as mentioned in Quick start, it is recommended to export AZULRETRIEVE_DATA=labs in the Datalabs (according to Named options).