Reducer

The reducer utility is designed to interact with the data organiser to reduce all data within a workspace, from the most basic calibration frames (bias, darks) through the data-reduction cascade to a final wavelength- and flux-calibrated object spectrum. Using the reducer utility and its command-line counterpart (soxspipe reduce) allows typical users to run the pipeline at a high level without worrying about individual recipe commands. Of course, those individual recipe commands are still available for ‘power-users’ to run with their own customised settings.

The algorithm used by the reducer util is shown in Fig. 34.

Fig. 34 The algorithm used by the reducer utility to process data through the entire data-reduction cascade.

The reducer utility first selects from the soxspipe.db database, all SOF files containing data still needing to be reduced and caches all recipe commands required to execute the recipes in order. It then executes the relevant recipes (in order) to generate the new data products. If, at any stage, a recipe fails to create a data product, the SOF files get rewritten to remove the failed product and insert the next-best product. The reducer utility stops when all the SOF files have been processed.

The pipeline can utilise multiprocessing to run recipes in parallel (by passing a --multiprocess flag. It is also possible to target specific science sof files so that the pipeline will only reduce the calibrations needed to process that specific science object.

Utility API

class reducer(log, workspaceDirectory, reductionTarget='all', settings=False, pathToSettings=False, quitOnFail=False, overwrite=False, daemon=False, verbose=False, refreshWorkspace=False)

Bases: object

*reduce all the data in a workspace, or target specific obs and files for reduction*

**Key Arguments:**

- ``log`` -- logger
- ``workspaceDirectory`` -- path to the root of the workspace
- ``reductionTarget`` -- target for reduction: "all", "sof", "ob" (default: "all")
- ``settings`` -- the settings dictionary
- ``pathToSettings`` -- path to the settings file.
- ``quitOnFail`` -- quit the pipeline on any recipe failure
- ``overwrite`` -- overwrite existing reductions. Default *False*.
- ``daemon`` -- run in daemon mode (no terminal output). Default *False*.
- ``verbose`` -- print verbose output to terminal. Default *False*.
- ``refreshWorkspace`` -- refresh the workspace before reducing to collect new files. Default *False*.

`

**Usage:**

```python
from soxspipe.commonutils import reducer
collection = reducer(
    log=log,
    workspaceDirectory="/path/to/workspace/root/",
    reductionTarget="all",
    settings=settings,
    pathToSettings="/path/to/settings.yaml"
)
collection.reduce()
```

Initialization

reduce(batch=False, multiprocess=False)

reduce the selected data

select_sof_files_to_process(recipe=False, reductionTarget=False, batch=False, arm=False)

select all of the SOF files still requiring processing

Key Arguments: - recipe – the name of the recipe to filter by (optional) - reductionTarget – target for reduction: “all”, “sof”, “ob” (default: False) - batch – number of SOF files to return (default: False, all) - arm – filter by arm (default: False, all)

Return:

• rawGroups – a dataframe of the containing a list of recipes and sof file paths

Usage:

rawGroups = reducer.select_sof_files_to_process()