`soxspipe.commonutils.reducer`¶

reduce all the data in a workspace, or target specific obs and files for reduction

Author: David Young
Date Created: January 17, 2024

Module Contents¶

Classes¶

reducer

*reduce all the data in a workspace, or target specific obs and files for reduction*

**Key Arguments:**

- ``log`` -- logger
- ``workspaceDirectory`` -- path to the root of the workspace
- ``reductionTarget`` -- target for reduction: "all", "sof", "ob" (default: "all")
- ``settings`` -- the settings dictionary
- ``pathToSettings`` -- path to the settings file.
- ``quitOnFail`` -- quit the pipeline on any recipe failure
- ``overwrite`` -- overwrite existing reductions. Default *False*.
- ``daemon`` -- run in daemon mode (no terminal output). Default *False*.
- ``verbose`` -- print verbose output to terminal. Default *False*.
- ``refreshWorkspace`` -- refresh the workspace before reducing to collect new files. Default *False*.

Functions¶

`run_recipe`	execute a pipeline recipe
`run_recipe_bulk`	execute a pipeline recipe in multiprocessing mode

API¶

class soxspipe.commonutils.reducer.reducer(log, workspaceDirectory, reductionTarget='all', settings=False, pathToSettings=False, quitOnFail=False, overwrite=False, daemon=False, verbose=False, refreshWorkspace=False)[source][source]¶

Bases: object

*reduce all the data in a workspace, or target specific obs and files for reduction*

**Key Arguments:**

- ``log`` -- logger
- ``workspaceDirectory`` -- path to the root of the workspace
- ``reductionTarget`` -- target for reduction: "all", "sof", "ob" (default: "all")
- ``settings`` -- the settings dictionary
- ``pathToSettings`` -- path to the settings file.
- ``quitOnFail`` -- quit the pipeline on any recipe failure
- ``overwrite`` -- overwrite existing reductions. Default *False*.
- ``daemon`` -- run in daemon mode (no terminal output). Default *False*.
- ``verbose`` -- print verbose output to terminal. Default *False*.
- ``refreshWorkspace`` -- refresh the workspace before reducing to collect new files. Default *False*.

**Usage:**

```python
from soxspipe.commonutils import reducer
collection = reducer(
    log=log,
    workspaceDirectory="/path/to/workspace/root/",
    reductionTarget="all",
    settings=settings,
    pathToSettings="/path/to/settings.yaml"
)
collection.reduce()
```

Initialization

reduce(batch=False, multiprocess=False)[source][source]¶: reduce the selected data

select_sof_files_to_process(recipe=False, reductionTarget=False, batch=False, arm=False)[source][source]¶

select all of the SOF files still requiring processing

Key Arguments: - recipe – the name of the recipe to filter by (optional) - reductionTarget – target for reduction: “all”, “sof”, “ob” (default: False) - batch – number of SOF files to return (default: False, all) - arm – filter by arm (default: False, all)

Return:

rawGroups – a dataframe of the containing a list of recipes and sof file paths

Usage:

rawGroups = reducer.select_sof_files_to_process()

soxspipe.commonutils.reducer.run_recipe(log, recipe, sof, settings, overwrite, command=False, verbose=False, turnOffMP=False)[source][source]¶

execute a pipeline recipe

Key Arguments:

recipe – the name of the recipe tp execute
sof – path to the sof file containing the files the recipe requires
command – the command used to run the recipe
settings – the settings dictionary
overwrite – overwrite existing reductions. Default False.
verbose – print verbose output to terminal. Default False.
turnOffMP – turn off multiprocessing mode. Default False.