soxspipe.commonutils.data_organiser

The SOXSPIPE Data Organiser

Author

David Young

Date Created

March 9, 2023

Module Contents

Classes

data_organiser

The soxspipe Data Organiser

API

class soxspipe.commonutils.data_organiser.data_organiser(log, rootDir, vlt=False, dbConnect=True)

Bases: object

The soxspipe Data Organiser

Key Arguments:

• log – logger

• rootDir – the root directory of the data to process

• vlt – prepare the workspace using the standard vlt /data directory

Usage:

To setup your logger, settings and database connections, please use the fundamentals package (see tutorial here https://fundamentals.readthedocs.io/en/master/initialisation.html).

To initiate a data_organiser object, use the following:

from soxspipe.commonutils import data_organiser
do = data_organiser(
    log=log,
    rootDir="/path/to/workspace/root/"
)
do.prepare()

Initialization

build_sof_files()

scan the raw frame table to generate the listing of products that are expected to be created and then write out all of the needed SOF files

Usage:

self.build_sof_files()

close()

close the database connection

Usage:

do.close()

get_incomplete_raw_frames_set()

get_raw_frames_and_groups(ttype=None, arm=None, tech=None, recipe=None, recipeOrder=None, filterName=None, unprocessedOnly=False)

Process raw frames to group and calculate mean MJD values.

Key Arguments: - ttype – optional data product eso dpr type to filter by - arm – optional instrument eso seq arm to filter by - tech – optional list of eso dpr tech to filter by - recipe – recipe name to assign to groups - recipeOrder – recipe reduction order - filterName – optional name to filter the groups - unprocessedOnly – if True, only return unprocessed raw frames

Return: - rawFrames – processed raw frames dataframe - rawGroups – grouped raw frames with calculated MJD values

Usage:

rawFrames, rawGroups = self.get_raw_frames_and_groups()

list_obs()

list all observation names and IDs in the current workspace

list_raw(sofFile)

list the all the raw frames associated with a given science object SOF file

list_sofs()

list all science object SOF files in the current workspace

predict_product_frames(productTypes, rawGroups, recipe)

Process product frames for a given set of product types and raw groups.

Key Arguments:

• productTypes – List of product types to process.

• rawGroups – DataFrame containing raw groups.

• recipe – Recipe name.

Return:

• incompleteProducts – Number of incomplete products.

prepare(refresh=False, report=True)

Prepare the workspace for data reduction by generating all SOF files and reduction scripts.

Key Arguments:

• refresh – trigger a complete refresh the workspace during preparation (delete database and do a complete prepare)

raw_frames_to_sof_map(rawGroups, containerSofs)

Generate the SOF map from raw groups and complete product SOFs.

Key Arguments:

• rawGroups – DataFrame containing raw frame groups.

• containerSofs – array of complete product SOFs.

Return:

• sofMapDF – DataFrame containing the generated SOF map.

session_create(sessionId=False)

create a data-reduction session with accompanying settings file and required directories

Key Arguments:

• sessionId – optionally provide a sessionId (A-Z, a-z 0-9 and/or _- allowed, 16 character limit)

Return:

• sessionId – the unique ID of the data-reduction session

Usage:

do = data_organiser(
    log=log,
    rootDir="/path/to/workspace/root/"
)
sessionId = do.session_create(sessionId="my_supernova")

session_list(silent=False)

list the sessions available to the user

Key Arguments:

• silent – don’t print listings if True

Return:

• currentSession – the single ID of the currently used session

• allSessions – the IDs of the other sessions

Usage:

from soxspipe.commonutils import data_organiser
do = data_organiser(
    log=log,
    rootDir="."
)
currentSession, allSessions = do.session_list()

session_refresh(silent=False, failure=True)

refresh a session’s SOF files (needed if a recipe fails)

Usage:

from soxspipe.commonutils import data_organiser
do = data_organiser(
    log=log,
    rootDir="."
)
do.session_refresh()

session_switch(sessionId)

switch to an existing workspace data-reduction session

Key Arguments:

• sessionId – the sessionId to switch to

Usage:

from soxspipe.commonutils import data_organiser
do = data_organiser(
    log=log,
    rootDir="."
)
do.session_switch(mySessionId)

use_vlt_environment_folders()

use vlt environment folders

Key Arguments: # -

Return: - None

Usage:

usage code

---

.. todo::

    - add usage info
    - create a sublime snippet for usage
    - write a command-line tool for this method
    - update package tutorial with command-line tool info if needed