set_of_files

In ESO parlance, a set-of-files (.sof) file contains a list of the input data (or ingredients) needed for a pipeline recipe. This is a plain-text file where each input file path is specified with an associated classification label (one per line). Here is an example of the contents of a sof file needed by the soxs-mdark recipe:

./raw/SOXS_GEN_DARK_NIR_122_0001.fits  DARK_NIR
./raw/SOXS_GEN_DARK_NIR_122_0002.fits  DARK_NIR
./raw/SOXS_GEN_DARK_NIR_122_0003.fits  DARK_NIR
./raw/SOXS_GEN_DARK_NIR_122_0004.fits  DARK_NIR
./raw/SOXS_GEN_DARK_NIR_122_0005.fits  DARK_NIR

The primary purpose of the set_of_files utility is to read and translate these sof files into a CCDProc ImageFileCollection, an in-memory database of the files and their header keywords. Alongside sof files, the utility also accepts a Python list of file paths or a path to a directory of FITS files as acceptable inputs.

Lines in a sof file beginning with a # are considered comments and, therefore, ignored by the pipeline. This is helpful to quickly remove a file from the recipe input by commenting out its line in the sof file or for adding user notes to the sof file.

Utility API

class set_of_files(log, settings=False, inputFrames=[], verbose=True, recipeName=False, ext=0, session=None)

Bases: object

The worker class for the sof module used to homogenize various frame input formats (sof file, directory of fits fits, list of fits file paths) into a CCDProc ImageFileCollection

Key Arguments:

• log – logger

• settings – the settings dictionary

• inputFrames – can be a directory, a set-of-files (SOF) file or a list of fits frame paths. Default []

• verbose – verbose. True or False. Default True

• recipeName – the name of the recipe. Default False

• ext – the data extension for the frame. Default 0.

Usage

To initiate a sof object, use the following:

# inputFrames = "/path/to/a/directory"
# inputFrames = ['/path/to/one.fits','/path/to/two.fits','/path/to/three.fits']
inputFrames = '/path/to/myfiles.sof'
from soxspipe.commonutils.set_of_files import set_of_files
sof = set_of_files(
    log=log,
    settings=settings,
    inputFrames=inputFrames,
    ext=0
)

inputFrames can be a directory, a list of fits filepaths or a set-of-files (SOF) file

Initialization

create_supplementary_file_dictionary(supplementaryFilepaths)

create supplementary file dictionary

Key Arguments:

• supplementaryFilepaths – the list of filepaths to generate the dictionary for

Return:

• supplementary_sof – a dictionary of non-fits files needed for recipe

get()

return the set-of-files as a CCDProc ImageFileCollection

Return:

• sof – a ccdproc ImageFileCollection of the frames

Usage

To generate a ImageFileCollection from a directory, a list of fits filepaths or a set-of-files (SOF) file try the following:

# inputFrames = "/path/to/a/directory"
# inputFrames = ['/path/to/one.fits','/path/to/two.fits','/path/to/three.fits']
inputFrames = '/path/to/myfiles.sof'
from soxspipe.commonutils.set_of_files import set_of_files
sof = set_of_files(
    log=log,
    settings=settings,
    inputFrames=inputFrames
)
sofFile, supplementarySof = sof.get()
print(sofFile.summary)

inputFrames can be a directory, a list of fits filepaths or a set-of-files (SOF) file.