Helper Utilities¶

add_recipe_logger¶

Add a recipe-specific handler to the default logger that writes the recipe logs adjacent to the recipe project.

soxspipe.commonutils.toolkit.add_recipe_logger(log, productPath)[source][source]

add a recipe-specific handler to the default logger that writes the recipe’s logs adjacent to the recipe project

Key Arguments:

log – original logger
productPath – path to the recipe product

Usage:

from soxspipe.commonutils.toolkit import add_recipe_logger
log = add_recipe_logger(log, productPath="/path/to/product")

create_dispersion_solution_grid_lines_for_plot¶

Use a dispersion solution table and 2D image map to generate a dispersion solution grid to add to QC plots.

soxspipe.commonutils.toolkit.create_dispersion_solution_grid_lines_for_plot(log, dispMap, dispMapImage, associatedFrame, kw, skylines=False, slitPositions=False, slit_length=11)[source][source]

given a dispersion solution and accompanying 2D dispersion map image, generate the grid lines to add to QC plots

Key Arguments:

log – logger
dispMap – path to dispersion map. Default False
dispMapImage – the 2D dispersion map image
associatedFrame – a frame associated with the reduction (to read arm and binning info).
kw – fits header kw dictionary
skylines – a list of skylines to use as the grid. Default False
slitPositions – slit positions to plot (else plot min and max)
slit_length – length of the slit to use for the dispersion map dataframe (default 11)

Returns:

orderPixelTable – DataFrame containing the pixel coordinates for grid lines to plot.
interOrderMask – Mask array indicating inter-order regions.

Usage:

from soxspipe.commonutils.toolkit import create_dispersion_solution_grid_lines_for_plot
gridLinePixelTable, interOrderMask = create_dispersion_solution_grid_lines_for_plot(
    log=log,
    dispMap=dispMap,
    dispMapImage=dispMapImage,
    associatedFrame=CCDObject,
    kw=kw,
    skylines=skylines
)

for l in range(int(gridLinePixelTable['line'].max())):
    mask = (gridLinePixelTable['line'] == l)
    ax.plot(gridLinePixelTable.loc[mask]["fit_y"], gridLinePixelTable.loc[mask]["fit_x"], "w-", linewidth=0.5, alpha=0.8, color="black")

cut_image_slice¶

Cut and return an N-pixel wide and M-pixel long image slice, centred on a given coordinate from the input image frame.

soxspipe.commonutils.toolkit.cut_image_slice(log, frame, width, length, x, y, sliceAxis='x', median=False, debug=False)[source][source]

cut and return an N-pixel wide and M-pixels long slice, centred on a given coordinate from an image frame

Key Arguments:

log – logger
frame – the data array to cut the slice from (masked array)
width – width of the slice (odd number)
length – length of the slice
x – x-coordinate
y – y-coordinate
sliceAxis – the axis along which slice is to be taken. Default x
median – collapse the slice to a median value across its width
debug – generate a plot of slice. Useful for debugging.

Usage:

from soxspipe.commonutils.toolkit import cut_image_slice
slice = cut_image_slice(log=self.log, frame=self.pinholeFlat.data,
                                width=1, length=sliceLength, x=x_fit, y=y_fit, plot=False)
if slice is None:
    return None

detector_lookup¶

Return a dictionary of detector characteristics and parameters.

class detector_lookup(log, settings=False)[source]

Bases: object

return a dictionary of detector characteristics and parameters

Key Arguments:

log – logger
settings – the settings dictionary

Usage:

To initiate a detector_lookup object, use the following:

from soxspipe.commonutils import detector_lookup
detector = detector_lookup(
    log=log,
    settings=settings
).get("NIR")
print(detector["science-pixels"])

Initialization

get(arm)[source]

return a dictionary of detector characteristics and parameters

Key Arguments:

arm – the detector parameters to return

dispersion_map_to_pixel_arrays¶

Use a dispersion solution to convert wavelength, slit-position and echelle order numbers to X, Y pixel positions.

soxspipe.commonutils.dispersion_map_to_pixel_arrays.dispersion_map_to_pixel_arrays(log, dispersionMapPath, orderPixelTable, removeOffDetectorLocation=True, trimColumns=False)[source][source]

Use a dispersion solution to convert wavelength, slit-position and echelle order numbers to X,Y pixel positions.

Return a line-list with x,y fits given a first guess dispersion map.*

Key Arguments:

log – logger
dispersionMapPath – path to the dispersion map
orderPixelTable – a data-frame including ‘order’, ‘wavelength’ and ‘slit_pos’ columns
removeOffDetectorLocation – if data points are found to lie off the detector plane then remove them from the results. Default True
trimColumns – if True, only return the specified columns. Default False

Usage:

from soxspipe.commonutils import dispersion_map_to_pixel_arrays
myDict = {
    "order": [11, 11, 11],
    "wavelength": [850.3, 894.3, 983.2],
    "slit_position": [0, 0, 0]
}
orderPixelTable = pd.DataFrame(myDict)
orderPixelTable = dispersion_map_to_pixel_arrays(
    log=log,
    dispersionMapPath="/path/to/map.csv",
    orderPixelTable=orderPixelTable
)

filenamer¶

Given a FITS object (HDU list and header), use the SOXS file-naming scheme to return a filename to be used to save the FITS object to disk.

soxspipe.commonutils.filenamer.filenamer(log, frame, keywordLookup=False, detectorLookup=False, settings=False)[source][source]

Given a FITS object, use the SOXS file-naming scheme to return a filename to be used to save the FITS object to disk

Key Arguments:

log – logger
frame – the CCDData object frame
keywordLookup – the keyword lookup dictionary (needed if settings not provided). Default False
detectorLookup – the detector parameters (needed if settings not provided). Default False
settings – the soxspipe settings dictionary (needed if keywordLookup and detectorLookup not provided). Default False

Return:

filename – stanardised name to for the input frame

frame = CCDData.read(filepath, hdu=0, unit=u.electron, hdu_uncertainty='ERRS',
        du_mask='QUAL', hdu_flags='FLAGS', key_uncertainty_type='UTYPE')

from soxspipe.commonutils import filenamer
filename = filenamer(
    log=log,
    frame=frame,
    settings=settings
)

generic_quality_checks¶

Measure some very basic quality checks on a frame and return the QC table with results appended

soxspipe.commonutils.toolkit.generic_quality_checks(log, frame, settings, recipeName, qcTable)[source][source]

measure very basic quality checks on a frame and return the QC table with results appended

Key Arguments:

log – logger
frame – CCDData object
settings – soxspipe settings
recipeName – the name of the recipe
qcTable – the QC pandas data-frame to save the QC measurements

Usage:

from soxspipe.commonutils.toolkit import generic_quality_checks
qcTable = generic_quality_checks(log=log, frame=myFrame, settings=settings, recipeName="my recipe", qcTable=qcTable)

get_calibration_lamp¶

Given a frame (CCDObject), determine which calibration lamp is used.

soxspipe.commonutils.toolkit.get_calibration_lamp(log, frame, kw)[source][source]

given a frame, determine which calibration lamp is being used

Key Arguments:

log – logger
frame – the frame to determine the calibration lamp for
kw – the FITS header keyword dictionary

Usage:

from soxspipe.commonutils.toolkit import get_calibration_lamp
lamp = get_calibration_lamp(log=log, frame=frame, kw=kw)

get_calibrations_path¶

Return the root path to the static calibrations (ships alongside code).

soxspipe.commonutils.toolkit.get_calibrations_path(log, settings)[source][source]

return the root path to the static calibrations

Key Arguments:

log – logger
settings – the settings dictionary

Usage:

from soxspipe.commonutils.toolkit import get_calibrations_path
calibrationRootPath = get_calibrations_path(log=log, settings=settings)

keyword_lookup¶

Given a tag (internal to the pipeline), and an optional keyword index, return the FITS Header keyword for the selected instrument.

class keyword_lookup(log, instrument=False, settings=False)[source]

Bases: object

The worker class for the keyword_lookup module

Key Arguments:

log – logger
settings – the settings dictionary. Default False
instrument – can directly add the instrument if settings file is not avalable. Default False

Usage

To initalise the keyword lookup object in your code add the following:

from soxspipe.commonutils import keyword_lookup
kw = keyword_lookup(
    log=log,
    settings=settings,
    instrument=False,
).get

After this it’s possible to either look up a single keyword using it’s alias:

kw("DET_NDITSKIP")
> "ESO DET NDITSKIP"

or return a list of keywords:

kw(["PROV", "DET_NDITSKIP"])
> ['PROV', 'ESO DET NDITSKIP']

For those keywords that require an index it’s possible to also pass the index to the kw function:

kw("PROV", 9)
> 'PROV09'

If a tag is not in the list of FITS Header keyword aliases in the configuration file a LookupError will be raised.

Initialization

get(tag, index=False)[source]

given a tag, and optional keyword index, return the FITS Header keyword for the selected instrument

Key Arguments:

tag – the keyword tag as set in the yaml keyword dictionary (e.g. ‘SDP_KEYWORD_TMID’ returns ‘TMID’). Can be string or list of sttings.
index – add an index to the keyword if not False (e.g. tag=‘PROV’, index=3 returns ‘PROV03’) Default False

Return:

keywords – the FITS Header keywords. Can be string or list of sttings depending on format of tag argument

Usage

See docstring for the class

predict_product_path¶

Predict the path of the recipe product from a given SOF file name.

soxspipe.commonutils.toolkit.predict_product_path(sofName, recipeName=False)[source][source]

predict the path of the recipe product from a given SOF name

Key Arguments:

log – logger,
sofName – name or full path to the sof file
recipeName – name of the recipe being considered. Default False.

Usage:

from soxspipe.commonutils import toolkit
productPath, startNightDate = toolkit.predict_product_path(sofFilePath)

qc_settings_plot_tables¶

Generate the QC and settings tables at the bottom of the QC plots.

soxspipe.commonutils.toolkit.qc_settings_plot_tables(log, qc, qcAx, settings, settingsAx)[source][source]

generate QC and settings table to be placed at the bottom of the QC plots

Key Arguments:

log – logger
qc – date frame of collected QCs
qcAx – the axis to add the QC table to
settings – settings to report in settings table
settingsAx – the axis to add the settings table to

Usage:

from soxspipe.commonutils.toolkit import qc_settings_plot_tables
qc_settings_plot_tables(log=log,qc=self.qc,qcAx=qcAx, settings=settings,settingsAx=settingsAx)

quicklook_image¶

Generate a quick look image of a CCDObject - useful for development/debugging.

soxspipe.commonutils.toolkit.quicklook_image(log, CCDObject, show=True, ext='data', stdWindow=3, title=False, surfacePlot=False, dispMap=False, dispMapImage=False, inst=False, settings=False, skylines=False, saveToPath=False)[source][source]

generate a quicklook image of a CCDObject - useful for development/debugging

Key Arguments:

log – logger
CCDObject – the CCDObject to plot
show – show the image. Set to False to skip
ext – the name of the the extension to show. Can be “data”, “mask” or “err”. Default “data”.
title – give a title for the plot
surfacePlot – plot as a 3D surface plot
dispMap – path to dispersion map. Default False
dispMapImage – the 2D dispersion map image
inst – provide instrument name if no header exists
skylines – mark skylines on image

from soxspipe.commonutils.toolkit import quicklook_image
quicklook_image(
    log=self.log, CCDObject=myframe, show=True)

read_spectral_format¶

Return a spectral format table for your selected instrument containing key parameters about the detector.

soxspipe.commonutils.toolkit.read_spectral_format(log, settings, arm, dispersionMap=False, extended=True, binx=1, biny=1)[source][source]

read the spectral format table to get some key parameters

Key Arguments:

log – logger
settings – soxspipe settings
arm – arm to retrieve format for
dispersionMap – if a dispersion map is given, the minimum and maximum dispersion axis pixel limits are computed
extended – the spectral format table can provide WLMIN/WLMAX (extended=False) or WLMINFUL/WLMAXFUL (extended=True)
binx – binning in the x-axis (from FITS header). Default 1
biny – binning in the y-axis (from FITS header). Default 1

Return:

orderNums – a list of the order numbers
waveLengthMin – a list of the maximum wavelengths reached by each order
waveLengthMax – a list of the minimum wavelengths reached by each order

Usage:

from soxspipe.commonutils.toolkit import read_spectral_format
# READ THE SPECTRAL FORMAT TABLE TO DETERMINE THE LIMITS OF THE TRACES
orderNums, waveLengthMin, waveLengthMax = read_spectral_format(
        log=self.log, settings=self.settings, arm=arm)

uncompress¶

Uncompress ESO fits.Z frames before processing them with the data-organiser.

soxspipe.commonutils.uncompress.uncompress(log, directory)[source][source]

uncompress ESO fits.Z frames

Key Arguments:

log – logger
directory – directory containing .Z file to uncompress

from soxspipe.commonutils import uncompress
uncompress(
    log=log,
    directory="/path/to/raw_data/"
)

spectroscopic_image_quality_checks¶

Perform some generic image quality checks and add to the QC output of the recipe.

soxspipe.commonutils.toolkit.spectroscopic_image_quality_checks(log, frame, orderTablePath, settings, recipeName, qcTable)[source][source]

measure and record spectroscopic image quailty checks

Key Arguments:

log – logger
frame – CCDData object
orderTablePath – path to the order table
settings – soxspipe settings
recipeName – the name of the recipe
qcTable – the QC pandas data-frame to save the QC measurements

Usage:

from soxspipe.commonutils.toolkit import spectroscopic_image_quality_checks
qcTable = spectroscopic_image_quality_checks(
        log=log, frame=myFrame, settings=settings, recipeName="this recipe", qcTable=qcTable, orderTablePath=orderTablePath)

twoD_disp_map_image_to_dataframe¶

Convert the 2D dispersion image map to a pandas dataframe.

soxspipe.commonutils.toolkit.twoD_disp_map_image_to_dataframe(log, slit_length, twoDMapPath, kw=False, associatedFrame=False, removeMaskedPixels=False, dispAxis='y')[source][source]

convert the 2D dispersion image map to a pandas dataframe

Key Arguments:

log – logger
twoDMapPath – 2D dispersion map image path
kw – fits keyword lookup dictionary
associatedFrame – include a flux column in returned dataframe from a frame associated with the dispersion map. Default False
removeMaskedPixels – remove the masked pixels from the associated image? Default False
dispAxis – x or y. Needed for pixel scale calculation

Usage:

from soxspipe.commonutils.toolkit import twoD_disp_map_image_to_dataframe
mapDF = twoD_disp_map_image_to_dataframe(log=log, twoDMapPath=twoDMap, associatedFrame=objectFrame, kw=kw)

unpack_order_table¶

Unpack an order location table and return an orderPolyTable dataframe containing the polynomial coefficients for the order centres and edges, an orderPixelTable dataframe containing the pixel-coordinates for each order centre and edges, and finally, an orderMetaTable dataframe giving metadata about the frame binning and format

soxspipe.commonutils.toolkit.unpack_order_table(log, orderTablePath, extend=0.0, pixelDelta=1, binx=1, biny=1, prebinned=False, order=False, limitToDetectorFormat=False)[source][source]

Key Arguments:

orderTablePath – path to the order table
extend – fractional increase to the order area in the y-axis (needed for masking)
pixelDelta – space between returned data points. Default 1 (sampled at every pixel)
binx – binning in the x-axis (from FITS header). Default 1
biny – binning in the y-axis (from FITS header). Default 1
prebinned – was the order-table measured on a pre-binned frame (typically only for mflats). Default False
order – unpack only a single order
limitToDetectorFormat – limit the pixels return to those limited by the detector format static calibration table

Usage:

# UNPACK THE ORDER TABLE
from soxspipe.commonutils.toolkit import unpack_order_table
orderPolyTable, orderPixelTable = unpack_order_table(
    log=self.log, orderTablePath=orderTablePath, extend=0.)