soxspipe.commonutils.toolkit

small reusable functions used throughout soxspipe

Author

David Young

Date Created

September 18, 2020

Module Contents

Classes

MaxFilter

Functions

add_recipe_logger	add a recipe-specific handler to the default logger that writes the recipe’s logs adjacent to the recipe project
add_snr_efficiency_qcs	add quality checks to the qc table
calculate_rolling_snr	Calculate the rolling Signal-to-Noise Ratio (SNR) for a given column in a pandas DataFrame.
create_dispersion_solution_grid_lines_for_plot	given a dispersion solution and accompanying 2D dispersion map image, generate the grid lines to add to QC plots
cut_image_slice	cut and return an N-pixel wide and M-pixels long slice, centred on a given coordinate from an image frame
extinction_correction_factor
frame_to_32	convert a given frame to 32-bit float format
generic_quality_checks	measure very basic quality checks on a frame and return the QC table with results appended
get_calibration_lamp	given a frame, determine which calibration lamp is being used
get_calibrations_path	return the root path to the static calibrations
plot_merged_spectrum_qc	Plot merged spectrum QC plot as a standalone function.
predict_product_path	predict the path of the recipe product from a given SOF name
qc_settings_plot_tables	generate QC and settings table to be placed at the bottom of the QC plots
quicklook_image	generate a quicklook image of a CCDObject - useful for development/debugging
read_spectral_format	read the spectral format table to get some key parameters
spectroscopic_image_quality_checks	measure and record spectroscopic image quailty checks
twoD_disp_map_image_to_dataframe	convert the 2D dispersion image map to a pandas dataframe
unpack_order_table	Unpack an order location table and return an orderPolyTable dataframe containing the polynomial coefficients for the order centres and edges, an orderPixelTable dataframe containing the pixel-coordinates for each order centre and edges, and finally, an orderMetaTable dataframe giving metadata about the frame binning and format.
utility_setup	setup some tools needed by most soxspipe utils

API

class soxspipe.commonutils.toolkit.MaxFilter(max_level)

Initialization

filter(record)

soxspipe.commonutils.toolkit.add_recipe_logger(log, productPath)

add a recipe-specific handler to the default logger that writes the recipe’s logs adjacent to the recipe project

Key Arguments:

• log – original logger

• productPath – path to the recipe product

Usage:

from soxspipe.commonutils.toolkit import add_recipe_logger
log = add_recipe_logger(log, productPath="/path/to/product")

soxspipe.commonutils.toolkit.add_snr_efficiency_qcs(log, spectrumDF, qcTable, orderJoins, recipeName, dateObs)

add quality checks to the qc table

Key Arguments:

• log – logger

• spectrumDF – the dataframe containing the extracted spectrum with a column named ‘SNR’ or ‘EFFICIENCY’ to calculate the checks from.

• qcTable – the qc table to which the SNR checks will be added

• orderJoins – a dictionary containing the wavelengths of order joins (if any) to be added as QCs.

• recipeName – name of the recipe to add to the QC entries

• dateObs – the observation date to add to the QC entries (in ISO format, e.g., “2024-06-01T00:00:00”)

Return:

• qcTable – the updated qc table with SNR checks added

Usage:

qcTable = add_snr_qcs(log, spectrumDF, qcTable, orderJoins)

soxspipe.commonutils.toolkit.calculate_rolling_snr(dataframe, flux_column, window_size)

Calculate the rolling Signal-to-Noise Ratio (SNR) for a given column in a pandas DataFrame.

This function computes the rolling SNR for a specified column in the DataFrame using a custom rolling window function. The SNR is calculated as the ratio of the median signal to the noise, where the noise is estimated using a robust statistical method.

Key Arguments:

- `dataframe`: The input pandas DataFrame containing the data.
- `flux_column`: The name of the column in the DataFrame for which the rolling SNR
    will be calculated.
- `window_size`: The size of the rolling window to use for the calculation.

Return:

- `dataframe`: The input DataFrame with an additional column 'SNR' containing the
calculated rolling SNR values.

Usage:

```python
from soxspipe.commonutils.toolkit import calculate_rolling_snr
df_with_snr = calculate_rolling_snr(dataframe=df, flux_column='flux', window_size=5)
```

soxspipe.commonutils.toolkit.create_dispersion_solution_grid_lines_for_plot(log, dispMap, dispMapImage, associatedFrame, kw, skylines=False, slitPositions=False, slit_length=11)

given a dispersion solution and accompanying 2D dispersion map image, generate the grid lines to add to QC plots

Key Arguments:

• log – logger

• dispMap – path to dispersion map. Default False

• dispMapImage – the 2D dispersion map image

• associatedFrame – a frame associated with the reduction (to read arm and binning info).

• kw – fits header kw dictionary

• skylines – a list of skylines to use as the grid. Default False

• slitPositions – slit positions to plot (else plot min and max)

• slit_length – length of the slit to use for the dispersion map dataframe (default 11)

Returns:

• orderPixelTable – DataFrame containing the pixel coordinates for grid lines to plot.

• interOrderMask – Mask array indicating inter-order regions.

Usage:

from soxspipe.commonutils.toolkit import create_dispersion_solution_grid_lines_for_plot
gridLinePixelTable, interOrderMask = create_dispersion_solution_grid_lines_for_plot(
    log=log,
    dispMap=dispMap,
    dispMapImage=dispMapImage,
    associatedFrame=CCDObject,
    kw=kw,
    skylines=skylines
)

for l in range(int(gridLinePixelTable['line'].max())):
    mask = (gridLinePixelTable['line'] == l)
    ax.plot(gridLinePixelTable.loc[mask]["fit_y"], gridLinePixelTable.loc[mask]["fit_x"], "w-", linewidth=0.5, alpha=0.8, color="black")

soxspipe.commonutils.toolkit.cut_image_slice(log, frame, width, length, x, y, sliceAxis='x', median=False, debug=False)

cut and return an N-pixel wide and M-pixels long slice, centred on a given coordinate from an image frame

Key Arguments:

• log – logger

• frame – the data array to cut the slice from (masked array)

• width – width of the slice (odd number)

• length – length of the slice

• x – x-coordinate

• y – y-coordinate

• sliceAxis – the axis along which slice is to be taken. Default x

• median – collapse the slice to a median value across its width

• debug – generate a plot of slice. Useful for debugging.

Usage:

from soxspipe.commonutils.toolkit import cut_image_slice
slice = cut_image_slice(log=self.log, frame=self.pinholeFlat.data,
                                width=1, length=sliceLength, x=x_fit, y=y_fit, plot=False)
if slice is None:
    return None

soxspipe.commonutils.toolkit.extinction_correction_factor(wave, extinctionTablePath, airmass)

soxspipe.commonutils.toolkit.frame_to_32(frame)

convert a given frame to 32-bit float format

Key Arguments:

• frame – the input frame

Returns:

• frame – the converted frame

Usage:

from soxspipe.commonutils.toolkit import frame_to_32
frame = frame_to_32(frame)

soxspipe.commonutils.toolkit.generic_quality_checks(log, frame, settings, recipeName, qcTable)

measure very basic quality checks on a frame and return the QC table with results appended

Key Arguments:

• log – logger

• frame – CCDData object

• settings – soxspipe settings

• recipeName – the name of the recipe

• qcTable – the QC pandas data-frame to save the QC measurements

Usage:

from soxspipe.commonutils.toolkit import generic_quality_checks
qcTable = generic_quality_checks(log=log, frame=myFrame, settings=settings, recipeName="my recipe", qcTable=qcTable)

soxspipe.commonutils.toolkit.get_calibration_lamp(log, frame, kw)

given a frame, determine which calibration lamp is being used

Key Arguments:

• log – logger

• frame – the frame to determine the calibration lamp for

• kw – the FITS header keyword dictionary

Usage:

from soxspipe.commonutils.toolkit import get_calibration_lamp
lamp = get_calibration_lamp(log=log, frame=frame, kw=kw)

soxspipe.commonutils.toolkit.get_calibrations_path(log, settings)

return the root path to the static calibrations

Key Arguments:

• log – logger

• settings – the settings dictionary

Usage:

from soxspipe.commonutils.toolkit import get_calibrations_path
calibrationRootPath = get_calibrations_path(log=log, settings=settings)

soxspipe.commonutils.toolkit.plot_merged_spectrum_qc(merged_orders, products, log, qcDir, filenameTemplate, noddingSequence, dateObs, arm, recipeName, orderJoins=False, debug=False, fluxCalibrated=False, qcTable=False, settings=False)

Plot merged spectrum QC plot as a standalone function.

Returns: products (pd.DataFrame): Updated products table. filePath (str): Path to the saved QC plot PDF.

soxspipe.commonutils.toolkit.predict_product_path(sofName, recipeName=False)

predict the path of the recipe product from a given SOF name

Key Arguments:

• log – logger,

• sofName – name or full path to the sof file

• recipeName – name of the recipe being considered. Default False.

Usage:

from soxspipe.commonutils import toolkit
productPath, startNightDate = toolkit.predict_product_path(sofFilePath)

soxspipe.commonutils.toolkit.qc_settings_plot_tables(log, qc, qcAx, settings, settingsAx)

generate QC and settings table to be placed at the bottom of the QC plots

Key Arguments:

• log – logger

• qc – date frame of collected QCs

• qcAx – the axis to add the QC table to

• settings – settings to report in settings table

• settingsAx – the axis to add the settings table to

Usage:

from soxspipe.commonutils.toolkit import qc_settings_plot_tables
qc_settings_plot_tables(log=log,qc=self.qc,qcAx=qcAx, settings=settings,settingsAx=settingsAx)

soxspipe.commonutils.toolkit.quicklook_image(log, CCDObject, show=True, ext='data', stdWindow=3, title=False, surfacePlot=False, dispMap=False, dispMapImage=False, inst=False, settings=False, skylines=False, saveToPath=False)

generate a quicklook image of a CCDObject - useful for development/debugging

Key Arguments:

• log – logger

• CCDObject – the CCDObject to plot

• show – show the image. Set to False to skip

• ext – the name of the the extension to show. Can be “data”, “mask” or “err”. Default “data”.

• title – give a title for the plot

• surfacePlot – plot as a 3D surface plot

• dispMap – path to dispersion map. Default False

• dispMapImage – the 2D dispersion map image

• inst – provide instrument name if no header exists

• skylines – mark skylines on image

from soxspipe.commonutils.toolkit import quicklook_image
quicklook_image(
    log=self.log, CCDObject=myframe, show=True)

soxspipe.commonutils.toolkit.read_spectral_format(log, settings, arm, dispersionMap=False, extended=True, binx=1, biny=1)

read the spectral format table to get some key parameters

Key Arguments:

• log – logger

• settings – soxspipe settings

• arm – arm to retrieve format for

• dispersionMap – if a dispersion map is given, the minimum and maximum dispersion axis pixel limits are computed

• extended – the spectral format table can provide WLMIN/WLMAX (extended=False) or WLMINFUL/WLMAXFUL (extended=True)

• binx – binning in the x-axis (from FITS header). Default 1

• biny – binning in the y-axis (from FITS header). Default 1

Return:

• orderNums – a list of the order numbers

• waveLengthMin – a list of the maximum wavelengths reached by each order

• waveLengthMax – a list of the minimum wavelengths reached by each order

Usage:

from soxspipe.commonutils.toolkit import read_spectral_format
# READ THE SPECTRAL FORMAT TABLE TO DETERMINE THE LIMITS OF THE TRACES
orderNums, waveLengthMin, waveLengthMax = read_spectral_format(
        log=self.log, settings=self.settings, arm=arm)

soxspipe.commonutils.toolkit.spectroscopic_image_quality_checks(log, frame, orderTablePath, settings, recipeName, qcTable)

measure and record spectroscopic image quailty checks

Key Arguments:

• log – logger

• frame – CCDData object

• orderTablePath – path to the order table

• settings – soxspipe settings

• recipeName – the name of the recipe

• qcTable – the QC pandas data-frame to save the QC measurements

Usage:

from soxspipe.commonutils.toolkit import spectroscopic_image_quality_checks
qcTable = spectroscopic_image_quality_checks(
        log=log, frame=myFrame, settings=settings, recipeName="this recipe", qcTable=qcTable, orderTablePath=orderTablePath)

soxspipe.commonutils.toolkit.twoD_disp_map_image_to_dataframe(log, slit_length, twoDMapPath, kw=False, associatedFrame=False, removeMaskedPixels=False, dispAxis='y')

convert the 2D dispersion image map to a pandas dataframe

Key Arguments:

• log – logger

• twoDMapPath – 2D dispersion map image path

• kw – fits keyword lookup dictionary

• associatedFrame – include a flux column in returned dataframe from a frame associated with the dispersion map. Default False

• removeMaskedPixels – remove the masked pixels from the associated image? Default False

• dispAxis – x or y. Needed for pixel scale calculation

Usage:

from soxspipe.commonutils.toolkit import twoD_disp_map_image_to_dataframe
mapDF = twoD_disp_map_image_to_dataframe(log=log, twoDMapPath=twoDMap, associatedFrame=objectFrame, kw=kw)

soxspipe.commonutils.toolkit.unpack_order_table(log, orderTablePath, extend=0.0, pixelDelta=1, binx=1, biny=1, prebinned=False, order=False, limitToDetectorFormat=False)

Unpack an order location table and return an orderPolyTable dataframe containing the polynomial coefficients for the order centres and edges, an orderPixelTable dataframe containing the pixel-coordinates for each order centre and edges, and finally, an orderMetaTable dataframe giving metadata about the frame binning and format.

Key Arguments:

• orderTablePath – path to the order table

• extend – fractional increase to the order area in the y-axis (needed for masking)

• pixelDelta – space between returned data points. Default 1 (sampled at every pixel)

• binx – binning in the x-axis (from FITS header). Default 1

• biny – binning in the y-axis (from FITS header). Default 1

• prebinned – was the order-table measured on a pre-binned frame (typically only for mflats). Default False

• order – unpack only a single order

• limitToDetectorFormat – limit the pixels return to those limited by the detector format static calibration table

Usage:

# UNPACK THE ORDER TABLE
from soxspipe.commonutils.toolkit import unpack_order_table
orderPolyTable, orderPixelTable = unpack_order_table(
    log=self.log, orderTablePath=orderTablePath, extend=0.)

soxspipe.commonutils.toolkit.utility_setup(log, settings, recipeName, startNightDate)

setup some tools needed by most soxspipe utils

Key Arguments:

• log – logger

• settings – the settings dictionary

• recipeName – name of the recipe as it appears in the settings dictionary

• startNightDate – YYYY-MM-DD date of the observation night. Default “”

Return:

• qcDir – the QC directory (created if missing)

• productDir – the product directory (created if missing)

Usage:

# Example usage with timezone-aware UTC datetime
from datetime import datetime, UTC
startNightDate = datetime.now(UTC).strftime("%Y-%m-%d")
qcDir, productDir = utility_setup(log=log, settings=settings, recipeName="my_recipe", startNightDate=startNightDate)