soxspipe ∞

The data-reduction pipeline for the SOXS instrument (a python package with command-line tools).

Documentation for soxspipe is hosted by Read the Docs (development version and master version). The code lives on github. Please report any issues you find here.

Installation ∞

The best way to install or upgrade soxspipe is to use conda to install the package in its own isolated environment, as shown here:

conda create -n soxspipe python=3.9 soxspipe -c conda-forge
conda activate soxspipe

If you have previously installed soxspipe, a warning will be issued stating that a conda environment already exists; select ‘y’ when asked to remove the existing environment.

To check installation was successful run soxspipe -v. This should return the version number of the install.

How to cite soxspipe ∞

If you use soxspipe in your work, please cite using the following BibTeX entry:

@software{Young_soxspipe,
    author = {Young, David R.},
    doi = {10.5281/zenodo.8038264},
    license = {GPL-3.0-only},
    title = ,
    url = {https://zenodo.org/doi/10.5281/zenodo.8038264}
}

Quickstart Guide ∞

Warning

This quickstart guide is subject to (much) change during the development of the pipeline. New features and ways of operating the pipeline are still being added. Current data taken in stare mode can be reduced to the point of sky subtraction.

Install ∞

The best way to install soxspipe is to use conda and install the package in its own isolated environment (using either Anaconda or Minicoda), as shown here:

conda create -n soxspipe python=3.9 soxspipe -c conda-forge
conda activate soxspipe

If you have previously installed soxspipe, a warning will be issued stating that a conda environment already exists; select y when asked to remove the existing environment. This has proven to be the cleanest way to upgrade soxspipe.

To check installation was successful run soxspipe -v. This should return the version number of the installation.

Demo Data ∞

The demo XShooter data (stare-mode) is of the X-ray binary SAX J1808.4-3658 taken during a 2019 outburst. You can download and unpack the data with the following commands:

curl -L "https://www.dropbox.com/s/t3adwc86bcwonkj/soxspipe-quickstart-demo-lite.tgz?dl=1" > soxspipe-quickstart-demo.tgz
tar -xzvf soxspipe-quickstart-demo.tgz

You may also retrieve the raw data directly from the ESO archive with the following parameters:

RA = 18 08 27.54
Dec = -36 58 44.3
Night = 2019 08 30
Spectroscopy = XSHOOTER/VLT
Science

Preparing the Data-Reduction Workspace ∞

Now you have a sample data set to work with, it is time to prepare the soxspipe-quickstart-demo workspace. Change into the soxspipe-quickstart-demo directory and run the soxspipe prep command:

cd soxspipe-quickstart-demo
soxspipe prep .

Once the workspace has been prepared, you should find it contains the following files and folders:

misc/: a lost-and-found archive of non-fits files
raw_frames/: all raw-frames to be reduced
sessions/: directory of data-reduction sessions
sof/: the set-of-files (sof) files required for each reduction step
soxspipe.db: a sqlite database needed by the data-organiser, please do not delete

soxspipe reduces data within a reduction session and an initial base session is automatically created when running the prep command.

Reduce the Data ∞

In most use case, you will want to reduce all of the raw frames contained within your workspace. To do this run the command:

soxspipe reduce all .

Logging ∞

When running a recipe, soxspipe writes informative logs to the terminal (stdout), allowing the user to keep track of the reduction progress in real time. For the sake of provenance, this same information is written to a log file adjacent to the recipe’s product file(s).

If the recipe happens to fail, a separate error log is written to the directory the product file should have been written to had the recipe succeeded. Error logs are named with a ”_ERROR.log” suffix.

Data Reduction Sessions ∞

soxspipe reduces data within a ‘reduction session’. These sessions are designed to be self-contained and isolated from other sessions, allowing a user to reduce the same set of raw data with multiple different pipeline settings. When a workspace is first prepared (using the soxspipe prep command), an initial base session is automatically created. Sessions are stored in the sessions directory of the workspace and contain their own settings file, products and QC directories. soxspipe will remember and use the latest session you were working with unless you create a new session or switch to another one.

To see which reduction session you are working within and all other sessions available, run the command:

soxspipe session ls

To create a new session, run the command:

soxspipe session new

The new session will be named with the date-time stamp of its creation date. Alternatively, you can also supply a memorable name for the session. The name can be up to 16 characters long and use alpha-numeric characters, - or _.

soxspipe session new my_supernova

Or to switch to an existing session, run the command:

soxspipe session <sessionId>

For user convenience, when you switch to a new session, the symbolic links found within the workspace root folder are automatically switched to point to the current session assets (products, qc, sof, soxspipe.yaml,soxspipe.db etc). If you run ls -lrt within your workspace root directory you will see these symlinks reported:

products -> ./sessions/base/products
qc -> ./sessions/base/qc
soxspipe.db -> ./sessions/base/soxspipe.db
soxspipe.yaml -> ./sessions/base/soxspipe.yaml
sof -> ./sessions/base/sof

`reduce` ∞

In the simplest use case, you will want to reduce all of the raw frames contained within your workspace. To do this, first change directory to the base of your workspace, and then run the commands:

soxspipe prep .
soxspipe reduce all . 

soxspipe

will now attempt to reduce all of the data in your workspace.

A Primer on SOXS Observation Modes ∞

The glow of thermal emission from the earth’s atmosphere, and even the telescope itself, start to dominate photon counts at NIR wavelengths, seriously contaminating signal from target sources. This thermal emission glow is typically dubbed ‘sky background’ (despite the fact it originates in the foreground of the target source).

The sole reason SOXS has multiple observation modes is to provide astronomers with flexibility and a choice of methods they can employ to attempt to isolate and remove the noise generated by the sky-background from their science frames.

The 3 main observation modes are (click links for more detail):

Stare mode where the on-source sky-background is modelled, fitted and removed with software. Although removal of the background is not as accurate as the other observation modes, the time lost in overheads is lower.
Nodding mode employs an observational technique to measure the sky-background of an empty patch of sky near in time and spatially close to the source. This spectral measurement of the sky-background flux is simply removed from the on-source spectrum. Although generating more overhead time than stare-mode, unlike offset-mode, this technique allows for source and sky to be observed on the same frames.
Offset mode employs the same basic ‘observe and subtract’ method to remove the sky-background as nod mode, but is typically favoured for extended sources where it is not possible to observe a blank patch of sky and the source within the same 11 arcsec slit. As sky and source are observed in separate frames this observation mode has the highest fraction of time lost to overheads.

Recipes ∞

SOXSPIPE borrows the informative concept of `recipes’ employed by ESO’s data reduction pipelines to define the modular components of the pipeline. These recipes can be strung together to create an end-to-end workflow that takes as input the raw and calibration frames from the instrument and telescope and processes them all the way through to fully reduced, calibrated, ESO Phase III compliant science products.

Standard Calibrations ∞

`soxs_mbias` ∞

A zero-second exposure will contain only read-noise and ~half of pixels within this Gaussian distribution centred around zero count will always contain negative flux. To avoid negative counts an offset bias voltage is applied at the amplifier stage so that even when no photons are detected the A/D converter will always register a positive value. This bias-voltage offset must be accounted for in the data reduction process.

The purpose of the soxs_mbias recipe is to provide a master-bias frame that can be subtracted from science/calibration frames to remove the contribution of pixel counts resulting from the bias-voltage.

Input ∞

Data Type	Content	Related OB
FITS images	raw bias frames (UV-VIS/AC exposures with exptime = 0)	`SOXS_img_cal_Bias`, `SOXS_gen_cal_VISBias`

Parameters ∞

Parameter	Description	Type	Entry Point	Related Util
frame-clipping-sigma	number of σ from the median frame flux beyond which pixel is added to the bad-pixel mask	float	settings file	`clip_and_stack`
clipping-iteration-count	number of sigma-clipping iterations to perform when added pixels to the bad-pixel mask	int	settings file	`clip_and_stack`
stacked-clipping-sigma	number of σ deviations from the median pixel flux beyond which pixel is excluded from stack	float	settings file	`clip_and_stack`
stacked-clipping-iterations	number of σ-clipping iterations to perform before stacking	float	settings file	`clip_and_stack`

Method ∞

The purpose of the soxs_mbias recipe is to stack raw bias-frames together (using the clip_and_stack utility) into master-bias frames and in the process clipping rogue pixels from the individual raw frames and reducing the read-noise contribution.

Output ∞

Data Type	Content
FITS image	Master bias frame (frame containing typical bias-voltage applied to the detector)

QC Metrics ∞

Metric	Description
TBC	…

Recipe API ∞

class soxs_mbias(log, settings=False, inputFrames=[], verbose=False, overwrite=False)[source] ∞

The soxs_mbias recipe is used to generate a master-bias frame from a set of input raw bias frames. The recipe is used only for the UV-VIS arm as NIR frames have bias (and dark current) removed by subtracting an off-frame of equal expsoure length.

Key Arguments

log – logger
settings – the settings dictionary
inputFrames – input fits frames. Can be a directory, a set-of-files (SOF) file or a list of fits frame paths.
verbose – verbose. True or False. Default False
overwrite – overwrite the prodcut file if it already exists. Default False

Usage

from soxspipe.recipes import soxs_mbias
mbiasFrame = soxs_mbias(
    log=log,
    settings=settings,
    inputFrames=fileList
).produce_product()

Todo

add a tutorial about soxs_mbias to documentation

verify_input_frames()[source] ∞

verify the input frame match those required by the soxs_mbias recipe

If the fits files conform to required input for the recipe everything will pass silently, otherwise an exception shall be raised.

produce_product()[source] ∞

generate a master bias frame

Return:

productPath – the path to the master bias frame

qc_bias_structure(combined_bias_mean)[source] ∞

calculate the structure of the bias

Key Arguments:

combined_bias_mean – the mbias frame

Return:

structx – slope of BIAS in X direction
structx – slope of BIAS in Y direction

Usage:

structx, structy = self.qc_bias_structure(combined_bias_mean)

qc_periodic_pattern_noise(frames)[source] ∞

calculate the periodic pattern noise based on the raw input bias frames

A 2D FFT is applied to each of the raw bias frames and the standard deviation and median absolute deviation calcualted for each result. The maximum std/mad is then added as the ppnmax QC in the master bias frame header.

Key Arguments:

frames – the raw bias frames (imageFileCollection)

Return:

- ``ppnmax``

Usage:

self.qc_periodic_pattern_noise(frames=self.inputFrames)

`soxs_mdark` ∞

Every raw CCD image contains counts resulting from a ‘dark current’, electrons released due to the thermal effects in the CCD material. For both the UVB-VIS (< 0.00012 e^-/s/pixel) and NIR detectors (< 0.005 e^-/s/pixel) the dark-current is almost negligible. Not all pixels will have the same dark-current, some will have a high than typical current. These are so-called ‘hot-pixels’ and it’s important that these are identified and recorded (using the create_noise_map utility).

The purpose of the soxs_mdark recipe is to generate a master-dark frame used to remove flux attributed to the dark-current from other frames.

Input ∞

Data Type	Content	Related OB
FITS images	raw dark frames (exposures with identical exposure time and detectors readout parameters)	`SOXS_gen_cal_VISDark`, `SOXS_gen_cal_NIRDark`, `SOXS_img_cal_Dark`

Parameters ∞

Parameter	Description	Type	Entry Point	Related Util
frame-clipping-sigma	number of σ from the median frame flux beyond which pixel is added to the bad-pixel mask	float	settings file	`clip_and_stack`
clipping-iteration-count	number of sigma-clipping iterations to perform when added pixels to the bad-pixel mask	int	settings file	`clip_and_stack`
stacked-clipping-sigma	number of σ deviations from the median pixel flux beyond which pixel is excluded from stack	float	settings file	`clip_and_stack`
stacked-clipping-iterations	number of σ-clipping iterations to perform before stacking	float	settings file	`clip_and_stack`

Method ∞

Stack raw dark-frames together (using the clip_and_stack utility) into master-dark frames and in the process clipping rogue pixels from the individual raw frames and reducing the read-noise contribution.

Output ∞

Data Type	Content
master dark frame	frame containing typical dark-current flux accumulated over the exposure time of the input frames

QC Metrics ∞

Metric	Description
TBC	…

Recipe API ∞

class soxs_mdark(log, settings=False, inputFrames=[], verbose=False, overwrite=False)[source] ∞

The soxs_mdark recipe

Key Arguments

log – logger

settings – the settings dictionary

inputFrames – input fits frames. Can be a directory, a set-of-files (SOF) file or a list of fits frame paths.

verbose – verbose. True or False. Default False

overwrite – overwrite the prodcut file if it already exists. Default False

Usage

from soxspipe.recipes import soxs_mdark
mdarkFrame = soxs_mdark(
    log=log,
    settings=settings,
    inputFrames=fileList
)..produce_product()

Todo

add a tutorial about soxs_mdark to documentation

verify_input_frames()[source] ∞

verify input frame match those required by the soxs_mdark recipe

If the fits files conform to required input for the recipe everything will pass silently, otherwise an exception shall be raised.

produce_product()[source] ∞

generate a master dark frame

Return:

productPath – the path to master dark frame

`soxs_mflat` ∞

The purpose of the soxs_mflat recipe is to create a single normalised master-flat frame used to correct for non-uniformity in response to light across the detector plain.

Sources of this non-uniformity include varying pixel sensitivities, obstructions in the optical path (e.g. dust or pollen grains), vignetting at the edges of the detector. A flat-frame is ideally an image taken where the illumination is uniform across the light collecting pixels of the detector. This evenly exposed image can be used to identify irregularities in the response in the detector.

Input ∞

Data Type	Content	Related OB
FITS images	raw flats frames (exposures with identical exposure time and detectors readout parameters). UV-VIS requires separate sets D-Lamp and QTH-Lamp flats.	`SOXS_slt_cal_NIRLampFlat`, `SOXS_slt_cal_NIRLampFlatAtt`, `SOXS_slt_cal_VISLampFlat`, `SOXS_slt_cal_VISLampFlatAtt`
FITS Image	Master Bias Frame (UV-VIS only)	-
FITS Table	order table containing coefficients to the polynomial fits describing the order centre locations. UV-VIS requires separate tables for D-Lamp and QTH-Lamp.

Parameters ∞

Parameter	Description	Type	Entry Point	Related Util
frame-clipping-sigma	number of σ from the median frame flux beyond which pixel is added to the bad-pixel mask	float	settings file	`clip_and_stack`
clipping-iteration-count	number of sigma-clipping iterations to perform when added pixels to the bad-pixel mask	int	settings file	`clip_and_stack`
stacked-clipping-sigma	number of σ deviations from the median pixel flux beyond which pixel is excluded from stack	float	settings file	`clip_and_stack`
stacked-clipping-iterations	number of σ-clipping iterations to perform before stacking	float	settings file	`clip_and_stack`
centre-order-window	the width of the slice to cut along the centre of each order when determining mean exposure level	int	settings file	-
slice-length-for-edge-detection	length of image slice to take across orders when detecting edges	int	settings file	`detect_order_edges`
slice-width-for-edge-detection	width of image slice to take across orders when detecting edges	int	settings file	`detect_order_edges`
min-percentage-threshold-for-edge-detection	minimum value flux can drop to as percentage of central flux and be counted as an order edge	int	settings file	`detect_order_edges`
max-percentage-threshold-for-edge-detection	maximum value flux can claim to as percentage of central flux and be counted as an order edge	int	settings file	`detect_order_edges`
disp-axis-deg	degree of dispersion axis component of polynomial fit to order edges	int	settings file	`detect_order_edges`
order-deg	degree of order component of polynomial fit to order edges	int	settings file	`detect_order_edges`
poly-fitting-residual-clipping-sigma	number of σ deviations from the median fit residual beyond which individual data points are removed when iterating towards a fit of order edges	int	settings file	`detect_order_edges`
poly-clipping-iteration-limit	number of sigma-clipping iterations to perform before settings on a polynomial fit for the order edges	int	settings file	`detect_order_edges`
low-sensitivity-clipping-sigma	number of σ deviations below the median flux of a master-flat frame beyond which a pixel is added to the bad-pixel mask	int	settings file	-

Method ∞

The individual flat field frames need to have bias and dark signatures removed before they are combined. This is achieved with the detrend utility. Here is an example of one such calibrated flat frame:

Normalising Exposure Levels in Individual Flat Frames ∞

Once calibrated, exposure-levels in the individual flat frames need to be normalised as total illumination will vary from frame-to-frame. The individual frame exposure levels are calculated in two stages.

In the first stage the mean inner-order pixel-value across the frame is used as a first approximation of an individual frame’s exposure level. To calculate this mean value, the order locations are used to identify a curved slice N-pixels wide centred on each of the order-centres and bad pixels are masked (see image below). The collected inner order pixel values are then sigma-clipped to excluded out-lying values and a mean value calculated.

Individual frames are then divided through by their mean inner-order pixel value in this first attempt to normalise the exposure-levels of the frames.

The normalised flat-frames are then combined using the clip_and_stack utility into a first-pass master-flat frame:

The second stage then is to divide each original dark and bias subtracted flat frame by this first-pass master flat (see example below). This removes the typical cross-plane illumination and so now the mean inner-order pixel-value across the frame will give a much better estimate of each frame’s intrinsic exposure level.

The mean inner-order pixel-value is calculated again on this frame and the original dark and bias subtracted flat is re-normalised by divided through by this accurate measurement of its intrinsic exposure level.

Building a Final Master-Flat Frame ∞

These re-normalised flats are then combined for a second time into a master-flat frame.

Finally order edges are located with the detect_order_edges utility and the inter-order area pixel value are set to 1.

Low-sensitivity pixels are flagged and added to the bad-pixel map and a final master-flat frame written to file.

UV Master Flat Frame Stitching ∞

As the UV-VIS uses a combination of D-Lamp and QTH-Lamp flat sets, a further step is required to stitch the best orders from each of these master-flats together into a dual lamp master-flat.

For both the D-Lamp and QTH-Lamp master-flat frames, we have for each order the number of pixel positions that contributed to the final order-edge fit. We use these numbers to decide which orders to slice and stitch from the D-Lamp to the QTH-Lamp master flat frame.

With a crossover order now selected, the median flux from a square window at the centre of this order in both D- and QTH frames is measured. Using the ratio of these fluxes the D-Lamp frame is scaled to the QTH-Lamp frame.

From the upper order-edge polynomial for the D-Lamp we define a curved, intra-order line 5 pixels above the upper edge of the crossover order selected previously. This line is used to slice and stitch the D-Lamp and QTH-Lamp orders together. This process is done on the flux images, error maps and bad-pixel maps. Typically the bluest orders from the D-Lamp will be selected with the remaining orders coming from the QTH-Lamp.

Finally, the combined normalised frames for both the D and QTH-Lamps are stacked to obtain a good level of flux in each order. This stacked frame is used to re-detect the order edges (the resulting order table is used going forward).

Output ∞

Data Type	Content
master flat frame	frame used correct for non-uniformity in response to light across the detector plain (including blaze)

QC Metrics ∞

Metric	Description
TBC	…

Recipe API ∞

class soxs_mflat(log, settings=False, inputFrames=[], verbose=False, overwrite=False)[source] ∞

The soxs_mflat recipe

Key Arguments

log – logger

settings – the settings dictionary

inputFrames – input fits frames. Can be a directory, a set-of-files (SOF) file or a list of fits frame paths.

verbose – verbose. True or False. Default False

overwrite – overwrite the prodcut file if it already exists. Default False

Usage

from soxspipe.recipes import soxs_mflat
recipe = soxs_mflat(
    log=log,
    settings=settings,
    inputFrames=fileList
)
mflatFrame = recipe.produce_product()

Todo

add a tutorial about soxs_mflat to documentation

verify_input_frames()[source] ∞

verify the input frames match those required by the soxs_mflat recipe

If the fits files conform to required input for the recipe everything will pass silently, otherwise an exception will be raised.

produce_product()[source] ∞

generate the master flat frames updated order location table (with egde detection)

Return:

productPath – the path to the master flat frame

calibrate_frame_set()[source] ∞

given all of the input data calibrate the frames by subtracting bias and/or dark

Return:

calibratedFlats – the calibrated frames

normalise_flats(inputFlats, orderTablePath, firstPassMasterFlat=False, lamp='')[source] ∞

determine the median exposure for each flat frame and normalise the flux to that level

Key Arguments:

inputFlats – the input flat field frames
orderTablePath – path to the order table
firstPassMasterFlat – the first pass of the master flat. Default False

- `lamp` -- a lamp tag for QL plots

Return:

normalisedFrames – the normalised flat-field frames (CCDData array)

mask_low_sens_pixels(frame, orderTablePath, returnMedianOrderFlux=False, writeQC=True)[source] ∞

add low-sensitivity pixels to bad-pixel mask

Key Arguments:

frame – the frame to work on
orderTablePath – path to the order table
returnMedianOrderFlux – return a table of the median order fluxes. Default False.
writeQC – add the QCs to the QC table?

Return:

frame – with BPM updated with low-sensitivity pixels
medianOrderFluxDF – data-frame of the median order fluxes (if returnMedianOrderFlux is True)

stitch_uv_mflats(medianOrderFluxDF, orderTablePath)[source] ∞

return a master UV-VIS flat frame after slicing and stitch the UV-VIS D-Lamp and QTH-Lamp flat frames

Key Arguments:

medianOrderFluxDF – data frame containing median order fluxes for D and QTH frames
orderTablePath – the original order table paths from order-centre tracing

Return:

stitchedFlat – the stitch D and QTH-Lamp master flat frame

Usage:

mflat = self.stitch_uv_mflats(medianOrderFluxDF)

find_uvb_overlap_order_and_scale(dcalibratedFlats, qcalibratedFlats)[source] ∞

find uvb order where both lamps produce a similar flux. This is the order at which the 2 lamp flats will be scaled and stitched together

Key Arguments:

qcalibratedFlats – the QTH lamp calibration flats.
dcalibratedFlats – D2 lamp calibration flats

Return:

order – the order number where the lamp fluxes are similar

Usage:

overlapOrder = self.find_uvb_overlap_order_and_scale(dcalibratedFlats=dcalibratedFlats, qcalibratedFlats=qcalibratedFlats)

Dispersion and Spatial Solutions ∞

There is a strong curvature in the traces of the NIR orders and spectral-lines do not run perpendicular to the dispersion direction, but are highly tilted. Therefore wavelength cannot be expressed as simply a function of pixel position, but instead detector pixel positions (\(X, Y\)) much be mapped as a function of:

wavelength \(\lambda\)
order number \(n\), and
slit position \(s\)

This 2D mapping function is determined incrementally via the soxs_disp_solution, soxs_order_centres and soxs_spatial_solution recipes. The soxs_straighten recipe can then be used to transform spectral images from detector pixel-space to wavelength and slit-position space

`soxs_disp_solution` ∞

The purpose of the soxs_disp_solution is to use a single-pinhole arc-lamp frame (example image above) to generate a first guess dispersion solution.

Input ∞

As input this recipes accepts the Pinhole Map file.

Data Type	Content	Related OB
FITS Image	Arc Lamp through single pinhole mask	`SOXS_slt_cal_VISArcsPinhole`, `SOXS_slt_cal_NIRArcsPinhole`
FITS Image	Master Dark Frame (VIS only)	-
FITS Image	Master Bias Frame (VIS only)	-
FITS Image	Dark frame (Lamp-Off) of equal exposure length as single pinhole frame (Lamp-On) (NIR only)	`SOXS_slt_cal_NIRArcsPinhole`
CSV File	Pinhole Map

Parameters ∞

Parameter	Description	Type	Entry Point	Related Util
pixel-window-size	the side-length (in pixels) of the square window used to search for arc-line detection	int	settings file	`create_dispersion_map`
order-deg	the order of polynomial used to fit spectral-orders of detected arc-lines	int	settings file	`create_dispersion_map`
wavelength-deg	the order of polynomial used to fit wavelengths of detected arc-lines	int	settings file	`create_dispersion_map`
poly-fitting-residual-clipping-sigma	sigma distance limit, where distance is the difference between the detected and polynomial fitted positions of an arc-line, outside of which to remove lines from the fit	float	settings file	`create_dispersion_map`
poly-clipping-iteration-limit	number of sigma-clipping iterations to perform before settings on a polynomial fit for the dispersion solution	int	settings file	`create_dispersion_map`

Method ∞

After preparing and calibrating the single-pinhole arc-lamp frame (using the detrend), the create_dispersion_map) util is employed to detect and measure the positions of the arc lines on the frame. Below you can see the bright arc-lines outshining the traces of the order-centres and the detection of one of these lines during the create_dispersion_map) util.

Once the line positions have been measured, a dispersion solution is generated by iteratively fitting a global polynomial against the observed line-positions (see create_dispersion_map) for details). The final product is a Dispersion Map file.

Output ∞

Data Type	Content
CSV file (subject to change)	First guess Dispersion Map

QC Metrics ∞

The typical solution for the soxs_disp_solution recipe has sub-pixel residuals.

Metric	Description
TBC	…

Recipe API ∞

class soxs_disp_solution(log, settings=False, inputFrames=[], verbose=False, overwrite=False)[source] ∞

generate a first approximation of the dispersion solution from single pinhole frames

Key Arguments

log – logger

settings – the settings dictionary

inputFrames – input fits frames. Can be a directory, a set-of-files (SOF) file or a list of fits frame paths.

verbose – verbose. True or False. Default False

overwrite – overwrite the prodcut file if it already exists. Default False

Usage

from soxspipe.recipes import soxs_disp_solution
disp_map_path = soxs_disp_solution(
    log=log,
    settings=settings,
    inputFrames=sofPath
).produce_product()

Todo

add a tutorial about soxs_disp_solution to documentation

verify_input_frames()[source] ∞

verify input frames match those required by the ``soxs_disp_solution`` recipe

If the fits files conform to required input for the recipe everything will pass silently, otherwise an exception shall be raised.

produce_product()[source] ∞

generate a fisrt guess of the dispersion solution

Return:

productPath – the path to the first guess dispersion map

`soxs_order_centres` ∞

The purpose of the soxs_order_centres recipe is to find and fit the order centres with low-level polynomials.

Input ∞

Data Type	Content	Related OB
FITS Image	Flat lamp through a single-pinhole mask	`SOXS_slt_cal_VISLampFlatPinhole`, `SOXS_slt_cal_NIRLampFlatPinhole`
FITS Image	Master Dark Frame (VIS only)	-
FITS Image	Master Bias Frame (VIS only)	-
FITS Image	Dark frame (Lamp-Off) of equal exposure length as single-pinhole frame (Lamp-On) (NIR only)	`SOXS_slt_cal_NIRLampFlatPinhole`
CSV File	First guess dispersion solution	-

Parameters ∞

Parameter	Description	Type	Entry Point	Related Util
order-sample-count	number of times along the order in the dispersion direction to measure the order-centre trace	int	settings file	detect_continuum utility
peak-sigma-limit	minimum value a peak must be above the median value of pixel to be considered for order-trace fitting	int	settings file	detect_continuum utility
disp-axis-deg	degree of dispersion axis component of polynomal fit to order-centre traces		int	settings file
order-deg	degree of order component of polynomal fit to order-centre traces		int	settings file
poly-fitting-residual-clipping-sigma	sigma distance limit, where distance is the difference between the detected and polynomial fitted positions of the order-trace, outside of which to remove lines from the fit	float	settings file	detect_continuum utility
poly-clipping-iteration-limit	number of sigma-clipping iterations to perform before settings on a polynomial fit for the order-centre traces	int	settings file	detect_continuum utility

Method ∞

Once the single-pinhole flat-lamp frame has had the bias, dark and background subtracted it is passed to the detect_continuum utility to fit the order centres.

Output ∞

Data Type	Content
CSV File	order table containing coefficients to the polynomial fits describing the order centre locations

QC Metrics ∞

Plots similar to the one below are generated after each execution of soxs_order_centres.

Metric	Description
TBC	…

Recipe API ∞

class soxs_order_centres(log, settings=False, inputFrames=[], verbose=False, overwrite=False)[source] ∞

The soxs_order_centres recipe

Key Arguments

log – logger

settings – the settings dictionary

inputFrames – input fits frames. Can be a directory, a set-of-files (SOF) file or a list of fits frame paths.

verbose – verbose. True or False. Default False

overwrite – overwrite the prodcut file if it already exists. Default False

Usage

from soxspipe.recipes import soxs_order_centres
order_table = soxs_order_centres(
    log=log,
    settings=settings,
    inputFrames=a["inputFrames"]
).produce_product()

Todo

add a tutorial about soxs_order_centres to documentation

verify_input_frames()[source] ∞

verify input frames match those required by the soxs_order_centres recipe

Return:

- ``None``

If the fits files conform to required input for the recipe everything will pass silently, otherwise an exception shall be raised.

produce_product()[source] ∞

generate the order-table with polynomal fits of order-centres

Return:

productPath – the path to the order-table

`soxs_spatial_solution` - PLANNED ∞

The purpose of this recipe is to further enhance the wavelength solution achieved with soxs_disp_solution by expanding the solution into the spatial dimension (along the slit). This 2-dimensional solution will then account for any tilt in the spectral lines.¹

Each pinhole in the multi-pinhole mask is \(\text{0.5"}\) in diameter and the 9 pinholes are evenly spaced along the \(\text{11"}\) slit with a \(\text{1.4"}\) gap between adjacent holes. This knowledge affords us the ability to now map the dispersion solution along the spatial direction.

Input ∞

Data Type	Content	Related OB
FITS Image	Arc Lamp through multi-pinhole mask	`SOXS_slt_cal_VISArcsMultiplePinhole`, `SOXS_slt_cal_NIRArcsMultiplePinhole`
FITS Image	Master Dark Frame (VIS only)	-
FITS Image	Master Bias Frame (VIS only)	-
FITS Image	Dark frame (Lamp-Off) of equal exposure length as multi-pinhole frame (Lamp-On) (NIR only)	`SOXS_slt_cal_NIRArcsMultiplePinhole`
CSV File	First-guess Dispersion Map table
CSV File	Pinhole Map

Method ∞

Having prepared the multi-pinhole frame the bias and dark signatures are removed and the frame is divided through by the master flat frame. The calibrated frame and the first-guess dispersion map are passed to the create_dispersion_map utility to produce a 2D dispersion solution covering both the spectral and spatial dimensions.

Output ∞

Data Type	Content
CSV File (subject to change)	Dispersion Map table giving coefficients of polynomials describing 2D dispersion/spatial solution

QC Metrics ∞

Metric	Description
TBC	…

Recipe API ∞

class soxs_spatial_solution(log, settings=False, inputFrames=[], verbose=False, overwrite=False, create2DMap=True, polyOrders=False)[source] ∞

The soxs_spatial_solution recipe

Key Arguments

log – logger

settings – the settings dictionary

inputFrames – input fits frames. Can be a directory, a set-of-files (SOF) file or a list of fits frame paths

verbose – verbose. True or False. Default False

overwrite – overwrite the prodcut file if it already exists. Default False

create2DMap – create the 2D image map of wavelength, slit-position and order from disp solution.

polyOrders – the orders of the x-y polynomials used to fit the dispersion solution. Overrides parameters found in the yaml settings file. e.g 345435 is order_x=3, order_y=4 ,wavelength_x=5 ,wavelength_y=4, slit_x=3 ,slit_y=5. Default False.

See produce_product method for usage.

Todo

add a tutorial about soxs_spatial_solution to documentation

verify_input_frames()[source] ∞

verify input frames match those required by the ``soxs_spatial_solution`` recipe

If the fits files conform to required input for the recipe everything will pass silently, otherwise an exception shall be raised.

produce_product()[source] ∞

generate the 2D dispersion map

Return:

productPath – the path to the 2D dispersion map

Usage

from soxspipe.recipes import soxs_spatial_solution
recipe = soxs_spatial_solution(
    log=log,
    settings=settings,
    inputFrames=fileList
)
disp_map = recipe.produce_product()

relative to the perpendicular of the dispersion direction ↩

`soxs_straighten` - PLANNED ∞

This recipe takes the full dispersion map given by soxs_spatial_solution and uses it to map images from their original representation of the detector surface to one that presents the signal in a wavelength by slit-position coordinate system.

Input ∞

Data Type	Content	Related OB
CSV File	Coefficients of polynomials providing a full dispersion-spatial solution
FITS Image	An associated spectral image requiring rectification	Many

Parameters ∞

Parameter	Description	Type	Entry Point	Related Util
`straighten_grid_res_wavelength`	size of the grid cell in wavelength dimension (nm)	float	settings file
`straighten_grid_res_split`	size of the grid cell in slit dimension (arcsec)	float	settings file

Method ∞

We now have a pair of polynomials that can be used to give the exact pixel on the detector containing flux resulting from a specific order, with a given wavelength and slit position.

\[\begin{split} X = \sum\limits_{ijk} c_{ijk} \times n^i \times \lambda^j \times s^k \\ \end{split}\]

\[\begin{split} Y = \sum\limits_{ijk} c_{ijk} \times n^i \times \lambda^j \times s^k \\ \end{split}\]

To begin we want to create a full wavelength and slit-position map; a 2D grid of wavelengths along one axis and slit-position along the other. Using the polynomial solutions above, we populate each cell in the grid with its corresponding detector pixel coordinate. The 2D grid is of fine enough resolution so that many cells in the grid are mapped to each individual detector pixel. With this map in hand we can now assign flux recorded in each detector pixel to the corresponding cells in the 2D wavelength and slit-position grid. The flux from each detector is evenly distributed between all cells found to be associate with that pixel; so if 9 cells are associated then each cell gets 1/9th of the pixel flux.

The error and bad-pixel extensions go through the same mapping process.

Output ∞

Data Type	Content
FITS Images	The straightened images containing flux represented in wavelength space; one for each order

QC Metrics ∞

Metric	Description
TBC	…

Recipe API ∞

Utilities ∞

The soxspipe utilities can be viewed as the tool-kit with which the soxspipe recipes are built. Many of these utilities are used across multiple recipes and they can either be stand-alone objects or methods of the recipe itself.

`clip_and_stack` ∞

clip_and_stack mean combines input frames after sigma-clipping outlying pixels using a median value with median absolute deviation (mad) as the deviation function.

Before combining the frames we want to ‘clip’ any outlying pixel values found in the individual frames that are to be stacked. We isolate and remove pixels from any averaging calculation (mean or median) that have a value that strays too far from the ‘typical’ pixel value.

Using the median pixel value as the ‘typical’ value and the median absolute deviation (MAD) as a proxy for the standard-deviation we can accurately identify rogue pixels. For any given set of pixel values:

\[ MAD = \frac{1}{N}\sum_{i=0}^N |x_i - \text{median}(x)|. \]

The clipping is done iteratively so newly found rogue pixels are masks, median values are recalculated and clipping repeated. The iterative process stops whenever either no more bad-pixels are to be found or the maximum number of iterations has been reached.

After the clipping has been completed individual frames are mean-combined, ignoring pixels in the individual bad-pixel masks. If a pixel is flagged as ‘bad’ in all individual masks it is added to the combined frame bad-pixel mask.

_base_recipe_.clip_and_stack(frames, recipe, ignore_input_masks=False, post_stack_clipping=True)[source] ∞

mean combine input frames after sigma-clipping outlying pixels using a median value with median absolute deviation (mad) as the deviation function

Key Arguments:

frames – an ImageFileCollection of the frames to stack or a list of CCDData objects
recipe – the name of recipe needed to read the correct settings from the yaml files
ignore_input_masks – ignore the input masks during clip and stacking?
post_stack_clipping – allow cross-plane clipping on combined frame. Clipping settings in setting file. Default True.

Return:

combined_frame – the combined master frame (with updated bad-pixel and uncertainty maps)

Usage:

This snippet can be used within the recipe code to combine individual (using bias frames as an example):

combined_bias_mean = self.clip_and_stack(
    frames=self.inputFrames, recipe="soxs_mbias", ignore_input_masks=False, post_stack_clipping=True)

`create_dispersion_map` ∞

The create_dispersion_map utility is used to search for arc-lines in the single/multi-pinhole arc-lamp frames and then iteratively fit a global polynomial dispersion solution (and spatial-solution in the case of multi-pinhole frame) with the observed line-positions. It is used by both the soxs_disp_solution) and soxs_spatial_solution) solution recipes.

In the static calibration suite we have Pinhole Maps listing the wavelength \(\lambda\), order number \(n\) and slit position \(s\) of the spectral lines alongside a first approximation of their (\(X, Y\)) pixel-positions on the detector.

If the input frame is a single-pinhole frame, we can filter the Pinhole Map to contain just the central pinhole positions. If however input is the multi-pinhole frame then we use the first guess Dispersion Map (created with soxs_disp_solution) to calculate the shift between the predicted and the observed line positions for the central pinholes. We then update the Pinhole Map by applying the same shift to the other pinholes.

For each line in the Pinhole Map line-list:

an image stamp centred on the predicted pixel-position (\(X_o, Y_o\)), of dimensions winX and winY, is generated from the pinhole calibration frame
a sigma-clipped median pixel value is calculated and then subtracted from each stamp
DAOStarFinder is employed to search for the observed detector position (\(X, Y\)) of the arc-line via 2D Gaussian profile fitting on the stamp

We now have a list of arc-line wavelengths and their observed pixel-positions and the order they were detected in. These values are used to iteratively fit two polynomials that describe the global dispersion solution for the detector. In the case of the single-pinhole frames these are:

\[X = \sum\limits_{ij} c_{ij} \times n^i \times \lambda^j\]

\[Y = \sum\limits_{ij} c_{ij} \times n^i \times \lambda^j\]

where \(\lambda\) is wavelength and \(n\) is the echelle order number.

In the case of the multi-pinhole we also have the slit position \(s\) and so adding a spatial solution to the dispersion solution:

\[X = \sum\limits_{ijk} c_{ijk} \times n^i \times \lambda^j \times s^k\]

\[Y = \sum\limits_{ijk} c_{ijk} \times n^i \times \lambda^j \times s^k\]

Upon each iteration the residuals between the fits and the measured pixel-positions are calculated and sigma-clipping is employed to eliminate measurements that stray to far from the fit. Once the maximum number of iterations is reach, or all outlying lines have been clipped, the coefficients of the polynomials are written to a Dispersion Map file.

2D Image Map ∞

The Dispersion Map is used to generate a triple extension FITS file with each extension image exactly matching the dimensions of the detector. The first extension contains the wavelength value at the centre of each pixel location, the second the slit-position and the third the order number. The solutions for these images are iteratively converged on in a brute force manner (see workflow diagram below). These image maps are used in sky-background subtraction and object extraction utilities.

class create_dispersion_map(log, settings, recipeSettings, pinholeFrame, firstGuessMap=False, orderTable=False, qcTable=False, productsTable=False, sofName=False, create2DMap=True)[source] ∞

detect arc-lines on a pinhole frame to generate a dispersion solution

Key Arguments:

log – logger
settings – the settings dictionary
recipeSettings – the recipe specific settings
pinholeFrame – the calibrated pinhole frame (single or multi)
firstGuessMap – the first guess dispersion map from the soxs_disp_solution recipe (needed in soxs_spat_solution recipe). Default False.
orderTable – the order geometry table
qcTable – the data frame to collect measured QC metrics
productsTable – the data frame to collect output products
sofName – name of the originating SOF file
create2DMap – create the 2D image map of wavelength, slit-position and order from disp solution.

Usage:

from soxspipe.commonutils import create_dispersion_map
mapPath, mapImagePath, res_plots, qcTable, productsTable = create_dispersion_map(
    log=log,
    settings=settings,
    pinholeFrame=frame,
    firstGuessMap=False,
    qcTable=self.qc,
    productsTable=self.products
).get()

get()[source] ∞

generate the dispersion map

Return:

mapPath – path to the file containing the coefficients of the x,y polynomials of the global dispersion map fit

get_predicted_line_list()[source] ∞

lift the predicted line list from the static calibrations

Return:

orderPixelTable – a panda’s data-frame containing wavelength,order,slit_index,slit_position,detector_x,detector_y

detect_pinhole_arc_line(predictedLine, iraf=True)[source] ∞

detect the observed position of an arc-line given the predicted pixel positions

Key Arguments:

predictedLine – single predicted line coordinates from predicted line-list
iraf – use IRAF star finder to generate a FWHM

Return:

predictedLine – the line with the observed pixel coordinates appended (if detected, otherwise nan)

write_map_to_file(xcoeff, ycoeff, orderDeg, wavelengthDeg, slitDeg)[source] ∞

write out the fitted polynomial solution coefficients to file

Key Arguments:

xcoeff – the x-coefficients
ycoeff – the y-coefficients
orderDeg – degree of the order fitting
wavelengthDeg – degree of wavelength fitting
slitDeg – degree of the slit fitting (False for single pinhole)

Return:

disp_map_path – path to the saved file

calculate_residuals(orderPixelTable, xcoeff, ycoeff, orderDeg, wavelengthDeg, slitDeg, writeQCs=False, pixelRange=False)[source] ∞

calculate residuals of the polynomial fits against the observed line positions

Key Arguments:

orderPixelTable – the predicted line list as a data frame

xcoeff – the x-coefficients

ycoeff – the y-coefficients

orderDeg – degree of the order fitting

wavelengthDeg – degree of wavelength fitting

slitDeg – degree of the slit fitting (False for single pinhole)

writeQCs – write the QCs to dataframe? Default False

pixelRange – return centre pixel and +- 2nm from the centre pixel (to measure the pixel scale)

Return:

residuals – combined x-y residuals
mean – the mean of the combine residuals
std – the stdev of the combine residuals
median – the median of the combine residuals

fit_polynomials(orderPixelTable, wavelengthDeg, orderDeg, slitDeg, missingLines=False)[source] ∞

iteratively fit the dispersion map polynomials to the data, clipping residuals with each iteration

Key Arguments:

orderPixelTable – data frame containing order, wavelengths, slit positions and observed pixel positions
wavelengthDeg – degree of wavelength fitting
orderDeg – degree of the order fitting
slitDeg – degree of the slit fitting (0 for single pinhole)
missingLines – lines not detected on the image

Return:

xcoeff – the x-coefficients post clipping
ycoeff – the y-coefficients post clipping
goodLinesTable – the fitted line-list with metrics
clippedLinesTable – the lines that were sigma-clipped during polynomial fitting

create_placeholder_images(order=False, plot=False, reverse=False)[source] ∞

create CCDData objects as placeholders to host the 2D images of the wavelength and spatial solutions from dispersion solution map

Key Arguments:

order – specific order to generate the placeholder pixels for. Inner-order pixels set to NaN, else set to 0. Default False (generate all orders)
plot – generate plots of placeholder images (for debugging). Default False.
reverse – Inner-order pixels set to 0, else set to NaN (reverse of default output).

Return:

slitMap – placeholder image to add pixel slit positions to
wlMap – placeholder image to add pixel wavelength values to

Usage:

slitMap, wlMap, orderMap = self._create_placeholder_images(order=order)

map_to_image(dispersionMapPath)[source] ∞

convert the dispersion map to images in the detector format showing pixel wavelength values and slit positions

Key Arguments:

dispersionMapPath – path to the full dispersion map to convert to images

Return:

dispersion_image_filePath – path to the FITS image with an extension for wavelength values and another for slit positions

Usage:

mapImagePath = self.map_to_image(dispersionMapPath=mapPath)

order_to_image(orderInfo)[source] ∞

convert a single order in the dispersion map to wavelength and slit position images

Key Arguments:

orderInfo – tuple containing the order number to generate the images for, the minimum wavelength to consider (from format table) and maximum wavelength to consider (from format table).

Return:

slitMap – the slit map with order values filled
wlMap – the wavelengths map with order values filled

Usage:

slitMap, wlMap = self.order_to_image(order=order,minWl=minWl, maxWl=maxWl)

convert_and_fit(order, bigWlArray, bigSlitArray, slitMap, wlMap, iteration, plots=False)[source] ∞

convert wavelength and slit position grids to pixels

Key Arguments:

order – the order being considered
bigWlArray – 1D array of all wavelengths to be converted
bigSlitArray – 1D array of all split-positions to be converted (same length as bigWlArray)
slitMap – place-holder image hosting fitted pixel slit-position values
wlMap – place-holder image hosting fitted pixel wavelength values
iteration – the iteration index (used for CL reporting)
plots – show plot of the slit-map

Return:

orderPixelTable – dataframe containing unfitted pixel info
remainingCount – number of remaining pixels in orderTable

Usage:

orderPixelTable = self.convert_and_fit(
        order=order, bigWlArray=bigWlArray, bigSlitArray=bigSlitArray, slitMap=slitMap, wlMap=wlMap)

create_new_static_line_list(dispersionMapPath)[source] ∞

using a first pass dispersion solution, use a line atlas to generate a more accurate and more complete static line list

Key Arguments:

- `dispersionMapPath` -- path to the first pass dispersion solution

Return:

- `newPredictedLineList` -- a new predicted line list (to replace the static calibration line-list)

`create_noise_map` - PLANNED ∞

`detect_continuum` ∞

The purpose of the detect_continuum utility is to find and fit the order centres with low-level polynomials.

The utility takes as input a calibrated single pinhole flat-frame which displays the traces of the order centre locations:

From a spectral format table (specific to the arm in question) we know the minimum and maximum wavelength values for each order. Then using the first guess dispersion map generated by soxs_disp_solution we generate an array of approximate pixel locations on each order centre between these wavelength limits.

Centred on each pixel position we take a single-pixel wide image slice in the cross-dispersion direction N-pixels long (N is a recipe parameter). A 1D gaussian is fitted against the pixel slice and the peak pixel-position is stored.

Finally, for each order, the set of gaussian peak pixel-positions is fitted with a low-order polynomial (X as a function of Y).

\[ X = \sum_{i=0}^n c_{i} \times Y^i \]

Where \(n\) is the degree of the polynomials. Polynomials are iteratively fitted while sigma-clipping pixel-positions with outlying residuals. Results are stored in an order table.

`detect_order_edges` – COMPLETED ∞

The purpose of the detect_order_edges utility is to detect the edges of each order on the detector.

The utility takes as input a fully illuminated slit image, that is, a calibrated flat-field frame or a stacked flat-field frame (preferable as cosmic ray hits will be removed from the stack frame). The second input is the order table generated by the soxs_order_centres recipe which provides an estimation of a. the order locations, and b. the order shapes.

For each order an array of central pixel positions is generated. At each pixel position a N-pixel long, M-pixel wide image slice in the cross-dispersion direction (N and M are recipe parameters) is cut from the flat-frame. The slice is collapsed to a 1D array by taking the median value across its width (ignoring masked pixels). Further median smoothing is applied along the length of the slice to compensate of any rogue pixel values. Below you can see an example of a slice with the grey dot representing the collapsed 1D slice pixel values and the red crosses showing the median smoothed values.

From the central 1D slice in each order the minimum and maximum fluxes are calculated to give a flux range. The absolute minimum and maximum flux thresholds are determined from the percentage thresholds given as recipe parameters. If the minimum threshold was set at 25% then the absolute flux would be:

\[ threshold = minvalue + (maxvalue - minvalue) * 0.25 \]

For each slice along each order the pixel-locations xmin and xmax along the order edge where the flux reaches this minimum flux threshold are detected and recorded (see red circles in the figure above). If the minimum flux is not recorded at any point along the order edge, the threshold is slowly incremented up to a maximum flux until the threshold is registered. Slices where both edges are not detected before the maximum flux threshold is reached are rejected.

Finally for each order the arrays of xmin and xmax pixel-positions are iteratively fitted with low order polynomials of Y as a function of X. The new order table is written to file and now includes the upper and lower-edge locations alongside the central location for each order.

`detector_lookup` ∞

The purpose of the detector_lookup utility is to act as reference in the code for the various characteristics of the soxs detectors.

When initiated detector_lookup reads the detector characteristics from a Detector Parameters file and can later serve these parameters to the soxspipe code when requested.

class detector_lookup(log, settings=False)[source] ∞

return a dictionary of detector characteristics and parameters

Key Arguments:

log – logger
settings – the settings dictionary

Usage:

To initiate a detector_lookup object, use the following:

from soxspipe.commonutils import detector_lookup
detector = detector_lookup(
    log=log,
    settings=settings
).get("NIR")
print(detector["science-pixels"])

get(arm)[source] ∞

return a dictionary of detector characteristics and parameters

Key Arguments:

arm – the detector parameters to return

`filenamer` ∞

The purpose of the filenamer utility is to implement the file-naming scheme as laid out here. This util is usually called to determine a file name before writing a frame/table/product to disk.

`keyword_lookup` ∞

The purpose of the keyword_lookup utility is to act as a lookup-reference in the code for specific SOXS FITS Header keywords.

When initiated keyword_lookup reads the keywords from a Keyword Dictionary file and can later serve the keywords names to the soxspipe code when requested.

class keyword_lookup(log, instrument=False, settings=False)[source] ∞

The worker class for the keyword_lookup module

Key Arguments:

log – logger
settings – the settings dictionary. Default False
instrument – can directly add the instrument if settings file is not avalable. Default False

Usage

To initalise the keyword lookup object in your code add the following:

from soxspipe.commonutils import keyword_lookup
kw = keyword_lookup(
    log=log,
    settings=settings,
    instrument=False,
).get

After this it’s possible to either look up a single keyword using it’s alias:

kw("DET_NDITSKIP")
> "ESO DET NDITSKIP"

or return a list of keywords:

kw(["PROV", "DET_NDITSKIP"])
> ['PROV', 'ESO DET NDITSKIP']

For those keywords that require an index it’s possible to also pass the index to the kw function:

kw("PROV", 9)
> 'PROV09'

If a tag is not in the list of FITS Header keyword aliases in the configuration file a LookupError will be raised.

get(tag, index=False)[source] ∞

given a tag, and optional keyword index, return the FITS Header keyword for the selected instrument

Key Arguments:

tag – the keyword tag as set in the yaml keyword dictionary (e.g. ‘SDP_KEYWORD_TMID’ returns ‘TMID’). Can be string or list of sttings.
index – add an index to the keyword if not False (e.g. tag=’PROV’, index=3 returns ‘PROV03’) Default False

Return:

keywords – the FITS Header keywords. Can be string or list of sttings depending on format of tag argument

Usage

See docstring for the class

`prepare_frames` ∞

The purpose of prepare_frames is to prepare the raw SOXS frames for data reduction.

Here’s the typical workflow for preparing the raw frames:

1. Trim Overscan

The first thing we need to do is trim off the overscan area of the image. The science-pixel regions for the detectors are read from the Detector Parameters file.

2. ADU to Electrons

Next the pixel data is converted from ADU to electron counts by multiplying each pixel value in the raw frame by the detector gain (the gain is read in units of electrons/ADU).

\[\rm{electron\ count} = \rm{adu\ count} \times \rm{gain}\]

3. Generating an Uncertainty Map

Next an uncertainty map is generated for the raw image and added as the ‘ERR’ extension of the image.

4. Bad Pixel Mask

The default detector bitmap is read from the static calibration suite and converted to a boolean mask, with values >0 becoming TRUE to indicate these pixels need to be masks. All other values are set to FALSE. This map is add as the ‘QUAL’ extesion of the image.

Finally the prepared frames are saved out into the intermediate frames location with the prefix pre_.

Viewing the image in DS9 (using the command ds9 -multiframe -tile columns pre_filename.fits to show all extensions as tiled frames) we can see the ‘FLUX’, ‘QUAL’ and ‘ERR’ extensions are now all present.

_base_recipe_.prepare_frames(save=False)[source] ∞

prepare raw frames by converting pixel data from ADU to electrons and adding mask and uncertainty extensions

Key Arguments:

save – save out the prepared frame to the intermediate products directory. Default False.

Return:

preframes – the new image collection containing the prepared frames

Usage

Usually called within a recipe class once the input frames have been selected and verified (see soxs_mbias code for example):

self.inputFrames = self.prepare_frames(
    save=self.settings["save-intermediate-products"])

`set_of_files` ∞

The set_of_files utility helps to translate and homogenise various recipe input-frame lists. This allows recipes to accept any of the following inputs:

an ESORex-like sof file,
a directory of FITS files
a list of fits file paths

Behind the scenes set_of_files converts the lists into a CCDProc ImageFileCollection.

Lines in a sof file beginning with a # are considered as comments and therefore ignored by the pipeline.

`subtract_background` ∞

The purpose of the subtract_background utility is to model the topology of the scattered background light within an image and then remove it.

Here’s the workflow for subtracting a frame’s background:

Here’s an example frame requiring the background scattered light to be fitted and removed:

Having unpacked the order location table, a mask is created containing pixels lying within the order-locations. The mask is extended in either direction along the y-axis by a given fraction (recipe parameter) of its original length to make sure all inner-order flux is masked.

The bad-pixel mask is merged with this inner-order mask. For each row in the masked frame, a bspline is fitted to the unmasked pixel fluxes to model the shape of the scattered background light along the row.

For each row, flux-values are generated for all pixels in the row using the bspline fit and added to a first pass model background image.

It’s essential that background fitting is accurate within the order locations, but not outside of these areas, so there is no cause for concern if the fit is poor at the edges and corners of the image.

A median-filter is applied to the image to remove the structure resulting from a bad row fits (light and dark lines along the x-axis).

Finally, the modelled background image is subtracted from the original frame:

`detrend` - SEMI-COMPLETED ∞

For the UVB-VIS arm we will often need to scale the master-dark frame to match the exposure time of our science/calibration frame. If this is the case than the master-bias frame needs to be subtracted from the master-dark frame before scaling. However, if the master-dark frame has the same exposure time as your science/calibration frame then it can be subtracted directly from frame as this serves to remove the bias and dark-current contributions simultaneously.

This logic is all housed within the detrend method.

Files ∞

Files generated by SOXSPIPE follow a strict naming scheme.

Product Files ∞

Static Calibration Files ∞

Pinhole Map ∞

The Pinhole Map is a static calibration file that provides a list wavelength \(\lambda\), order number \(n\) and slit position \(s\) of the arc-lines from a multi-pinhole exposure alongside a first approximation of their (\(X, Y\)) pixel-positions on the detector.

Detector Parameters ∞

This is a yaml file hosting the fixed characteristics of the soxs detectors such as gain, science-pixel indexes and image orientations.

Having these characteristics in a plain-text yaml file allows them to be abstracted from the main code and helps maintain a single-source of truth.

Spectral Format Table ∞

The spectral format tables are static calibration files that provide some overview information about how each order typically presents itself on the plain of the detector. Useful information includes the minimum and maximum wavelength limits for each order. Here is an example of the XShooter UVB spectral format table contents:

ORDER,LAMP,WLMINFUL,WLMIN,WLMAX,WLMAXFUL,DISP_MIN,DISP_MAX,LFSR,UFSR
13,QTH,536.57666,544.63275,590.0,603.15594,19.0,2999.0,547.52716,591.32935
14,QTH,499.57135,505.7841,550.3,558.4777,120.0,2999.0,509.7667,547.52716
...

Intermediate Files ∞

Master Bias ∞

The master bias frames are the product of the soxs_mbias recipe. The master-bias frame is to be subtracted from science/calibration frames to remove the contribution to pixel counts resulting from the bias-voltage.

Master Dark ∞

The master dark frames are the product of the soxs_mdark recipe. The master-dark frame is to be subtracted from science/calibration frames to remove the contribution to pixel counts resulting from the dark-current.

Prepared Frame ∞

Dispersion Map ∞

In the case of soxs_disp_solution the output dispersion map hosts the coefficients (\(c_{ij}\)) to two polynomials that describe the global dispersion solution for the entire detector frame:

\[\begin{split} X = \sum\limits_{ij} c_{ij} \times n^i \times \lambda^j \\ \end{split}\]

\[\begin{split} Y = \sum\limits_{ij} c_{ij} \times n^i \times \lambda^j \\ \end{split}\]

soxs_spatial_solution, building from this dispersion solution, provides global dispersion and spatial solution. The dispersion map output by this recipe hosts the coefficients (\(c_{ijk}\)) to two polynomials (note now the inclusion of slit position):

\[\begin{split} X = \sum\limits_{ijk} c_{ijk} \times n^i \times \lambda^j \times s^k \\ \end{split}\]

\[\begin{split} Y = \sum\limits_{ijk} c_{ijk} \times n^i \times \lambda^j \times s^k \\ \end{split}\]

Order Table ∞

The order table gets built over soxs_order_centres and soxs_mflat recipes. The final product contains polynomial fits for the order centres and the upper and lower edges of the order locations on the detector.

Here is an example of the content of the NIR order table from the soxs_order_centres recipes.

order,degy,CENT_c0,CENT_c1,CENT_c2,CENT_c3
11,3,787.9616,0.387875383,-0.000175467653,7.67345562e-10
12,3,714.084224,0.392816612,-0.000177468713,1.24058095e-09
...

Todo

add order table example after soxs_mflat

Release Notes ∞

v0.10.2 - April 23, 2024 ∞

ENHANCEMENT: the calibration lamp name is now added to the sof filenames, and hence the product file names
ENHANCEMENT: file summary now shows which calibration lamps are used
ENHANCEMENT: adding bad-pixel maps for SOXS detectors (currently blank)
ENHANCEMENT: pipeline can select default settings for either soxs or xsh (previously there was only one default settings file)
FIXED: SOXS VIS darks are now getting split by EXPTIME in data-organiser

v0.10.1 - April 11, 2024 ∞

FEATURE: the data-organiser has been ‘plumbed’ to work with SOXS data (will now work with Xshooter or SOXS data).
ENHANCEMENT: clipping of entire MPH set based on their combined RMS scatter from their predicted locations. MPH sets with large scatter are consider poor and removed before polynomial fitting.
ENHANCEMENT: option added to relevant recipes settings to allow toggling of fitting and subtracting of intra-order scattered background light (subtract_background)
REFACTOR: added arm and lamp to QC plot titles.
REFACTOR recipe settings can now be set independently for each arm.
REFACTOR: fitting of the scatter background light is now much more robust.
REFACTOR: The scattered light background images are now saved as QC PDFs instead of FITS frames.
FIXED: fixed issue where logs were getting duplicated.
FIXED: the scaling and stitching together of the UVB D2 and QTH lamp flats.

v0.10.0 - February 20, 2024 ∞

a bootstrap_dispersion_solution has been added to the advanced settings. It this is set to True, the pipeline will attempt to bootstrap the initial dispersion solution (using the static line list) with lines from a line-atlas. The line-atlas contains more lines and lines with more precise wavelength measurements.
FEATURE: a new ‘reducer’ module and terminal command replace the old _reduce_all/sh script. This allows the data-organiser to dynamically self-correct if a recipe fails.
ENHANCEMENT: robustness fixes and updates.
pinhole_fwhm_px_min and pinhole_fwhm_px_max settings added to soxs-spatial-solution. Detected pinholes with a FWHM below/above these values get clipped.
FIXED: The bad-pixel mask from the noise map of the mbias frame is now injected mbias product. The Xshooter UVB electron trap is now clearly visible in the master bias quality extension.
mph_line_set_min setting added to soxs-spatial-solution. Full multi-pinholes sets (same arc line) with fewer than mph_line_set_min lines detected get clipped.

v0.9.9 - January 24, 2024 ∞

FIXED: bug fix logger

v0.9.8 - January 19, 2024 ∞

FIXED: bug fix in collecting settings files from the default location

v0.9.7 - December 7, 2023 ∞

ENHANCEMENT: the instrument name is now included in the SOF & product filename.
ENHANCEMENT: setting bad pixels to zero in sky-subtracted product frames.
FIXED: blocking filters now taken into account when building the master-flats and determining the order-edges.
FIXED: master flats taken with blocking filters are no longer matched with multi-pinhole frames fro the spatial solution recipe.
FIXED: master flats with identical slit-widths now matched against science frames within the data-organiser when building SOF files.
FIXED: the order of the columns in the extracted & merged spectrum tables is now WAVE, FLUX (was FLUX, WAVE).
FIXED: specutils dependency added to conda-forge requirements.

v0.9.4 - December 5, 2023 ∞

REFACTOR: orders are now clipped so that only the pixels deemed to be within the flux receiving regions of the order are extracted (according to the static calibration spectral format table).
REFACTOR:soxspipe prep will now warn user if no FITS files are found in the workspace directory and quit before moving any files (safer).
REFACTOR:soxspipe session commands will look for a sessions directory before creating any new files and folders (cleaner).
REFACTOR:read_spectral_format function can now return limits to the usable region of each spectral order if a dispersion map is given.
FIXED: fixes to make detect_continuum more robust.

v0.9.2 - November 29, 2023 ∞

ENHANCEMENT: intra-order background (scattered light) fits are now being written to FITS image files in the QC directories and reported at the end of a recipe run.
ENHANCEMENT: added a create_dispersion_solution_grid_lines_for_plot function to allow adding dispersion solution grid to QC plots. This is extremely useful for quickly diagnosing problems with the fits.
REFACTOR: All product FITS files now pass fitverify without error or warnings. All issues were due to using ‘-‘ instead of underscores in FITS binary table column names.
REFACTOR: bad-pixel values set to 0 in data extensions of products
REFACTOR: nans have been replaced by zero in FITS image product
FIXED: a mismatch between daofind results and the original input pixel table was causing dispersion solution to break (a recent bug introduced during code optimisations)
FIXED: the internal soxspipe logger was being interfered with by astropy so that logs were sometimes getting redirected to the wrong place

v0.9.0 - October 11, 2023 ∞

FEATURE: added a predict_product_path function to determine the product path from a recipe’s input sof file
FEATURE: Merging of individual order extracted spectra from object frame into a single spectrum for each arm
FEATURE: Object spectra are now extracted from the sky-subtracted frames using the Horne 86 method
FEATURE: Real SOXS data is now included in the unit-test suite (starting to replace simulated data unit-tests). soxs-disp-solu recipe so far.
FEATURE: SOXS NIR Xe line-lists added to static-calibration suite (single and multi pinhole).
FEATURE: when running a recipe, soxspipe writes informative logs to stdoutAND to a log file adjacent to the recipe’s product file(s). Error logs are also written if a recipe fails (see docs).
ENHANCEMENT: recipe timing added to the end of the logs
ENHANCEMENT: fitted lines from the dispersion solution are written out to file as a QC product
ENHANCEMENT: flux (and other daostarfinder metrics) are now recorded in the detected line-list QC file. This will help measure degradation of arc-lamps over time.
ENHANCEMENT: FWHM and pixel-scale added to fitted lines from the dispersion solution
ENHANCEMENT: legends added to many of the QC plots
ENHANCEMENT: OB ids now getting add to the data-organiser database tables.
ENHANCEMENT: object trace FITS binary table added to stare-mode products (alongside complimentary QC plot)
ENHANCEMENT: products and QC outputs are differentiated in the table reported upon recipe completion (see label column).
ENHANCEMENT: verifying the master flat used to calibrate object/std spectra has the same slit-witdh as used to take the science frames
REFACTOR:init command has been subsumed into the prep command. The prep command will generate a settings file to live within the prepared workspace.
REFACTOR:misc/ directory created by data-organiser even if empty
REFACTOR: close matplotlib plot after writing plots to file
REFACTOR: command-line startup speeds improved
REFACTOR: continuum fitting code made more robust against edge cases (orders of the fit are automatically reduced if fit does not converge)
REFACTOR: soxspipe now has a ‘full’ and a ‘lite’ test-suite. Using the lite suite will speed up deploying of new releases.
DOCS: updated docs with a more robust SOXSPIPE upgrade path (users having issue with conda update ...)
FIXED: sky-subtraction code and data-organiser fixed to work with binned data

v0.8.0 - May 18, 2023 ∞

FEATURE: we now have a data-organiser to sort data, prepare the required SOF files and generate reduction scripts.
ENHANCEMENT: ‘.db’, ‘.yaml’, ‘.sh’ and ‘.log’ extensions skipped when moving items to the misc folder
ENHANCEMENT: move information printed to STDOUT when preparing a workspace to inform the user of how the data is organised
ENHANCEMENT: code can automatically adjust polynomial fitting parameters to find a dispersion solution if those provided in the settings file fail.
ENHANCEMENT: uncompression of fits.Z files (if any) occurs before data-organising
REFACTOR: speed & robustness improvements to dispersion solution to 2D image map conversion.
REFACTOR: much fast check for product existence so recipes are quickly skipped if they have already run.
REFACTOR: removed the intermediate-data-root setting renamed to a more accurate workspace-root-dir
REFACTOR: removed the reduced-data-root setting.
REFACTOR: updating all depreciated pandas commands so pipeline is now compatible with 1.X and 2.X versions of pandas
FIXED pandas 1.X and pandas 2.X were doing different things when renaming columns in data-frames. Both 1.X and 2.X now work in the pipeline.

v0.7.2 - March 3, 2023 ∞

REFACTOR: Big improvements on sky-subtraction
REFACTOR: UV order-edge detection more robust
REFACTOR: changed quickstart guide compress to gzipped tar
REFACTOR: updated default settings to be more robust

v0.7.1 - November 4, 2022 ∞

FEATURE: UV D-Lamp and QTH-Lamp master flats now being stitched together
FEATURE: errors in error maps now being treated correctly and propagating to combined images
FEATURE: Pipeline can now ‘remember’ where it left off in the reduction cascade. If it has run a recipe before it will exit with a message to the user informing them how to force the recipe to rerun.
FEATURE: added a twoD_disp_map_image_to_dataframe function to toolkit
ENHANCEMENT: PRO CATG now written to product FITS header
ENHANCEMENT: Handling of binned images when generating flats and order-locations
ENHANCEMENT: Where possible, product files are given the same name as the SOF file used to generate them (replacing .sof extension with .fits)
ENHANCEMENT: SOF files can now contain a file ‘tag’ to allow users to read the SOF file contents and know exactly which files are being passed to the recipe (e.g. MASTER_BIAS_UVB, LAMP,DORDERDEF_UVB … )
ENHANCEMENT: dispersion solution now working with simulated NIR SOXS data
ENHANCEMENT: quicklook now renders dispersion solution grid
ENHANCEMENT: ~40% speed gain in combining images.
REFACTOR: 2D Map generation now ~6-8 times faster (seeding solutions with nearest neighbour with cubic spline method)
REFACTOR: SOF filenames reworked to contain the UTC observation date instead of MJD (more in-line with ESO ecosystems)
REFACTOR: updated workflow for master bias combination
REFACTOR: updated workflow for master dark combination
REFACTOR: QC PDF plots now added to their own directory separate from the products
REFACTOR: products now sub-divided into recipe directories (e.g. ./products/soxs-mbias/)
DOCS: mflat docs brought up-to-date
DOCS: mflat docs brought up-to-date
FIXED: mflat recipe now exits if flat frames are not of a consistent exptime.

v0.6.2 - April 13, 2022 ∞

ENHANCEMENT: quickstart guide added for calibration recipes
FEATURE: QCs added for dispersion solution and order centre recipes
REFACTOR: clean up of stdout information

v0.6.1 - April 11, 2022 ∞

FEATURE: shipping static calibration files with the code (one less thing for end-users to install and set-up)

v0.6.0 - April 10, 2022 ∞

This is only a summary of some of the updates included in this release:

ENHANCEMENT: All CSV files moved to FITS binary tables - metadata very useful for developing data organiser
FEATURE: 2D image map now created by create_dispersion_solution subtract_calibrations util renamed to detrend and added ability to flat correct
FEATURE: 2D image map of wavelength values, slit-position values and order values written alongside polynomial solutions of full dispersion solution
FEATURE: soxspipe now on conda
FEATURE: QCs now being written to FITS header
FEATURE: adding QC and product collection in mbias recipe
ENHANCEMENT RON and bias structure QCs now reported by mbias
ENHANCEMENT nan ignored when scaling quicklook images
ENHANCEMENT RON and bias structure QCs now reported by mdark
ENHANCEMENT: QCs have an option to NOT (to_header) write to FITS header (default is to write)
REFACTOR: better treatment of masked pixels when stacking images (e.g. in mbias and mdark)
REFACTOR: removed raw frame reports and neater QC table
REFACTOR: fits header keywords neatly sorted before writing to file
FIX: Correct management of mask when determining RON on bias and darks

v0.5.1 - September 29, 2021 ∞

FEATURE: recipes now have a qc and products attribute. These are pandas data frames used to collect QCs and generated products throughout the life-time of the recipe. They are printed to STDOUT at the end of the recipe (can be used in the future to send post request to health monitor API with JSON content in request body).
ENHANCEMENT added code-base to conda-forge
ENHANCEMENT added bottleneck to the install requirement (makes image combination more efficient)
ENHANCEMENT masked pixel now coloured red in quicklook plots (easier to differentiate from good pixels)
ENHANCEMENT low-sensitivity pixels in lamp-flats now identified and added to bad-pixel mask
ENHANCEMENT add a verbosity flag to the command-line and a verbose parameter to each recipe
REFACTOR inter-order pixel value in flats now set to unity (instead of running background fitting and subtraction)
REFACTOR: recipes now have their recipe name as a recipeName attribute

v0.5.0 - June 10, 2021 ∞

FEATURE Added a new filenamer module that implements a strict intermediate and reduced file-naming scheme
FEATURE:soxs_mflat recipe now included
FEATURE:soxs_spatial_solution recipe is now included
FEATURE:subtract_background utility added
FEATURE: added a detect_order_edges object
FEATURE: Added a dispersion_map_to_pixel_arrays function to convert from order-based and wavelength arrays to pixel arrays (first guess dispersion map only so far)
FEATURE: added a quicklook function in toolkit to quickly visualise a frame
FEATURE: added a toolkit module for small functions used throughout soxspipe
FEATURE: added function in toolkit to unpack an order table into lists of coordinates, one list per order.
FEATURE: added image slice tool to toolkit
ENHANCEMENT Added a -o <outputDirectory> switch to the command-line to optionally override the ‘intermediate-data-root’ setting in the settings file.
ENHANCEMENT: added a fraction of a second tolerance when matching exptimes between darks and science/calibration frames
ENHANCEMENT: y limits now added to the order table to show limits of order locations on detector
REFACTOR: Change the “SOXSPIPE PRE” date stamp keyword to “SXSPRE” to future-proof for phase III (8 character keyword limit)
REFACTOR: Pandas tables are now used through-out code to pass line-lists between methods
REFACTOR: refactoring of polynomial fitting has made creation of dispersion maps ~50 times faster
REFACTOR: removed OBID from file names and added readout mode. This information is more helpful at the glance.
FIXED: correct binning reported in product file names
FIXED: lines in a sof file beginning with a # are considered as comments and therefore ignored by the pipeline.

v0.4.1 - September 15, 2020 ∞

FEATURE: add command-line util for soxs order_centres recipe
FEATURE added the detect_continuum utility to fit order centre locations in single pinhole flat frames.
ENHANCEMENT: added a supplementary file list for non-fits input files in set-of-file util
ENHANCEMENT: adding more information residual plots & visualisation of fitting for disp solution
ENHANCEMENT: check that files in the sof files exist before proceeding.
ENHANCEMENT: added spectral format table lookup to detector settings file
REFACTOR: moved chebyshev order/wavelength polynomials into its own class - decoupled from create_dispersion_map class

v0.4.0 - September 3, 2020 ∞

FEATURE: added create_dispersion_map class to be used in soxs_disp_solution and soxs_spatial_solution
FEATURE: added a detrend method to subtract calibration frames (bias and dark) from an input frame
FEATURE: added the dispersion solution recipe and unit tests
FEATURE: added the disp_solution command-line tool
DOCS: major docs overhaul
ENHANCEMENT: added predicted lines lists to detector parameter file
ENHANCEMENT: DPR CATG and DPR TECH added to metadata of sof imagefilecollection objects
ENHANCEMENT: wcs copied from a single frame into the combined frames during clip and stack
REFACTOR: bad-pixel map paths abstracted to detector settings files
REFACTOR: renaming of unit-testing test data directories
REFACTOR: only filenames reported by sof summaries when files are found in the same directory (easier to read on terminal)
FIXED: fixed detector science pixels for UVB

v0.3.1 - August 25, 2020 ∞

FEATURE: recipe & command-line tool for master dark creation (mdark)
ENHANCEMENT: default binning add to detector settings file
ENHANCEMENT: added mixed exposure time unit test for dark-frames
ENHANCEMENT: added default values for gain and ron in the detector settings files. Default values can be overwritten if correct GAIN and RON are found in fits-headers (overwritten for UVB and VIS but not NIR for XShooter)
ENHANCEMENT: can now interrupt “~” as home directory in sof file path
FIXED: binning factor used when trimming frames
FIXED: the trimming dimensions of NIR frames - bad-pixel map now aligns correctly with data frame
FIXED: science pixels for all 3 xshooter detectors in parameters file

v0.3.0 - August 18, 2020 ∞

FEATURE: added a write method to the _base_recipe to write frames to disk (renames extensions to ESO preferred naming scheme)
FEATURE: detector lookup class added alongside yaml files to host detector specific parameters (rotation, science-pixels etc). Code has been updated to remove hard-wired detector values.
FEATURE: added a cleanup method to remove intermediate file once recipe completes
ENHANCEMENT: parameters for clip and stack method added to the settings files
ENHANCEMENT: added strict typing of data and variables with astropy units to avoid silent mistakes in frame arithmetic
ENHANCEMENT: added mixing of readout speeds to input frame verification checks
ENHANCEMENT: added readnoise and gain to the list of keyword values to check during frame verification
ENHANCEMENT: inject a ‘SOXSPIPE PRE’ keyword with timestamp value into prepared frames
ENHANCEMENT: check frames for ‘SOXSPIPE PRE’ keyword before preparing - raises exception if found
ENHANCEMENT: ron and gain are added to the recipe’s detector lookup dictionary during frame verification (so they don’t need read again later)
REFACTOR: moved stacking code to it own clip_and_stack method hosted in the _base_recipe
REFACTOR: moved basic input frame verifications to the _base_recipe - so not to repeat code
REFACTOR: removed python 2.7 support - not feasible with CCDProc
DOCS: added workflow diagrams to the documentation for many of the methods implemented (prepare_frames(), clip_and_stack)

v0.2.0 - February 27, 2020 ∞

FEATURE added keyword lookups - abstracting exact keyword names from code

Modules ∞

`soxspipe.commonutils`	common tools used throughout package
`soxspipe.recipes`	The pipeline recipes
`soxspipe.commonutils.polynomials`	definition of polynomial functions needed throughout code
`soxspipe.commonutils.toolkit`	small reusable functions used throughout soxspipe
`soxspipe.utKit`	Unit testing tools

Classes ∞

`soxspipe.commonutils.create_dispersion_map`	detect arc-lines on a pinhole frame to generate a dispersion solution
`soxspipe.commonutils.data_organiser`	The worker class for the data_organiser module
`soxspipe.commonutils.detect_continuum`	*find and fit the continuum in a pinhole flat frame with low-order polynomials.
`soxspipe.commonutils.detect_order_edges`	using a fully-illuminated slit flat frame detect and record the order-edges
`soxspipe.commonutils.detector_lookup`	return a dictionary of detector characteristics and parameters
`soxspipe.commonutils.flux_calibration`	The worker class for the flux_calibration module
`soxspipe.commonutils.horne_extraction`	perform optimal source extraction using the Horne method (Horne 1986)
`soxspipe.commonutils.keyword_lookup`	The worker class for the keyword_lookup module
`soxspipe.commonutils.polynomials.chebyshev_order_wavelength_polynomials`	the chebyshev polynomial fits for the single frames; to be iteratively fitted to minimise errors
`soxspipe.commonutils.polynomials.chebyshev_order_xy_polynomials`	the chebyshev polynomial fits FIX ME
`soxspipe.commonutils.polynomials.chebyshev_xy_polynomial`	the chebyshev polynomial fits for the pinhole flat frame order tracing; to be iteratively fitted to minimise errors
`soxspipe.commonutils.reducer`	reduce all the data in a workspace, or target specific obs and files for reduction
`soxspipe.commonutils.response_function`	The worker class for the response_function module
`soxspipe.commonutils.subtract_background`	fit and subtract background flux from scattered light from frame
`soxspipe.commonutils.subtract_sky`	Subtract the sky background from a science image using the Kelson Method
`soxspipe.commonutils.toolkit.MaxFilter`
`soxspipe.recipes.soxs_disp_solution`	generate a first approximation of the dispersion solution from single pinhole frames
`soxspipe.recipes.soxs_mbias`	The `soxs_mbias` *recipe is used to generate a master-bias frame from a set of input raw bias frames.
`soxspipe.recipes.soxs_mdark`	The soxs_mdark recipe
`soxspipe.recipes.soxs_mflat`	The soxs_mflat recipe
`soxspipe.recipes.soxs_order_centres`	The soxs_order_centres recipe
`soxspipe.recipes.soxs_spatial_solution`	The soxs_spatial_solution recipe
`soxspipe.recipes.soxs_stare`	The soxs_stare recipe
`soxspipe.recipes.soxs_straighten`	The soxs_straighten recipe

Functions ∞

`soxspipe.commonutils.dispersion_map_to_pixel_arrays`	use a first-guess dispersion map to append x,y fits to line-list data frame.
`soxspipe.commonutils.filenamer`	Given a FITS object, use the SOXS file-naming scheme to return a filename to be used to save the FITS object to disk
`soxspipe.commonutils.getpackagepath`	Get the root path for this python package
`soxspipe.commonutils.toolkit.add_recipe_logger`	add a recipe-specific handler to the default logger that writes the recipe’s logs adjacent to the recipe project
`soxspipe.commonutils.toolkit.create_dispersion_solution_grid_lines_for_plot`	give a dispersion solution and accompanying 2D dispersion map image, generate the grid lines to add to QC plots
`soxspipe.commonutils.toolkit.cut_image_slice`	cut and return an N-pixel wide and M-pixels long slice, centred on a given coordinate from an image frame
`soxspipe.commonutils.toolkit.generic_quality_checks`	measure very basic quality checks on a frame and return the QC table with results appended
`soxspipe.commonutils.toolkit.get_calibration_lamp`	given a frame, determine which calibration lamp is being used
`soxspipe.commonutils.toolkit.get_calibrations_path`	return the root path to the static calibrations
`soxspipe.commonutils.toolkit.predict_product_path`	predict the path of the recipe product from a given SOF name
`soxspipe.commonutils.toolkit.quicklook_image`	generate a quicklook image of a CCDObject - useful for development/debugging
`soxspipe.commonutils.toolkit.read_spectral_format`	read the spectral format table to get some key parameters
`soxspipe.commonutils.toolkit.spectroscopic_image_quality_checks`	measure and record spectroscopic image quailty checks
`soxspipe.commonutils.toolkit.twoD_disp_map_image_to_dataframe`	convert the 2D dispersion image map to a pandas dataframe
`soxspipe.commonutils.toolkit.unpack_order_table`	*unpack an order table and return a top-level `orderPolyTable` data-frame and a second `orderPixelTable` data-frame with the central-trace coordinates of each order given
`soxspipe.commonutils.uncompress`	uncompress ESO fits.Z frames

A-Z Index ∞

Acknowledgements ∞

SOXSPIPE makes use of ccdproc, an Astropy package for image reduction (Craig et al. 2021).

soxspipe ∞

Installation ∞

How to cite soxspipe ∞

Quickstart Guide ∞

Install ∞

Demo Data ∞

Preparing the Data-Reduction Workspace ∞

Reduce the Data ∞

Logging ∞

Data Reduction Sessions ∞

reduce ∞

A Primer on SOXS Observation Modes ∞

Recipes ∞

Standard Calibrations ∞

soxs_mbias ∞

Input ∞

Parameters ∞

Method ∞

Output ∞

QC Metrics ∞

Recipe API ∞

soxs_mdark ∞

Input ∞

Parameters ∞

Method ∞

Output ∞

QC Metrics ∞

Recipe API ∞

soxs_mflat ∞

Input ∞

Parameters ∞

Method ∞

Output ∞

QC Metrics ∞

Recipe API ∞

Dispersion and Spatial Solutions ∞

soxs_disp_solution ∞

Input ∞

Parameters ∞

Method ∞

Output ∞

QC Metrics ∞

Recipe API ∞

soxs_order_centres ∞

Input ∞

Parameters ∞

Method ∞

Output ∞

QC Metrics ∞

Recipe API ∞

soxs_spatial_solution - PLANNED ∞

Input ∞

Method ∞

Output ∞

QC Metrics ∞

Recipe API ∞

soxs_straighten - PLANNED ∞

Input ∞

Parameters ∞

Method ∞

Output ∞

QC Metrics ∞

Recipe API ∞

Utilities ∞

clip_and_stack ∞

create_dispersion_map ∞

2D Image Map ∞

create_noise_map - PLANNED ∞

detect_continuum ∞

detect_order_edges – COMPLETED ∞

detector_lookup ∞

filenamer ∞

keyword_lookup ∞

prepare_frames ∞

set_of_files ∞

subtract_background ∞

detrend - SEMI-COMPLETED ∞

Files ∞

Product Files ∞

Static Calibration Files ∞

`reduce` ∞

`soxs_mbias` ∞

`soxs_mdark` ∞

`soxs_mflat` ∞

`soxs_disp_solution` ∞

`soxs_order_centres` ∞

`soxs_spatial_solution` - PLANNED ∞

`soxs_straighten` - PLANNED ∞

`clip_and_stack` ∞

`create_dispersion_map` ∞

`create_noise_map` - PLANNED ∞

`detect_continuum` ∞

`detect_order_edges` – COMPLETED ∞

`detector_lookup` ∞

`filenamer` ∞

`keyword_lookup` ∞

`prepare_frames` ∞

`set_of_files` ∞

`subtract_background` ∞

`detrend` - SEMI-COMPLETED ∞