soxspipe.commonutils.create_dispersion_map

detect arc-lines on a pinhole frame to generate a dispersion solution

Author

Marco Landoni & David Young

Date Created

September 1, 2020

Module Structure

This module contains the main create_dispersion_map class and helper functions:

Main Class:

• create_dispersion_map - Generates dispersion solutions from pinhole arc frames

Helper Functions:

• measure_line_position() - Detect and measure arc line positions on image stamps

• straighten_mph_sets() - Straighten multi-pinhole sets by projecting onto fitted line

• find_largest_cluster_center() - Find center of largest cluster using DBSCAN

• _plot_slit_index_comparisons() - Debug visualization of slit position residuals

Workflow:

1. Load predicted line positions from static calibrations

2. Detect arc lines on pinhole frame iteratively

3. Calculate shifts between predicted and observed positions

4. Fit polynomial dispersion solution (wavelength, order, slit)

5. Generate QC plots and write output files

Module Contents

Classes

create_dispersion_map

detect arc-lines on a pinhole frame to generate a dispersion solution

Functions

find_largest_cluster_center	Find the center of the largest cluster in (x, y) data using DBSCAN
measure_line_position	measure the line position on a given stamp
straighten_mph_sets	Straighten multi-pinhole sets by projecting points onto fitted line

API

class soxspipe.commonutils.create_dispersion_map.create_dispersion_map(log, settings, recipeSettings, pinholeFrame, firstGuessMap=False, orderTable=False, qcTable=False, productsTable=False, sofName=False, create2DMap=True, lineDetectionTable=False, startNightDate='', arcFrame=None, debug=False, turnOffMP=False)

Bases: object

detect arc-lines on a pinhole frame to generate a dispersion solution

Key Arguments:

• log – logger

• settings – the settings dictionary

• recipeSettings – the recipe specific settings

• pinholeFrame – the calibrated pinhole frame (single or multi)

• firstGuessMap – the first guess dispersion map from the soxs_disp_solution recipe (needed in soxs_spat_solution recipe). Default False.

• orderTable – the order geometry table

• qcTable – the data frame to collect measured QC metrics

• productsTable – the data frame to collect output products

• sofName – name of the originating SOF file

• create2DMap – create the 2D image map of wavelength, slit-position and order from disp solution.

• lineDetectionTable – the list of arc-lines detected on the pinhole frame (used only for pipeline tuning)

• startNightDate – YYYY-MM-DD date of the observation night. Default “”

• arcFrame – the calibrated arc frame used to determine spectral resolution. Default None

• debug – debug mode. Default False

• turnOffMP – turn off multiprocessing. Default False.

Usage:

from soxspipe.commonutils import create_dispersion_map
mapPath, mapImagePath, res_plots, qcTable, productsTable = create_dispersion_map(
    log=log,
    settings=settings,
    pinholeFrame=frame,
    firstGuessMap=False,
    qcTable=self.qc,
    productsTable=self.products,
    sofName=sofName,
    create2DMap=True,
    recipeSettings=recipeSettings,
    startNightDate=startNightDate,
    arcFrame=arcFrame,
    debug=debug,
    turnOffMP=turnOffMP
).get()

Initialization

calculate_residuals(orderPixelTable, xcoeff, ycoeff, orderDeg, wavelengthDeg, slitDeg, writeQCs=False, pixelRange=False)

calculate residuals of the polynomial fits against the observed line positions

Key Arguments:

• orderPixelTable – the predicted line list as a data frame

• xcoeff – the x-coefficients

• ycoeff – the y-coefficients

• orderDeg – degree of the order fitting

• wavelengthDeg – degree of wavelength fitting

• slitDeg – degree of the slit fitting (False for single pinhole)

• writeQCs – write the QCs to dataframe? Default False

• pixelRange – return centre pixel and ± 2nm from the centre pixel (to measure the pixel scale)

Return:

• residuals – combined x-y residuals

• mean – the mean of the combine residuals

• std – the stdev of the combine residuals

• median – the median of the combine residuals

convert_and_fit(order, bigWlArray, bigSlitArray, slitMap, wlMap, iteration, plots=False)

convert wavelength and slit position grids to pixels

Key Arguments:

• order – the order being considered

• bigWlArray – 1D array of all wavelengths to be converted

• bigSlitArray – 1D array of all split-positions to be converted (same length as bigWlArray)

• slitMap – place-holder image hosting fitted pixel slit-position values

• wlMap – place-holder image hosting fitted pixel wavelength values

• iteration – the iteration index (used for CL reporting)

• plots – show plot of the slit-map

Return:

• orderPixelTable – dataframe containing unfitted pixel info

• remainingCount – number of remaining pixels in orderTable

Usage:

orderPixelTable = self.convert_and_fit(
        order=order, bigWlArray=bigWlArray, bigSlitArray=bigSlitArray, slitMap=slitMap, wlMap=wlMap)

create_new_static_line_list(dispersionMapPath)

using a first pass dispersion solution, use a line atlas to generate a more accurate and more complete static line list

Key Arguments:

• dispersionMapPath – path to the first pass dispersion solution

Return:

• newPredictedLineList – a new predicted line list (to replace the static calibration line-list)

create_placeholder_images(order=False, plot=False, reverse=False)

create CCDData objects as placeholders to host the 2D images of the wavelength and spatial solutions from dispersion solution map

Key Arguments:

• order – specific order to generate the placeholder pixels for. Inner-order pixels set to nan, else set to 0. Default False (generate all orders)

• plot – generate plots of placeholder images (for debugging). Default False.

• reverse – Inner-order pixels set to 0, else set to nan (reverse of default output).

Return:

• slitMap – placeholder image to add pixel slit positions to

• wlMap – placeholder image to add pixel wavelength values to

Usage:

slitMap, wlMap, orderMap = self._create_placeholder_images(order=order)

detect_pinhole_arc_lines(orderPixelTable, iraf=True, sigmaLimit=3, iteration=False, brightest=False, exclude_border=False, returnAll=False)

detect the observed position of an arc-line given the predicted pixel positions

Key Arguments:

• orderPixelTable – the initial line list

• iraf – use IRAF star finder to generate a FWHM

• sigmaLimit – the lower sigma limit for arc line to be considered detected

• iteration – which detect and shift iteration are we on?

• brightest – find the brightest source?

• exclude_border – exclude border pixels from detection?

Return:

• predictedLine – the line with the observed pixel coordinates appended (if detected, otherwise nan)

fit_polynomials(orderPixelTable, wavelengthDeg, orderDeg, slitDeg, missingLines=False)

iteratively fit the dispersion map polynomials to the data, clipping residuals with each iteration

Key Arguments:

• orderPixelTable – data frame containing order, wavelengths, slit positions and observed pixel positions

• wavelengthDeg – degree of wavelength fitting

• orderDeg – degree of the order fitting

• slitDeg – degree of the slit fitting (0 for single pinhole)

• missingLines – lines not detected on the image

Return:

• xcoeff – the x-coefficients post clipping

• ycoeff – the y-coefficients post clipping

• goodLinesTable – the fitted line-list with metrics

• clippedLinesTable – the lines that were sigma-clipped during polynomial fitting

get()

generate the dispersion map

Return:

• mapPath – path to the file containing the coefficients of the x,y polynomials of the global dispersion map fit

Workflow:

1. LOAD PREDICTED LINE POSITIONS FROM CALIBRATION FILES

2. PREPARE PINHOLE FRAME (MASKING, STATISTICS)

3. ITERATIVELY DETECT ARC LINES ON FRAME

4. FIT POLYNOMIAL DISPERSION SOLUTION

5. WRITE OUTPUTS (LINE LISTS, MAP FILES, QC PLOTS)

get_predicted_line_list()

lift the predicted line list from the static calibrations

Return:

• orderPixelTable – a panda’s data-frame containing wavelength,order,slit_index,slit_position,detector_x,detector_y

map_to_image(dispersionMapPath, orders=False)

convert the dispersion map to images in the detector format showing pixel wavelength values and slit positions

Key Arguments:

• dispersionMapPath – path to the full dispersion map to convert to images

• orders – orders to add to the image map. List. Default False (add all orders)

Return:

• dispersion_image_filePath – path to the FITS image with an extension for wavelength values and another for slit positions

Usage:

mapImagePath = self.map_to_image(dispersionMapPath=mapPath)

order_to_image(orderInfo)

convert a single order in the dispersion map to wavelength and slit position images

Key Arguments:

• orderInfo – tuple containing the order number to generate the images for, the minimum wavelength to consider (from format table) and maximum wavelength to consider (from format table).

Return:

• slitMap – the slit map with order values filled

• wlMap – the wavelengths map with order values filled

Usage:

slitMap, wlMap = self.order_to_image(order=order,minWl=minWl, maxWl=maxWl)

update_static_line_list_detector_positions(originalOrderPixelTable, dispersionMapPath)

using a first pass dispersion solution, update the original static line list

Key Arguments:

• originalOrderPixelTable – original order pixel table

• dispersionMapPath – path to the first pass dispersion solution

Return:

• updatedLineList – updated static line list

write_map_to_file(xcoeff, ycoeff, orderDeg, wavelengthDeg, slitDeg)

write out the fitted polynomial solution coefficients to file

Key Arguments:

• xcoeff – the x-coefficients

• ycoeff – the y-coefficients

• orderDeg – degree of the order fitting

• wavelengthDeg – degree of wavelength fitting

• slitDeg – degree of the slit fitting (False for single pinhole)

Return:

• disp_map_path – path to the saved file

soxspipe.commonutils.create_dispersion_map.find_largest_cluster_center(x, y, eps=2.0, min_samples=5)

Find the center of the largest cluster in (x, y) data using DBSCAN

Key Arguments:

• x – x-coordinates of points

• y – y-coordinates of points

• eps – maximum distance between points in a cluster

• min_samples – minimum samples required to form a cluster

Returns:

• center_x, center_y – coordinates of largest cluster center, or (None, None) if no cluster found

soxspipe.commonutils.create_dispersion_map.measure_line_position(stampInfo, log, windowHalf, iraf, sigmaLimit, iteration, brightest=False, exclude_border=False, multipinhole=False, returnAll=True, debug=False)

measure the line position on a given stamp

Key Arguments:

• stampInfo – stamp pixels, xlow, xup, ylow, yup

• log – logger

• windowHalf – half of the stamp side

• iraf – run IRAF source detection to get FWHM?

• sigmaLimit – stamp source detection minimum sigma

• brightest – find the brightest source?

• multipinhole – is this a multipinhole frame?

Returns:

• list of detected line positions with properties

soxspipe.commonutils.create_dispersion_map.straighten_mph_sets(group)

Straighten multi-pinhole sets by projecting points onto fitted line

Key Arguments:

• group – dataframe group containing observed_x and observed_y coordinates

Returns:

• group – group with added tilt_corrected_x and tilt_corrected_y columns