SAO/TDC Reticon Spectrum Archive Format

Harvard-Smithsonian Center for Astrophysics
Optical and Infrared Division

M E M O R A N D U M
August 2, 1990

To: Doug Mink, Charles Maxson, John Huchra, Susan Tokarz
Subject: Yet Again REVISED Archive format for Reticon reduced files
From: Bill Wyatt

This documents the archive format for Reticon reduced files on the Telescope Data Center system. The formats for the analysis-related record are set out below.

  1. HEADER (variable length) ASCII
    This record consists of ASCII lines in the form Keyword = Parameter. It is similar to FITS except that the lines are not any particular length and are terminated with a newline (a linefeed on Unix systems). The minimal set of parameters are below. Other parameters, comments, and history lines may be added. As in FITS, the RA, DEC and other strings are enclosed in single quotes.

    Minimum parameters:

         keyword        format           what
         RFN        =   integer        Reduced File Number
         OBJECT     =   'name'         File name
         RA         =   'hh:mm:ss.s'   right ascension
         DEC        =   'dd:mm:ss.s'   declination
         EPOCH      =   float          epoch of RA, Dec in years (e.g. 1950.0)
         EXPTIME    =   float          integration time, seconds
         JDN        =   float          Julian Day Number at end of exposure,
                                       topocentric
         GJDN       =   float          Julian Day Number at middle of
                                       exposure, topocentric
         HJDN       =   float          Julian Day Number at middle of
                                       exposure, heliocentric
         UT         =   'hh:mm:ss'     Universal Time at middle of exposure
         DATE-OBS   =   'dd/mm/yy'     Date of observation
         PI         =   'name'         Principal Investigator
         PROGRAM    =   'string'       Program name or number
         OBSERVER   =   'name'         observer at telescope
         FILTER     =   'string'       Filter name
         END                           Header ending line
    
    
  2. REDUCESUMMARY 96 binary
    This contains various parameters, including some duplicated in the HEADER record, above.
         format        what
         int*4       Reduced File Number
         float*4     RA, radians
         float*4     Dec, radians
         float*4     Epoch of coordinates, years (e.g. 1950.0)
         float*8	Julian Day Number (but see note to HEADER section)
         float*4     Integration time, seconds
         float*4     Heliocentric Velocity correction, km/sec
         int*2       telescope number:
                         0 = unknown
                         1 = FLWO 61cm,
                         2 = FLWO 1.5m
                         3 = MMT
                         4 = ORO 1.5m
                         5 = FLWO 1.2m (48-inch)
                         6 = MMT 6.5m upgraded
         int*2       Grating number (600 lines/mm, etc.)
         int*2       Image tube number
         int*2       Object Category
         float*4     observatory longitude, radians
         float*4     observatory latitude, radians
         int*4       Number of left-slit files minus number of
                     right-slit files (i.e., should be zero)
         float*4*4   Ncheck   RMS, 2-pattern, 4-pattern, 8-pattern
         float*4     Beginning-end comparison shift, pixels
         float*4     Left-right comparison shift, pixels
         float*4     Mean comparison line width
         float*4     Mean sky line width
         float*4     Hour angle, radians
         float*4     Sidereal time, radians
         float*4     air mass
    

  3. REDUCESUMMARY2 120 binary
    This contains various parameters, including some duplicated in the HEADER record, above.
         format        what
         int*4       Reduced File Number
         float*4     RA, radians
         float*4     Dec, radians
         float*4     Epoch of coordinates, years (e.g. 1950.0)
         float*8     HJDN = Heliocentric Julian Day Number
                     (but see note to HEADER section)
         float*4     Integration time, seconds
         float*4     HCV = Heliocentric Velocity correction, km/sec
         int*2       telescope number:
                         0 = unknown
                         1 = FLWO 61cm,
                         2 = FLWO 1.5m
                         3 = MMT
                         4 = ORO 1.5m
                         5 = FLWO 1.2m (48-inch)
                         6 = MMT 6.5m upgraded
         int*2       Grating number (600 lines/mm, etc.)
         int*2       Image tube number
         int*2       Object Category
         float*4     observatory longitude, radians
         float*4     observatory latitude, radians
         int*4       Number of  left-slit  files  minus  number  of
                     right-slit files (i.e., should be zero)
         float*4*4   Ncheck RMS, 2-pattern, 4-pattern, 8-pattern
         float*4     Beginning-end comparison shift, pixels
         float*4     Left-right comparison shift, pixels
         float*4     Mean comparison line width
         float*4     Mean sky line width
         float*4     Hour angle, radians
         float*4     Sidereal time, radians
         float*4     air mass
         float*8     GJDN = Topocentric Julian Day Number
                     (but see note to HEADER section)
         float*4     BCV = Barycentric Velocity correction, km/sec
         float*4     altitude of observatory, meters.
         char*8      spare bytes
    
  4. SKELETON (variable length) ASCII
    Like the HEADER record, the SKELETON record consists of lines of the form Keyword = Parameter. The keywords control the reduction of the raw data, and can be quite extensive. Fortunately, most of the reduction parameters are repeated time after time, so these are gathered up into a so-called `template' file. This skeleton record then refers to a template and contains only the unique information, such as file names, that need be specified for this reduced file.

    The full documentation of the keywords and their effects on reduction will be in another document. Meanwhile, a minimal set is documented below:

         keyword         format         what
         rfn         =   integer      Reduced File Number
         template    =   'pathname'   Template file for defaults
         r5w         =   'pathname'   File to read preliminary  5th-order
                                      wavelength solution from
         rawdir      =   'pathname'   Directory containing raw data files
         telescope   =   integer      Telescope number (used only  if  no
                                      TELESCOP keyword in FITS header)
         grating     =   integer      Grating number
         itube       =   integer      Image tube number
         objtype     =   integer      Object  type  (e.g.,  star  =   32,
                                                  Redshift object = 2)
         incandN     =   'pathname'   Incandescent normalization file  N,
                                      where  N  = 1, 2, etc., relative to
                                      rrrraaaawwwwddddiiiirrrr
         compN       =   'pathname'   Comparison file N (N = 1, 2,  ...),
                                      relative to rrrraaaawwwwddddiiiirrrr
         objectN     =   'pathname'   Object data file N (N = 1, 2, ...),
                                      relative to rrrraaaawwwwddddiiiirrrr
         END                          End marker for skeleton record
    

  5. COMMENTS (variable length) ASCII
    Each line beginning with `COMMENT' in the raw data file FITS header is copied to this record (without the leading `COMMENT'). Leading and trailing blanks are removed and each line is terminated with a newline. The comments end with an `END ' line.

  6. DISTORTION (variable length) binary
    This record, if present, contains the right-to-left distortion removal polynomial determined during the reduction and the basic fitting results.
         format        what
         int *4      Number of lines in fit (i.e. matched  on
                     both sides)
         float*4     RMS residual of fit
         int*4       dimension of polynomial
         pointer*4   Pointer to coefficients in the in-core C
                     poly   structure   (i.e.  irrelevant  to
                     archive file)
         float*8     Midpoint   of    independent    variable
                     (subtracted  off  first  when evaluating
                     the polynomial)
         float*8     Scaling factor for independent  variable
                     (divided    through    after    midpoint
                     subtraction    when    evaluating    the
                     polynomial; zero means no scaling done)
         float*8*N   N coefficients of the polynomial, lowest
                     order term first
    

  7. COARSEWAVER (variable length) binary
    This contains the polynomial used during the wavelength-table-matching phase of the reductions. By default this is a 5th-order polynomial (dimension 6) for a record length of 164 bytes, but if a lower order fit was selected in the reduction, then that will be shorter. The format is similar to that for the distortion polynomial, except the wavelength limits and RMS of the fit are also in the record.
         format        what
         float*4     bluest wavelength line identified
         float*4     reddest wavelength line identified
         float*4     RMS of wavelength fit, Angstroms
         float*4     RMS of wavelength fit, pixels
         int*4       Number of lines in fit
    
                     Pixel -> Wavelength polynomial (WAVER)
         int*4       dimension of WAVER polynomial (nominally 6)
         pointer*4   Pointer to coefficients in the in-core C poly
                     structure (i.e. irrelevant to archive file)
         float*8     Midpoint of independent variable (subtracted off
                     first when evaluating the polynomial), pixels
         float*8     Scaling factor for independent  variable (divided
                     through after midpoint subtraction when evaluating
                     the polynomial; zero means no scaling done)
         float*8*N   N coefficients of the WAVER polynomial (nominally 6),
                     lowest order term first
    
                     Wavelength -> Pixel polynomial (IWAVER)
         int*4       dimension of IWAVER polynomial (nominally 6)
         pointer*4   Pointer to coefficients  in  the  in-core  C poly
                     structure  (i.e. irrelevant to archive file)
         float*8     Midpoint of independent variable (subtracted
                     off  first  when evaluating the polynomial), Angstroms.
         float*8     Scaling  factor  for  independent   variable (divided
                     through after midpoint subtraction when evaluating the
                     polynomial;  zero  means no scaling done)
         float*8*N   N coefficients  of  the  IWAVER  polynomial
                     (nominally 6), lowest order term first
    
  8. FINEWAVER 196 binary
    This contains the polynomial used for the final fitting and line-rejection pass of the reduction phase. The default is a 7th-order polynomial (dimension 8) for a length of 196 bytes, but if a lower order fit was selected in the reduction, then the length given above will be shorter. The format is similar to that for the COARSEWAVER polynomial, except the wavelength limits are for the reddest and bluest lines actually used in the fit.
         format      what
         float*4     bluest wavelength line used
         float*4     reddest wavelength line used
         float*4     RMS of wavelength fit, Angstroms
         float*4     RMS of wavelength fit, pixels
         int*4       Number of lines in fit
    
                     Pixel -> Wavelength polynomial (WAVER)
         int*4       dimension of WAVER polynomial (nominally 8)
         pointer*4   Pointer to coefficients  in  the  in-core  C
                     poly  structure  (i.e. irrelevant to archive file)
         float*8     Midpoint of independent variable (subtracted off
                     first  when evaluating the polynomial), pixels
         float*8     Scaling  factor  for  independent   variable
                     (divided  through after midpoint subtraction
                     when evaluating the polynomial;  zero  means
                     no scaling done)
         float*8*N   N  coefficients  of  the  WAVER   polynomial
                     (nominally 8), lowest order term first
    
                     Wavelength -> Pixel polynomial (IWAVER)
         int*4       dimension of IWAVER polynomial (nominally 8)
         pointer*4   Pointer to coefficients  in  the  in-core  C
                     poly  structure  (i.e. irrelevant to archive
                     file)
         float*8     Midpoint of independent variable (subtracted
                     off  first  when evaluating the polynomial),
                     Angstroms.
         float*8     Scaling  factor  for  independent   variable
                     (divided  through after midpoint subtraction
                     when evaluating the polynomial;  zero  means
                     no scaling done)
         float*8*N   N  coefficients  of  the  IWAVER  polynomial
                     (nominally 8), lowest order term first
    

  9. COMPLINES (variable length) binary
    This record contains all the comparison and sky lines found by the reduction program, even those not matched to a line in the wavelength table. (N.B.: archive files converted from Nova files are limited to 64 lines, and only include those lines that matched a table line.) The record consists of some summary information and then the parameters for each line:
         format        what
         int*4       Total number of lines in table, N
         int*4       Total number of lines matched (comparison + sky)
         int*4       Number of skyline matched
         float*4     Mean comparison line width, pixels
         float*4     Mean sky fit residual, Angstroms
         line*28*N   N lines, structure as below
    
     Line records:
    
         format      what
         float*4
                   Center of line, pixels. If negative, this was rejected by
                   the  7th order fit (see also the rejection flag), and the
                   absolute value is the true line center.
         float*4
                   Table  Wavelength  of  line,  Angstroms.  If   zero,   no
                   wavelength   match  was  made.   If  negative,  this  was
                   rejected by the 7th order fit  (see  also  the  rejection
                   flag).
         float*4   Height of line, counts
         float*4   Width of line, pixels
         float*4   Continuum level of line, counts
         float*4   Slope of continuum at line center, counts
         int*4     Rejection flag:
                    >=  0 : line used in 7th order fit (i.e. not rejected)
                     = -1 : line never matched to a wavelength; never used
                    <= -2 : line rejected by wavelength window criterion in
                            the 5th order fit, or, if the line center is negative,
                            by the 7th order fit.
    

  10. SPECTRUM (variable length) BITS nbits type DIM naxes xlen ylen ...
    This record contains the spectral or image data. The BITS parameter given in the label gives the number of bits in the data, as with FITS. The type parameter is either IIII for integer or FFFF for IEEE floating point (both network byte order). If the value is 8 or 16, the data must be integer.

    The values following the DIM keyword give the number of data axes and their sizes, as in FITS, with the xlen coordinate incrementing fastest.

  11. ANALYSISSUMMARY 40 binary
    Summarizes the overall object velocity, including the separately determined emission and correlation velocities. Both are not always calculated.
         format        what
         int*4       Quality control status:
                        0 - Not reviewed yet
                        1 - Inconclusive velocity determination
                        2 - Insufficient wavelength coverage
                        3 - Incorrect redshift velocity
                        4 - Correct redshift velocity
         float*4*3   Overall CZ8, error, confidence
         float*4*3   Correlation CZ8, error, R value
         float*4*3   Emission CZ8, error, scatter x82
    

  12. CORRELATION (variable length) binary
    Correlations against various templates.
         format         what
         float*4*2    LW1, LW2: Blue and Red ends of template8|-
         float*4      RMS of prepared object spectrum
         int*2        Nt, Number of templates following
         int*2        Nc, Number of emission lines chopped
         int*2*2*Nc   ln(L) of line center, +/- range in pixels chopped.
         corr*44*Nt   Nt template correlations, where the structure is as below
    
    Template correlation structure:
    
         format      what
         char*16   Template name, space padded
         float*4
                               Center of correlation  peak,  lnL,  where  z  =
                               exp(c/AA)-1
         float*4   Height of correlation peak
         float*4   width (fwhm) of correlation peak
         float*4   Antisymmetric RMS => Rvalue = height/(2 * Arms)
         float*4   Correlation RMS
         float*4   Template shift, km/sec.
         float*4   AA = pixels per lnL = (N - 1)/(ln LW2 - ln LW1)
    
    |- Note that the LW1 and LW2 wavelength limits are integers on the original Nova files, and are meaningful only for the Z-Machine data, where all templates were specified to have the same limits. The limits are not saved for the echelle templates, but the AA value

  13. CORRVELOCITYDISPERSION 32 binary
    Summarizes the dispersion velocity calculated from the stellar template correlation width. This record is not present if the dispersion is not calculated.
         format        what
         float*4*3   Velocity dispersion, +68%, -68% confidence errors
         float*4     Velocity width RMS of stellar template
         float*4*4   Correlation  width   to   dispersion   conversion
                     coefficients, 3rd order polynomial
    

  14. EMISSIONLINES (variable length) binary
    Gives the emission line data found by the analysis program. This record is not present if emission lines are not fitted.
         format        what
         int*2       Number of emission lines found.
         int*2       Number of emission lines contributing to emission CZ8.
         line*76*N   N emission lines, structure as below
    
    Emission line records:
    
         format        what
         float*4     Rest wavelength, Angstroms.
         float*4     Pixel center
         float*4     Line height
         float*4     Line width
         float*4     Continuum level
         float*4     Continuum slope
         float*4*2   Line center, +/- error from Gaussian fit
         float*4*2   Line height, +/- error from Gaussian fit
         float*4*2   Line width, +/- error from Gaussian fit
         float*4*3   Parabolic fit to continuum (during Gaussian fit)
         float*4*2   Equivalent width, error (Angstroms)
         float*4     Chi^2
         int*2       Degrees of freedom
         int*2       Weight in average
                     (zero means not used in emission velocity average).
    
    contains most of the necessary information.

  15. EQUIVALENTWIDTH (variable length) MMMMEEEETTTTHHHHOOOODDDD type binary
    Equivalent widths of various lines. If the MMMMEEEETTTTHHHHOOOODDDD is type `NOVA', these are copied from the Nova reduced files, and the equivalent widths are not calculated in a standard way.
         format        what
         line*16*N   N equivalent widths, structure as below
    
    Line records:
    
         format        what
         char*8      Line name (e.g. MgH, CaK, etc.), blank padded
         float*4*2   Equivalent width, +/- error, milliAngstoms
    
    The changes from previous formats are: Note that the keyword names will not change.

    Every record starts with an ASCII 48-character descriptive label giving at minimum a name and the length in free-field format, followed by the data record. This record is not a priori in any particular format; that is determined strictly on a record-by-record basis. The length given in the 48-character label is used by routines wishing to skip the data without further ado.

    Note that a given archive file need not have the records in the same order as given here, and that applica- tion or instrument specific records can be added at any time. All programs accessing these files should simply skip or copy unknown records as appropriate.