FAST Reduction Manual Susan Tokarz August 15, 2003 Nathalie Martimbeau January 22, 2008

There are also a number of distribution programs written by Bill Wyatt that have an adjunct role in the FAST pipeline. These programs are used in two main ways. The first is as a filter to screen and distribute the night's raw data that will not be reduced by roadrunner. The second use of the distribution programs is to distribute the reduced 1-D and partially reduced (up to extraction) 2-D data at the end of each FAST run. A full description of the distribution programs and their parameters can be found at: /home/wyatt/mine/Hectospec/Scripts/Usage.

After each night's data have been processed, the reduced 1-D files are copied to the FAST archive and then entered into the FAST database, Fast.db. The raw data is automatically written to tape at the end of the day.

Programs

First, log in to the fast account.

After each night, data are automatically transferred from Mt Hopkins to Cambridge via cron jobs run on the tdc2 computer. This transfer often can not be complete until 11:30 a.m. EST, depending on the time of sunrise at Mt Hopkins and whether DARK exposures were taken at the end of the night.

The final automatic transfer is not made until just after 12:00 pm MST so that all files are guaranteed to be stored in the pervious night's data directory. If you are reducing FAST data before this time, it is prudent to manually run the transfer program to be sure all data is in Cambridge:

fastmirror -off -1

No files left to get from ...

/data/fast/rawdata/<year>.<month><day>

where year is four digits, month and day are two each.

Before reducing the data, cd to the transfer directory, for example, in /data/fast/rawdata/2003.0925, and type:

mkfastlog

Next, execute:

gather.raw

fast_rawscan.db	all files
fast_rawscan.rawdist.db	files to be distributed
fast_rawscan.rem.db	files to be removed because they are not standard reductions
fast_rawscan.reduce.db	files to be reduced

If, however, there are files in the fast_rawscan.rem.db, you will have to distribute them. You might actually want to look at the files and can do so by typing:

vurawscan -b < fast_rawscan.rem.db | more or
vurawscan -a < fast_rawscan.rem.db | more

-h "pi program"

To move the files to the directory from which the P.I. can pick them up, type:

distrib.raw -i fast_rawscan.rawdist.db

This puts them in dated subdirectories in /data/fastdist/distribute/fastraw/ ProgNNN/<year>.<month><day>/ where NNN is the program number. You will note that fast_rawscan.rawdist.db has more objects than does fast_rawscan.rem.db. That is because the rawdist files include BIAS and DARK frames as well as photometric and velocity standards. Susan usually edited this file to remove the standards and anything else she knew the P.I. did not need. However, it was not intended to be edited and it is not expected you to edit it. Once the files have been placed in the raw data distribution directory, you can notify the P.I. that they are there to be picked up. You do this with the command inform "-r date-range", for example,

inform -r 2003.0925 or
inform -r 2003.0925-2003.0927

The inform command creates and sends email that contains a script which will allow the P.I. or her delegate to copy the data to her local disk. The scripts are created because the program distribution directories, with the exception of velocity and photometric standards, are closed to the P.I.s. This was done to maintain security without assigning a different group name to each separate program. This means the P.I. cannot go to the data distribution directories, type an "ls" and see what is there. However, if the P.I. knows the path name, which includes file name and number, she can pick up the data. Thus, a list of the files with pathnames is included in the email sent by the inform program. The raw data files remain on the distribution disk in this location for a period of 6 months before they are removed. The email sent to the P.I. notes this fact.

The list of program numbers, names, P.I. and the person or people to whom the data will be distributed are kept in the fastprogs.db file, which is currently located in /home/fast/progdb. Note that the fastprogs.db file is a tab-separated file. It will occasionally need editing, usually to add new programs. If the program number and name do not match the P.I. listed in fastprogs.db, gather.raw will fail. It will produce an error file and a fix file. Often it is easiest to edit the error file to fix the headers. Whatever you edit, your fixfile should end up with lines looking something like:

sethead 0151.sn2003gh.fits P.I.="Kirshner"
sethead 0151.sn2003gh.fits PROGRAM="#2 SN Observations"

chmod +x fixfile
fixfile

Before leaving the transfer directory and when there are files to be reduced by the P.I., create a listing of those file names from the fast_rawscan.rem.db. To do this, type:

headoff FILENAME < fast_rawscan.rem.db > remlist

Summary of the gather commands for raw data:

gather.raw
vurawscan -b < fast_rawscan.rawdist.db (optional)
distrib.raw -i fast_rawscan.rawdist.db
inform -r date range

1. To load IRAF first make sure you are in an xterm-type window, then execute

   ecl

   This is an alias that will put you into the correct IRAF login directory and start up IRAF with roadrunner loaded.

2. CD to your reduction home directory, /home/fast/ared, and create the night's directory:

   mkdir <year>.<month><day>

   for example, mkdir 2003.0925; mkdir can be run either in or outside of IRAF.

3. Copy files: cd to your reduction directory, for example

   cd reddir$2003.0925   or
   cd ~/ared/2003.0925   (if outside IRAF)

   and copy the raw data into that directory. I prefer to do this in a window outside of IRAF with the command:

   cp /data/fast/rawdata/<year>.<month><day>/*.fits .

   But you can use the imcopy command instead of cp if you are in an IRAF window.

   Also, copy the remlist (files to be removed) for that night with the command:

   cp /data/fast/rawdata/<year>.<month><day>/remlist .

   Then, in the IRAF window, but same subdirectory, remove the files that will not be reduced with the command,

   imdel @remlist

4. Outside of IRAF, after deleting the files that will not be reduced (see "remlist" in the previous section), check and see if necessary keywords are correct so the data will run through roadrunner easily. To do this use Doug Mink's gethead command:

   gethead *.fits aperture disperse tiltpos naxis2 | more
   or
   gethead *.fits aperture disperse tiltpos naxis2 > indata

5. Edit indata to remove lines that look ok; for instance, find all lines with "3 300 610" on them and remove those lines. If anything is left over, it may be that the header is wrong or the setup is nonstandard.

6. If there was a grating change in the middle of the night, say from 300 to 600 back to 300, the 300 l/mm data should be reduced in 2 groups. As mentioned above, roadrunner will not automatically separate the sets; you have to do it. Split the data into set1 and set2 by manually copying each set of data to a subdirectory (I call them set1dir and set2dir). These subdirectories should be created with mkdir in the reduction directory for that night. So, for example, you end up with a subdirectory with the path /data/mc4/fast/reductions/<year>.<month><day>/set1dir. Remember you will use the same BIAS and DARK files for both data sets.

7. In the IRAF window and the reduction subdirectory, i.e., /data/mc4/fast/reductions/<year>.<month><day>, load the fastpack package. Type:

   fastpack

8. Now you can start the roadrunner program. Use the command:

   roadr > road.log

   The > road.log diverts the roadrunner output from the screen to the file, road.log, so you have a log of the reduction process.

9. Monitor the log. In another window, outside of IRAF, run the command,

   tail -f road.log

   This allows you, if you wish, to see what is happening. There is no graceful exit from this program; you have to Control C out of it after roadrunner has finished in the IRAF window Be careful that you are not in the IRAF window when you use Control C as you don't want to interrupt the roadrunner program. You can also check road.log after it has finished running by typing:

   more road.log

   In checking this log there are a number of things to look for:
   • There should be no zeros in the imstat of the biases. If there are, you should look at the image to make certain it is a bias and that there are no problems.

   • Review the imstat of the darks -- again, does any one file stand out. If so, look at it.

   • Carefully check the comp assignment list. The algorithm usually works in assigning the correct comp to the object for which it was taken but there are exceptions, generally when you are observing one or more nearby objects. If there is an incorrect assignment of comp to object, you first delete the TF.fits file with

     imdel 0123.nameTF,

     and then run transform with the correct comp:

     transform input=0123.<name> output=0123.<name>TF fitnames=0124.COMP

10. Roadrunner also creates another log, reidlog that shows the number of lines found and the residuals for the 2-D wavelength calibration. Although there is a flag that appears in road.log if the wavelength residuals are too high, it is nevertheless a good idea to inspect reidlog. Generally the residuals are on the order of 0.06 to 0.09 Angstroms and you expect to get from 49 to 55 lines. A residual over 0.1 Angstrom wouldn't necessarily be cause for alarm, but certainly if it reaches 0.2, there is a problem with the calibration of the henear.

11. If the data have been reduced in two or more sets, this is the time to copy them to the same directory. To do this in one of the set subdirectories:

    imcopy *TF.fits ../

    or alternatively, in the reduction directory, above the set directories, type:

    imcopy set1dir/*TF.fits .

    This allows the one dimensional spectra for a night to be extracted together, in the same directory. It makes archiving and distribution significantly easier.

12. If road.log and reidlog look fine, you are ready to extract the data with beepbeep. Parts of beepbeep are interactive so you won't be running it in the background. beepbeep first goes through each TF file (transformed file, which is the output from roadrunner) and automatically extracts a best guess of the spectrum and background while displaying a plot of each cross section and what has been extracted. After extracting all the files to be done automatically, beepbeep then turns over control to you and again brings up each cross-section plot for you to approve or change. Since the plot is a cut through the spatial direction, objects will show up as peaks. If you are happy with the peak and background chosen, type q and go on to the next one. If not, you can change it as you are now in an interactive plotting program.

13. If the automated program has chosen the wrong object (peak), you move the cursor to the correct peak, type p. To change the size of the profile chosen, type f and then .15 or.2; you might want to experiment to see how changing the number changes the size of the aperture extracted. If you don't like the background, you can type b and then something like 2 or 3. Once you have made a change you like, type d which will delete the old file and reextract the object using the new, changed parameters.

14. At the end of the list of automated objects, the program brings up targets that you should do by hand, such as supernovae, N7331, M31. Now you are in IRAF's plotting package (the previous plotting package was Doug Mink's) and the way to change peaks and sizes of background is different. If the peak chosen is incorrect, move the cursor to the correct peak and type s, followed by ab. If you are unhappy with the size of the profile chosen, put your cursor on the right hand side you want as a limit and type u (for upper). For the left side, type l (for lower). If the background chosen isn't a good one (say, it has another object in it instead of just sky), you can go into the background program by typing b. A t will get rid of the current selected background and you mark the new background with the s key. So, after typing t, you put your cursor on the point on the plot where you want to begin marking the left hand side background section, type s, move the cursor so you have some length that doesn't include another object, and type s again. You have now marked one side of the background. Do the same thing on the other side of the object peak. To get out of the background program, type q, then another q will take you out of the plot for that object and go to the next file.

15. After you have reviewed and corrected when necessary all of the object files observed for the night, the program pops up the extracted spectrum for each file so you can review it and remove any huge cosmic ray hits. You remove cosmic rays by positioning the cursor on the left side of the cosmic ray, type x, move the cursor to the right side and type x again. To overwrite the file with this new, corrected image type i and the program will ask you if you really want to overwrite the file -- type yes. Of course, you must be really careful when removing cosmic rays not to remove emission lines. I've become more and more conservative in this respect and usually only remove whopping cosmics. This review of the spectra also gives you a chance to notice if any of the spectra haven't been extracted correctly (say half the continuum isn't there) and you may want to reextract that spectrum with apall after beepbeep has finished running.

16. beepbeep next calls xcsao to compute velocities and this segment of the program is not interactive.

17. After the velocities are measured, beepbeep turns you over to the interactive program qplot where you mark whether the velocity is correct,Q, incorrect, X or probably correct but you're not absolutely sure, ?. A "Q" is labeled by typing "y", "X" by typing "n" and "?" by typing "j". Note that a ? doesn't mean you don't know if the velocity is correct, it means you think it is most likely correct but the S/N isn't such that you are absolutely sure.

    The qplot plot marks the absorption and emission lines for the velocity of the best template. You will notice that we run the object against ten templates and the template with the highest r value is the one chosen. There are times when you may wish to force the velocity to a different template or to the results of the emsao program. To force a different template, type t and then the number of the template you want, such as 3. The templates are listed from highest r value to lowest and the number of that sequence is the number to use when you want to select a different template. If you like what you see, type s and then x. The x stands for correlation and this will put your chosen correlation velocity in the header keyword, VELOCITY.

    The emission line program, emsao is a program to find a velocity based on emission lines only. I seldom run emsao because the emission line template, femtemp, will most likely have the highest r value when there are good emission lines in the spectrum. Unlike the correlation program run by xcsao that matchs the Fourier transform of the templates against the Fourier transform of the object, emsao actually fits the emission lines and comes up with a final velocity based on the velocities of the lines. You may want to run emsao when emission lines are present, but are weak and not marked. In such a case, you would either want to confirm that the emission line velocity matchs the absorption line correlation velocity or find an emission velocity if the correlation velocities aren't correct. Sometimes too, the femtemp template has an incorrect velocity because it has mistaken the N line, at a resting wavelength of 6583 Angstroms, for the H Alpha line, with a resting wavelength is 6563 Angstroms.

    To force the emsao velocity to be the final velocity, type s and then e.

18. After you have finished with the quality check, you are out of beepbeep and may want to reextract any poorly extracted object. Type bye to get out of the fastpack package, then load twod and apextract. Now, you will use apall to extract your object; the advantage here is that you can actually see the trace which is often important if it was missed. Also, the S/N of the extraction of a faint source seems to improve by extracting "by hand" with apall.

19. Occasionally you may want to combine two or more observations of the same object. To combine, for example, files 0124 and 0125, type:

    scombine 0124.<name>.ms,0125.<name>.ms 2425.<name>.sum

    If you have a summed file, make sure you add it to the outfile list created by beepbeep by editing outfile. I don't combine everything that has more than one exposure. If both have high r values, I usually don't bother; if one is terrible, but the other is ok, again I don't do it. However, if both are marginal, say with r values in the 4 range, you might be able to get a better velocity by combining them and then running xcsao on the combined file, above, 2425.<name>.sum.fits. You should also run hedit on the new file 2425.<name>.sum to change the title to <name>.sum. As you have noticed, we don't have a good numbering system for combined files. I try to use some part of the file numbers that are combined; nevertheless you should be aware that the keyword, DISKFILE, will be that of the first file to be combined, here 0124.

20. Once you have finished with the extractions, you might want to check the velocities to make sure you are happy with everything. To do this, type:

    gethead -ht -n 2 *ms.fits velocity czerr czxcr velqual besttemp | justify > zout

    This creates the file zout which is a list of the objects you reduced with their velocities, errors, r-values, quality check, and template used. Among the things I look for are:
    • Is the number of objects listed correct? Have I missed something?

    • Do all the observations have a quality check?

    • If there are two or more observations of the same object, are their velocities consistent?

    • Is the velocity of a velocity standard at or near the published velocity?
    Next, outside of the IRAF window, I make a hard copy of zout by typing:

    enscript -d(printer) -r zout

    I don't know if you will want to do this, but if so, this is not a list that should be casually shared; it really is for your use only. FAST data are proprietary and we have to take the security of that data seriously. I suggest, instead, that you create a directory owned by fast where you keep the zout files. Permissions on this directory, and the files within it, should be set so that only user fast and group fast can read it.

21. A note about the final ms (multispec) spectra. You may notice that each ms file contains four spectra: the first is the variable weighted spectrum. To be able to reject most of the cosmic rays, the clean algorithm in apall is turned on. This removes many of the cosmic rays, but also occasionally some parts of emission lines. This is the spectrum used to get velocities. The second spectrum in the ms file is the uncleaned spectrum. It will have many horrible cosmics but it is the spectrum that should be used to calculate equivalent widths of emission lines. The third spectrum is the background spectrum and the fourth is the sigma spectrum. You can look at each of these four spectra by typing:

    splot 0000.<name>.ms

    and then hitting the ) key. The ( key takes you in the other direction through the four spectra.

    Saving/Storing Reductions

    After you have finished with the night's reductions you will want to copy the .ms.fits files to the FAST archive. The archive currently is located on /data/mc4/fast/archive/ and the files are stored by night, each night's directory having the form, <year>.<month><day>, for example, 2003.0925.

    There are scripts set up for the transfer. In the reduction directory, e.g. /data/mc4/fast/reductions/2003.0925/, type

    s1 = date

    for example,

    s1 = 2003.0925

    Then type

    cl < ../mvlist

    This creates that night's directory in the archive and copies the data into it. It also copies a few other files that may be helpful if someone ever wants to look back and see how the data were reduced. Of particular importance is the database directory.

    If you have reduced the data in 2 or more sets, you will also want to copy the database and log files created by the roadrunner program in each set directory to the final archive. So, after you have done the above transfer, go into each of the subdirectories to copy those files to the archive. Go into the first subdirectory and type:

    cp /data/mc4/fast/reductions/mvlistold .

    Edit the mvlistold so that it has the correct date. Then type:

    cl < mvlistold

    Then go into the other subdirectories and do the same thing, editing the set1 designation (to set2 for example) so each logfile will have a unique name and won't overwrite the logfile you copied earlier.

    Once the data are in the archive, go to that file and do a quick count of the fits files to make sure all of that night's data really got transferred into the archive. After that check, cd to the directory above, /data/mc4/fast/archive/. Run chmod to set permissions:

    chmod -R 750 2003.0925 (for example)

    The -R, which is recursive, changes the permissions on the directory, in the above example, 2003.0925, as well as on the files within it.

    Next, copy the data from the archive into the Fast database. To do this, first type:

    scanfast 2003.0901-2003.0916 (date range of run)

    This creates a database file, called Scanfast.db, and a summary of the files for that night by p.i. and program number are shown on the screen. If this looks ok, you are ready to put the night's data into the FAST database, Fast.db. If it doesn't, you might need to fix p.i or program number and then run scanfast again.

    To enter the data from Scanfast.db into Fast.db, type:

    addscan

    This takes a few moments to run because, in addition to adding the files to the database, it is indexing them by ra, dec, title, etc. for faster retrieval. The program will show you the number added and the total number of files in the database.

    Finally please note that the FAST archive and the FAST database files are not open to anyone other than the fast group members; the security of this data is a high priority. Also, while the database can be rebuilt from the FAST archive, the FAST archive itself is the final storage for reduced FAST files. The Computer Facility automatically backs up the disks where these files reside; I do not make a backup.

    Distribution

    At the end of a run, the .ms.fits and TF.fits files are distributed to a disk where they can be picked up by the P.I.s This is done by the same set of programs that make distribution of raw data to P.I.s easy.

    Distribution of 1-D data.

    I'll use concrete examples but the input, set with -i, and output, set with -o, names are arbitrary and the date range would actually be the date range of the run.

    Go to the archive file, /data/mc4/fast/archive/, and type, for example:

    gather -o gather.sept03.db 2003.0901-2003.0916

    This creates the database file "gather.sept03.db" Of course you may name the output file whatever you want, but as this file will eventually be moved to: /data/mc4/fast/archive/gatherdb.dir/ to keep a record of what was done, it is important to differentiate the runs by date.

    Next:

    greport -i gather.sept03.db -o gatherlog.sept03

    This gives you a summary of what you have already seen from running gather but in a form you can easily keep and this file will be moved to /data/mc4/fast/archive/gather.log.dir/

    Then:

    distribute -i gather.sept03.db -1

    The -1 on the distribute command line puts the data into the fast1d directory which is:

    /data/fastdist/distribute/fast1d

    The other distribution directories are:

    /data/fastdist/distribute/fast2d and
    /data/fastdist/distribute/fastraw
    

    Finally,

    inform -1 2003.0901-2003.0916

    which sends, to every p.i. with data from the run, a script with a list of the files, with full path names, to enable each p.i. to copy her reduced data.

    Distribution of 2-D Data

    Go to your reduction home directory, /data/mc4/fast/reductions/ and repeat the above with the following modifications:

    gather -X TF.fits -o gather.sept03.db 2003.0901-2003.0916
    greport -i gather.sept03.db -o report.sept03
    distribute -i gather.sept03.db -2 (for 2-D)
    inform -2 2003.0901-2003.0916
    

    That's All Folks!