Suggestions for P band data reduction

The AIPS Cookbook contains detailed discussions on how to analyze the observations from a synthesis telescope like the VLA. The following guidelines are intended to supplement the Cookbook (the tasks mentioned are specific to AIPS). An understanding of standard synthesis data analysis is a pre-requisite to follow these guidelines.

• Load the data with FILLM (see the notes on FILLM). Specify DOWEIGHT = 1 to calculate UV-weights using the nominal sensitivity of each baseline, IF, and polarization (use DOUVCOMP = -1 to leave your data "uncompressed" because the compressed format only retains one weight per baseline at each time stamp; i.e. multiple IFs and polarizations would all be assigned the same weight). Subsequently, always use DOCAL = 2 to apply the weights. NOTE: At P band, the VLA on-line system can generate unnecesary flags on reasonably good data. If this is your case, re-load the data setting cparm(2)=1, and cparm(3) = 16 to turn off all on-line flags. Truly bad data will need to be re-flagged in the subsequent analysis but this is generally a small percentage.


• Delete the CH 0 dataset. It is generated by the VLA on-line system prior to Hanning-smoothing and often contains excessive RFI as well as the result of potentially serious instrumental phase slopes. A new one can be calculated later.


• Print out a log of your observation with LISTR (opcode='scan') and the antenna file with PRTAN.


• Edit the data using a variety of tasks like QUACK, TVFLG, SPFLG, UVFLG, CLIP, and FLGIT. Examine the data for all of the calibrators to check for RFI. Pay particular attention to the bandpass calibrator(s). Note channels that contain RFI. Run TVFLG and look at the deviations from running averages. Flag any obviously dead or bad antennas. Be sure to loop over polarizations and channels. If the RFI looks particularly bad or does not seem restricted to a few channels, run SPFLG on your bandpass calibrator(s) at this point.


• Run BPASS on your bandpass calibrator(s). Choose a small number of RFI-free channels as reference to do the calibration in order to avoid averaging over any noticeable phase slopes. This is generally not much of a problem if you use a 3 MHz band but can be important when using a 6 MHz band. Set BPASSPRM(5)=2 (do not normalize by an external channel-0), BPASSPRM(9)=1 (interpolate over flagged channels), BPASSPRM(10)=3 (normalize the spectrum using ICHANSEL channels), and BPASSPRM(11)=1 (weights independent of channel); set ICHANSEL to select a few channels free of RFI (e.g. 12,18,1,1,0); cmethod='dft'

  For the first run of BPASS set solint=0. This computes a BP solution for each scan which should be examined with BPLOT to look for signs of RFI in the solutions and changes of the bandpass with time on the different scans. (a) If everything looks good, run BPASS again but with solint=-1 (averaging the scans). (b) If a particular scan looks bad, flag that timerange and go to (a). (c) If a few channels look bad flag those channels with UVFLG, be sure BPASSPRM(9)=1 and go to (a). (d) if any antennas look bad on all scans, flag these antennas and go to (a). Do a final check of the BP solutions with POSSM (aparm(8)=2). Also run POSSM with aparm=0, source='bp calibrator', and doband=1. Check that after applying the solutions to the calibrators, the spectra are flat and the phases are near zero and show no slopes.



• Run SPLAT to apply the zeroth-order bandpass calibration (obtained with solint=-1) and eliminate the phase slopes (if present). Do not calibrate the weights at this point (that should be done later when applying the amplitude calibration). Use flagver=-1 to keep the flagging table but not apply it yet.


• Run INDXR on the output of SPLAT. This is the file you will work on from now on in the standard way.


• Run AVSPC to generate a CH 0 file. Sources= ' '; aparm(1)=3; For 31 channels use something like bchan=5; echan=27 (look at the BP solutions to see where the roll-off of the bandpass occurs); doband=1; flagver=-1 (flagver=-1 is important to ensure that the current flag table is copied to the new CH 0 but not applied. This flag table might receive more entries in the next step and will be copied later to the line dataset via TACOP. If the flag table is not copied in SPLAT, do so with TACOP).


• Look at visibility plots of all the calibrators with UVPLT in the CH 0 dataset (both polarizations). Any major variations from the expected behaviour (generally constant visibilities) should be investigated further with TVFLG on CH 0. Be sure to flag both polarizations (stokes flag) and set it to flag all channels.


• Run SETJY on the flux calibrator.


• Run CALIB in the usual way for all of the calibrators.


• Run GETJY on the secondary calibrator(s).


• Run CLCAL, source= ''; calsour=sour; calcode '*'; interpol='self'.


• Run LISTR optype='matx'; sour='flux cal' 'phase cal'; docal=2; flagver=1; gainuse=2; dparm=5,0; docrt=1. Check that the amplitudes match the output of SETJY and GETJY and that the phases are near zero. Due to source confusion in the wide field and ionospheric fluctuations, phases are often no better than +/- 10 degrees -- this is OK. Look for trends and unusually high/low phases/amplitudes. If necessary delete SN tables; CL table 2; go back to the UVPLT setp and repeat.


• Once you are happy with CL table 2, rerun CLCAL with source='your source'; calsour='phase cal'; interpol='simp'.


• If you have observed your bandpass calibrator multiple times throughout your run, it might be advantageous to do bandpass interpolation. If you wish to do so, run BPASS on the LINE file that you obtained with SPLAT and use solint=0, to get scan averages.<\li>
  
• Copy the CL tables from your CH 0 file and apply the calibration with SPLIT, using DOCAL=2, DOBAND=3 (if you do bandpass interpolation). Loop over sources and their corresponding calibrators in the usual manner.


• Run AVSPC on the SPLITed data set in order to generate a calibrated continuum database. This file is useful for making uv-plots, finding the flanking fields that are needed to represent strong sources, etc. Run UVPLT on this file and edit the data using TVFLG, UVFND, UVFLG, and CLIP. Flagging on Stokes 'V' may be helpful to find RFI; also consider examining the weights with UVPLT. Be sure to apply all flagging commands to the line data set. Other flagging utilities like SPFLG and FLGIT may also be useful. However, you will obtain the best continuum images using the spectral visibilities as IMAGR will grid each visibility using the correct cell.


• For wide-field, 3D imaging, one needs an inner dense grid of facets that cover the main lobe of the primary beam calculated so that 3D effects are small enough to be ignored. In addition, outlying fields are needed to cover the bright confusing sources which lie in the first few sidelobes. These can be found best by using external catalogs like the NVSS and WENSS. The task SETFC can be used to create both the "fly's eye" of overlapping facets in the region of the main lobe of the primary beam and the outlying facets on NVSS/WENSS fields. The positions of the facets can be checked by the task CHKFC. Because the sidelobes of the primary beam at P band are not well known, a first imaging pass with all candidate outlying fields (often several hundred) is necessary to determine which ones are relevant to your data. Subsequent imaging steps need only use the relevant flanking fields (usually less than one hundred). You might have to cast a wide net depending on your sensitivity. Often, 3C sources have to be imaged even if they are ~10 degrees from your nominal pointing position.


• Besides the external weights, it is necessary to choose the weighting scheme of the gridded visibilities that will result in the best compromise between beam size and noise characteristics. This requires trying several different sets of parameters. The weighting scheme is usually set by specifying ROBUST and UVTAPER and the best solution depends on the detailed uv sampling. The goal is to obtain a dirty beam whose main beam is as close to the Gaussian (with which the CLEAN components will be convolved) as possible with rapidly decreasing sidelobes. Since the CLEAN components are convolved with the corresponding Gaussian and added to the residuals which have been convolved with the dirty beam, the above procedure helps to keep the difference in the treatment of CLEAN components and that of noise to a minimum. This requires trying several different sets of parameters before one starts. The best solution depends on the detailed uv sampling, so it is not possible in general to decide this in advance.


• Once the full grid is set up, an initial set of images cleaned to a modest level is needed to determine the position of the sources.


• Without BOXES to limit the areas that CLEAN can consider, CLEAN will scatter power outside the regions containing real sources to optimize its fit to the data. This will bias the noise low, bias the source flux densities low, and take longer to run than if one restricts the algorithm. It will also produce a non-optimum model for the eventual self-calibration. There are other ways to solve this problem besides BOXES such as rejecting isolated clean components by some method before restoring the image but clean boxes seem to be the most straight-forward, even if it is expensive in real time to the user.

  As a first try, one can use the task BOXES which will generate CLEAN boxes containing all NVSS/WENSS (or, any other user supplied catalog) sources above a certain user-specified threshold flux density that fall in the images specified by the data and the fields specified by the adverb BOXFILE. Subsequently, these boxes can be modified suitably. It is also possible to generate CLEAN boxes interactively. To do so, display the image (or part of it) on the TV-display and use the graphics overlays to mark the positions of the BOXES. The interactive program FILEBOX writes the marked positions to the same ASCII file that stores the field centers and sizes. This file can also be edited by hand, although this process can be very time consuming, however.

• After generating the BOXES, rerun IMAGR starting from scratch.


• Self-calibrate the data using all the clean components and re-image the field. Usually, the first step should be a phase-only selfcal, followed by further imaging, perhaps repeating the process. Check the clean boxes after the first self-calibration; usually, quite a few can be added at this point. Eventually, a phase and amplitude selfcalibration will lead to the final images. This amplitude and phase self-calibration is usually done with a longer solution interval since the algorithm is determining twice the number of more variables than for the phase-only case. Besides fixing poor calibration the amplitude and phase step also forces all IFs to be on the same flux density scale, which minimizes errors for bright sources.


• After determining a good model of the observed field, it is possible to subtract the sources and further edit bad data using the residual visibilities. Such a subtraction can be done using UVSUB. Then use UVPLT to plot the data and CLIP to remove remaining high points. TVFLG can be used as well. Copy the FG table to the original (unsubtracted) data set or restore the MODEL to the edited residual visibilities with UVSUB.


• Depending on the complexity of the sources being imaged, it is possible to use multi-resolution CLEAN in IMAGR. Alternatively, it is possible to use VTESS which performs the deconvolution using the principles of the Maximum Entropy Method. Both multi-resolution CLEAN and VTESS are particularly useful for complex and extended sources. Both of these tasks require quite a bit of experimentation to produce the best images.

Several changes have improved FILLM, beginning with the 31DEC01 version of AIPS. The BPARM adverb specifies corrections made to the initial CL table that is attached to the data. These currently include corrections for opacity and antenna gain, both as a function of elevation and time. The defaults (BPARM(n) = 0) will get our current best estimate for these corrections. To turn off these corrections, set BPARM(1) = -1; BPARM(2) = -1. This will result in running FILLM as it used to pre-31DEC01. Note that if you forget to turn off these corrections, and decide that you do not want them, there is no need to run FILLM again - simply use EXTDEST to get rid of CL table version 1 (that created by FILLM), and run INDXR on the dataset.

Please note that FILLM puts the gain and opacity information into the initial CL table. This means that later calibration steps require setting DOCALIB = 2 even when GAINUSE = 1.

If you are not sure that those corrections have been made, check your CL-1 table by plotting the amplitudes using SNPLT. If those values are different from 1.0, corrections have been made.

Within AIPS, uv-data need to be weighted based on the measured VLA "nominal sensitivities" (which are related to the system temperatures). Data weighting can be important at P band because different antennas have different system temperatures which vary somewhat as a function of elevation. Because the weights are derived from system parameters measured during the observations, this method produces good estimates of the true weights irrespective of the source strength, frequency, or observing technique. However, data weighting results in somewhat less uniform uv-coverage. Thus, images of complex sources might be better with uniform weighting.

In the AIPS 31DEC01 versions and later, the following is considered to be the standard data reduction method for all observing bands.

1. Fill data with FILLM using DOWEIGHT > 0 (or equivalently cparm(2) = 8).
   
   UV-weights will be calculated using the nominal sensitivity on each baseline, IF, and polarization. Note that weights are not fully retained if the data are stored in compressed form (in compressed format only one weight is stored foa all spectral channels, polarizations, and IFs for each time/baseline sample). If you want to use these weights, then specify uncompressed data (DOUVCOMP = -1) when running FILLM. The alternative, DOWEIGHT < 1, will assign weights based on the integration time.
   
   
2. Always use DOCAL = 2 in CALIB and SPLIT.
   
   Once you load the weights with FILLM, you must calibrate and apply them by setting DOCAL = 2 on any of the calibration tasks. If you try to split the data without calibrating the weights, the uncalibrated weights might lead to very messy results. If you decide that you do not wish to use the weights, you must re-run FILLM. CALIB solutions will depend on the weights: If the weights on a baseline show that the data are very noisy, CALIB allows the solution on that baseline to vary more. This way, a bad antenna won't "drag around" other antennas. If you do not have the weights in CALIB, then you will tend to get more failed solutions on bad data (rather than having relatively poor solutions on down-weighted data). Weights are not used in the bandpass calibration; all spectral line channels have the same weight. If you do not split the calibrated data before imaging, set DOCAL = 2 in IMAGR to apply the calibration to the uv data and the weights.



3. Check to ensure that everything is OK.
   
   There are several routines that can be used to examine the weights:
   • PRTUV/UVPRT to look at the weights directly. UVPRT can apply the calibration.
   • UVPLT to plot the weights versus elevation, time, etc., with and without DOCAL = 2 to examine the effect.
   • VPLOT to plot the weights for each baseline on separate pages.
   
   
   
4. NOTE on imaging weighted data: Exercise caution when imaging weighted data. If a uv point from a relatively poor baseline is isolated in a cell in the uv plane, then Uniform weighting will increase the weight in this cell and negate all the hard work done to down-weight this uv point in the first place. Thus, Uniform weighting is generally not recommended for data that has been weighted using VLA nominal sensitivities. This is especially true for VLA A or B array data where the uv plane is relatively sparsely sampled. Thus, Natural or Robust weighting during the imaging process is generally better.

When combining two datasets, pay attention to the weights. Earlier versions of AIPS did not have the DOWEIGHT = 1 option in FILLM, so if you want to combine an older data set with a newer you must either re-reduce your old data using weights, or fill your new data with DOWEIGHT < 0.

After splitting each of your two data sets, you can combine those sets using DBCON. At this point it is very important that you check the relative weighting between the two datasets. Ideally, if the datasets were produced in an identical way (observationally and calibrationally), the resulting weights would be comparable. Therefore you would expect observations during bad weather to have lower weights in general. However, real data probably differ from this ideal situation, so it is important that you check that the weights make sense between the two datasets (using PRTUV, UVPRT or UVPLT). If the 'bad ' data have lower weights it is probably ok to DBCON your data with no re-weighting. If you however need to down-weight the bad data you must re-weight in DBCON - by some factor that you consider appropriate.

At other occasions you might want some datasets to contribute more to the final image (e.g. a resolution issue), in which case you can also adjust the relative weighting using 'reweight' in DBCON. To determine the relative weighting, you may want to run IMAGR on your two sets before DBCON. Using the exact same parameters (UVWTFN, CELLSIZE, IMSIZE etc), IMAGR will report the 'sum of gridding weights' for the data sets respectively.

Copyright © 2009 Associated Universities, Inc.
The National Radio Astronomy Observatory is a facility of the National Science Foundation operated under cooperative agreement by Associated Universities, Inc.