VLBA data calibration pipeline processing (using AIPS)

VLBA data calibration service and pre-calibrated VLBA data sets


This web document is intended to provide information on the VLBA data calibration pipeline. That is, we are currently running data of selected VLBA projects correlated at the VLBA correlator in Socorro through an automated process using AIPS to provide a best effort a-priori calibration of the data.

General information on the VLBA, how to propose for observations with the VLBA, how to schedule VLBA observations, and how to reduce VLBA data can be obtained through the VLBA and AIPS homepages.

Topics in this document include:

  Pipeline ideology What is the VLBA data calibration pipeline?
  Copyrights, etc. Restrictions for use of the VLBA data calibration pipeline
  Current status Current possibilities and limitations of the VLBA data calibration pipeline
  Scripts & releasesUpdates and older versions of the VLBA data calibration pipeline scripts and 'do it yourself' hints
  Pipelined projects How to obtain the pipelined data for your project
  Points of contact NRAO scientific staff involved in the VLBA data calibration pipeline
  Last but not leastThe acknowledgments section

What is the VLBA data calibration pipeline?

The VLBA data calibration pipeline is a 'runfile' in AIPS which is primarily based on the AIPS VLBAUTIL procedures. The VLBAUTIL procedures are described in Appendix C of the AIPS Cookbook. The VLBA data calibration pipeline performs calibration tasks and makes lots of plots and some simple images to find out whether the calibration has worked - the VLBA data calibration pipeline is not intended to produce high fidelity images. The VLBA data calibration pipeline will be available to AIPS users as part of the AIPS distribution and the Midnight Job.

At the moment the VLBA data calibration pipeline works reasonably well for simple VLBA-only experiments correlated in Socorro, at frequencies between about 1 and 15 GHz, both for continuum and spectral line experiments. It has been successfully run on higher and lower frequencies, but we're not advertising that for general use - in particular the spectral RFI flagging must be improved.

At the moment the VLBA data calibration pipeline works according to the (bit outdated) flow chart as shown here in ps or in gif.


Restrictions for use of the VLBA data calibration pipeline

The VLBA data calibration pipeline is an effort to streamline and simplify customary calibration of data from the Very Long Baseline Array (VLBA). The VLBA is operated by the National Radio Astronomy Observatory (NRAO) as a facility of the National Science Foundation (NSF) operated under cooperative agreement by Associated Universities, Inc. (AUI).

The Astronomical Imaging Processing System (AIPS) is NRAO's software package for calibration, data analysis, image display, plotting, and a variety of ancillary tasks on (mostly interferometric, radio-) astronomical data. The VLBA data calibration pipeline is an NRAO product (like AIPS itself), comes as an AIPS 'runfile' (with an associated 'helpfile'), and therefore assumes the terms and conditions applicable to using AIPS; this is mostly the GNU General Public License (GNU GPL). AIPS is copyright (c) 1995 through 2012 by AUI.

To make use of the VLBA data calibration pipeline, you must have a relatively current installation of AIPS on your computer (AIPS version 31DEC09 or later), preferably run the AIPS 'Midnight Job' (MNJ), and follow the instructions as provided in the VLBA data calibration pipeline 'helpfile'. If you do not run the MNJ, you may be restricted to using an older version. Please report any bugs you may find after making sure you are using the latest versions (the bug might already have been fixed).

Note that the VLBA data calibration pipeline comes as is, without warranty or guarantee that it works (although we try to make it work)!

[That is, we do not assume any responsibility for whatever happens (probably nothing malignant) in case you decide to use the scripts.]

When you have calibrated your data using the VLBA data calibration pipeline, it would be appreciated if you would mention the VLBA pipeline next to an acknowledgment to NRAO in the publication of your scientific results; e.g. "The VLBA data was calibrated using NRAO's "VLBA data calibration pipeline" in AIPS (Sjouwerman, Mioduszewski & Greisen, in preparation).


Current possibilities and limitations of the VLBA data calibration pipeline

Below we have collected some current 'can-do' and 'cannot-do' features of the VLBA data calibration pipeline, which will be updated after new releases.
Some general tasks that the VLBA data calibration pipeline does or can do:
  • Calibrate VLBA data at multiple intermediate radio frequencies (1-15GHz guaranteed)
    for amplitude, bandpass, parallactic angle, some instrumental polarization,
    pulse-cal (continuum) or manual p-cal (spectral-line)
    on data sets that include calibration transfer files (post April 1999)
    [or use VLBALOAD first and append the tables with ANTAB before starting]
  • Including some (yet non-robust, non-complete) flagging of bad data
  • Application of ionospheric correction with JPL's TEC-models
  • Fringe-fitting of calibrators and strong sources
  • Including phase-referencing schemes for up to 15 target sources
  • Imaging of continuum and spectral-line features
    with an optional self-cal for strong sources (fragile!)
  • Including survey work (large number of relatively strong sources)
    (watch this space for updates of capabilities)

    Wish list for future implementation, roughly in order of priority:
    • Finish spectral-line and polarization calibration (some loose ends)
      [e.g. calibration of bandpass phases after FRINGE and L-R calibration]
    • Copy tables to second pass from first pass, then split and image 2nd pass
    • Find a more robust and/or improve non-interactive flagging algorithm
    • Append VLBA calibration from a text file (aka 'manual calibration transfer')
      [i.e. use VLBALOAD first and append the tables with ANTAB before starting]
    • Include single dish VLA (phased array needs source fluxes)
      [i.e. use VLBALOAD first and append the tables with ANTAB before starting]
    • Include EB_VLBA and GB_VLBA (should be relatively easy)
      [i.e. use VLBALOAD first and append the tables with ANTAB before starting]
    • Expand to high (>15GHz) and low (<1GHz) frequencies
      [it usually works already for experiments with well designed observations]
    • Investigate inclusion of tropospheric correction (high frequency)
    • JPL ionospheric TEC-models attached as table (external file already)
    • Include other VLBI stations (EVN, DSN, etc, will be relatively difficult)
    • Pipeline in aips++ ?? (very long-term plan at the moment)
      Of course any other suggestions for the wish list are very welcome as well.

      The VLBA data calibration pipeline is not supposed to handle, or cannot do:
      • Correct for bad observing weather/phase-coherence/RFI/tactic/techniques!
      • Data from other correlators than the VLBA correlator (ie. in Socorro NM, USA)
        Note: data from the JIVE correlator can be processed with an EVN pipeline.
      • High-fidelity flagging of visibilities in single (and multi-) source files
      • Advanced imaging (multi-field & -resolution, deep cleaning, MEM, etc)
      • Advanced hybrid-mapping (high-fidelity self-cal iterations, tricks & kicks)
      • Apply doppler-tracking for individual spectral line sources (aka CVEL)

        Updates and older versions of the VLBA data calibration pipeline scripts

        We intend to archive older versions of the VLBA data calibration pipeline, along with older versions of the VLBAUTIL scripts. This should accommodate sites not running the AIPS 'Midnight Job' (MNJ) and make it possible to recreate 'features' (that is, check the results in one version with a different version if one does not trust the results). We will provide the collected older versions and keep track of new updates and their revision history at this spot in the near future. We anticipate however, that the sites running the MNJ get updates and new releases through the MNJ so that they always have the hottest versions (and their bugs? ;-) without any special intervention.

        Revision history of the VLBA data calibration pipeline scripts

        Latest pipeline version in AIPS:updates with DOUVCOMP, messages not suppressed, new HELP, PLOT index
        Pipeline version before 25NOV02:updates, and NPOINTS->NCHAV in IMAGR; first MNJ version
        Pipeline version before 25JUN02:overhaul to new VLBAUTIL (frequency ID split)
        Pipeline version before 25MAR02:added pol/spectral-line, surveys
        Pipeline version before 25JAN02:plain continuum version, including self-cal imaging

        How to obtain the pipelined data for your project

        Please contact Loránt Sjouwerman if you have data in the correlator queue (i.e. your data has been observed recently) and you would like to obtain your pipeline calibrated data. Highest priority will be given to projects of novice users for which the VLBA data calibration pipeline is known to work; though the latter may change on a regular basis! Lowest priority will be projects out of range of the current capabilities, projects of experienced users, and projects observed in the more distant past.

        You can try to pipeline your own data with these instructions

        If you are running the MNJ, you are almost set to go. If not, you have to get the two VLBAPIPE files in the table below and put them in the appropriate places. We highly recommend that you use a local environment to place your aips pipeline files. However, if you don't want to keep the files in your local environment (see section 3.10 (page 3-16) of the AIPS Cookbook) and make the VLBA data calibration pipeline script available to other users at your institute, it is easiest to source the AIPS LOGIN.[C]SH file, and then type $CDTST to find the variable $RUNFIL and $HLPFIL (usually, if your AIPS installation is, for example, in /home/AIPS, then $RUNFIL is usually /home/AIPS/RUN, and $HLPFIL would be /home/AIPS/31DEC02/HELP). HOWEVER, the latter is highly discouraged in case you want to edit the procedure to your own likings and/or run the MNJ - always ONLY do that in your local environment! (You also wisely may not have the privileges to do that). Alternatively, pre-defined variables such as $HOME, $FITS, $DA01 (your first AIPS data disk), etc, can be used as a local environment (but a separate AIPS-runfile directory is preferred). After placing the files in one of these directories, define the non-predefined directory as a variable (e.g. in csh: setenv MYAREA `pwd` before starting AIPS!) - skip this step for HOME, FITS, etc, and if you have the MNJ enabled. Start up AIPS as usual, and if you do not use the default $RUNFIL area, but e.g. your $MYAREA area to place the pipeline scripts, remember to initialize the 'local version'. That is, within AIPS you make the files in the local directory available to AIPS by typing (e.g.) version 'myarea' or version 'home'. Then type explain vlbapipe within AIPS to read the help available.

        The AIPS 'helpfile' contains information on how to use the 'runfile' and is maintained properly. In case it bombs on you, it may be necessary to also fetch the 'frozen utilities' - a version of the VLBAUTIL procedures that are known to work with the VLBAPIPE script. If it still fails, it may be that the individual underlying AIPS tasks have changed. Either update your AIPS installation, or send us a bug report (depending on your AIPS version and latest overhaul of the AIPS MNJ version) - if you use an AIPS version older than 31DEC02, don't expect the pipeline 'runfile' to work.

        Current VLBA pipeline runfile script:VLBAPIPE.001Put this in your $MYAREA (or $RUNFIL) directory
        Current VLBA pipeline helpfile text: VLBAPIPE.HLPPut this in your $MYAREA (or $HLPFIL) directory
        Temporary frozen VLBAUTIL procedures:VLBAUTIL.001You may need this too (in $MYAREA, or $RUNFIL)
        For both the MNJ and newly fetched files, you have to type run vlbautil and run vlbapipe (each on a separate command line) in your AIPS window to make sure you are using the latest 'runfiles'. If you forget this, either you might end up using an older version, or it will complain about not finding or recognizing the command. The same will happen if you forget to place the 'helpfile' or 'runfiles' in the appropriate directories (or misplace them).

        After defining the procedure in AIPS, read the 'helpfile' for its use (type explain vlbapipe) and type inputs vlbapipe to review its current parameter settings. Invoke the VLBA data calibration pipeline by simply typing vlbapipe; it will check the inputs for possible problems, and if none are found, tell it that the tape is ready to be read (by typing 0 when the tape drive has settled down) and/or sit back and let the VLBA data calibration pipeline do it's work. Do not forget to check the plots and images it has made, as they may point to specific problems with your data set (dead IF channels, RFI, failing fringe-fitting, etc). If something is definitely wrong, usually these problems are solved by flagging the bad data (in flag/FG table one!) and rerunning the VLBA data calibration pipeline. Other common problems are that you selected a time range for your bandpass calibrator in which it has too few visibilities (e.g. when it is late on source), in which the source is below the horizon for one or more antennas, or the timerange does not correspond to the scan on your bandpass calibrator - change either or both. For other problems contact Loránt Sjouwerman

        Archive of previous "releases" (frozen versions) of the VLBA data calibration pipeline scripts

        Pipeline version before 25NOV02:VLBAPIPE.001_25nov02 VLBAPIPE.HLP_25nov02 VLBAUTIL.001_25nov02
        Pipeline version before 25JUN02:VLBAPIPE.001_25jun02 VLBAPIPE.HLP_25jun02 VLBAUTIL.001_25jun02
        Pipeline version before 25MAR02:VLBAPIPE.001_25mar02 VLBAPIPE.HLP_25mar02 VLBAUTIL.001_25mar02
        Pipeline version before 25JAN02:VLBAPIPE.001_25jan02 VLBAPIPE.HLP_25jan02 VLBAUTIL.001_25jan02

        NRAO scientific staff involved in the VLBA data calibration pipeline

        Please consult one of the following if you have any questions regarding parts of the VLBA data calibration pipeline. Most likely you want to contact Loránt Sjouwerman, but if your question obviously regards one of the other items listed in the table below, the relevant addresses are given as well.
        Loránt Sjouwermanlsjouwer@nrao.edu VLBA data calibration pipeline scripts & data
        Amy Mioduszewski daip@nrao.edu General VLBA utility (VLBAUTIL) scripts
        Cast of AIPSAIPS Cookbook IndexVLBA data reduction in AIPS
        Eric Greisen daip@nrao.edu AIPS data reduction package
        Claire Chandler cchandle@nrao.edu VLBA scientific services
        Data analysts analysts@nrao.edu VLBA logistics (schedules/correlation/shipments)

        The acknowledgments section

        Again, when you have calibrated your data using the VLBA data calibration pipeline, it would be appreciated if you would mention the VLBA pipeline next to an acknowledgment to NRAO in the publication of your scientific results; e.g. "The VLBA data was calibrated using NRAO's "VLBA data calibration pipeline" in AIPS (Sjouwerman, Mioduszewski & Greisen, in preparation). Thanks!

        We on our side acknowledge the efforts of the Cast of AIPS, in particular Eric Greisen, for making and maintaining the AIPS-tasks and environment needed to reduce VLBI data in AIPS over the years. Chris Flatters started collecting the tasks into the VLBAUTIL procedures, which was continued by Amy Mioduszewski; the VLBAPIPE procedures heavily depend on these VLBAUTIL procedures. Finally, Greg Taylor allowed me to work on this pipeline.




        Pipeline ideology / Copyrights, etc. / Current status / Scripts & releases / Pipelined projects / Points of contact

        VLBA Home Page

        The VLBA data calibration pipeline processing webpage originated June 5th, 2002.
        Maintained by Loránt Sjouwerman: lsjouwerman@nrao.edu.