Skip to end of metadata
Go to start of metadata

Go to the website

Register

You will need to register to create an account. There are two ways to register (and then log in) with the MWA ASVO:

Choose How to Login / Register

Option 1: Log in / Register with your own Institution Account

This allows you to use your existing logon that you use at your institution. You do not need to remember a new username or password.

(info) If you are an existing MWA Collaboration member and have already enrolled, you can just log in without having to register again (assuming you use the same logon/provider).


The MWA ASVO supports thousands of login providers from around the world, including:

             

For more information about these providers, please see: MWA Federated Identity Information.

  • Click on "Register with your own institution account".
  • You will then have to find you institution in the list. Start typing your institution's name or just type in your email address- this should be enough to find your institution.
  • (info) NOTE: if you cannot find your institution in the list or have an issue using it, you can try:
  • Select your institution. This will take you to their sign in page. Log in to your institution as normal.
  • Fill in and submit the registration form.
  • You will be sent an email to verify your email address.
  • Confirm your email address by clicking on the link in the email.
  • An administrator will review the registration and then activate your account.
  • You will receive an email confirming your account is active. Until then you will not be able to log in to the system.


The MWA ASVO also supports logging in via our partner ASVO node: Data Central:

For more information about Data Central, please visit their website.

  • Click on "Register with your AAO data central account"
  • This will take you to their sign in page. Log in to Data Central as normal.
  • You will then be automatically logged in to the MWA ASVO!

Option 2: Log in / Register by creating an MWA ASVO username and password

  • Click on "Register for an MWA ASVO account"
  • Fill in and submit the registration form.
  • You will be sent an email to verify your email address.
  • Confirm your email address by clicking on the link in the email.
  • An administrator will review the registration and then activate your account..
  • You will receive an email confirming your account is active. Until then you will not be able to log in to the system.

Logging In

Once your account is activated, you will now be able to login to the system. Click "Login" in the top right of the home page.

If you chose Option 1 above, then click on "Login through your own institution account", otherwise click "Login with MWA ASVO account".

Searching for Observations

The first thing you'll want to do is to search for observations via the Search for Observations page. Click on "Search".

  • You can specify parameters on the left-hand panel (MWA ASVO Cone Search Query)
  • OR, if you are an advanced user, write your own ADQL query against the obscore schema.

Review the list of results in the table at the bottom of the page. You may do any of the following:

  • Create a conversion job or download job for a single observation through this web interface (see Create a Data Job section below)
  • Select any number of rows in the table (or use the "Select All/None" buttons), then click:
  • Export Conversion: this will allow you to generate a CSV file based on the conversion job options and observations you have selected, which can be used as input to the manta-ray-client (the command-line download tool)
  • Export Visibility: this will do the same as "Export Conversion", except for visibility download job.

Create a Data Job

Start a new data job

While in the Query Results panel, clicking on an observation ID (obsid) will take you to the Create a Data Job page where you can specify how you want your data.
Alternatively, if you already know the obsid you want to download, you go Data Jobs. Then, click on Create a Data Job button. Next, you must decide if you want to run a Conversion job or a Visibility Download job:

Conversion Job

This will retrieve raw visibilities from the MWA Archive and convert the MWA-specific file format into a CASA measurement set or uvfits file format using the cotter pre-processing utility. Optionally you may opt for an appropriate calibration solution to be found and applied to your data. You must also specify the frequency & time averaging options, as well as the edge-width and other settings. (See below)

  • Time Resolution (s): Average N seconds of time steps together before writing output.
  • Freq. Resolution (kHz): Average N kHz bandwidth of channels together before writing output. When averaging: flagging, collecting statistics and cable length fixes are done at the highest resolution. UVW positions are recalculated for new time steps.
  • Edge Width (kHz): Flag the given width of edge channels of each coarse channel (default: 80 kHz)
  • Calibrate: The system will try to find an appropriate calibration solution for this observation. The solution is applied before averaging. If you select this and it cannot find a calibration solution, the job will fail, however most observations will have a calibration solution available by Q2 2019. For more information on how the "calibrate" feature works, please see MWA ASVO: Calibration Option.

Now you submit your data job. If there are any problems with your parameters you will see an appropriate message on the page.

Visibility Download Job

This will retrieve either just the metadata files associated with the observation and/or the raw visibilities (still in their MWA-specific format). This option is intended for advanced users who are familiar with MWA tools such as cotter, the MWA RTS, etc and want more fine-grained control. Now you submit your data job. If there are any problems with your parameters you will see an appropriate message on the page.

Monitor Jobs

After submitting your data job, or if you want to come back and check on the status of your job(s), go to the Data Jobs page. You will see all of your current jobs and you can take different actions depending on their status. MWA ASVO data jobs go through the following statuses:

  • Queued: Your data job is submitted and is awaiting processing. You may cancel your data job during this phase using the Cancel button.
  • Processing: Your data job has been picked up by a worker node and the data job is running. The worker process will be downloading the data from the MWA Archive which is located at the Pawsey Supercomputing Centre, processing a data conversion job using cotter (if you selected a data conversion job) and then placing the resultant data product in a data retrieval area on our servers. You may cancel your data job during this phase using the Cancel button.
  • Complete: Your data job completed successfully and the data product is ready to download (see below).
  • Error: Something went wrong and the system could not produce the output you requested. The error message may provide you with details of what went wrong. If you are having difficulty processing an observation please contact MWA ASVO Technical Support and provide the obsid and job_id you are having trouble with.

Download Data

If your data job completed successfully, there will be a hyperlink to your data product in the "Product" column of the table on the Data Jobs page. MWA ASVO downloads can be very large, so we recommend copying the link url and using tools such as wget to download your data. See below for an example:

wget "https://asvo.mwatelescope.org:8778/api/download?job_id=JJJJ&file_name=NNNNNNNNNN_ms.zip" -O my_obs.zip

Your download will available for 7 days from completion, after which it will be removed from the MWA ASVO servers. 

Conversion Job Contents

After a conversion job (obsid_ms.zip or obsid_uvfits.zip) and you unzip the contents you will have the following files:

Output type: CASA Measurement Set

FilenameExampleDescription

\obsid.ms (directory)

\1104585920.msThis is the measurement set itself. If you asked for a calibrated dataset, then it will have had the calibration solution applied. In addition any tiles which failed to calibrate will have been flagged.
obsid.metafits1104585920.metafitsThis is the metafits file containing the latest metadata that was used by cotter to describe the raw data and the state of the instrument. If you asked for a calibrated dataset, this metafits file already has the tiles that failed to calibrate (see below) flagged.
calobsid.bin1104578264.binIf you asked for a calibrated dataset, this is the calibration solution file which is passed to cotter. NOTE: the calobsid is the calibrator observation ID selected by the system. This file is included for your information.
obsid_flagged_tiles.txt1104585920_flagged_tiles.txtThis is the list of flagged tiles (based on the index in the metafits file) which failed quality control during calibration. This is for your information only as these tiles have already been flagged in your CASA measurement set.

Output type: UVFITS

FilenameExampleDescription

obsid.uvfits

1104585920.uvfitsThis is the UVFITS file itself. If you asked for a calibrated dataset, then it will have had the calibration solution applied. In addition any tiles which failed to calibrate will have been flagged.
obsid.metafits1104585920.metafitsThis is the metafits file containing the latest metadata that was used by cotter to describe the raw data and the state of the instrument. If you asked for a calibrated dataset, this metafits file already has the tiles that failed to calibrate (see below) flagged.
calobsid.bin1104578264.binIf you asked for a calibrated dataset, this is the calibration solution file which is passed to cotter. NOTE: the calobsid is the calibrator observation ID selected by the system. This file is included for your information.
obsid_flagged_tiles.txt1104585920_flagged_tiles.txtThis is the list of flagged tiles (based on the index in the metafits file) which failed quality control during calibration. This is for your information only as these tiles have already been flagged in your uvfits file.

Download Job Contents

FilenameExampleDescription

obsid_datetime_gpuboxGG_NN.fits

1107707512_20150211163138_gpubox01_00.fits

1107707512_20150211163138_gpubox02_00.fits

...

1107707512_20150211163138_gpubox23_01.fits

1107707512_20150211163138_gpubox24_01.fits

These fits files are the raw visibility files. There are 24 correlator servers (gpuboxes) each handling one coarse channel of data (01-24 represented by "GG"). For long observations, each coarse channel may be split into sections ("NN" from 0-n).
obsid.metafits1107707512.metafitsThis is the metafits file containing the latest metadata that describes the raw data and the state of the instrument.
obsid_flags.zip1107707512_flags.zipIf available, this is a zip file containing cotter flag files (*.mwaf) for each coarse channel. These can then be used when running cotter or other pre-processing pipelines such as the MWA RTS.
obsid_metafits_ppds.fits1107707512_metafits_ppds.fitsThis is a potentially out of date metafits file, archived with the observation, containing the metadata that describes the raw data and the state of the instrument. Most importantly it contains extra HDUs not found in the obsid.metafits file which provide PPD plots of power vs frequency of all tiles at the start of the observation.

Bulk Downloads

Once you have submitted and downloaded a few jobs via the web interface you might be interested in a more efficient method to submit jobs and download data so you can do it in bulk. It is recommended that you use the manta-ray-client command line tool and API which makes submitting and downloading batches of jobs easier. Please see: MWA ASVO: Command Line Client (manta-ray-client) for more information.

  • No labels