Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

and this will help you determine the correct GPS times to pass to the -b/-e options. This applies for all jobs that have an optional beginning and end times.

Recombine
Anchor
recombine
recombine

As noted above in Downloading Data, the vcs_download.nf script includes the recombine step, so manually recombining will only be necessary for situations where the Nextflow option is either unavailable or undesirable. (Note that with the recent switch to using ASVO as the data downloading server, vcs_download.nf is now a bit of a misnomer – it can still be used to do the recombining, even if the data are already downloaded.) Other methods of recombining are detailed at the Recombine page.

Incoherent sums
Anchor
inco_sums
inco_sums

...

The total data volume downloaded will vary, but for maximum duration VCS observations this can easily be ~40 TB of just raw data. It is therefore important to keep in mind the amount of data you are processing and the disk space your are consuming. If only the raw voltages have been downloaded then you will need to recombine the data yourself, which doubles the amount of data (see next section).

...

Note that this step should be performed automatically by the vcs_download.nf script.

Recombine takes data spread over 32 files per second (each file contains 4 fine channels from one quarter of the array) and recombines them to 24+1 files per second (24 files with 128 fine channels from the entire array and one incoherent sum file); this is done on the GPU cluster ("gpuq") on Galaxy. When downloading the data, if you retrieved the "Processed" (i.e. recombined) data, then ignore this step as it has already been done on the NGAS server.

To recombine all of the data, use

Code Block
languagetext
process_vcs.py -m recombine -o <obs ID> -a

or, for only a subset of data, use

Code Block
languagetext
process_vcs.py -m recombine -o <obs ID> -b <starting GPS second> -e <end GPS second>

If you want to see the progress, then use:

Code Block
languagetext
squeue -p gpuq -u $USER

Generally, this processing should not take too long, typically ~few hours.

Checking the recombined data

As before, it is a good idea to check at this stage to make sure that all of the data were recombined properly. To do this, use:

Code Block
languagetext
checks.py -m recombine -o <obs ID>

This will check that there are all the recombined files are present and of the correct size. If there are missing raw files the recombining process will make zero-padded files and leave gaps in your data. If you would like to do a more robust check, beamform and splice the data (using the following steps) and then run:

Code Block
languagetext
prepdata -o recombine_test -nobary -dm 0 <fits files>

Then you can look through the produced .dat file for gaps using:

Code Block
languagetext
exploredat <.dat file>

Once you are happy that the data have been recombined correctly then you should delete the raw voltages (as they are no longer used in the pipeline and are a massive drain on storage resources).

Beamforming (old python method)

...