Containers on Garrawarla (and elsewhere):

The Garrawarla cluster doesn't mount /group so you'll have to ensure that your code is in /astro if you want to access it. Since most of our software was in /group we have started porting the software to singularity containers in /pawsey/mwa, which is mounted on Garrawarla.

To use containers:

Containers and their contents for people who are migrating away from MWA_tools module:

The following containers are likely to be useful to MWASCI members as a replacement for the MWA_Tools module that used to work on Magnus/Zeus. MWA_Tools module doesn't work any more because of changes that pawsey have made to the available modules and versions (mostly python issues). The MWA_Tools module will not be resurrected.

Container imagesoftware within container
/pawsey/mwa/singularity/casa/casa.imgcasa casaviewer 
/pawsey/mwa/singularity/cotter/cotter_latest.sifaoflagger aoqplot aoquality cotter fixmwams rfigui
/pawsey/mwa/singularity/wsclean/wsclean.imgwsclean  aoflagger casacore chgcentre  python2.7 python3.6 taql
/pawsey/mwa/singularity/mwa-reduce/mwa-reduce.img note-1addimg addimg aegean2model apparently applybeam applyion applysolutions autoprocess bbs2model beam calibrate cluster editmodel fitsmodel flagantennae flagbaselines flagionsolutions flagmwa flagsolutions flagsubbands ionpeel matchsources mrc2model mwafinfo pbaddimg pbcorrect peel phasecal regridimg render scaleimage sedcombine solutiontool storetime subtrmodel vo2model
/pawsey/mwa/singularity/python/python_latest.sifstilts python3 numpy cython scipy matplotlib astropy aipy h5py healpy ipython pandas pyyaml tqdm colorama patsy statsmodels AegeanTools mwa-hyperbeam pyuvdata PyBDSF
/pawsey/mwa/singularity/rfi_seeker/rfi_seeker.imgRFISeeker python3 astropy numpy scipy matplotlib wcstools
/pawsey/mwa/singularity/robbie/robbie-next.sifRobbie-next topcat stilts python3 AegeanTools astropy healpy numpy scipy lmfit pandas matplotlib 
/pawsey/mwa/singularity/manta-ray-client/manta-ray-client_latest.sif MWA client

Notes:

  1. mwa-reduce relies on the mwaprimary beam model from the "mwapy" package, which is NOT included in the singularity container because it's rather large. To invoke this container and have it work properly you should use:
    1. >singularity exec -B /pawsey/mwa:/usr/lib/python3/dist-packages/mwapy/data /pawsey/mwa/singularity/mwa-reduce/mwa-reduce.img <command>

Setup:

Best practice:

How to check your group quota:

If the mwa{sci,eor,ops} quota on /astro or /group is exceeded then all users in the given group will suddenly be unable to write files to the given device. This will cause jobs to fail in unexpected ways. To get a quick view of the current usage for your default group you should use:

pawseyAccountBalance --storage [-project=<group>]

Cross cluster jobs:

It is possible to submit jobs to the Zeus copy queue whilst logged into Galaxy. Just use sbatch -M Zeus -p copyq <script.sh> from the command line.

It is important that your job script has the following two lines in the header or cross-cluster job submission will give you some strange errors:

#!/bin/bash -l
#SBATCH --export=NONE

Moving data:

Here is a script for copying data from /group to /astro.


#!/bin/bash -l

#SBATCH --export=NONE
#SBATCH --account=mwasci
#SBATCH --clusters=zeus
#SBATCH --time=12:00:00                                                                                                    \
#SBATCH --nodes=1
#SBATCH --tasks=4
#SBATCH --cpus-per-task=1 
#SBATCH --partition=copyq

module load mpifileutils

# Edit this if your files are kept elsewhere
mydir=/group/mwasci/$USER

# Edit this to change which directory gets copied
dirtocopy=projectstuff

# Edit this if you want to put your files somewhere else
destdir=/astro/mwasci/$USER/destination

if [[ ! -d $destdir ]]
then
    mkdir $destdir
fi

# Note that in order to use four tasks, you have to explicitly use --tasks=4 in the SBATCH header.
mpirun -np 4 dcp -p $mydir/$dirtocopy $destdir/$dirtocopy

Put those instructions in a script (e.g. copyscript.sh) and then run (e.g.) sbatch -M zeus copyscript.sh .

Note that this will work in the copyq for up to 48 hours. I have copied about 20TB of data in less than 24h, so this should be suitable for most purposes. Note that this does NOT remove the old data! See the next entry to the guide.

Also note time limit may default to something short if you do not include it.

Deleting data:

From: https://support.pawsey.org.au/documentation/pages/viewpage.action?pageId=29263957

Using the standard Linux command rm to delete multiple files on a Lustre filesystem is not recommended. The rm command will generate a stat() operation for each file it removes, meaning all of the attributes of a file (filetype, owner, permission, modification time, etc.) will be returned. A large number of stat() operations can place an increased load on the metadata server, resulting in instabilities with the filesystem. Instead, users should use munlink, a Lustre-specific command that will simply delete a file without performing a stat() operation. Below is an example of how to remove files using munlink:

find ./processor0 -type f -print0 | xargs -0 munlink

Here's an overview of each step in that command:
find ./processor0 -type f
The find command will search the directory processor0 and all subdirectories for anything that is a file (-type f) as opposed to directories (-type d).
-print0
This dictates how the returned list of files is formatted, and ensures that they are in the correct format for the next step (xargs).
| xargs -0 munlink
The list of files is passed to the xargs command using the pipe, or |, command. xargs will then convert the list of files, line by line, into an argument for whatever command is specified at the end (in our case munlink). The -0 flag is related to the format of the listed files; if you use -print0 in the find command you must use -0 in the xargs command. Once all of the files are deleted you can remove directories with a similar command:
find ./processor0 -depth -type d -empty -delete
Again, the find command will search the directory processor0 and all subdirectories for any empty directories and delete them. The depth flag instructs finddepth Process each directory's contents before the directory itself. The -delete action also implies -depth. to process a directories contents before the directory itself. Please note that the flags passed to find are evaluated as an expression, so if you pass -delete first, find will attempt to delete below your starting directory.

Using the HSM:

You should get a confirmation email if your allocation is approved. By default you'll get 10TB, but you can argue for more if you have a convincing case.

See https://support.pawsey.org.au/documentation/display/US/Working+with+the+HSM for details on how to copy data to and retrieve data from the long-term storage.

A short version is:

The documentation has all of the details about the tools to use to check the status of your files. PLEASE read this very carefully before logging in and doing anything.

Since this is a tape system, I highly recommend tarring directories like measurement sets, since otherwise you will have to recall zillions of nested files, which will be messy (and not good for the robot). A good way of doing this is to use the workq and the command

module load pigz

tar -I pigz -cf sometarball.tar.gz somedirectory.ms/

pigz will do a multi-threaded zip which is faster than single-thread.