Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The following containers are likely to be useful to MWASCI members as a replacement for the MWA_Tools module that used to work on Magnus/Zeus. MWA_Tools module doesn't work any more because of changes that pawsey have made to the available modules and versions (mostly python issues). The MWA_Tools module will not be resurrected.

Container imagesoftware within container
/pawsey/mwa/singularity/casa/casa.imgcasa casaviewer 
/pawsey/mwa/singularity/cotter/cotter_latest.sifaoflagger aoqplot aoquality cotter fixmwams rfigui
/pawsey/mwa/singularity/wsclean/wsclean_2.9.2.imgwsclean  aoflagger casacore chgcentre  python2.7 python3.6
/pawsey/mwa/singularity/mwa-reduce/mwa-reduce.imgaddimg addimg aegean2model apparently applybeam applyion applysolutions autoprocess bbs2model beam calibrate cluster editmodel fitsmodel flagantennae flagbaselines flagionsolutions flagmwa flagsolutions flagsubbands ionpeel matchsources mrc2model mwafinfo pbaddimg pbcorrect peel phasecal regridimg render scaleimage sedcombine solutiontool storetime subtrmodel vo2model
/pawsey/mwa/singularity/python/python_latest.sifstilts python3 numpy cython scipy matplotlib astropy aipy h5py healpy ipython pandas pyyaml tqdm colorama patsy statsmodels AegeanTools mwa-hyperbeam pyuvdata PyBDSF
/pawsey/mwa/singularity/rfi_seeker/rfi_seeker.imgRFISeeker python3 astropy numpy scipy matplotlib wcstools
/pawsey/mwa/singularity/robbie/robbie-next.sifRobbie-next topcat stilts python3 AegeanTools astropy healpy numpy scipy lmfit pandas matplotlib 

Setup:

  • Remove the module load and export commands from your .bashrc and .profile files and use the example files at /group/mwa/software/_RC_FILES as a guide for making new ones.Copy /group/mwa/software/_RC_FILES/mwasci.casarc over the top of your ~/.casarc file so that casacore will work
  • See README on the manta-ray client github for how to setup asvo access via a batch script.
  • Contact your admin rep (Paul/Chris/Sammy) before contacting help@pawsey as there are some things that we can fix faster.
  • Only members of the group mwaadmin can build software and edit the module files in /group/mwa/software. We have automated the build process for future reference. If you require software that is not in there, and which is to be used by more than 1-2 people, then contact one your admin rep and we will work something out.

...

  • Don't rely on .profile or .bashrc to load the modules that you need within a batch script
  • Use containers as described above.
  • To ensure a 'clean' environment at the start of your batch script you should use:
    • source /group/mwa/software/module-reset.sh # to reset the modules to some default/minimal list. NOTE: you must source this script, don't run it directly.
  • At the start of each job script load the modules that you need using:
    • module use /group/mwa/software/modulefiles
    • module load MWA_Tools/mwa-sci # will give you access to the MWA software stack.
    • module load python # will give you python/astropy/numpy/scipy (included in MWA_Tools)
    • module load stilts # will give you java and stilts (included in MWA_Tools)
  • Data use:
    • /group and /astro should be used for storing code and data. Intermediate/long term data products should be on /group and /astro should only be used for temp/short term data as it's not backed up.
    • /home/ or ~ should be used only for config files like .profile .bash_aliases etc..
    • DO NOT store scripts, logfiles, or code in your home directory.
    • if you make symlinks in your home directory for convenience, don't use these links in your job scripts, it will put a strain on this file system for all users.
    • downloading and moving of data should be done via the Zeus copyq (for data <1GB you may get away with the occasional wget/scp)
    • copying data FROM pawsey should be done via hpc-data.pawsey.org.au

...