Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Warning
titleForecast Outages




StatusDate and Time (AWST) of OutageDate and Time (AWST) of Latest UpdateDescriptionAffected ProjectsReasonNotes


1100h

21 Dec 2020

1800h1600h

21 22 Dec 2020

MRO site wide power outageAll MWA, SKA and external instrumentsUnknown

Update 22 Dec 2020

The source of the power outage was a blown EM filter on a LV supply that provided power to the correlator room.  The correlator room lost power and some HV breakers tripped across the site.  Fortunately, the Curtin site area did not lose power.  Only equipment within the Curtin racks of the correlator room lost power.

All MWA, external instruments, AAVS and EDA2 field equipment did not lose power and a quick remote check has identified no obvious damage.  It is recommended all instrument managers check their field instruments and advise of any concerns to operations@lists.mwatelescope.org

Correlator room equipment and machines are now being brought online.  Should anyone have any concerns or issues with their equipment in the correlator room please email operations@lists.mwatelescope.org.

There will be no further updates regarding this power outage.

Update 21 Dec 2020

As at Australian WST1100h 21 Dec 2020, power was lost to all instruments, telescopes and equipment at the MRO.  On site CSIRO staff were unable to identify the cause of the fault.  Tomorrow 22 Dec 2020, electrical contractors and other CSIRO staff will travel to the MRO to support the repair effort.

Next update: WST 1800h 22 Dec 2020.

Status
colourGreen
titleCLOSED

07-11 Dec 202003 Nov 2020Site wide HV electrical maintenance and Cooling system maintenance in the correlator roomAll MWA, SKA and external instrumentsScheduled maintenance and fault repairs

Dec 7-11, 2020 CSIRO will be conducting HV electrical maintenance works and cooling system maintenance in the correlator room.  Power supply to all instruments and equipment within the Curtin site area will be cut approximately midday Monday 7th Dec and restored ~midday 11 Dec, 2020.  All equipment and instrument managers are to safely shutdown their equipment prior to midday Monday 07 Dec 2020. CIRA Operations staff will be onsite MRO to assist in the local management of any equipment.

Status
colourGreen
titleCLOSED

TBA by separate email on occurrence0730h 22 Sept 2020MWA maintenance operations on siteAll MWA, SKA and external instruments
MWA staff will be onsite MRO 21 Sept - 03 Oct inclusive.  Operations staff will be conducting routine maintenance tasks on all telescopes within the Curtin site licence area.  Impact on equipment and instruments will be minimal.  RFI will be experienced during daytime hours as a result of the activities of staff on site.  Individual telescopes will be without power during small periods of time as a safety precaution for technicians only

Status
colourGreen
titleCLOSED

Nil1300h 28 aug 2020MWA operations site tripAll MWA, SKA and external instruments
MWA staff will be onsite MRO 30 Aug - 10 Sept inclusive.  Operations staff will be conducting routine maintenance tasks on all telescopes within the Curtin site licence area.  Impact on equipment and instruments will be minimal.  RFI will be experienced during daytime hours as a result of the activities of staff on site.
Status
colourGreen
titleCLOSED


Ongoing from 23 July 2020, 1500h

26 Aug 2020, 1145h

Loss of power in correlator roomAll MWA, SKA and external instruments and equipment within the MRO correlator room

Unknown - under investigation

The MRO Correlator room will remain without power for an unknown period of time.  The cause and significance of the power outage remains under investigation.

The next information update will occur before COB Wed 29th July. 

Should you have any questions or concerns, please contact operations@lists.mwatelescope.org.

Please do not contact CSIRO / MRO staff directly.  The staff members are very busy and are cognisant of the implications of the power outage.

UPDATE: 30 Jul 2020

The power outage in the correlator room was caused by the activation of fire suppression systems.  The fire suppression was activated 23 July in response to a small component failure on a PCB that emitted particulate and smoke into the correlator room environment.  The cause for the activation of the fire systems has been investigated and identified. 

CSIRO are currently focussed on reinstalling and re-certifying all the components necessary to bring the fire suppression systems back online.  Power was quickly restored to the c-room but without fire control systems, equipment and instruments cannot be returned to operational status.

CSIRO have advised this process is likely to be complete by the 14th August. 

UPDATE: 14 Aug 2020

CSIRO have advised that works continue on recertifying the correlator room fire suppression systems but it is unlikely that instruments and equipment will be back online 17 Aug 2020.  The next update will be provided 21 Aug 2020 unless further information from CSIRO is provided in the interim.  CSIRO are aware of the impact this is having on all MRO users and are working with contractors to bring the MRO back online as quickly as possible.

UPDATE: 25 Aug 2020

CSIRO have advised that the fire suppression systems are being repaired.  However, during the repairs one of the gas canisters containing fire suppression gasses activated.  This has prevented the correlator room returning to operational status.  CSIRO are currently conducting a risk assessment to determine the level of risk associated with a reduced capacity to suppress a fire in the correlator room.  The outcome of this risk assessment will determine when and if the correlator room is returned to operational status this week.

UPDATE: 26 Aug 2020

CSIRO have advised they are willing to accept the reduced fire suppression system risk. The equipment in the correlator room is in the process of being brought online.

Status
titleCLOSED

03 Sept 2020, 0700-1200h25 Aug 2020Loss of connectivity to MROAll MWA, AAVS and external instrumentsCable installation works on the fibre cable between Mullewa and the MRO

A new gold mine is to be established at Yuin station, 150km south of the MRO.  The gold mine is proposed for an area where the current fibre communication cable between Boolardy and Mullewa travels.  A new cable is currently being laid around the proposed mine site.  

03 Sept 0700-1200 (WST) the fibre link between Mullewa and the MRO on the old cable will be cut and replaced by the new cable.  This process is expected to take 6 hours.

Status
titleCLOSED

27 Jul - 10 August 2020
Livestock mustering, whole of Boolardy stationAll MWA, AAVS and external instrumentsMustering of livestock During this period, up to four light planes and helicopters supported by as many as ten vehicles will be conducting a livestock muster on Boolardy station.  The muster will occur during daylight hours and will start at the northern boundary of the station moving south. Increased RF activity onsite will occur due to UHF radio use, vehicles and light aircraft. 

Status
titleCLOSED

9-10 July 202015 July 2020Unscheduled correlator room outageAll MWA, SKA and external instruments and equipment within the MRO correlator roomCooling system failure

Cooling systems failures in the MRO Correlator room continued overnight.  Automated shutdowns of equipment and instruments also occurred.

Cooling has been re-established to racks and equipment.  Equipment and instruments can be brought back online.

The source of the fault is known, the cause of the fault remains unknown.  Updates will be distributed as information becomes available.

UPDATE PM 10 Jul 2020

The cooling system issue remains unresolved. 

ASKAP is powering down and will not operate over the weekend.  The repair contractors are based in Adelaide and are attempting to remote in but not experiencing any positive outcomes for their efforts to identify the fault and repair the fault.

CSIRO will depart site at 1400h WAST.  They will not return to site till ~1200h Mon 13 Jul.  Based on this timeline and the complexity of this fault, all instrument managers are unlikely to have equipment and instruments online before AM Wed 15 Jul. 

MWA Ops will be provide further updates PM Mon and PM Tues on any new developments.  Unless the updates provide a different timeline, all instruments managers will NOT have access to their instruments or equipment before AM Wed 15 July 2020.

As always, any issues or queries please send to operations@lists.mwateelscope.org

UPDATE 15 JUL 2020

The cooling system issue remains unresolved.

ASKAP remains powered down and is only operating intermittently.

CSIRO and the equipment contractor are continuing to work on a solution to the fault.

At this stage, access to all instruments and equipment on site and in the correlator room remains restricted.  access is only provided on an as required basis.  This will prevent equipment and instruments experiencing damage whilst operating in circumstances of high temperature. 

At this stage, instrument managers should not expect access to their equipment or instruments before Wed 22 Jul 2020.

Status
titleCLOSED

8-9 July 20209 July 2020, 1000hUnscheduled correlator room outageAll MWA, SKA and external instruments and equipment within the MRO correlator roomCooling system failure

Overnight, the cooling systems for the correlator room failed at the MRO.  This caused significant overheating of equipment and triggered automated shutdowns of equipment and instruments.

As at 09 Jul 2020, 1000h (WAST) the cooling system has been repaired and cooling re-established to all correlator room equipment. 

Status
titleCLOSED

24-26 June 2020
MWA Site area power outageAll MWA, SKA and external instruments on the Curtin site area of the MROAnnual servicing works on the site area HVTXDuring this period, power will be switched off to all site instruments and equipment within the Curtin site licence area.  

Status
titleCLOSED

22-26 June 2020
Correlator room power outageAll MWA, SKA and external instruments on the Curtin site area of the MROMaintenance works on LV cabling within the correlator roomDuring this period, power will be switched off in the correlator room in order to replace some LV cables.  It is expected that this work will take five days.

Status
titleCLOSED

24-25 May 2020
Forecasted weather eventUnknownTo the west of the MRO, a late season tropical low from the north is expected to merge with a cold front coming form the south producing strong winds, heavy rains and areas of localised high rainfalls.  The impact and intensity of the event on the MRO is unknown.

Status
titleCLOSED

17-21 February 2020
Repair works on Correlator room cooling systemsPossibly all ASKAP, MWA and Curtin external instrumentsRepairs to damaged / faulty equipmentEarly Feb 2020 the correlator cooling equipment failed due to damaged / faulty equipment.  Of the three cooling systems only two have been successfully repaired and back on line.  Feb 17-21 attempts will be made to repair all three systems to return to normal operations.  Repairs are not expected to impact correlator room function but all MRO users should be aware this work is underway.

Status
titleCLOSED

Jan-Feb 2020
Proposed upgrades to EDGESAll ASKAP and MWA / Curtin external instrumentsInstallation of a new EDGES instrument

The proposal forecasts the installation of a new EDGES instrument.  The installation will involve establishment of a new ground plane.  Installation of the new ground plane will involve a period of welding conducted during daytime hours.

Status
titleCLOSED

25-29 November 2019
MRO Electrical safety testingAll ASKAP and MWA / Curtin external instrumentsHV RCD testing and LV RCD testing Site wide power outage including the correlator room.  All instruments should expect to be without power from Mon 25 - Fri 29 Nov

Status
titleCLOSED

20-22 November 2019
AAVS2 / EDA2 site tripAAVS2 / EDA2 Installation and commissioning of new servers to support SKA signal chainAAVS 2 and EDA2 arrays will be powered down whilst new servers and network architecture is installed.  

Status
titleCLOSED

12-17 November 2019
Lightning strikesAll MWA, Curtin site area external instruments and ASKAPWeatherNumerous lightning round strikes were recorded on the MRO during this period.  Nil damage identified in MWA / Curtin site area instruments.  

Status
titleCLOSED

4-10 November 2019
SMART box installationAAVS2 / EDA2 Installation of 16 SMART boxes on EDA2.  Final array commissioning. Increased RF activity onsite due to UHF radio use, vehicles and handheld equipment.  Overnight RFI due to filming activity 6-8 Nov inclusive.

Status
titleCLOSED

30 Sept - 8 Oct 2019
AAVS2 antenna installationAAVS2 / EDA2 Installation of 256 INAF SKALA 4.1 antennasIncreased RF activity onsite due to UHF radio use, vehicles and handheld equipment.  Overnight RFI due to filming activity 2-4 Oct.

Status
titleCLOSED

5-13 Sept 2019
MILSTD 461F testing on siteAll MWA, Curtin site area external instruments and ASKAPTesting of the new SKA Field IT processing facility in order to confirm its suitability for operations on the MRO.  Testing will also occur of the T-Hut and a random receiver.

Frequency ranges and tests conducted will be posted to this page at the conclusion of tests.

Status
titleCLOSED

19-31 Aug 2019
MWA ReconfigurationAll MWA and external instrumentsReconfiguration of the telescope from extended to compact array

During week Aug 19-23 there will be no power to the telescope due to the HV transformer power upgrade.  

During week Aug 26-31 fault finding and maintenance of the compact array will be occurring.  During this week some tiles and receivers will be powered down for maintenance. 

Status
titleCLOSED

19-31 Aug 2019
Power outage site-wideAll MWA and external instrumentsUpgrade of HV transformer

Outage does not include the correlator room. Outage will effect all instruments within the MWA site area

Status
titleCLOSED

29 July - 4 Aug 2019
Fibre rolloutAll MWA and external instrumentsLaying of new fibre cables to support SKA array trialsNew 288 core fibre cables will be laid from the site office area to the EDA2 and INAF SKALA 4.1 arrays.  This will involve the use of heavy machinery (telehandler).  

Status
titleCLOSED

13-27 June 2019
MRO access road repairsAll MWA and external instrumentsRoad maintenance

The contractor has extended the dates for the completion of roadworks on access roads leading to the MRO.  The previous finish date was 17 Jun 19.

During this period heavy earth moving equipment and water trucks will be used to rebuild and repair the access road to the MRO.  Works will during daylight hours only.  A generator will be used to provide power to a water pump based at the intersection of Beringarra rd and the MRO access road.  The works crew will be using UHF channel 40 during daylight hours. These tests may cause spurious RFI emissions in observations taken during this time period. 

Status
titleCLOSED

3-7 June 2019, 0700-1700h daily
RFI activities on siteAll MWA and external instruments

Testing of EDA 2 and AAVS 1.5 antennas using drone based transmitters

A drone testing campaign will be run on behalf of SKA.  Details of the drone campaign can be reviewed here. These tests may cause spurious RFI emissions in observations taken during this time period. 


...