|Status||Date and Time (AWST) of Outage||Date and Time (AWST) of Latest Update||Description||Affected Projects||Reason||Notes|
|07-11 Dec 2020||03 Nov 2020||Site wide HV electrical maintenance and Cooling system maintenance in the correlator room||All MWA, SKA and external instruments||Scheduled maintenance and fault repairs|
Dec 7-11, 2020 CSIRO will be conducting HV electrical maintenance works and cooling system maintenance in the correlator room. Power supply to all instruments and equipment within the Curtin site area will be cut approximately midday Monday 7th Dec and restored ~midday 11 Dec, 2020. All equipment and instrument managers are to safely shutdown their equipment prior to midday Monday 07 Dec 2020. CIRA Operations staff will be onsite MRO to assist in the local management of any equipment.
|TBA by separate email on occurrence||0730h 22 Sept 2020||MWA maintenance operations on site||All MWA, SKA and external instruments||MWA staff will be onsite MRO 21 Sept - 03 Oct inclusive. Operations staff will be conducting routine maintenance tasks on all telescopes within the Curtin site licence area. Impact on equipment and instruments will be minimal. RFI will be experienced during daytime hours as a result of the activities of staff on site. Individual telescopes will be without power during small periods of time as a safety precaution for technicians only|
|Nil||1300h 28 aug 2020||MWA operations site trip||All MWA, SKA and external instruments||MWA staff will be onsite MRO 30 Aug - 10 Sept inclusive. Operations staff will be conducting routine maintenance tasks on all telescopes within the Curtin site licence area. Impact on equipment and instruments will be minimal. RFI will be experienced during daytime hours as a result of the activities of staff on site.|
Ongoing from 23 July 2020, 1500h
26 Aug 2020, 1145h
|Loss of power in correlator room||All MWA, SKA and external instruments and equipment within the MRO correlator room|
Unknown - under investigation
The MRO Correlator room will remain without power for an unknown period of time. The cause and significance of the power outage remains under investigation.
The next information update will occur before COB Wed 29th July.
Should you have any questions or concerns, please contact firstname.lastname@example.org.
Please do not contact CSIRO / MRO staff directly. The staff members are very busy and are cognisant of the implications of the power outage.
UPDATE: 30 Jul 2020
The power outage in the correlator room was caused by the activation of fire suppression systems. The fire suppression was activated 23 July in response to a small component failure on a PCB that emitted particulate and smoke into the correlator room environment. The cause for the activation of the fire systems has been investigated and identified.
CSIRO are currently focussed on reinstalling and re-certifying all the components necessary to bring the fire suppression systems back online. Power was quickly restored to the c-room but without fire control systems, equipment and instruments cannot be returned to operational status.
CSIRO have advised this process is likely to be complete by the 14th August.
UPDATE: 14 Aug 2020
CSIRO have advised that works continue on recertifying the correlator room fire suppression systems but it is unlikely that instruments and equipment will be back online 17 Aug 2020. The next update will be provided 21 Aug 2020 unless further information from CSIRO is provided in the interim. CSIRO are aware of the impact this is having on all MRO users and are working with contractors to bring the MRO back online as quickly as possible.
UPDATE: 25 Aug 2020
CSIRO have advised that the fire suppression systems are being repaired. However, during the repairs one of the gas canisters containing fire suppression gasses activated. This has prevented the correlator room returning to operational status. CSIRO are currently conducting a risk assessment to determine the level of risk associated with a reduced capacity to suppress a fire in the correlator room. The outcome of this risk assessment will determine when and if the correlator room is returned to operational status this week.
UPDATE: 26 Aug 2020
CSIRO have advised they are willing to accept the reduced fire suppression system risk. The equipment in the correlator room is in the process of being brought online.
|03 Sept 2020, 0700-1200h||25 Aug 2020||Loss of connectivity to MRO||All MWA, AAVS and external instruments||Cable installation works on the fibre cable between Mullewa and the MRO|
A new gold mine is to be established at Yuin station, 150km south of the MRO. The gold mine is proposed for an area where the current fibre communication cable between Boolardy and Mullewa travels. A new cable is currently being laid around the proposed mine site.
03 Sept 0700-1200 (WST) the fibre link between Mullewa and the MRO on the old cable will be cut and replaced by the new cable. This process is expected to take 6 hours.
|27 Jul - 10 August 2020||Livestock mustering, whole of Boolardy station||All MWA, AAVS and external instruments||Mustering of livestock ||During this period, up to four light planes and helicopters supported by as many as ten vehicles will be conducting a livestock muster on Boolardy station. The muster will occur during daylight hours and will start at the northern boundary of the station moving south. Increased RF activity onsite will occur due to UHF radio use, vehicles and light aircraft. |
|9-10 July 2020||15 July 2020||Unscheduled correlator room outage||All MWA, SKA and external instruments and equipment within the MRO correlator room||Cooling system failure|
Cooling systems failures in the MRO Correlator room continued overnight. Automated shutdowns of equipment and instruments also occurred.
Cooling has been re-established to racks and equipment. Equipment and instruments can be brought back online.
The source of the fault is known, the cause of the fault remains unknown. Updates will be distributed as information becomes available.
UPDATE PM 10 Jul 2020
The cooling system issue remains unresolved.
ASKAP is powering down and will not operate over the weekend. The repair contractors are based in Adelaide and are attempting to remote in but not experiencing any positive outcomes for their efforts to identify the fault and repair the fault.
CSIRO will depart site at 1400h WAST. They will not return to site till ~1200h Mon 13 Jul. Based on this timeline and the complexity of this fault, all instrument managers are unlikely to have equipment and instruments online before AM Wed 15 Jul.
MWA Ops will be provide further updates PM Mon and PM Tues on any new developments. Unless the updates provide a different timeline, all instruments managers will NOT have access to their instruments or equipment before AM Wed 15 July 2020.
As always, any issues or queries please send to email@example.com
UPDATE 15 JUL 2020
The cooling system issue remains unresolved.
ASKAP remains powered down and is only operating intermittently.
CSIRO and the equipment contractor are continuing to work on a solution to the fault.
At this stage, access to all instruments and equipment on site and in the correlator room remains restricted. access is only provided on an as required basis. This will prevent equipment and instruments experiencing damage whilst operating in circumstances of high temperature.
At this stage, instrument managers should not expect access to their equipment or instruments before Wed 22 Jul 2020.
|8-9 July 2020||9 July 2020, 1000h||Unscheduled correlator room outage||All MWA, SKA and external instruments and equipment within the MRO correlator room||Cooling system failure|
Overnight, the cooling systems for the correlator room failed at the MRO. This caused significant overheating of equipment and triggered automated shutdowns of equipment and instruments.
As at 09 Jul 2020, 1000h (WAST) the cooling system has been repaired and cooling re-established to all correlator room equipment.
|24-26 June 2020||MWA Site area power outage||All MWA, SKA and external instruments on the Curtin site area of the MRO||Annual servicing works on the site area HVTX||During this period, power will be switched off to all site instruments and equipment within the Curtin site licence area. |
|22-26 June 2020||Correlator room power outage||All MWA, SKA and external instruments on the Curtin site area of the MRO||Maintenance works on LV cabling within the correlator room||During this period, power will be switched off in the correlator room in order to replace some LV cables. It is expected that this work will take five days.|
|24-25 May 2020||Forecasted weather event||Unknown||To the west of the MRO, a late season tropical low from the north is expected to merge with a cold front coming form the south producing strong winds, heavy rains and areas of localised high rainfalls. The impact and intensity of the event on the MRO is unknown.|
|17-21 February 2020||Repair works on Correlator room cooling systems||Possibly all ASKAP, MWA and Curtin external instruments||Repairs to damaged / faulty equipment||Early Feb 2020 the correlator cooling equipment failed due to damaged / faulty equipment. Of the three cooling systems only two have been successfully repaired and back on line. Feb 17-21 attempts will be made to repair all three systems to return to normal operations. Repairs are not expected to impact correlator room function but all MRO users should be aware this work is underway.|
|Jan-Feb 2020||Proposed upgrades to EDGES||All ASKAP and MWA / Curtin external instruments||Installation of a new EDGES instrument|
The proposal forecasts the installation of a new EDGES instrument. The installation will involve establishment of a new ground plane. Installation of the new ground plane will involve a period of welding conducted during daytime hours.
|25-29 November 2019||MRO Electrical safety testing||All ASKAP and MWA / Curtin external instruments||HV RCD testing and LV RCD testing ||Site wide power outage including the correlator room. All instruments should expect to be without power from Mon 25 - Fri 29 Nov|
|20-22 November 2019||AAVS2 / EDA2 site trip||AAVS2 / EDA2 ||Installation and commissioning of new servers to support SKA signal chain||AAVS 2 and EDA2 arrays will be powered down whilst new servers and network architecture is installed. |
|12-17 November 2019||Lightning strikes||All MWA, Curtin site area external instruments and ASKAP||Weather||Numerous lightning round strikes were recorded on the MRO during this period. Nil damage identified in MWA / Curtin site area instruments. |
|4-10 November 2019||SMART box installation||AAVS2 / EDA2 ||Installation of 16 SMART boxes on EDA2. Final array commissioning. ||Increased RF activity onsite due to UHF radio use, vehicles and handheld equipment. Overnight RFI due to filming activity 6-8 Nov inclusive.|
|30 Sept - 8 Oct 2019||AAVS2 antenna installation||AAVS2 / EDA2 ||Installation of 256 INAF SKALA 4.1 antennas||Increased RF activity onsite due to UHF radio use, vehicles and handheld equipment. Overnight RFI due to filming activity 2-4 Oct.|
|5-13 Sept 2019||MILSTD 461F testing on site||All MWA, Curtin site area external instruments and ASKAP||Testing of the new SKA Field IT processing facility in order to confirm its suitability for operations on the MRO. Testing will also occur of the T-Hut and a random receiver.|
Frequency ranges and tests conducted will be posted to this page at the conclusion of tests.
|19-31 Aug 2019||MWA Reconfiguration||All MWA and external instruments||Reconfiguration of the telescope from extended to compact array|
During week Aug 19-23 there will be no power to the telescope due to the HV transformer power upgrade.
During week Aug 26-31 fault finding and maintenance of the compact array will be occurring. During this week some tiles and receivers will be powered down for maintenance.
|19-31 Aug 2019||Power outage site-wide||All MWA and external instruments||Upgrade of HV transformer|
Outage does not include the correlator room. Outage will effect all instruments within the MWA site area
|29 July - 4 Aug 2019||Fibre rollout||All MWA and external instruments||Laying of new fibre cables to support SKA array trials||New 288 core fibre cables will be laid from the site office area to the EDA2 and INAF SKALA 4.1 arrays. This will involve the use of heavy machinery (telehandler). |
|13-27 June 2019||MRO access road repairs||All MWA and external instruments||Road maintenance|
The contractor has extended the dates for the completion of roadworks on access roads leading to the MRO. The previous finish date was 17 Jun 19.
During this period heavy earth moving equipment and water trucks will be used to rebuild and repair the access road to the MRO. Works will during daylight hours only. A generator will be used to provide power to a water pump based at the intersection of Beringarra rd and the MRO access road. The works crew will be using UHF channel 40 during daylight hours. These tests may cause spurious RFI emissions in observations taken during this time period.
|3-7 June 2019, 0700-1700h daily||RFI activities on site||All MWA and external instruments|
Testing of EDA 2 and AAVS 1.5 antennas using drone based transmitters
A drone testing campaign will be run on behalf of SKA. Details of the drone campaign can be reviewed here. These tests may cause spurious RFI emissions in observations taken during this time period.