|Status||Date and Time (AWST) of Outage||Date and Time (AWST) of Latest Update||Description||Affected Projects||Reason||Notes|
RFI effects only
01-03 Feb 2022
|1700h 27 Jan 2022||C-building maintenance and repairs||All MRO telescopes.||Repair works are being undertaken on the c-room sub floor water leak. The floor will be lfited in sections to remove the water and ventilate the sub-floor. Doors will be open in the c-building 0700h-1700h daily||Over the Xmas break a significant subfloor water leak occurred in the c-building. A significant amount of water is trapped in the c-building sub floor and needs to be removed by pumps. The sub floor space needs to be ventilated and dried.|
|28 Feb - 11 Mar||04 Jan 2022||Reconfiguration of MWA telescope from compact to extended observations.||MWA|
|title||OPENCLOSED||10-14, 17-21 Jan||04 Jan 2022||Upgrades to MWA telescope||All MRO instruments||To reduce the effects of the extreme heat common to the Murchison at this time of the year, the upgrade program will start 0500h Tues - Fri. This may impact on early morning observations . |
|28th June - 16 July||24 Mat 2021||Routine site trip||Nil||Nil|
|Tues 29th - Wed 30th June||24 May 2021||HVTX maintenance||All instruments and equipment within the Curtin Site Licence area. Any equipment within the corrleator room will not be effected.||Annual safety maintenance||During the dates specified, all power will be shutdown to the Curtin site licence area. The High Voltage Transformer (HVTX) will undergo annual safety inspection.|
|31 May - 13 July ||21 Jun 2021||Documentary film production||AAVS2, EDA2 and MWA|
During the dates specified, three separate film crews will be undertaking filming of different aspects of instruments and equipment within the Curtin site licence area and ASKAP. Filming within the AAVS / EDA2 instruments will be daytime only. Filming within the MWA area will be at night. During daylight hours, AAVS2 and EDA2 arrays will be powered down in order to prevent any possible damage to instruments and equipment during filming.
UPDATE 31 May
Over the last three days, significant levels of rainfall have been received within the Murchison area. All access roads between Geraldton and the MRO are now closed. It is unknown when the roads will be open again. Consequently, the filming activities have not been able to commence. At this stage, CSIRO and film crews are waiting for road access conditions to change before making decisions about when / if filming will take place
UPDATE 04 Jun
The film crews were able to make it to site effective 04 Jun. Filming will commence between 04 - 13 Jun. AAVS2 , EDA2 and MWA will remain powered down whilst filming and drone flights take place. Filming will be night and day.
UPDATE 21 Jun 2021
Due to poor weather, the majority of night filming activities did not occur. The filming activities have been rescheduled for the following dates:
- July 05-09 inclusive: 0500-1900h daily
- Jul 09 - 13: night lapse photography 1600-0800h
|23rd April||1400h 29 April|
Power outage in C-room
|All MRO instruments|
The 23rd May correlator room outage was triggered by a release of fire suppression gas. The gas release triggered a power shutdown. The sudden loss of load on the power generation facility resulted in the power generation facility shutting down for approximately one second. The power generation facility came back online via diesel generators.
CIRA operations staff have not identified any damage to equipment or instruments as a result of the rapid power cycle.
However, all equipment in the correlator room within the Curtin racks is not protected by a gaseous fire suppression system in the event of an actual fire. Therefore, as a safety measure ALL equipment and instruments within the correlator room will be powered down by 1600h WST 29 April. It is expected that all equipment will remain offline to at least the 10 May.
MWA correlator will remain offline pending the outcome of the causes of equipment failure. All other instruments and equipment can continue to operate as normal
|05 Jul - 13 Aug||21 Jun 2021||Maintenance to Boolardy Station Boundary fencing||All MRO instruments||repair / replace degraded fence line in order to control animal movements on Boolardy station|
The yellow line on the above map indicates the section of fencing involved in the maintenance activity. The red line indicates the boundary of the MRO within Boolardy Station. It is a considerable length of boundary fence line (probably close to 80km) with Meerberrie station and will take 2 months to complete. Works will occur in in daylight hours only. Works will involve the use of petrol driven equipment such as chainsaws, jackhammers and pole drivers. Tractors, trucks and utility vehicles will also be used to move personnel and materiel. All instruments may observe spurious RF emissions during daytime observations.
UPDATE 24 May 2021
Originally these works were due to commence mid May and be concluded by Late Jun. the works have yet to commence and have been rescheduled for dates post June. New dates will be forwarded as soon as possible.
UPDATE 21 Jun 2021
The fencing works will commence 05 Jul and are expected to be completed on or about 13 Aug.
|Mon 19 - Wed 21 Apr||29 Mar 2021||Inspection of vermin damaged FOBOT at Curtin fibre aggregation point||All instruments except AAVS2 and EDA2||Inspection and assessment||Vermin have entered FOBOT#2 of the main trunk cable between the Curtin fibre aggregation point and the correlator building. The vermin have damaged several fibre connections within the FOBOT. In order to remove and inspect the FOBOT, all fibre optic cables within the cabinet need to be disconnected and removed. It is assessed that the outage will take 2-3 days.|
|Mon 19 Apr - Fri 23 Apr||29 Mar 21||MRO Maintenance site trip||All Curtin Site area instruments||General maintenance||Routine maintenance activities are planned for Apr 19-23. All activities will occur during daylight hours only. Spurious RFI may be observed by instruments.|
|Fri PM Mar 19- Thurs PM Mar 25||15 Mar 21||MRO Power outage and cooling system shutdown ||All MWA, SKA and external instruments||Repairs to chiller#3 in correlator room damaged during power outage Dec 22; repairs to SESKA power supply system|
Mar 22-26 maintenance works will be undertaken on the MRO correlator cooling system and the MRO power station. During this period all instruments and equipment must be powered down and not used.
In order to protect equipment from any power surges or temperature fluctuations, all site equipment will be powered down PM Fri 19 Mar. All site circuit breakers will be flicked and rack equipment will be isolated at distribution boards. Equipment will be powered back up Thurs PM 25th Mar.
21 Dec 2020
22 Dec 2020
|MRO site wide power outage||All MWA, SKA and external instruments||Unknown|
Update 22 Dec 2020
The source of the power outage was a blown EM filter on a LV supply that provided power to the correlator room. The correlator room lost power and some HV breakers tripped across the site. Fortunately, the Curtin site area did not lose power. Only equipment within the Curtin racks of the correlator room lost power.
All MWA, external instruments, AAVS and EDA2 field equipment did not lose power and a quick remote check has identified no obvious damage. It is recommended all instrument managers check their field instruments and advise of any concerns to firstname.lastname@example.org
Correlator room equipment and machines are now being brought online. Should anyone have any concerns or issues with their equipment in the correlator room please email email@example.com.
There will be no further updates regarding this power outage.
Update 21 Dec 2020
As at Australian WST1100h 21 Dec 2020, power was lost to all instruments, telescopes and equipment at the MRO. On site CSIRO staff were unable to identify the cause of the fault. Tomorrow 22 Dec 2020, electrical contractors and other CSIRO staff will travel to the MRO to support the repair effort.
Next update: WST 1800h 22 Dec 2020.
|07-11 Dec 2020||03 Nov 2020||Site wide HV electrical maintenance and Cooling system maintenance in the correlator room||All MWA, SKA and external instruments||Scheduled maintenance and fault repairs|
Dec 7-11, 2020 CSIRO will be conducting HV electrical maintenance works and cooling system maintenance in the correlator room. Power supply to all instruments and equipment within the Curtin site area will be cut approximately midday Monday 7th Dec and restored ~midday 11 Dec, 2020. All equipment and instrument managers are to safely shutdown their equipment prior to midday Monday 07 Dec 2020. CIRA Operations staff will be onsite MRO to assist in the local management of any equipment.
|TBA by separate email on occurrence||0730h 22 Sept 2020||MWA maintenance operations on site||All MWA, SKA and external instruments||MWA staff will be onsite MRO 21 Sept - 03 Oct inclusive. Operations staff will be conducting routine maintenance tasks on all telescopes within the Curtin site licence area. Impact on equipment and instruments will be minimal. RFI will be experienced during daytime hours as a result of the activities of staff on site. Individual telescopes will be without power during small periods of time as a safety precaution for technicians only|
|Nil||1300h 28 aug 2020||MWA operations site trip||All MWA, SKA and external instruments||MWA staff will be onsite MRO 30 Aug - 10 Sept inclusive. Operations staff will be conducting routine maintenance tasks on all telescopes within the Curtin site licence area. Impact on equipment and instruments will be minimal. RFI will be experienced during daytime hours as a result of the activities of staff on site.|
Ongoing from 23 July 2020, 1500h
26 Aug 2020, 1145h
|Loss of power in correlator room||All MWA, SKA and external instruments and equipment within the MRO correlator room|
Unknown - under investigation
The MRO Correlator room will remain without power for an unknown period of time. The cause and significance of the power outage remains under investigation.
The next information update will occur before COB Wed 29th July.
Should you have any questions or concerns, please contact firstname.lastname@example.org.
Please do not contact CSIRO / MRO staff directly. The staff members are very busy and are cognisant of the implications of the power outage.
UPDATE: 30 Jul 2020
The power outage in the correlator room was caused by the activation of fire suppression systems. The fire suppression was activated 23 July in response to a small component failure on a PCB that emitted particulate and smoke into the correlator room environment. The cause for the activation of the fire systems has been investigated and identified.
CSIRO are currently focussed on reinstalling and re-certifying all the components necessary to bring the fire suppression systems back online. Power was quickly restored to the c-room but without fire control systems, equipment and instruments cannot be returned to operational status.
CSIRO have advised this process is likely to be complete by the 14th August.
UPDATE: 14 Aug 2020
CSIRO have advised that works continue on recertifying the correlator room fire suppression systems but it is unlikely that instruments and equipment will be back online 17 Aug 2020. The next update will be provided 21 Aug 2020 unless further information from CSIRO is provided in the interim. CSIRO are aware of the impact this is having on all MRO users and are working with contractors to bring the MRO back online as quickly as possible.
UPDATE: 25 Aug 2020
CSIRO have advised that the fire suppression systems are being repaired. However, during the repairs one of the gas canisters containing fire suppression gasses activated. This has prevented the correlator room returning to operational status. CSIRO are currently conducting a risk assessment to determine the level of risk associated with a reduced capacity to suppress a fire in the correlator room. The outcome of this risk assessment will determine when and if the correlator room is returned to operational status this week.
UPDATE: 26 Aug 2020
CSIRO have advised they are willing to accept the reduced fire suppression system risk. The equipment in the correlator room is in the process of being brought online.
|03 Sept 2020, 0700-1200h||25 Aug 2020||Loss of connectivity to MRO||All MWA, AAVS and external instruments||Cable installation works on the fibre cable between Mullewa and the MRO|
A new gold mine is to be established at Yuin station, 150km south of the MRO. The gold mine is proposed for an area where the current fibre communication cable between Boolardy and Mullewa travels. A new cable is currently being laid around the proposed mine site.
03 Sept 0700-1200 (WST) the fibre link between Mullewa and the MRO on the old cable will be cut and replaced by the new cable. This process is expected to take 6 hours.
|27 Jul - 10 August 2020||Livestock mustering, whole of Boolardy station||All MWA, AAVS and external instruments||Mustering of livestock ||During this period, up to four light planes and helicopters supported by as many as ten vehicles will be conducting a livestock muster on Boolardy station. The muster will occur during daylight hours and will start at the northern boundary of the station moving south. Increased RF activity onsite will occur due to UHF radio use, vehicles and light aircraft. |
|9-10 July 2020||15 July 2020||Unscheduled correlator room outage||All MWA, SKA and external instruments and equipment within the MRO correlator room||Cooling system failure|
Cooling systems failures in the MRO Correlator room continued overnight. Automated shutdowns of equipment and instruments also occurred.
Cooling has been re-established to racks and equipment. Equipment and instruments can be brought back online.
The source of the fault is known, the cause of the fault remains unknown. Updates will be distributed as information becomes available.
UPDATE PM 10 Jul 2020
The cooling system issue remains unresolved.
ASKAP is powering down and will not operate over the weekend. The repair contractors are based in Adelaide and are attempting to remote in but not experiencing any positive outcomes for their efforts to identify the fault and repair the fault.
CSIRO will depart site at 1400h WAST. They will not return to site till ~1200h Mon 13 Jul. Based on this timeline and the complexity of this fault, all instrument managers are unlikely to have equipment and instruments online before AM Wed 15 Jul.
MWA Ops will be provide further updates PM Mon and PM Tues on any new developments. Unless the updates provide a different timeline, all instruments managers will NOT have access to their instruments or equipment before AM Wed 15 July 2020.
As always, any issues or queries please send to email@example.com
UPDATE 15 JUL 2020
The cooling system issue remains unresolved.
ASKAP remains powered down and is only operating intermittently.
CSIRO and the equipment contractor are continuing to work on a solution to the fault.
At this stage, access to all instruments and equipment on site and in the correlator room remains restricted. access is only provided on an as required basis. This will prevent equipment and instruments experiencing damage whilst operating in circumstances of high temperature.
At this stage, instrument managers should not expect access to their equipment or instruments before Wed 22 Jul 2020.
|8-9 July 2020||9 July 2020, 1000h||Unscheduled correlator room outage||All MWA, SKA and external instruments and equipment within the MRO correlator room||Cooling system failure|
Overnight, the cooling systems for the correlator room failed at the MRO. This caused significant overheating of equipment and triggered automated shutdowns of equipment and instruments.
As at 09 Jul 2020, 1000h (WAST) the cooling system has been repaired and cooling re-established to all correlator room equipment.
|24-26 June 2020||MWA Site area power outage||All MWA, SKA and external instruments on the Curtin site area of the MRO||Annual servicing works on the site area HVTX||During this period, power will be switched off to all site instruments and equipment within the Curtin site licence area. |
|22-26 June 2020||Correlator room power outage||All MWA, SKA and external instruments on the Curtin site area of the MRO||Maintenance works on LV cabling within the correlator room||During this period, power will be switched off in the correlator room in order to replace some LV cables. It is expected that this work will take five days.|
|24-25 May 2020||Forecasted weather event||Unknown||To the west of the MRO, a late season tropical low from the north is expected to merge with a cold front coming form the south producing strong winds, heavy rains and areas of localised high rainfalls. The impact and intensity of the event on the MRO is unknown.|
|17-21 February 2020||Repair works on Correlator room cooling systems||Possibly all ASKAP, MWA and Curtin external instruments||Repairs to damaged / faulty equipment||Early Feb 2020 the correlator cooling equipment failed due to damaged / faulty equipment. Of the three cooling systems only two have been successfully repaired and back on line. Feb 17-21 attempts will be made to repair all three systems to return to normal operations. Repairs are not expected to impact correlator room function but all MRO users should be aware this work is underway.|
|Jan-Feb 2020||Proposed upgrades to EDGES||All ASKAP and MWA / Curtin external instruments||Installation of a new EDGES instrument|
The proposal forecasts the installation of a new EDGES instrument. The installation will involve establishment of a new ground plane. Installation of the new ground plane will involve a period of welding conducted during daytime hours.
|25-29 November 2019||MRO Electrical safety testing||All ASKAP and MWA / Curtin external instruments||HV RCD testing and LV RCD testing ||Site wide power outage including the correlator room. All instruments should expect to be without power from Mon 25 - Fri 29 Nov|
|20-22 November 2019||AAVS2 / EDA2 site trip||AAVS2 / EDA2 ||Installation and commissioning of new servers to support SKA signal chain||AAVS 2 and EDA2 arrays will be powered down whilst new servers and network architecture is installed. |
|12-17 November 2019||Lightning strikes||All MWA, Curtin site area external instruments and ASKAP||Weather||Numerous lightning round strikes were recorded on the MRO during this period. Nil damage identified in MWA / Curtin site area instruments. |
|4-10 November 2019||SMART box installation||AAVS2 / EDA2 ||Installation of 16 SMART boxes on EDA2. Final array commissioning. ||Increased RF activity onsite due to UHF radio use, vehicles and handheld equipment. Overnight RFI due to filming activity 6-8 Nov inclusive.|
|30 Sept - 8 Oct 2019||AAVS2 antenna installation||AAVS2 / EDA2 ||Installation of 256 INAF SKALA 4.1 antennas||Increased RF activity onsite due to UHF radio use, vehicles and handheld equipment. Overnight RFI due to filming activity 2-4 Oct.|
|5-13 Sept 2019||MILSTD 461F testing on site||All MWA, Curtin site area external instruments and ASKAP||Testing of the new SKA Field IT processing facility in order to confirm its suitability for operations on the MRO. Testing will also occur of the T-Hut and a random receiver.|
Frequency ranges and tests conducted will be posted to this page at the conclusion of tests.
|19-31 Aug 2019||MWA Reconfiguration||All MWA and external instruments||Reconfiguration of the telescope from extended to compact array|
During week Aug 19-23 there will be no power to the telescope due to the HV transformer power upgrade.
During week Aug 26-31 fault finding and maintenance of the compact array will be occurring. During this week some tiles and receivers will be powered down for maintenance.
|19-31 Aug 2019||Power outage site-wide||All MWA and external instruments||Upgrade of HV transformer|
Outage does not include the correlator room. Outage will effect all instruments within the MWA site area
|29 July - 4 Aug 2019||Fibre rollout||All MWA and external instruments||Laying of new fibre cables to support SKA array trials||New 288 core fibre cables will be laid from the site office area to the EDA2 and INAF SKALA 4.1 arrays. This will involve the use of heavy machinery (telehandler). |
|13-27 June 2019||MRO access road repairs||All MWA and external instruments||Road maintenance|
The contractor has extended the dates for the completion of roadworks on access roads leading to the MRO. The previous finish date was 17 Jun 19.
During this period heavy earth moving equipment and water trucks will be used to rebuild and repair the access road to the MRO. Works will during daylight hours only. A generator will be used to provide power to a water pump based at the intersection of Beringarra rd and the MRO access road. The works crew will be using UHF channel 40 during daylight hours. These tests may cause spurious RFI emissions in observations taken during this time period.
|3-7 June 2019, 0700-1700h daily||RFI activities on site||All MWA and external instruments|
Testing of EDA 2 and AAVS 1.5 antennas using drone based transmitters
A drone testing campaign will be run on behalf of SKA. Details of the drone campaign can be reviewed here. These tests may cause spurious RFI emissions in observations taken during this time period.