Network Events

Future / Scheduled events

There are no future events.

Currently affecting the system

Edge Router Issue

Date: 06 Dec 17 07:54 - None
Affected services: DSL

Update 14:38: We are still experiencing (twice now at 11:13 and 13:22) the original issue of the Cisco becoming unresponsive and we are continuing our investigation into this.

Update 9:30: At approximately 4:14 this morning our Cisco edge router to which our ADSL, VDSL, HSNS and a select few CityLink customers connect to became unresponsive for ~2 minutes. Again at 4:28 it became unresponsive, this time for ~12 minutes.
The unit showed nothing untoward in its internal logs and its syslog output. The unit had been up for 5 1/2 years so we power cycled the unit at 5:23 and it was back online at 5:26.
We updated one of our hot standby routers with a current configuration, and when the existing unit became unresponsive again at 7:29 we cut over to the replacement unit which we had online by 7:39.
We were then noticing packet loss on the replacement unit and troubleshooting eventually led us to a failing memory module. We down powered the unit at 8:57, replaced both memory modules and the unit was back online at 9:10 with no packet loss either to or through the unit.
We will continue close monitoring of this but are confident there will be no further issues.
We thank you all for your patience and understanding.

We are currently experiencing poor connectivity through our Edge Router servicing ADSL, VDSL and HSNS connections.
We are actively troubleshooting this.

Historic events

Room 1 UPS Replacements

Date: 25 Nov 17 07:00 - 26 Nov 17 21:00
Affected services: Hosted

Update 26th 13:57: The second UPS replacement has now been successfully completed.

Update 25th 16:03: The first UPS replacement has been successfully completed and we will be replacing the second UPS tomorrow.

Update: We will now be replacing the UPS's next weekend (the 25th and 26th).

This weekend (the 18th and 19th) we will be replacing both UPS's and batteries for our Data Centre Room 1.
We will be replacing the first UPS & battery string on Saturday and the second UPS & battery string on the Sunday.
We will be running our entire facility on an industrial 1MW backup power generator for the period of the changeovers, and our own generators and UPS's will still be in play, so we do not envisage an interruption or any other concerns regarding power supply.
We will have our industrial electricians running the project and the UPS suppliers technical team will also be on site.
If there are any last minute concerns we will be postponing the project to next weekend (the 25th and 26th).
If you do experience any suspected loss of power or connectivity to your equipment over the weekend please do not hesitate to call us direct (Dion Hallam/021 755 873 or Mike Lee/021 755 872)

Finally, we apologies for the short notice but have been waiting on confirmation from all the involved parties.
We will be updating the NOC event notifications during the weekend as the upgrades are completed.

Co-Location Gateway Migration

Date: 20 Oct 17 06:00 - 20 Oct 17 12:00
Affected services: Hosted

On Friday the 20th at 6:00 AM we will be migrating our Co-Location customer gateways to new hardware. The process should be seamless but some customer firewalls might experience an outage if their ARP tables do not update efficiently.
If you have any connectivity issues after 6 AM please do not hesitate to call us on 04 474 0770 or follow your standard support process.

Urgent Spark Wholesale Outage

Date: 23 Oct 15 01:00 - 23 Oct 15 02:00
Affected services: Cable, CityLink, DSL, E-Mail, Hosted, Wireless

Spark Wholesale have advised us of an urgent outage on our upstream national/international circuits, beginning at 1 AM this Friday the 23rd, for up to 1 hour. This is to replace a failing wavelength router card and clean the fibre terminations.
Xtreme connections may notice a 2-3 minute loss of service as traffic going over our Spark Wholesale connections fails over to our Vodafone Wholesale connections.
No other impact should be noticed unless data is sent to another ISP that depends solely on Spark Wholesale traffic through the failing card for their upstream connectivity, as they will be unavailable for the entire duration of the outage.

Vodafone Packet Loss

Date: 24 Sep 15 10:00 - 24 Sep 15 12:00
Affected services: Cable, CityLink, DSL, Hosted, Wireless

At approximately 10 AM we started experiencing around 40% packet loss on our international Vodafone Wholesale upstream circuit.
To minimise impact we have shut this connection down so all our international traffic is now going through our Spark Wholesale/GGIS circuit.
When Vodafone resolve this we will bring the connection back up.

Update 12:00 Vodafone have advised they had power related issues at Northcote and these have now been resolved.

CityLink Issues

Date: 10 Dec 12 08:35 - 10 Dec 12 09:07
Affected services: CityLink

CityLink had issues in the Featherston/Johnston Street area due to a faulty UPS. This was resolved around 9:00 AM

Spam filtering issues

Date: 12 Oct 12 05:00 - 12 Oct 12 12:15
Affected services: E-Mail

At 5:00 this morning a spam filter update introduced a new set of checks which caused a lot of false positives (rejects of legitimate email). In all cases the sender would have received a notification that the message was not delivered.

Our attention was first drawn to this at 10:00 and we disabled the primary spam filter system at 10:30 as the full extent of the issue became apparent. This resolved the incorrect rejects, but unfortunately allowed more spam through.

We isolated the problem and restored functional spam filtering at 12:15 PM.

We sincerely apologise for any trouble this caused and are investigation options to ensure this does not happen again.

Major CityLink Outage

Date: 02 May 12 07:03 - 02 May 12 16:10
Affected services: CityLink

CityLink have advised that they have a fibre down at the intersection of Wakefield Streetand Cambridge Terrace. The fibre appears to have been brought down by a truck.
Engineering staff have been dispatched to site to investigate.
CityLink are unable to provide an ETA for full resolution at this time, but customers and buildings will regain connectivity as individual fibres are repaired.

7:47 Update:
The damaged fibre is being removed and a replacement section installed. Once the replacement section is ready it will be spliced onto the undamaged sections.

CityLink anticipate having the replacement section ready for splicing by 10am. They will then be prioritising the splicing to get the in use fibres repaired first. All services should be restored by 2pm.

Cable modem issues

Date: 27 Aug 10 09:14 - 23 Sep 10 16:00
Affected services: Cable

Cable modem connections (primarily in, but not isolated to) the Kilbernie area are experiencing loss of connectivity that may come and go. The cable modem network provider has advised that an update on a single piece of equipment that should not have caused any issues was performed early yesterday morning, no further information has been provided as yet.
Connecting a laptop directly to the cable modem with the connections IP information makes the connection work again, but may not be a permanent fix. Please contact us if you or a customer is having any issues.

23rd September: We believe the issues have been resolved by rolling back an upgrade, but have not received any official report from the cable modem network provider.

Core Firewall Upgrade

Date: 24 Jun 09 16:00 - 26 Jun 09 06:00
Affected services: Cable, CityLink, DSL, E-Mail, Hosted, Wireless

On Friday morning between 4:30 and 5:30 we will replacing our core firewalls, this will result in our web, mail and DNS servers being unavilable for a short period of time.
If everything goes smoothly these services should be unavailable for no longer than one minute.
If you have any concerns on Friday morning please don't hesitate to call us on 04 474 0770 or 0800 469 873.

ADSL Outage

Date: 31 Mar 09 12:05 - 31 Mar 09 14:00
Affected services: DSL

Telecom have advised of issues resulting in loss of connectivity for some Wellington and South Island connections. They currently have no ETR.

Update on Monday's Power Outage

Date: 28 May 08 08:11 - 28 May 08 08:11
Affected services:

At 11:58 Monday morning the top end of Wellington lost power. Within 30 seconds our generator started and Xtreme itself experienced no downtime. We ran on generator until 12:35 and then cut back to normal mains power. Unfortunately, CityLink’s equipment located at AT&T House did not last through the power cut.
CityLink worked franticly to bring the network up piece by piece and the CityLink network was fully restored at 4PM.

We are meeting with CityLink next week to discuss what they will be doing and what else we can do to ensure future power cuts have less impact on the CityLink network and our customers.

Thank you all for your patience and understanding on Monday.

Smartlinx3 in the Hutt Valley

Date: 05 Feb 08 20:20 - 05 Feb 08 22:00
Affected services: Wireless

Smartlinx3 are currently experincing issues in the Hutt Valley resulting in loss of connectivity, they hope to have this resolved by 10:00 PM