Author Topic: (RESOLVED) Forum issue - April 3, 2018  (Read 2969 times)

Online Chris Bergin

(RESOLVED) Forum issue - April 3, 2018
« on: 04/03/2018 11:00 am »
Overnight, about 5 am UTC, a major data center that we use had a big issue resulting in (unknown number) of hosts/sites falling over.

The news site was unaffected at all times, but the forum was affected. We have a major chunk of the forum hosted by IBM in Dallas.

>At 03-Apr-2018 5:03 UTC the Network Operation Center was alerted to communication failures for the backend private networks in the Dallas Region. Network Engineers are investigating at this time. We will provide more detail as it becomes available.<

>At 06:48 UTC network engineers performed a supervisor failover on ------ which restored backend connectivity for affected hosts. A reboot and disk check may be required for virtual hosts which use SAN.<


We came back, but with a few random issues (like uploading attachments).

When I got in this morning I made Mark aware and he went through a full reboot of the forum, which required the data center's assistance - and they were obviously very busy with all the other sites coming back from the data center issue.

We're back now and we're looking good, but please note in this thread if you notice any issues.

Very, very rare we go down these days, but when we do we always note the reasons.
« Last Edit: 04/03/2018 11:07 am by Chris Bergin »
Support NSF via L2 -- Help improve NSF -- Site Rules/Feedback/Updates
**Not a L2 member? Whitelist this forum in your adblocker to support the site and ensure full functionality.**


Advertisement NovaTech
Advertisement Northrop Grumman
Advertisement Margaritaville Beach Resort South Padre Island
Advertisement Brady Kenniston
Advertisement NextSpaceflight
Advertisement Nathan Barker Photography