Datacenter issue - July 23, 2019

[Chris Bergin]

So we were down for about 20 minutes due to a datacenter at the forum host going down.

"Our networking team has identified an issue with networking in our ------ regions that is currently causing Droplet connectivity issues and event processing errors. We will post an update as additional information is available."

The forum is back, but it will be a bit shakey while they fully fix it.

[Zed_Noir]

So we were down for about 20 minutes due to a datacenter at the forum host going down.

"Our networking team has identified an issue with networking in our ------ regions that is currently causing Droplet connectivity issues and event processing errors. We will post an update as additional information is available."

The forum is back, but it will be a bit shakey while they fully fix it.

Being getting 522 time out error messages in the last little while. Messages indicated problem connecting with the NSF server. 

Note - why is the thread title got "May 23, 2019" in it?

[Chris Bergin]

So we were down for about 20 minutes due to a datacenter at the forum host going down.

"Our networking team has identified an issue with networking in our ------ regions that is currently causing Droplet connectivity issues and event processing errors. We will post an update as additional information is available."

The forum is back, but it will be a bit shakey while they fully fix it.

Being getting 522 time out error messages in the last little while. Messages indicated problem connecting with the NSF server. 

Note - why is the thread title got "May 23, 2019" in it?

Thread title is because I'm an idiot

Yeah, as noted, they got the sites back, but not to full speed it seems. Just got another e-mail saying resolved and monitoring, so it should improve now!

[Chris Bergin]

Still seeing some 522 timeout errors, so clearly still not 100 percent fixed. Will update when they update.

[catdlr]

Got a 502right now:

Cloudflare Ray ID: 4fb2f93d9b7053d6 • Your IP: 104.174.123.107

Ray ID: 4fb2f93d9b7053d6 • 2019-07-24 03:58:35 UTC

Response time on the entire site is very slow (10 - 15 sec)

[catdlr]

another batch;

Error 522 Ray ID: 4fb32f797ef0d37a • 2019-07-24 04:35:37 UTC
Connection timed out

Error 522 Ray ID: 4fb332c95f05d37a • 2019-07-24 04:37:52 UTC
Connection timed out

I would say I'm erroring about 1out of every 10-15 threads.  The response is very good except when I get the Timeout.

[Chris Bergin]

Really should be OK now:

DO earlier: "This network issue resurfaced and our Network Engineers have identified the issue and are working to fix this. We will share the details of the issue and progress on fixing the issue here :

Once this issue is resolved, we believe the 522 error should also get resolved. We apolagize for the inconvenience caused in this case. "

An hour ago:
"Incident resolved
Our engineering team has resolved the issue with networking in our ______ regions. Droplet connectivity issues and event processing errors should now be operating normally."

[mtakala24]

Getting errors again.
4fb6169a28ac5b43

[RocketLover0119]

Had issues getting on this morning, kept getting 522's.

[Chris Bergin]

Yeah, that 522 error was fixed about 8am UTC. Looks they just patched the entire fix just now with some bad gateways. Should be fine now!