Author Topic: Master Thread for NASASpaceflight.com and NSF Forum Outages, Upgrades and Error Message Reporting (Read 258989 times)

AARON17 · « **Reply #80 on:** 10/01/2014 11:48 pm »

I assume this is to protect against Shellshock? You guys seem very on the ball, as I remember you patched against Heartbleed before it even hit the news.

Chris Bergin · « **Reply #81 on:** 10/02/2014 12:25 am »

Quote from: AARON17 on 10/01/2014 11:48 pm

I assume this is to protect against Shellshock? You guys seem very on the ball, as I remember you patched against Heartbleed before it even hit the news.

We always try to be as much 'on the ball' as possible. We patched the servers as soon as the Shellshock story broke, today the infrastructure (VM's and storage) needs to be rebooted to mitigate a security issue in the VM infrastructure.

(I sound clever there, don't I!................................... Mark wrote that!

)

Chris Bergin · « **Reply #82 on:** 10/02/2014 03:53 am »

Obviously not happened yet. We're next, however!

bubbagret · « **Reply #83 on:** 10/02/2014 05:57 am »

The site went out for me around a half hour ago.... Aaaaand, you're back!

ChrisWilson68 · « **Reply #84 on:** 10/02/2014 05:58 am »

That wasn't too bad!

Chris Bergin · « **Reply #85 on:** 10/02/2014 06:26 am »

Nice! That looked pretty painless!

Chris Bergin · « **Reply #86 on:** 03/02/2015 04:35 pm »

Our host has informed us they will be patching and rebooting servers to counter the latest security thing those naughty people are cooking up. This will affect our Dallas servers at 7am UTC (2am Eastern) on March 5. Apparently it will be a short period (the last time it was less than 30 minutes and I think that was more work).

This is a necessary step and our host seems ahead of the game on this sort of thing. Thankfully there isn't a SpaceX launch or an event at that time!

Anyway, so you know. The period is pretty much the quietest time of day for the site, so that's helpful at least.

Update:

So it will be a short downtime, within a four hour window that opens at 7am - as they'll be rebooting one pod at a time:

>
Per data center maintenance will be performed one pod at a time. Individual hosts will be rebooted on a rolling basis within the pod. Maintenance will begin in the next pod after the previous pod's maintenance is completed. We estimate the process to take 4 hours per data center.<

In other words we could be up past 7am UTC and go down for a bit at 9am UTC.

Quote

System Engineers have completed all host server reboots in DAL01 at this time, and VSIs in this datacenter should be back online. I

The Amazing Catstronaut · « **Reply #87 on:** 03/03/2015 07:08 am »

Thanks for notifying us!

A lot of website admins (for, of course, other, less infallible websites) would have just let the downtime happen without considering it significant enough to warn people over.

Chris Bergin · « **Reply #88 on:** 03/04/2015 01:27 pm »

Updated the opening post. Split some of the other stuff into the feedback thread.

Two datacenters completed. I think we're next, about at about 7am UTC. Window looks like it'll be a lot shorter, as the completed a larger datacenter in 75 mins (the full datacenter, all of the pods), so the pods will be less than the entire datacenter.

Anyhoo, will try and be up for when this happens. If not, I'd be interested if anyone is around at that time to let us know how long we were dark.

NSF Webmaster · « **Reply #89 on:** 03/05/2015 06:25 am »

Awaiting official confirmation from our host it seems we are back. Downtime : 10 minutes.

[edit]And confirmed.

Quote

System Engineers have completed all host server reboots in DAL01 at this time, and VSIs in this datacenter should be back online.

Mark

Chris Bergin · « **Reply #90 on:** 03/05/2015 11:36 am »

Well that was painless. And looking in my garage, all the NSF servers seem to be humming away nicely!

catdlr · « **Reply #91 on:** 03/06/2015 03:12 am »

Quote from: Chris Bergin on 03/05/2015 11:36 am

Well that was painless. And looking in my garage, all the NSF servers seem to be humming away nicely!

Chris...Your garage is big enough to service a Shuttle.

NSF Webmaster · « **Reply #92 on:** 03/23/2015 11:32 am »

Next wednesday (25/3) we will perform some maintenance and upgrade tasks on the forum. While we try to prevent any downtime, a couple of error messages may occur during this window which will start at 03:00 EST (08:00 UTC) and end 60 minutes later. Probably the time need will be in the 10-20 minute range, but better safe then sorry.

Mark

NSF Webmaster · « **Reply #93 on:** 03/24/2015 06:11 pm »

Well, the wonders of software :-) .. Maintenance was completed already without any downtime. If you experience something out of the ordinary, don't hesitate to report it here or by PM.

Mark

Chris Bergin · « **Reply #94 on:** 05/07/2015 12:36 pm »

I still seem to have access, but I know we have a DNS issue - and the site is quiet. We are getting it resolved.

Bargemanos · « **Reply #95 on:** 05/07/2015 12:43 pm »

Up and Running here again, The Netherlands.

ehb · « **Reply #96 on:** 05/07/2015 12:45 pm »

FYI - It is working again for me.
Was concerned when I couldn't access for a few minutes and then was concerned that I was a NSF junkie

edit: location Texas

Chris Bergin · « **Reply #97 on:** 05/07/2015 12:46 pm »

Yeah, site getting busier, so must be coming back. Might need to propitiate across the internet. Funny how it never became a problem for me! Thanks to those who let me know, otherwise I'd have been sat here twiddling my thumbs wondering why the site was quiet!

ehb · « **Reply #98 on:** 05/07/2015 12:49 pm »

Quote from: Chris Bergin on 05/07/2015 12:46 pm

Thanks to those who let me know, otherwise I'd have been sat here twiddling my thumbs wondering why the site was quiet!

What is the recommended way to let you know when the site is down ?

R7 · « **Reply #99 on:** 05/07/2015 12:49 pm »

North back on track. Used UK proxy for a while.