Author Topic: Site Demand During Major SpaceX Events  (Read 49163 times)

Online Chris Bergin

Site Demand During Major SpaceX Events
« on: 04/20/2013 12:32 pm »
*This thread will be moved around to the top of sections that have a major event.*

(Old posts are before we added more solutions. Newer posts are more to do with increasing capability)

=====

Despite the large volume of people who access the site during big launches and events, our super duper servers - paid for by our super loyal L2 members - can usually cope.

However, as seen during Dragon's troublesome early minutes on orbit during CRS-2 (https://forum.nasaspaceflight.com/index.php?topic=31195.msg1019736#msg1019736), a massive peak can strain the servers and slow the site down (and even knock the site down for a few minutes, as one solution is to reboot the servers).

It's rare, but per the above example, we were getting hit hundreds and hundreds of times per second by thousands of visitors, and many of those visitors were mashing the refresh like that arcade game Track and Field (remember that? http://tinyurl.com/c3tf9kn ;D). Only mainstream sites that have a room of in-house servers can cope with such demand.

Again, we have very capable servers here thanks to the L2 subscriptions funding them (massive database, good bandwidth, etc - remember, the forum side is heavy bandwidth), but an action we will now take, if we see the forum slowing down, is to move to member only as a temp measure. In other words, stopping guest access to the forum, if only for a few hours.

So IF that is ever required, it's a good idea to make sure you are logged into the forum BEFORE such an action is required. In other words, be logged in for major events.

To gain a membership/login, it's very simple, it's free and it even allows you to post on the threads, etc.

WE DO NOT USE YOUR E-MAIL FOR ANYTHING DODGY. The e-mail you use to register is simply used to send you a confirmation e-mail. It's never sold or passed on or used for anything and only I can see it as admin, no one else. You will get one e-mail and that's it. You do not need to set up a "[email protected]" e-mail address to carry out the above process. Using a working e-mail address also means you can set it up for notifications on threads or PMs, only if you wish (it's not default), recover a lost password and even be one step away from it being upgraded to L2. No one has ever been spammed by registering with this site's forum.

Note, once you set up your username and password, your account will be active. You do not have to wait for an e-mail and click an authorization link, etc.

So bottom line is, guests are always welcome here, but to avoid any potential disappointment of losing access to the forum during a major event due to demand, make sure you set yourself up with a membership and be logged in that day.
« Last Edit: 05/28/2020 12:25 am by Chris Bergin »
Support NSF via L2 -- Help improve NSF -- Site Rules/Feedback/Updates
**Not a L2 member? Whitelist this forum in your adblocker to support the site and ensure full functionality.**

Online Chris Bergin

This is far less of a concern after our latest servers coped very well with the Cassiope launch, with zero downtime and all the guests still on the forum - via what was the second highest visitation rate since the site begun.

However, I'm keeping this place holder, as we can't foresee what may have during future events, such as the reason we were overloaded during CRS-2 is because everyone pounced on the site when Dragon had its hiccup after staging. So that's an example as to where we would remove guests for a temp period.....and as such it's still best you get yourself a membership.

So, good news. Top notch servers proved their worth on Super Sunday, but always best to avoid disappointment, per the opening post.
« Last Edit: 10/02/2013 03:34 pm by Chris Bergin »
Support NSF via L2 -- Help improve NSF -- Site Rules/Feedback/Updates
**Not a L2 member? Whitelist this forum in your adblocker to support the site and ensure full functionality.**

Online Chris Bergin

I should update this as that small lull of Cassiope was a one off. We became very, very busy on the following launches and we do need now need to be logged into the forum for around a SpaceX launch.

Again, it's free. No one can see your e-mail (apart from me if I wanted ;) ) and we never, ever sell your e-mail or spam it.
Support NSF via L2 -- Help improve NSF -- Site Rules/Feedback/Updates
**Not a L2 member? Whitelist this forum in your adblocker to support the site and ensure full functionality.**

Online Chris Bergin

So DSCOVR attempt 1 was the first test of beefed up servers and they coped well with large demand all the way through the count.

When the talk of a scrub was coming, inside the terminal, the peak (which was occurring as per usual) really spiked - to crazy levels. Slowing us down....although we amazingly stayed up (the servers actually coped), but it was slow loading (forum interface was overloading), causing a bottleneck. Within 30 seconds of seeing that, I had removed guests and we sped back up again.

Mark showed me what the peak looked like - see attachment screenshot.

For attempt 2 we will keep guests on as long as possible to test the latest improvements he installed yesterday. I will - as always - reserve the option to remove guests for a short period if we get bottlenecked, so this is all still in effect.
Support NSF via L2 -- Help improve NSF -- Site Rules/Feedback/Updates
**Not a L2 member? Whitelist this forum in your adblocker to support the site and ensure full functionality.**

Offline mheney

  • The Next Man on the Moon
  • Global Moderator
  • Full Member
  • *****
  • Posts: 780
  • Silver Spring, MD
  • Liked: 398
  • Likes Given: 199
Re: GUEST ACCESS TO THE SITE'S FORUM DURING MAJOR EVENTS
« Reply #4 on: 02/10/2015 03:47 pm »
FWIW, I did get a 500 error about two minutes after the scrub - but it was a one-shot deal.  I reloaded, and the page came up fine.

Online Chris Bergin

Wow. So Mark has really gotten us improved!

We never removed guests throughout this latest launch and we were up and smooth throughout - bar a 30 second timeout (though I was watching real time visitations and people were still getting in during that 30 seconds). Was after the first burn, so we go through the peak nicely!

Looks like we're heading the right way to deal with the massive demand of these events!
Support NSF via L2 -- Help improve NSF -- Site Rules/Feedback/Updates
**Not a L2 member? Whitelist this forum in your adblocker to support the site and ensure full functionality.**

Online Chris Bergin

For CRS-7 we were coping very well with very high demand. I simply had to remove guests after the failure as I knew there would be an unprecedented surge - and there was. Site stayed up and strong throughout.

Guests back on soon.
Support NSF via L2 -- Help improve NSF -- Site Rules/Feedback/Updates
**Not a L2 member? Whitelist this forum in your adblocker to support the site and ensure full functionality.**

Online Chris Bergin

I'm using a continuation of this thread as it's a great sign of how we've improved the servers and site in the years since, despite the continued increase in visitors. Major thanks to L2 members for allowing us to pay for this continued evolution in hosting. Remember, we're not owned by a media group or Mark Zuckerberg.....and there are very few large independent sites around these days.

To put this into context per the improvements, our cruising level (average use of the forum during a day even when there isn't a launch etc.) is the level that would have crashed us during the aforementioned "Bad Gateway!" days. However, I think we all know Falcon Heavy is going to create unprecedented demand and we've been working towards this with improved hosting and numerous other changes to at least give us a fighting chance.

Some pointers on where surges in demand come from during major launch events:

1) Prelaunch - people checking the internet for info ahead of the webcast. When the webcast starts a good amount of people concentrate on that.
2) Holds, delays - people rushing to get more info on what's caused it.
3) Post launch (but still live coverage) - an example is booster landing but shaky webcast of status, coast phase, etc.
4) Post-Webcast - this is where we see incredible surges as people rush on to the internet to discuss what they've just watched.
5) Failures, issues. Which is why CRS-7 was the last time we were hitting problems and had to cull guests for a while.
6) Other sites being knocked off line via high demand and their visitors heading to us and other sites to keep up.

Interesting sidenote, I believe our "record" for the number of people on site at the same time was Curiosity's landing, but everyone was watching the webcast and not hammering NSF. One of those lifetime moments where we all wanted to be in "the same room" during that historic moment.

For this event we'll keep the forum as it is until we reach a certain level, then we may use the option to remove guest access to the forum to help us on the demand side. The problem with that is a lot of guests then quickly surge to sign up to gain a login and that is demanding too, so we'll potentially cease new registrations during that period.

While I'd love NSF to be able to cope with thousands of people mashing F5 (not that even major sites can cope with that), its primary mission is to document live events. It's actually more about before and after launches per its news and usefulness....but documenting launches as they happen so people can refer back to them, with visuals, screenshots, photos and videos is where our strength is, especially in this era of text based Web 2.0. Being different is what makes you valuable. We're bandwidth heavy by nature.

That's the main reason we protect the site during surges, to allow for that live documentation of the event so people can refer back to it with text and visual resources, all posted live, but available for years to come. The stats show some people - to this day - regularly go through our live flight days of Shuttle missions, for example.

Also, if we do struggle, there are alternatives, especially ones owned by big companies who can cope with huge demands, such as Facebook and Reddit.

Unofficial sites, but I'm sure you all know them, but still worth bookmarking.
SpaceX Reddit: https://www.reddit.com/r/spacex/
SpaceX Facebook https://www.facebook.com/groups/spacexgroup/

Who knows, maybe our improvements are at such a level we'll be able to cope, but I'd rather post this before the event and not as some "sorry the site went down", post.

The site will be heavily monitored and Mark will be waving his magic wand around, so let's see how we do, but note the potential mitigation if you're reading NSF as a lurker/guest, which is by far the majority of visitors here.
« Last Edit: 02/05/2018 07:14 pm by Chris Bergin »
Support NSF via L2 -- Help improve NSF -- Site Rules/Feedback/Updates
**Not a L2 member? Whitelist this forum in your adblocker to support the site and ensure full functionality.**

Online Chris Bergin

(Falcon Heavy Maiden Launch)

Very busy with three hours to go. Mark's got the forum spread over several resources and we're looking good so far. You can all ignore the "users online" counter because of the spread. I'll report back with the actual stats at the end of the event.

IF and only IF there's major problems, follow @nasaspaceflight as I do some coverage in there too....which I like as it's a nice summary resource, live.
« Last Edit: 05/28/2020 12:27 am by Chris Bergin »
Support NSF via L2 -- Help improve NSF -- Site Rules/Feedback/Updates
**Not a L2 member? Whitelist this forum in your adblocker to support the site and ensure full functionality.**

Online Chris Bergin

(Falcon Heavy pre-launch with 90 mins to go).

Ok then. Many apologies about that. We were coping absolutely fine with very large demand. Everything we had implemented was working. Then without any notice we fell over (not the news site). It wasn't load, it was something else, so we think we can continue. Mark was also driving home, terribly unfortunate timing all round.

Onwards!

(Was fine from T-60 through to launch and beyond!)
« Last Edit: 05/28/2020 12:29 am by Chris Bergin »
Support NSF via L2 -- Help improve NSF -- Site Rules/Feedback/Updates
**Not a L2 member? Whitelist this forum in your adblocker to support the site and ensure full functionality.**

Online Chris Bergin

Well we survived after the above, and through demand that was massively in excess of the above server hamster fatality, so that was an unrelated fault (to load demand). We had the biggest ever surge, on what has been a record day for the site, after SpaceX's webcast ended and had to remove guests for a while as we were getting sluggish (we didn't go down).

Will post the stats in the morning, but my goodness! ;D
Support NSF via L2 -- Help improve NSF -- Site Rules/Feedback/Updates
**Not a L2 member? Whitelist this forum in your adblocker to support the site and ensure full functionality.**

Online Chris Bergin

Got to say, that surge when the booster was spinning was crazy, but we stayed strong and with guests on! Amazing work from Mark to get us to this capability and we're adding more powerful cloud servers ahead of next year's big events!

And thanks again to the L2 members who without we'd never be able to spend this much on the hosting side!
Support NSF via L2 -- Help improve NSF -- Site Rules/Feedback/Updates
**Not a L2 member? Whitelist this forum in your adblocker to support the site and ensure full functionality.**

Online Chris Bergin

Re: Site Demand During Major SpaceX Events
« Reply #12 on: 05/28/2020 12:22 am »
These old posts make me chuckle as we would have been taken down by 10 percent of today's demand noted in those earlier posts.

Anyway, DM-2's demand with about an hour to go was crazy! It did impact us for a few mins before getting it back properly. We beat Falcon Heavy's maiden launch for demand, even despite DM-2 not actually launching, so we need to add yet more hamsters to the set up. We'll be rebooting things in the morning (UTC, when the site is pretty quiet compared to most of the day).

More:

https://forum.nasaspaceflight.com/index.php?topic=51046.0

But as noted above, thanks to our L2 members who literally allow us to afford these crazy hosting set ups.
« Last Edit: 05/28/2020 12:29 am by Chris Bergin »
Support NSF via L2 -- Help improve NSF -- Site Rules/Feedback/Updates
**Not a L2 member? Whitelist this forum in your adblocker to support the site and ensure full functionality.**

Tags:
 

Advertisement NovaTech
Advertisement Northrop Grumman
Advertisement
Advertisement Margaritaville Beach Resort South Padre Island
Advertisement Brady Kenniston
Advertisement NextSpaceflight
Advertisement Nathan Barker Photography
1