No notification emails since yesterday's Gmail outage

[Ken the Bin]

Chris, Mark, et.al.

Yesterday Gmail had an outage of most services, including Gmail.  Reportedly some emails sent to Gmail email addresses were rejected with 5xy errors, which would indicate to the sending email server that the account is no longer valid.

I suspect that happened with my NSF forum notification emails and the NSF system thinks that my email address is no longer valid.  Gmail is working again.  I have verified that incoming emails now work correctly (I sent an email from a separate non-Gmail email account and it came through immediately).  But I'm still not getting notification emails.  I can see on my profile notifications page what topics have notifications so I know that I should be getting emails.  I use Thunderbird, but I checked on the Gmail website and verified that there aren't any emails hung up there.

I don't see any way to revalidate my email account.  I did an update to Account Settings without changing anything in the hope that the software would be smart enough to see that I did not change the email address but that it is (presumably) marked as invalid and would send a new email address verification, but alas, that didn't happen.

Is there something that I can do (hopefully short of changing to a different email address and then changing back)?

Or can you mark my email address as valid or trigger the email verification process?

Considering how many people use Gmail, there are probably other NSF users in the same situation.  That's what I'm creating a forum topic instead of sending a PM.

Thanks.

[gemmy0I]

Can confirm - same here. No notification emails since yesterday.

(Note that it's not just @gmail.com addresses; mine, for instance, is @googlemail.com, an alternate domain Gmail used to assign to some users in the past. I imagine other domains serviced through "hosted Gmail" would be affected as well.)

[Ken the Bin]

Can confirm - same here. No notification emails since yesterday.

(Note that it's not just @gmail.com addresses; mine, for instance, is @googlemail.com, an alternate domain Gmail used to assign to some users in the past. I imagine other domains serviced through "hosted Gmail" would be affected as well.)

It's my understanding your googlemail.com email address is also a gmail.com email address.  You should be able to just change the domain and carry on.

But yes, you are very probably correct about hosted domains being impacted.  I used to have an email address of [email protected][redacted].org for a non-profit organization that I was a key volunteer for.  It was hosted by Gmail and very probably would also have been impacted.

[Ken the Bin]

It sounds like Gmail may be having the same or a similar outage today:

December 15, 2020 1:29:00 PM PST

We're aware of a problem with Gmail affecting a significant subset of users. The affected users are able to access Gmail, but are seeing error messages, high latency, and/or other unexpected behavior. We will provide an update by December 15, 2020 2:30:00 PM PST detailing when we expect to resolve the problem. Please note that this resolution time is an estimate and may change.

[Ken the Bin]

A more recent update:

December 15, 2020 2:18:00 PM PST

Our team is continuing to investigate this issue. We will provide an update by December 15, 2020 3:00:00 PM PST with more information about this problem. Thank you for your patience. The affected users are able to access Gmail, but are seeing error messages, high latency, and/or other unexpected behavior.

[vaporcobra]

Same issue here 😩

[Ken the Bin]

Here are three additional updates from Google.  Today's issue is now resolved.  It's possible that users who didn't have their NSF notification emails shut down by yesterday's issue may have had that happen now because of today's issue.

December 15, 2020 2:44:00 PM PST

Gmail service has already been restored for some users, and we expect a resolution for all users in the near future. Please note this time frame is an estimate and may change. The affected users are able to access Gmail, but are seeing error messages, high latency, and/or other unexpected behavior.

December 15, 2020 3:09:00 PM PST

We expect to resolve the problem affecting a significant subset of users of Gmail at December 15, 2020 4:00:00 PM PST. Please note that this time frame is an estimate and may change. The affected users are able to access Gmail, but are seeing error messages, high latency, and/or other unexpected behavior.

December 15, 2020 3:51:00 PM PST

The problem with Gmail has been resolved. We apologize for the inconvenience and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are making continuous improvements to make our systems better. If you are still experiencing an issue, please contact us via the Google Help Center.

[Eylrid]

*Raises hand* I'm not getting notification emails either.

[Ken the Bin]

Chris, Mark, et.al.

Did someone do something?  I just received a "Topic reply" notification email.

Is anyone else that wasn't getting them now getting them again?

[Ken the Bin]

Per an article in TechCrunch, today's Gmail issue was generating 5xy errors like yesterday's, leading to the likelihood of additional users with invalidated email addresses.

(Emphasis in the original.)

One TechCrunch writer, meanwhile, noticed that emails he was sending to Gmail accounts appeared to immediately bounce, with Gmail’s server responding with an error reading “550-5.1.1 The email account that you tried to reach does not exist.”

[Eylrid]

Chris, Mark, et.al.

Did someone do something?  I just received a "Topic reply" notification email.

Is anyone else that wasn't getting them now getting them again?

I am

[rsdavis9]

I'm getting a few now. Not as many as there should be.

[gemmy0I]

I'm getting notification emails again. 👍 (Only for topics that have been updated since the emails started working again - notifications that were sent out during the downtime were lost and won't be regenerated until I visit the thread again - but I expected as much. I went through my list of notified topics in Settings and visited threads for which I'd missed emails to get things rolling again.)

I'm not so sure this was ever an issue with users' email addresses getting "invalidated" by the forum software in response to 5xx errors. That may be an RFC-acceptable response to 5xx errors, but it would be up to the forum software (SMF) to decide whether and how to act upon that. Given that the system evidently doesn't retain failed emails in a "backlog" to be tried again, simply doing nothing in response to a bounce and leaving the address in the system to attempt future notifications would be a reasonable design choice (as without a "backlog" there isn't a risk of bounces creating a runaway queue situation).

But that's just speculation on my part...I do not have experience as a SMF administrator so I don't know how it behaves or what options it provides (and/or if the NSF admins did something to "fix" the problem like manually clearing invalid flags in the database).

[Ken the Bin]

I'm not so sure this was ever an issue with users' email addresses getting "invalidated" by the forum software in response to 5xx errors. That may be an RFC-acceptable response to 5xx errors, but it would be up to the forum software (SMF) to decide whether and how to act upon that. Given that the system evidently doesn't retain failed emails in a "backlog" to be tried again, simply doing nothing in response to a bounce and leaving the address in the system to attempt future notifications would be a reasonable design choice (as without a "backlog" there isn't a risk of bounces creating a runaway queue situation).

But that's just speculation on my part...I do not have experience as a SMF administrator so I don't know how it behaves or what options it provides (and/or if the NSF admins did something to "fix" the problem like manually clearing invalid flags in the database).

I also do not have experience as an SMF administrator, but I have had some limited experience with administering a mailing list.  It is a common feature for mailing list software to invalidate email addresses that get repeated 5xy errors, and to not retain failed emails.  Otherwise if an email address is actually deleted, it would make for a very messy situation.

In this specific instance, it was a long time after the outage was cleared that I started receiving notification emails again, and I know that in the interim there were many replies which should have generated notification emails.

It is possible that the software automatically started trying again after some period of time.  The lack of any official response from the NSF adminstrators leaves us just guessing.  But as long as it keeps working, I'm not going to look the gift equus ferus caballus in the oral orifice.

[Navier–Stokes]

I have not received any email notifications today and some of the threads I have flagged were updated this afternoon.

[Ken the Bin]

I have not received any email notifications today and some of the threads I have flagged were updated this afternoon.

That's surprising.  I've only had one definite instance of a missing notification email since they starting coming through again.  I received a notification email for your post that I'm responding to.

Hopefully TPTB are watching this topic and will check on your situation. ❓❓❓

[gemmy0I]

I have not received any email notifications today and some of the threads I have flagged were updated this afternoon.

For what it's worth, if you would have received a notification from a particular thread during the outage, but that notification was lost due to the outage, the forum software likely marked it down as "sent notification" and thenceforward (per normal behavior) would not send you any further notifications for that thread until you visit the thread again.

That's what I observed on my end: when the outage ended, I only started getting notifications again for threads that I hadn't missed a notification for during the outage (or that I was otherwise caught up on, e.g. from manually visiting the threads to read their new posts during the outage period).

If you go to Profile -> Modify Settings -> Notifications, you will see the full list of all threads for which you are subscribed to notifications. Any thread in that list with the "new" icon (indicating you aren't caught up on it) is one that the system generally won't send (additional) notifications for until you visit it again. What I did was use that list to catch up on all the threads for which I missed notifications, and going forward I've received notifications normally.

(This is assuming you're using the default notification mode, i.e. to be notified "Instantly - but only for the first unread reply". In that case, the system thinks you have already been notified for the first unread reply - even though that notification was lost - and doesn't think it needs to send more.)

[the_other_Doug]

I have never set up to get notification e-mails, but I've been getting a pretty standard number of Pinterest e-mails to my gmail account.  Those and Expedia & Lyft ads, and Google & YouTube messages, are what I mostly get on gmail.  Everything else goes to my Comcast address.

So, it seems that the outage either didn't include me, or did but not in a way that I'd notice.  In any event, except for not allowing 5xy errors to remove the affected e-mail address from the forum software, I guess I don't see how this is a problem with the forum.  You wouldn't send a forum bug report if your internet connectivity was going in and out, since the forum software can do nothing about that.  It also can't do much about the gmail outages.

But yeah, you don't want the forum software turning off your notification settings due to a glitch in your mail servers...

[Ken the Bin]

I have never set up to get notification e-mails, but I've been getting a pretty standard number of Pinterest e-mails to my gmail account.  Those and Expedia & Lyft ads, and Google & YouTube messages, are what I mostly get on gmail.  Everything else goes to my Comcast address.

So, it seems that the outage either didn't include me, or did but not in a way that I'd notice.  In any event, except for not allowing 5xy errors to remove the affected e-mail address from the forum software, I guess I don't see how this is a problem with the forum.  You wouldn't send a forum bug report if your internet connectivity was going in and out, since the forum software can do nothing about that.  It also can't do much about the gmail outages.

But yeah, you don't want the forum software turning off your notification settings due to a glitch in your mail servers...

The forum software had no way of knowing it was a temporary glitch.  A 5xy error indicates a permanent error with the email.  (A 4xy error would indicate a temporary error.)

My reason for creating the topic was two-fold:

1. To find if there is some way I as a user could revalidate my email address (short of changing to a different email address and changing back), or if not, if there was some way that the administrators could revalidate it for me.

2. To provide a place for other users having the same issue to report it, which is why I created a topic rather than sending a PM.

[Chris Bergin]

"The lack of any official response from the NSF adminstrators leaves us just guessing"

This thread was in the wrong section of the forum. Moving it to where the team read issues.

[NSF Webmaster]

@All I will dig into this later today and report back. SMF doesn't block or invalidate any mail addresses itself but I think our upstream provider was triggered by this Gmail outage. Hang on

[NSF Webmaster]

On 12/15 and 12/16 we see a spike in supressed or failed messages but since then the system is showing a healthy stream of outbound (and delivered) messages.

@Ken Also your emailaddress is getting all "Delivered" confirmations for the last few days, so do you still have issues ?

[Ken the Bin]

On 12/15 and 12/16 we see a spike in supressed or failed messages but since then the system is showing a healthy stream of outbound (and delivered) messages.

@Ken Also your emailaddress is getting all "Delivered" confirmations for the last few days, so do you still have issues ?

I'm no longer having issues.  On December 16 around 05:30 UTC notifications started coming through again.

I wonder if your upstream provider did a mass reactivation of gmail email addresses because of the outage.  If so, good on them!