> If the client SHOULD do something and doesn't, and your server does not know why, you SHOULD disconnect and move on.

That is not a rule.

In this situation the server can reject any message if it wants to, and not doing a SHOULD tests the server's patience, but it's still ultimately in the "server wanted to" category, not the "RFC was violated" category.

> If the server has considered fully the implications

The server "considers" nothing. The considerations are for the human implementers to make when building their software. And they can never presume to know why the software on the other side is working a certain way. Only that the RFC didn't make something mandatory.

The rejection isn't to be compliant with the RFC, it's a choice made by the server implementers.

That clearly means it’s not required.

How does Google know whether or not the sender has a valid reason? They cannot know that so for them to reject an email for it means they would reject emails that have valid reasons as well.

Either the server must explicitly confirm to servers or the clients must accept everything. Otherwise message delivery is not guaranteed. In the context of an email protocol, this often is a silent failure which causes real-world problems.

I don’t care what the protocol rfc says, the client arbitrarily rejecting an email from the server for some missing unimportant header (for deduction detection?) is silly.

Most of the time, in my experience, when one encounters a situation like this in Internet tech (i.e. "why is this suggestion treated like a hard requirement?"), this is the answer: "because attackers found a way to exploit the lack of the suggestion's implementation in the wild, so it is now a hard requirement."

The standards, to my observation, tend to lag the CVEs.

Side-note: If someone has built a reverse-database that annotates RFCs with overriding CVEs that have invalidated or rendered harmful part of the spec, I'd love to put that in my toolbox. It'd be nice-to-have in the extreme if it hasn't been created yet.

> … when dealing with ancient protocols

I’m doing some work with an email company at the moment. The company has been in the email space for decades. Holy moly email is just full of stuff like this. There is an insane amount of institutional knowledge about how email actually works - not what the specs say but what email servers need to actually do to process real emails and deal with real email clients and servers.

The rfcs try to keep up, but they’re missing a lot of details and all important context for why recommendations are as they are, and what you actually need to do to actually process real email you see in the wild. (And talk to the long tail of email software).

This conversation makes me think about cornering some of the engineers with a microphone. It’d be great to talk through the specs with them, to capture some of that missing commentary.

Happily, the meanings in RFCs are clearly specified, see https://www.rfc-editor.org/rfc/rfc2119.

Note "the full implications must be understood and carefully weighed before choosing a different course". Gmail and the other big hosters have full-time spam teams who spend a lot of time weighing implications, so I assume the implications of this was weighed.

I think it's the latter. But, in either case, you're right in that you get the same result.

Now, let's assume that if it is the latter (it's spam related), and Google were to accept the message, but then internally bin the message, it would be worse. At least in this case, they are bouncing the message. Because of this, the sender is at least aware that the message wasn't delivered.

Also, the author was able to get their mail delivered to a personal gmail.com address. The issue was with a Google Workspace custom email domain. This further makes me think of this as a security/spam related issue. Google is clearly capable of processing the message without a Message-id, they are just refusing for business customers.

My takeaway is that I think that Google is doing the least-wrong thing. And by being explicit in how they are handling it, it at least made the debugging for the author possible.

Also note: in a quick reading of RFC5321 (SMTP), rejecting messages for "policy reasons" is an acceptable outcome. I'm not sure if it applies completely here. The author should probably also be taking into account RFC5321 (SMTP) instead of just 5322 (message format).

Does it even matter when in reality it's more likely that this is intentional anti-competitive behavior by Google?

They once made all emails from my very reputable small German email provider (a company that has existed and provided email services long before Google existed) go into a black whole - not bounce them back or anything like that, mind you, their servers accepted them and made them disappear forever. I was in contact with the technicians then to get the problem fixed and they told me it's very difficult for them to even reach anyone at Google. It took them several days to get the problem fixed.

Of course, no one will ever be able to prove an intention behind these kind of "technical glitches." Nothing of significance ever happened when Google had large optics fiber connections with NSA installed illegally and claimed to have no knowledge of it, so certainly nothing will happen when small issues with interoperability occur and drive more people to Gmail.

Google is rejecting it to ensure incoming messages aren't spam. SHOULD means "you should do this unless you have a really, really good reason not to." Do they have a good reason not to? It doesn't seem so, meaning Viva is in the wrong here.

The RFC says a SHOULD is to be treated like a MUST, but well-justified exceptions are allowed.

I SHOULD have 8 hours of sleep every night. It's RECOMMENDED. However, there are times where it's best I don't (e.g. because of work, or travel, or needing to take someone to the hospital, etc.). It's definitely not that I MUST sleep 8 hours every night.

Hospitals understand that despite the fact that people SHOULD be sleep at night, they may have reason not to be, and remain able to serve people throughout the night. People understand that they SHOULD be sleep at night, so they are more often than not.

The robustness principle: "be conservative in what you do, be liberal in what you accept from others".

SHOULD generally means: some people might require it. implement it for best results

backward compatibility makes it hard to add MUST. using SHOULD is a good alternative

Typically, MUST means that if you don't do that then something will break at the protocol level.

SHOULD means that if you don't that, bad things are likely to happen, but it will not immediately break at the protocol level and during discussion in the IETF some people thought there could be valid reasons to violate the SHOULD.

Typically, IETF standards track RFCs consider the immediate effects of the protocol but often do not consider operational reality very well.

Sometimes operational reality is that a MUST gets violated because the standard is just wrong. Sometimes a SHOULD becomes required, etc.

Certainly for email, there is a lot you have to do to get your email accepted that is not spelled out in the RFCs.

MUST means omission is unacceptable. SHOULD means MUST unless you have a good, well-reasoned excuse.

“When jump getting over a wall, you SHOULD use three points of contact.”

For most cases you should use three points of contact. However, there may be other situations for example if someone is giving you a leg up, or you can pole vault, where another solution is preferred.

> This is 100 percent the case, and why these things are this way.

But gmail accepts emails without message-id on personal mailboxes apparently.

I think a mail 2.0 would be notify and pull based.... you notify a recipient's mail server that there's a message from <address> for them, then that server connects to the MX of record for the domain of <address> and retrieves <message-id> message.

Would this make mass emails and spam harder, absolutely. Would it be a huge burden for actual communications with people, not so much. From there actual white/black listing processes would work all that much better.

https://jmap.io

Ah, so it's just the evil bit, or lack of.