Empty Message-ID in SMTP Test email?

matt0023

@johnpoz oh thanks for testing, I wonder what I'm doing wrong.
Ah I thought pfSense needed a relay server (?)

So instead of gmail, I used an address local to my relay server, the message is accepted (I have my source network added as allowed there). But there is no message ID, not even null, just nothing in the header.

I'll keep poking around thanks!

johnpoz

@matt0023 said in Empty Message-ID in SMTP Test email?:

Ah I thought pfSense needed a relay server (?)

Why would you think that? I send notifications to my gmail address, so I send direct to gmail - why have a middle man? You also notice I send to 2 other email addresses as well, yahoo and icloud via gmail.

Gertjan

@johnpoz

Or : full TLS :

as gmail 'likes' 465 also.

@matt0023 The Message-ID is there :

johnpoz

@Gertjan 587 is Explicit to use tls, while 465 is implicit.

Received: from sg4860.home.arpa ([209.snipped])
        by smtp.gmail.com with ESMTPSA id e9e14a558f8ab-3a6f9890bacsm11013405ab.70.2024.11.10.14.30.56
        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
        Sun, 10 Nov 2024 14:30:58 -0800 (PST)

587 is the standard, did 465 ever even make it into the RFC?

Gertjan

@johnpoz said in Empty Message-ID in SMTP Test email?:

587 is Explicit to use tls, while 465 is implicit.

Isn't that the other way around ?
Port 587 starts non encrypted. After the HELO, the mail server lists its capabilities, and among them there can be 'STARTTLS'.
The mail client sends the STARTS command , and from then on the connection '587' is using TLS.
465 is explicit TLS from byte 0.

A mail client, if it's old and stupid enough, can decide not to switch the TLS, thus not sending STARTTLS, and send the mail non encrypted to the server.
Be ware : the mail, but also the user ID and the mail password ...

But hey, could me interpreting something wrong here, be aware : non native language user here ^^

@johnpoz said in Empty Message-ID in SMTP Test email?:

587 is the standard, did 465 ever even make it into the RFC?

Yeah, for a long time I was really convinced that 587 was to be ditched, as, my way of thinking, port 80 for web servers is also going to duty very soon now.
It's all 'TLS' these day.
Recnetly, I actually discovered that it is '587' that is still used, and '465' never really made it into the RFC as the method to be used.
Still, gmail, microsoft, my own ISP, (and my own mail servers) etc etc still propose it.

johnpoz

@Gertjan yes 587 does start as non encrypted.. But it is the standard, 465 never was a RFC that I am aware of.

If 465 is still supported, then sure you can use that - but 587 is backed by rfc and to me is the better choice.

matt0023

Thanks for the info, folks, yeah @johnpoz it looks like: no, the documentation does not say you must have a relay, but having configured multiple things like this over the years, I didn't think it an outlandish direction to go

That said, in this case gmail is the relay server but with auth enabled. Yeah I have no problem dropping on gmail directly. I assume you need an 'app auth' credential for that?

Re: what @Gertjan was saying, the documentation re: support for SMTP over SSL vs STARTTLS could be clearer IMO: Screenshot 2024-11-11 at 08.40.29.png
I've tried using port 25 and 587 now, and pfSense does not seem to do a STARTTLS at all.

I have no idea why pfSense doesn't create a Message ID with my test but I'm assuming if I do something like relay directly to Gmail with auth it will be proper. My mailserver is running 3.7.11-0+deb12u1 so it's not exactly ancient

matt0023

@matt0023 Actually re-reading these docs for the Nth time it sounds like pfSense won't attempt STARTTLS without expecting to also authenticate into the target server. That's sort of counterintuitive since those are two different things. I've seen it elsewhere discussed but it does sort of make me wonder, why isn't there a lightweight SMTP relay service in pfSense itself?

Gertjan

@matt0023 said in Empty Message-ID in SMTP Test email?:

I've tried using port 25 and 587 now, and pfSense does not seem to do a STARTTLS at all.

Don't stay in the dark.
Connect to your mail server yourself :

[24.03-RELEASE][root@pfSense.hf.tld]/root: telnet smtp.orange.fr 587
Trying 80.12.26.33...
Connected to smtp.orange.fr.
Escape character is '^]'.
220 opmta1mto19nd1 smtp.orange.fr ESMTP server ready
EHLO whatthef#howisit gointhere
250-opmta1mto19nd1 hello [82.127.26.108], pleased to meet you
250-HELP
250-AUTH LOGIN PLAIN
250-SIZE 46000000
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-8BITMIME
250-STARTTLS
250 OK
STARTTLS
220 2.0.0 Ready to start TLS
.....

I entered the

EHLO somethinghere

and then the mail server (smtp.ornage.fr) presented me the list with its capabilities.
As you can see, STARTTLS is listed.
So I entered

STARTTLS

The connections witched to TLS .... and that's where I had to bail out as my (my hands and keyboard) manual TLS bit-stream capabilities are close to none ^^

edit : when using port 587, the mail server that picks up can (should) offer STARTTLS. It is not mandatory.
A bit like a web site that only supports http, not https. They still exist (I think ?!)

matt0023

@Gertjan said in Empty Message-ID in SMTP Test email?:

The connections witched to TLS .... and that's where I had to bail out as my (my hands and keyboard) manual TLS bit-stream capabilities are close to none ^^

Heh you're not handcrafting encryption on the fly? tsk tsk

Yes my relay server is happily awaiting STARTTLS but pfSense is not interested. I'm assuming now that per the docs, it won't attempt it without expecting to also do a user-level authentication.

[ip addresses and server name obscured]

Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250-flux.example.net
Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250-PIPELINING
Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250-SIZE 10240000
Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250-VRFY
Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250-ETRN
Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250-STARTTLS
Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250-ENHANCEDSTATUSCODES
Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250-8BITMIME
Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250-DSN
Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250-SMTPUTF8
Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250 CHUNKING
Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: smtp_stream_setup: maxtime=300 enable_deadline=0 min_data_rate=0
Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: < unknown[255.255.99.205]: MAIL FROM:<exampleguy@gmail.com>
Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 530 5.7.0 Must issue a STARTTLS command first
Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: smtp_stream_setup: maxtime=300 enable_deadline=0 min_data_rate=0
Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: < unknown[255.255.99.205]: RSET
Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 530 5.7.0 Must issue a STARTTLS command first
Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: smtp_stream_setup: maxtime=300 enable_deadline=0 min_data_rate=0
Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: < unknown[255.255.99.205]: QUIT
Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 221 2.0.0 Bye

matt0023

ok just to get things working, I've gone ahead and started using smtp.gmail.com, port 587 and a Google App Password Thanks for the suggestion, @johnpoz !

One thing that's interesting, the App Password is stored in plaintext in the pfSense backup config.xml file. Definitely not ideal, but I guess not too worrying for me, I don't leave my backups in an insecure place. But it seems odd since the other passwords in the config.xml file are hashed.

I guess the workaround is checking "Encrypt this configuration file". Will do!

johnpoz

@matt0023 you can use either 587 or if like @Gertjan mentioned you can use 465.. I just changed mine to 465 and working.. I need to do some more research, seems there is a rfc that calls for implicit port use of 465

https://datatracker.ietf.org/doc/html/rfc8314#page-7

   It is desirable to migrate core
   protocols used by MUA software to Implicit TLS over time, for
   consistency as well as for the additional reasons discussed in
   Appendix A.  However, to maximize the use of encryption for
   submission, it is desirable to support both mechanisms for Message
   Submission over TLS for a transition period of several years.  As a
   result, clients and servers SHOULD implement both STARTTLS on
   port 587 and Implicit TLS on port 465 for this transition period.
   Note that there is no significant difference between the security
   properties of STARTTLS on port 587 and Implicit TLS on port 465 if
   the implementations are correct and if both the client and the server
   are configured to require successful negotiation of TLS prior to
   Message Submission.

Either way the submission of the email and the auth would and should be encrypted - so from that point of view either or works.. It comes down to which one where you sending supports, and what your client supports I guess.

And you should be using tls 1.3 for sure - which you can see from posted header info that is currently used by pfsense notification via email.. I have currently moved mine to 465, I doubt gmail is going to stop use of that port no matter what any rfcs - atleast for many man years..