Empty Message-ID in SMTP Test email?
-
Thanks for the info, folks, yeah @johnpoz it looks like: no, the documentation does not say you must have a relay, but having configured multiple things like this over the years, I didn't think it an outlandish direction to go
That said, in this case gmail is the relay server but with auth enabled. Yeah I have no problem dropping on gmail directly. I assume you need an 'app auth' credential for that?
Re: what @Gertjan was saying, the documentation re: support for SMTP over SSL vs STARTTLS could be clearer IMO:
I've tried using port 25 and 587 now, and pfSense does not seem to do a STARTTLS at all.
I have no idea why pfSense doesn't create a Message ID with my test but I'm assuming if I do something like relay directly to Gmail with auth it will be proper. My mailserver is running 3.7.11-0+deb12u1 so it's not exactly ancient
-
@matt0023 Actually re-reading these docs for the Nth time it sounds like pfSense won't attempt STARTTLS without expecting to also authenticate into the target server. That's sort of counterintuitive since those are two different things. I've seen it elsewhere discussed but it does sort of make me wonder, why isn't there a lightweight SMTP relay service in pfSense itself?
-
@matt0023 said in Empty Message-ID in SMTP Test email?:
I've tried using port 25 and 587 now, and pfSense does not seem to do a STARTTLS at all.
Don't stay in the dark.
Connect to your mail server yourself :[24.03-RELEASE][root@pfSense.hf.tld]/root: telnet smtp.orange.fr 587 Trying 80.12.26.33... Connected to smtp.orange.fr. Escape character is '^]'. 220 opmta1mto19nd1 smtp.orange.fr ESMTP server ready EHLO whatthef#howisit gointhere 250-opmta1mto19nd1 hello [82.127.26.108], pleased to meet you 250-HELP 250-AUTH LOGIN PLAIN 250-SIZE 46000000 250-ENHANCEDSTATUSCODES 250-PIPELINING 250-8BITMIME 250-STARTTLS 250 OK STARTTLS 220 2.0.0 Ready to start TLS .....I entered the
EHLO somethinghereand then the mail server (smtp.ornage.fr) presented me the list with its capabilities.
As you can see, STARTTLS is listed.
So I enteredSTARTTLSThe connections witched to TLS .... and that's where I had to bail out as my (my hands and keyboard) manual TLS bit-stream capabilities are close to none ^^
edit : when using port 587, the mail server that picks up can (should) offer STARTTLS. It is not mandatory.
A bit like a web site that only supports http, not https. They still exist (I think ?!) -
@Gertjan said in Empty Message-ID in SMTP Test email?:
The connections witched to TLS .... and that's where I had to bail out as my (my hands and keyboard) manual TLS bit-stream capabilities are close to none ^^
Heh you're not handcrafting encryption on the fly? tsk tsk
Yes my relay server is happily awaiting STARTTLS but pfSense is not interested. I'm assuming now that per the docs, it won't attempt it without expecting to also do a user-level authentication.
[ip addresses and server name obscured]
Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250-flux.example.net Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250-PIPELINING Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250-SIZE 10240000 Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250-VRFY Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250-ETRN Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250-STARTTLS Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250-ENHANCEDSTATUSCODES Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250-8BITMIME Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250-DSN Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250-SMTPUTF8 Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250 CHUNKING Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: smtp_stream_setup: maxtime=300 enable_deadline=0 min_data_rate=0 Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: < unknown[255.255.99.205]: MAIL FROM:<exampleguy@gmail.com> Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 530 5.7.0 Must issue a STARTTLS command first Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: smtp_stream_setup: maxtime=300 enable_deadline=0 min_data_rate=0 Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: < unknown[255.255.99.205]: RSET Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 530 5.7.0 Must issue a STARTTLS command first Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: smtp_stream_setup: maxtime=300 enable_deadline=0 min_data_rate=0 Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: < unknown[255.255.99.205]: QUIT Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 221 2.0.0 Bye -
ok just to get things working, I've gone ahead and started using smtp.gmail.com, port 587 and a Google App Password
Thanks for the suggestion, @johnpoz !One thing that's interesting, the App Password is stored in plaintext in the pfSense backup config.xml file. Definitely not ideal, but I guess not too worrying for me, I don't leave my backups in an insecure place. But it seems odd since the other passwords in the config.xml file are hashed.
I guess the workaround is checking "Encrypt this configuration file". Will do!
-
@matt0023 you can use either 587 or if like @Gertjan mentioned you can use 465.. I just changed mine to 465 and working.. I need to do some more research, seems there is a rfc that calls for implicit port use of 465
https://datatracker.ietf.org/doc/html/rfc8314#page-7
It is desirable to migrate core protocols used by MUA software to Implicit TLS over time, for consistency as well as for the additional reasons discussed in Appendix A. However, to maximize the use of encryption for submission, it is desirable to support both mechanisms for Message Submission over TLS for a transition period of several years. As a result, clients and servers SHOULD implement both STARTTLS on port 587 and Implicit TLS on port 465 for this transition period. Note that there is no significant difference between the security properties of STARTTLS on port 587 and Implicit TLS on port 465 if the implementations are correct and if both the client and the server are configured to require successful negotiation of TLS prior to Message Submission.Either way the submission of the email and the auth would and should be encrypted - so from that point of view either or works.. It comes down to which one where you sending supports, and what your client supports I guess.
And you should be using tls 1.3 for sure - which you can see from posted header info that is currently used by pfsense notification via email.. I have currently moved mine to 465, I doubt gmail is going to stop use of that port no matter what any rfcs - atleast for many man years..
-
J johnpoz referenced this topic on
-
@matt0023 Aside from sending to gmail directly, did you ever figure out how to add message-id's? It's not a huge problem as these emails are sent to the local smtp which is set to ignore noncompliance from local hosts, but would be nice to get it right.
I came up with the following;
After line 488 insert the following;
/* generate message_id - added 20250113 by j */ $jmessage_id_rnd = uniqid('', true); $jhostname = gethostname(); $jmessage_id = "<" . $jmessage_id_rnd . "@" . $jhostname . ">";This generates the message-id. Then after line 494 in the array, insert;
"Message-ID" => $jmessage_idResulting array looks like
$headers = array( "From" => $from, "To" => $to, "Subject" => $subject, "Date" => date("r"), "Message-ID" => $jsmessage_id );Email headers now contain
Message-ID: <6785a220aa0593.67966469@pfs.local.domain>Sequence before the decimal point is a random hex string. After the decimal point appears to just be numeric. Either way, sufficiently random enough for me.
There may be a more elegant way of doing this but my php is practically nonexistent.
-
@GPz1100 said in Empty Message-ID in SMTP Test email?:
Message-ID
The message-ID mail header is added by the originating mail server. The mail client, like Outlook Office 365, Thunderbird etc don't need to add one.
That is, that what I make of it when reading the first 10 suggestions or so from here mail header Message-IDI did found this : Message could not be delivered due to Missing valid MessageID Header but it boiled down to a missing SPF ( !? that's like sending a letter without a postal stamp ).
This is a mail my pfSense send to myself this morning :
Return-Path: <gertjan.xxxxx@gmail.com> Received: from pfSense.bhf.tld ([2a01:cb19:907:a600:92ec:77ff:fe29:392a]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-436e2e92f7bsm198698155e9.38.2025.01.13.22.01.12 for <gertjan.xxxxx@gmail.com> (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Jan 2025 22:01:13 -0800 (PST) Message-ID: <6785fda9.050a0220.1ea7d1.3a86@mx.google.com> From: gertjan.xxxxx@gmail.com X-Google-Original-From: pfsense@bhf.tld To: gw.kroeb@gmail.com Subject: pfSense.bhf.tld - Notification Date: Tue, 14 Jan 2025 07:01:11 +0100 Notifications in this message: 1 ================================ 07:01:10 The following updates are available and can be installed using System > Package Manager: System_Patches: 2.2.20 ==> 2.2.20_1 Some packages are part of the base system and will not show up in Package Manager. If any such updates are listed below, run `pkg upgrade` from the shell to install them: pfSense-pkg-System_Patches: 2.2.20 -> 2.2.20_1 [pfSense]The message ID was added by the receiving server (the originating server - and also the destination) :
Message-ID: <6785fda9.050a0220.1ea7d1.3a86@mx.google.com>Btw : aha, their is a new Patches pfSense package
-
@Gertjan This is interesting. According to this wiki page, this header can be added either by the client or the first email server.
But according to postfix (search for always_add_missing_headers), adding missing headers can break dkim. Thinking more about this, I can see dkim getting broken if a RECEIVING server adds the missing header. However the sending server is the one which generates the dkim hash in the first place. In my case the sending and receiving server is the same server.
This setting is not enabled in my postfix config (meaning missing headers are not added). More so, looking at notification emails from other servers such as truenas, freepbx, nextcloud, sophos utm (previous firewall), and proxmox, they all contain the message-id header. I can't find a RFC requirement (says its optional) for this, but does appear to be convention to have it generated by the entity sending the message, at least when it comes to server like entities.
Even thunderbird generates it's own message id. I can't find a setting to mangle the numbers to the left of @, but the domain portion can be adjusted by creating a string key such as
mail.identity.id{X}.FQDN={some domain}in the advanced properties.Message-ID: <4c4e5bba-c65d-1284-4fd1-09fc91973fff@zzzzz.zzzzz>for a message sent to myself after changing the above.I don't use gmail/hotmail/yahoo/etc other than for burner account use. Been running my own mail stack for nearly 2 years now. Once the bugs got worked out it's been relatively maintenance free other than periodic updates. Even without a proper PTR for the domain, I can't remember the last time an outbound message bounced. There is proper spf/dkim/dmarc configured.
-
@GPz1100 said in Empty Message-ID in SMTP Test email?:
But according to postfix (search for always_add_missing_headers), adding missing headers can break dkim. Thinking more about this, I can see dkim getting broken if a RECEIVING server adds the missing header. However the sending server is the one which generates the dkim hash in the first place. In my case the sending and receiving server is the same server.
This setting is not enabled in my postfix config (meaning missing headers are not added).
Interesting.
I had a look at my own main.cf, and found at the top :# testing - 'Date' was missing in headers ... Not good for my filters http://www.postfix.org/cleanup.8.html #always_add_missing_headers = yeswhich means to me, that I testing this paramter in the past (Normally I added more dates, comments etc, as the number of postfix settings is plain huge).
Probably because Outlook (the mail client) doesn't add "Date" mail headers while sending self-(Oulook) generated test mails, And then, dono who, Outlook itself, or the mail sever, rightfully, complains.Breaking DKIM : if the cleanup, the sub postfix process that adds missing headers if needed, does this after a DKIM milter has signed the mail, then yeah, things break. That's probably the reason I've disabled "always_add_missing_headers "myself.
@GPz1100 said in Empty Message-ID in SMTP Test email?:
There is proper spf/dkim/dmarc configured
And when you think your done, check with, for example, this.
These days, its also DNSSEC, DANE and more. -
@Gertjan said in Empty Message-ID in SMTP Test email?:
And when you think your done, check with, for example, this.
These days, its also DNSSEC, DANE and more.Went down quite the rabbit hole with this one. I have now have working DANE support too.
How are you handling the TLSA rollover? For now I've set this up with the --reuse-key in certbot so the private key does not change. Not sure how I feel about this. This is for the mail server which handles its own certs.
It would be nice to have pf manage all the LE certs then push them to the respective servers after update. The latter can be accomplished with a script, but i'm not sure how to handle the rollover scenario.
It's possible to use cloudflare api to push the new certificate hash to an existing TLSA record, or just have it create a new one. Is it valid to have multiple TLSA records, even if only one of them is actually the corresponding to the one on the smtp server?
-
@GPz1100 said in Empty Message-ID in SMTP Test email?:
How are you handling the TLSA rollover? For now I've set this up with the --reuse-key in certbot so the private key does not change. Not sure how I feel about this. This is for the mail server which handles its own certs.
I'm using installed acme also on my mail server, a bare bone, no GUI Debian old school setup.
As pfSense uses acme.sh, I started to use "acme" on all my servers.Afaik : I published the four known CA certificates :
_25._tcp TLSA 2 1 1 ( 025490860B498AB73C6A12F27A49AD5FE230FAFE3AC8 F6112C9B7D0AAD46941D ) TLSA 2 1 1 ( 2BBAD93AB5C79279EC121507F272CBE0C6647A3AAE52 E22F388AFAB426B4ADBA ) TLSA 2 1 1 ( 6DDAC18698F7F1F7E1C69B9BCE420D974AC6F94CA8B2 C761701623F99C767DC7 ) TLSA 2 1 1 ( 8D02536C887482BC34FF54E41D2BA659BF85B341A0A2 0AFADB5813DCFBCF286D ) TLSA 2 1 1 ( 919C0DF7A787B597ED056ACE654B1DE9C0387ACF349F 73734A4FD7B58CF612A4 ) TLSA 2 1 1 ( F1647A5EE3EFAC54C892E930584FE47979B7ACD1C76C 1271BCA1C5076D869888 )Based upon : Provisioning DANE-TA(2) TLSA records for Let's Encrypt CAs.
@GPz1100 said in Empty Message-ID in SMTP Test email?:
It would be nice to have pf manage all the LE certs then push them to the respective servers after update
Yeah. I know what you mean.
I've a domain name strictly reserved for my pfSense LAN network(s) only.
With the pfsense acme.sh package I ask for a wild card certificates, so I can use it like
pfsense .my-domain.tld (192.168.1.1)
portal .my-domain.tld -192.168.2.1) as I use the captive portal on 192.168.2.1
I have also two synology NAS devices, so I'm using the same cert on them. Like :
nas1.my-domain.tld
nas2.my-domain.tldI found a document that I used to run a script on pfSense (acme.sh) :
The script :
#!/bin/sh # # Copy certificate files to temporary directory on Synology NAS: scp -i /root/.ssh/nas1-openssh-private -O -P 22 /conf/acme/V2_domain_name.tld.crt root@nas1.domain_name.tld:/usr/syno/etc/certificate/_archive/certs/domain_name.tld/cert.pem scp -i /root/.ssh/nas1-openssh-private -O -P 22 /conf/acme/V2_domain_name.tld.key root@nas1.domain_name.tld:/usr/syno/etc/certificate/_archive/certs/domain_name.tld/privkey.pem scp -i /root/.ssh/nas1-openssh-private -O -P 22 /conf/acme/V2_domain_name.tld.fullchain root@nas1.domain_name.tld:/usr/syno/etc/certificate/_archive/certs/domain_name.tld/fullchain.pem # # Update certificate on Synology NAS remotely: ssh -i /root/.ssh/nas1-openssh-private -p 22 root@nas1.domain_name.tld "sudo /usr/syno/etc/certificate/_archive/update-cert.sh 'domain_name.tld'" # # Delete temporary certificate files: ssh -i /root/.ssh/nas1-openssh-private -p 22 root@nas1.domain_name.tld "rm -rf /usr/syno/etc/certificate/_archive/certs/domain_name.tld/*" # ssh -i /root/.ssh/nas1-openssh-private -p 22 root@nas1.domain_name.tld "/usr/syno/bin/synosystemctl restart nginx"It copies the cert info the a special place on my nas1.
Then, I execute a remote script on my nas1.
To finish, I delete the copied cert files (cleanup).On the nas1 side, I've placed this ":
#!/bin/sh # # Copy certificate files to temporary directory on Synology NAS: scp -i /root/.ssh/nas1-openssh-private -O -P 22 /conf/acme/V2_domain_name.tld.crt root@nas1.domain_name.tld:/usr/syno/etc/certificate/_archive/certs/domain_name.tld/cert.pem scp -i /root/.ssh/nas1-openssh-private -O -P 22 /conf/acme/V2_domain_name.tld.key root@nas1.domain_name.tld:/usr/syno/etc/certificate/_archive/certs/domain_name.tld/privkey.pem scp -i /root/.ssh/nas1-openssh-private -O -P 22 /conf/acme/V2_domain_name.tld.fullchain root@nas1.domain_name.tld:/usr/syno/etc/certificate/_archive/certs/domain_name.tld/fullchain.pem # # Update certificate on Synology NAS remotely: ssh -i /root/.ssh/nas1-openssh-private -p 22 root@nas1.domain_name.tld "sudo /usr/syno/etc/certificate/_archive/update-cert.sh 'domain_name.tld'" # # Delete temporary certificate files: ssh -i /root/.ssh/nas1-openssh-private -p 22 root@nas1.domain_name.tld "rm -rf /usr/syno/etc/certificate/_archive/certs/domain_name.tld/*" # # Reboot Synology NAS: ## ssh -i /root/.ssh/nas1-openssh-private -p 22 root@nas1.domain_name.tld "sudo reboot now" # Restart the Syno nginx boot Synology ssh -i /root/.ssh/nas1-openssh-private -p 22 root@nas1.domain_name.tld "/usr/syno/bin/synosystemctl restart nginx"" script file, and the helper python file.I've also a 'big' scanner/printer, but it's SSH access offers me no such cert transfer capabilities, so on that one, I copy the cert over every 60 days manually. And when I forgot to do, not a big problem.
@GPz1100 said in Empty Message-ID in SMTP Test email?:
It's possible to use cloudflare api to push the new certificate hash to an existing TLSA record, or just have it create a new one. Is it valid to have multiple TLSA records, even if only one of them is actually the corresponding to the one on the smtp server?
True.
Rotated "3 1 1" TLSA records based upon actual, and upcoming, certificates would be better.
As mentioned above, right now I publish (the hashes) of all the 4 current LE signing certificates, so I don't need to do anything, as long as LE doesn't change its certs it uses to create mine. Every x years or so, I have to change them, as LE certs aren't eternal.
So, I check ones in a while with the LE support pages for upcoming big changes, and do what is needed when time arrives.No "help me" PM's please. Use the forum, the community will thank you.
Edit : and where are the logs ?? -
@GPz1100 hey that's cool, nice work! Nope I haven't changed anything, still Gmail-ing it directly.
But I'd like to try out your patch so I can relay it locally -
@Gertjan said in Empty Message-ID in SMTP Test email?:
Rotated "3 1 1" TLSA records based upon actual, and upcoming, certificates would be better.
As mentioned above, right now I publish (the hashes) of all the 4 current LE signing certificates, so I don't need to do anything, as long as LE doesn't change its certs it uses to create mine. Every x years or so, I have to change them, as LE certs aren't eternal.
So, I check ones in a while with the LE support pages for upcoming big changes, and do what is needed when time arrives.I decided to go a similar route, but more so based on the actual LE cert generated. I have as part of my LE update script, to also create the new tlsa record based on the new cert. Until I figure out how to parse json content, the old tlsa record will remain. Periodically (once a year?), I'll log in to CF and delete the older records.
As I understand it, so long as there's at least one valid tlsa record, then it's all good?
-
@GPz1100 said in Empty Message-ID in SMTP Test email?:
As I understand it, so long as there's at least one valid tlsa record, then it's all good?
That's what I do, I publish the four (5 ?) "2.1.1" hashes that could be used by LE to sign my certificate. As long as one of them matches, the TLSA validation will work out : example :
