Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Empty Message-ID in SMTP Test email?

    Scheduled Pinned Locked Moved General pfSense Questions
    23 Posts 4 Posters 2.2k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • GertjanG
      Gertjan @matt0023
      last edited by Gertjan

      @matt0023 said in Empty Message-ID in SMTP Test email?:

      I've tried using port 25 and 587 now, and pfSense does not seem to do a STARTTLS at all.

      Don't stay in the dark.
      Connect to your mail server yourself :

      [24.03-RELEASE][root@pfSense.hf.tld]/root: telnet smtp.orange.fr 587
      Trying 80.12.26.33...
      Connected to smtp.orange.fr.
      Escape character is '^]'.
      220 opmta1mto19nd1 smtp.orange.fr ESMTP server ready
      EHLO whatthef#howisit gointhere
      250-opmta1mto19nd1 hello [82.127.26.108], pleased to meet you
      250-HELP
      250-AUTH LOGIN PLAIN
      250-SIZE 46000000
      250-ENHANCEDSTATUSCODES
      250-PIPELINING
      250-8BITMIME
      250-STARTTLS
      250 OK
      STARTTLS
      220 2.0.0 Ready to start TLS
      .....
      

      I entered the

      EHLO somethinghere
      

      and then the mail server (smtp.ornage.fr) presented me the list with its capabilities.
      As you can see, STARTTLS is listed.
      So I entered

      STARTTLS
      

      The connections witched to TLS .... and that's where I had to bail out as my (my hands and keyboard) manual TLS bit-stream capabilities are close to none ^^

      edit : when using port 587, the mail server that picks up can (should) offer STARTTLS. It is not mandatory.
      A bit like a web site that only supports http, not https. They still exist (I think ?!)

      No "help me" PM's please. Use the forum, the community will thank you.
      Edit : and where are the logs ??

      M 1 Reply Last reply Reply Quote 0
      • M
        matt0023 @Gertjan
        last edited by matt0023

        @Gertjan said in Empty Message-ID in SMTP Test email?:

        The connections witched to TLS .... and that's where I had to bail out as my (my hands and keyboard) manual TLS bit-stream capabilities are close to none ^^

        Heh you're not handcrafting encryption on the fly? tsk tsk ๐Ÿ˜†

        Yes my relay server is happily awaiting STARTTLS but pfSense is not interested. I'm assuming now that per the docs, it won't attempt it without expecting to also do a user-level authentication.

        [ip addresses and server name obscured]

        Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250-flux.example.net
        Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250-PIPELINING
        Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250-SIZE 10240000
        Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250-VRFY
        Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250-ETRN
        Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250-STARTTLS
        Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250-ENHANCEDSTATUSCODES
        Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250-8BITMIME
        Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250-DSN
        Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250-SMTPUTF8
        Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 250 CHUNKING
        Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: smtp_stream_setup: maxtime=300 enable_deadline=0 min_data_rate=0
        Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: < unknown[255.255.99.205]: MAIL FROM:<exampleguy@gmail.com>
        Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 530 5.7.0 Must issue a STARTTLS command first
        Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: smtp_stream_setup: maxtime=300 enable_deadline=0 min_data_rate=0
        Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: < unknown[255.255.99.205]: RSET
        Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 530 5.7.0 Must issue a STARTTLS command first
        Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: smtp_stream_setup: maxtime=300 enable_deadline=0 min_data_rate=0
        Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: < unknown[255.255.99.205]: QUIT
        Nov 10 16:36:03 flux postfix/submission/smtpd[1325256]: > unknown[255.255.99.205]: 221 2.0.0 Bye
        
        1 Reply Last reply Reply Quote 0
        • M
          matt0023
          last edited by

          ok just to get things working, I've gone ahead and started using smtp.gmail.com, port 587 and a Google App Password ๐Ÿ˜„ Thanks for the suggestion, @johnpoz !

          One thing that's interesting, the App Password is stored in plaintext in the pfSense backup config.xml file. Definitely not ideal, but I guess not too worrying for me, I don't leave my backups in an insecure place. But it seems odd since the other passwords in the config.xml file are hashed.

          I guess the workaround is checking "Encrypt this configuration file". Will do!

          johnpozJ 1 Reply Last reply Reply Quote 0
          • johnpozJ
            johnpoz LAYER 8 Global Moderator @matt0023
            last edited by

            @matt0023 you can use either 587 or if like @Gertjan mentioned you can use 465.. I just changed mine to 465 and working.. I need to do some more research, seems there is a rfc that calls for implicit port use of 465

            https://datatracker.ietf.org/doc/html/rfc8314#page-7

               It is desirable to migrate core
               protocols used by MUA software to Implicit TLS over time, for
               consistency as well as for the additional reasons discussed in
               Appendix A.  However, to maximize the use of encryption for
               submission, it is desirable to support both mechanisms for Message
               Submission over TLS for a transition period of several years.  As a
               result, clients and servers SHOULD implement both STARTTLS on
               port 587 and Implicit TLS on port 465 for this transition period.
               Note that there is no significant difference between the security
               properties of STARTTLS on port 587 and Implicit TLS on port 465 if
               the implementations are correct and if both the client and the server
               are configured to require successful negotiation of TLS prior to
               Message Submission.
            

            Either way the submission of the email and the auth would and should be encrypted - so from that point of view either or works.. It comes down to which one where you sending supports, and what your client supports I guess.

            And you should be using tls 1.3 for sure - which you can see from posted header info that is currently used by pfsense notification via email.. I have currently moved mine to 465, I doubt gmail is going to stop use of that port no matter what any rfcs - atleast for many man years..

            An intelligent man is sometimes forced to be drunk to spend time with his fools
            If you get confused: Listen to the Music Play
            Please don't Chat/PM me for help, unless mod related
            SG-4860 24.11 | Lab VMs 2.8, 24.11

            1 Reply Last reply Reply Quote 0
            • johnpozJ johnpoz referenced this topic on
            • GPz1100G
              GPz1100 @matt0023
              last edited by GPz1100

              @matt0023 Aside from sending to gmail directly, did you ever figure out how to add message-id's? It's not a huge problem as these emails are sent to the local smtp which is set to ignore noncompliance from local hosts, but would be nice to get it right.

              I came up with the following;

              https://github.com/pfsense/pfsense/blob/6bf3e080f56facab1f00e29acd24dff62d5bd707/src/etc/inc/notices.inc#L488

              After line 488 insert the following;

               /* generate message_id - added 20250113 by j */
                      $jmessage_id_rnd = uniqid('', true);
                      $jhostname = gethostname();
                      $jmessage_id = "<" . $jmessage_id_rnd . "@" . $jhostname . ">";
              

              This generates the message-id. Then after line 494 in the array, insert;

              "Message-ID" => $jmessage_id
              

              Resulting array looks like

              $headers = array(
                              "From"    => $from,
                              "To"      => $to,
                              "Subject" => $subject,
                              "Date"    => date("r"),
                              "Message-ID" => $jsmessage_id
                      );
              

              Email headers now contain
              Message-ID: <6785a220aa0593.67966469@pfs.local.domain>

              Sequence before the decimal point is a random hex string. After the decimal point appears to just be numeric. Either way, sufficiently random enough for me.

              There may be a more elegant way of doing this but my php is practically nonexistent.

              GertjanG M 2 Replies Last reply Reply Quote 0
              • GertjanG
                Gertjan @GPz1100
                last edited by

                @GPz1100 said in Empty Message-ID in SMTP Test email?:

                Message-ID

                The message-ID mail header is added by the originating mail server. The mail client, like Outlook Office 365, Thunderbird etc don't need to add one.
                That is, that what I make of it when reading the first 10 suggestions or so from here mail header Message-ID

                I did found this : Message could not be delivered due to Missing valid MessageID Header but it boiled down to a missing SPF ( !? that's like sending a letter without a postal stamp ).

                This is a mail my pfSense send to myself this morning :

                Return-Path: <gertjan.xxxxx@gmail.com>
                Received: from pfSense.bhf.tld ([2a01:cb19:907:a600:92ec:77ff:fe29:392a])
                        by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-436e2e92f7bsm198698155e9.38.2025.01.13.22.01.12
                        for <gertjan.xxxxx@gmail.com>
                        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
                        Mon, 13 Jan 2025 22:01:13 -0800 (PST)
                Message-ID: <6785fda9.050a0220.1ea7d1.3a86@mx.google.com>
                From: gertjan.xxxxx@gmail.com
                X-Google-Original-From: pfsense@bhf.tld
                To: gw.kroeb@gmail.com
                Subject: pfSense.bhf.tld - Notification
                Date: Tue, 14 Jan 2025 07:01:11 +0100
                
                Notifications in this message: 1
                ================================
                
                07:01:10 The following updates are available and can be installed using System > Package Manager:
                
                System_Patches: 2.2.20 ==> 2.2.20_1
                
                Some packages are part of the base system and will not show up in Package Manager. If any such updates are listed below, run `pkg upgrade` from the shell to install them:
                
                pfSense-pkg-System_Patches: 2.2.20 -> 2.2.20_1 [pfSense]
                

                The message ID was added by the receiving server (the originating server - and also the destination) :

                Message-ID: <6785fda9.050a0220.1ea7d1.3a86@mx.google.com>
                

                Btw : aha, their is a new Patches pfSense package ๐Ÿ‘

                No "help me" PM's please. Use the forum, the community will thank you.
                Edit : and where are the logs ??

                GPz1100G 1 Reply Last reply Reply Quote 0
                • GPz1100G
                  GPz1100 @Gertjan
                  last edited by GPz1100

                  @Gertjan This is interesting. According to this wiki page, this header can be added either by the client or the first email server.

                  But according to postfix (search for always_add_missing_headers), adding missing headers can break dkim. Thinking more about this, I can see dkim getting broken if a RECEIVING server adds the missing header. However the sending server is the one which generates the dkim hash in the first place. In my case the sending and receiving server is the same server.

                  This setting is not enabled in my postfix config (meaning missing headers are not added). More so, looking at notification emails from other servers such as truenas, freepbx, nextcloud, sophos utm (previous firewall), and proxmox, they all contain the message-id header. I can't find a RFC requirement (says its optional) for this, but does appear to be convention to have it generated by the entity sending the message, at least when it comes to server like entities.

                  Even thunderbird generates it's own message id. I can't find a setting to mangle the numbers to the left of @, but the domain portion can be adjusted by creating a string key such as mail.identity.id{X}.FQDN={some domain} in the advanced properties.

                  Message-ID: <4c4e5bba-c65d-1284-4fd1-09fc91973fff@zzzzz.zzzzz> for a message sent to myself after changing the above.

                  I don't use gmail/hotmail/yahoo/etc other than for burner account use. Been running my own mail stack for nearly 2 years now. Once the bugs got worked out it's been relatively maintenance free other than periodic updates. Even without a proper PTR for the domain, I can't remember the last time an outbound message bounced. There is proper spf/dkim/dmarc configured.

                  GertjanG 1 Reply Last reply Reply Quote 0
                  • GertjanG
                    Gertjan @GPz1100
                    last edited by Gertjan

                    @GPz1100 said in Empty Message-ID in SMTP Test email?:

                    But according to postfix (search for always_add_missing_headers), adding missing headers can break dkim. Thinking more about this, I can see dkim getting broken if a RECEIVING server adds the missing header. However the sending server is the one which generates the dkim hash in the first place. In my case the sending and receiving server is the same server.

                    This setting is not enabled in my postfix config (meaning missing headers are not added).

                    Interesting.
                    I had a look at my own main.cf, and found at the top :

                    # testing - 'Date' was missing in headers ... Not good for my filters http://www.postfix.org/cleanup.8.html
                    #always_add_missing_headers = yes
                    

                    which means to me, that I testing this paramter in the past (Normally I added more dates, comments etc, as the number of postfix settings is plain huge).
                    Probably because Outlook (the mail client) doesn't add "Date" mail headers while sending self-(Oulook) generated test mails, And then, dono who, Outlook itself, or the mail sever, rightfully, complains.

                    Breaking DKIM : if the cleanup, the sub postfix process that adds missing headers if needed, does this after a DKIM milter has signed the mail, then yeah, things break. That's probably the reason I've disabled "always_add_missing_headers "myself.

                    @GPz1100 said in Empty Message-ID in SMTP Test email?:

                    There is proper spf/dkim/dmarc configured

                    And when you think your done, check with, for example, this.
                    These days, its also DNSSEC, DANE and more.

                    No "help me" PM's please. Use the forum, the community will thank you.
                    Edit : and where are the logs ??

                    GPz1100G 1 Reply Last reply Reply Quote 0
                    • GPz1100G
                      GPz1100 @Gertjan
                      last edited by

                      @Gertjan said in Empty Message-ID in SMTP Test email?:

                      And when you think your done, check with, for example, this.
                      These days, its also DNSSEC, DANE and more.

                      Went down quite the rabbit hole with this one. I have now have working DANE support too.

                      How are you handling the TLSA rollover? For now I've set this up with the --reuse-key in certbot so the private key does not change. Not sure how I feel about this. This is for the mail server which handles its own certs.

                      It would be nice to have pf manage all the LE certs then push them to the respective servers after update. The latter can be accomplished with a script, but i'm not sure how to handle the rollover scenario.

                      It's possible to use cloudflare api to push the new certificate hash to an existing TLSA record, or just have it create a new one. Is it valid to have multiple TLSA records, even if only one of them is actually the corresponding to the one on the smtp server?

                      GertjanG 1 Reply Last reply Reply Quote 0
                      • GertjanG
                        Gertjan @GPz1100
                        last edited by

                        @GPz1100 said in Empty Message-ID in SMTP Test email?:

                        How are you handling the TLSA rollover? For now I've set this up with the --reuse-key in certbot so the private key does not change. Not sure how I feel about this. This is for the mail server which handles its own certs.

                        I'm using installed acme also on my mail server, a bare bone, no GUI Debian old school setup.
                        As pfSense uses acme.sh, I started to use "acme" on all my servers.

                        Afaik : I published the four known CA certificates :

                        _25._tcp		TLSA	2 1 1 (
                        				025490860B498AB73C6A12F27A49AD5FE230FAFE3AC8
                        				F6112C9B7D0AAD46941D )
                        			TLSA	2 1 1 (
                        				2BBAD93AB5C79279EC121507F272CBE0C6647A3AAE52
                        				E22F388AFAB426B4ADBA )
                        			TLSA	2 1 1 (
                        				6DDAC18698F7F1F7E1C69B9BCE420D974AC6F94CA8B2
                        				C761701623F99C767DC7 )
                        			TLSA	2 1 1 (
                        				8D02536C887482BC34FF54E41D2BA659BF85B341A0A2
                        				0AFADB5813DCFBCF286D )
                        			TLSA	2 1 1 (
                        				919C0DF7A787B597ED056ACE654B1DE9C0387ACF349F
                        				73734A4FD7B58CF612A4 )
                        			TLSA	2 1 1 (
                        				F1647A5EE3EFAC54C892E930584FE47979B7ACD1C76C
                        				1271BCA1C5076D869888 )
                        

                        Based upon : Provisioning DANE-TA(2) TLSA records for Let's Encrypt CAs.

                        @GPz1100 said in Empty Message-ID in SMTP Test email?:

                        It would be nice to have pf manage all the LE certs then push them to the respective servers after update

                        Yeah. I know what you mean.
                        I've a domain name strictly reserved for my pfSense LAN network(s) only.
                        With the pfsense acme.sh package I ask for a wild card certificates, so I can use it like
                        pfsense .my-domain.tld (192.168.1.1)
                        portal .my-domain.tld -192.168.2.1) as I use the captive portal on 192.168.2.1
                        I have also two synology NAS devices, so I'm using the same cert on them. Like :
                        nas1.my-domain.tld
                        nas2.my-domain.tld

                        I found a document that I used to run a script on pfSense (acme.sh) :

                        5de1a385-c70c-4d2f-ba23-43d3dcfb228d-image.png

                        The script :

                        #!/bin/sh
                        #
                        # Copy certificate files to temporary directory on Synology NAS:
                        scp -i /root/.ssh/nas1-openssh-private -O -P 22 /conf/acme/V2_domain_name.tld.crt root@nas1.domain_name.tld:/usr/syno/etc/certificate/_archive/certs/domain_name.tld/cert.pem
                        scp -i /root/.ssh/nas1-openssh-private -O -P 22 /conf/acme/V2_domain_name.tld.key root@nas1.domain_name.tld:/usr/syno/etc/certificate/_archive/certs/domain_name.tld/privkey.pem
                        scp -i /root/.ssh/nas1-openssh-private -O -P 22 /conf/acme/V2_domain_name.tld.fullchain root@nas1.domain_name.tld:/usr/syno/etc/certificate/_archive/certs/domain_name.tld/fullchain.pem
                        #
                        # Update certificate on Synology NAS remotely:
                        ssh -i /root/.ssh/nas1-openssh-private -p 22 root@nas1.domain_name.tld "sudo /usr/syno/etc/certificate/_archive/update-cert.sh 'domain_name.tld'"
                        #
                        # Delete temporary certificate files:
                        ssh -i /root/.ssh/nas1-openssh-private -p 22 root@nas1.domain_name.tld "rm -rf /usr/syno/etc/certificate/_archive/certs/domain_name.tld/*"
                        #
                        ssh -i /root/.ssh/nas1-openssh-private -p 22 root@nas1.domain_name.tld "/usr/syno/bin/synosystemctl restart nginx"
                        

                        It copies the cert info the a special place on my nas1.
                        Then, I execute a remote script on my nas1.
                        To finish, I delete the copied cert files (cleanup).

                        On the nas1 side, I've placed this ":

                        #!/bin/sh
                        #
                        # Copy certificate files to temporary directory on Synology NAS:
                        scp -i /root/.ssh/nas1-openssh-private -O -P 22 /conf/acme/V2_domain_name.tld.crt root@nas1.domain_name.tld:/usr/syno/etc/certificate/_archive/certs/domain_name.tld/cert.pem
                        scp -i /root/.ssh/nas1-openssh-private -O -P 22 /conf/acme/V2_domain_name.tld.key root@nas1.domain_name.tld:/usr/syno/etc/certificate/_archive/certs/domain_name.tld/privkey.pem
                        scp -i /root/.ssh/nas1-openssh-private -O -P 22 /conf/acme/V2_domain_name.tld.fullchain root@nas1.domain_name.tld:/usr/syno/etc/certificate/_archive/certs/domain_name.tld/fullchain.pem
                        #
                        # Update certificate on Synology NAS remotely:
                        ssh -i /root/.ssh/nas1-openssh-private -p 22 root@nas1.domain_name.tld "sudo /usr/syno/etc/certificate/_archive/update-cert.sh 'domain_name.tld'"
                        #
                        # Delete temporary certificate files:
                        ssh -i /root/.ssh/nas1-openssh-private -p 22 root@nas1.domain_name.tld "rm -rf /usr/syno/etc/certificate/_archive/certs/domain_name.tld/*"
                        #
                        # Reboot Synology NAS:
                        ## ssh -i /root/.ssh/nas1-openssh-private -p 22 root@nas1.domain_name.tld "sudo reboot now"
                        # Restart the Syno nginx boot Synology 
                        ssh -i /root/.ssh/nas1-openssh-private -p 22 root@nas1.domain_name.tld "/usr/syno/bin/synosystemctl restart nginx"" script file, and the helper python file.
                        

                        I've also a 'big' scanner/printer, but it's SSH access offers me no such cert transfer capabilities, so on that one, I copy the cert over every 60 days manually. And when I forgot to do, not a big problem.

                        @GPz1100 said in Empty Message-ID in SMTP Test email?:

                        It's possible to use cloudflare api to push the new certificate hash to an existing TLSA record, or just have it create a new one. Is it valid to have multiple TLSA records, even if only one of them is actually the corresponding to the one on the smtp server?

                        True.
                        Rotated "3 1 1" TLSA records based upon actual, and upcoming, certificates would be better.
                        As mentioned above, right now I publish (the hashes) of all the 4 current LE signing certificates, so I don't need to do anything, as long as LE doesn't change its certs it uses to create mine. Every x years or so, I have to change them, as LE certs aren't eternal.
                        So, I check ones in a while with the LE support pages for upcoming big changes, and do what is needed when time arrives.

                        No "help me" PM's please. Use the forum, the community will thank you.
                        Edit : and where are the logs ??

                        GPz1100G 1 Reply Last reply Reply Quote 0
                        • M
                          matt0023 @GPz1100
                          last edited by

                          @GPz1100 hey that's cool, nice work! Nope I haven't changed anything, still Gmail-ing it directly.
                          But I'd like to try out your patch so I can relay it locally ๐Ÿ‘

                          1 Reply Last reply Reply Quote 0
                          • GPz1100G
                            GPz1100 @Gertjan
                            last edited by GPz1100

                            @Gertjan said in Empty Message-ID in SMTP Test email?:

                            Rotated "3 1 1" TLSA records based upon actual, and upcoming, certificates would be better.
                            As mentioned above, right now I publish (the hashes) of all the 4 current LE signing certificates, so I don't need to do anything, as long as LE doesn't change its certs it uses to create mine. Every x years or so, I have to change them, as LE certs aren't eternal.
                            So, I check ones in a while with the LE support pages for upcoming big changes, and do what is needed when time arrives.

                            I decided to go a similar route, but more so based on the actual LE cert generated. I have as part of my LE update script, to also create the new tlsa record based on the new cert. Until I figure out how to parse json content, the old tlsa record will remain. Periodically (once a year?), I'll log in to CF and delete the older records.

                            As I understand it, so long as there's at least one valid tlsa record, then it's all good?

                            GertjanG 1 Reply Last reply Reply Quote 0
                            • GertjanG
                              Gertjan @GPz1100
                              last edited by

                              @GPz1100 said in Empty Message-ID in SMTP Test email?:

                              As I understand it, so long as there's at least one valid tlsa record, then it's all good?

                              That's what I do, I publish the four (5 ?) "2.1.1" hashes that could be used by LE to sign my certificate. As long as one of them matches, the TLSA validation will work out : example :

                              039e2d13-3531-42af-b85e-674d67acd371-image.png

                              No "help me" PM's please. Use the forum, the community will thank you.
                              Edit : and where are the logs ??

                              1 Reply Last reply Reply Quote 0
                              • First post
                                Last post
                              Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.