SMTP doesn't work to Comcast servers through pfSense

danhanisch

So I have pfSense setup as a pass through firewall between my main router and the switch serving the computers in the network. I have pfBlockerNG setup for geolocated IP blocking to keep traffic mainly in the US. I've had this issue where sending emails to Comcast addresses has been hit and miss, mostly miss. They just sit in the outbound queue from my Mail Enable. Since they eventually go through, I didn't address it as a major priority because of the rarity of sending to Comcast emails.

Anyhow, I performed the update from 2.4.5 to 2.5.2 and hit a crash, leading me to removing the firewall from the network and setting it back up, reinstalling and restoring config files.

I noticed that suddenly Comcast emails worked while the firewall was down. So here's where it gets weird. I put the firewall in and did a test and they stopped working again.

I did a test from my desktop (Win 10) using "telnet mx1.comcast.net 25" and was able to get a response.

I then did a test from my Server 2016 box and my connection times out.

Both machines are going through the pfSense.

The configuration of the pfSense is very basic. I have LAN bridged to WAN for pass-through. I don't have any firewall rules setup for any special cases and only am running pfBlockerNG for the geoIP blocking.

Thinking that it may be pfBlockerNG, I removed that and just had it as a pass-through firewall with no rules.

My win10 machine still can connect, but my Server2016 box cannot.

If I go into Systen > Advanced > Firewall & Nat > Disable all packet filtering, the Server 2016 box can telnet test and successfully send emails to Comcast. If I uncheck that box, it starts again. Out of curiousity, I tested it on another Server based box and get the same result, but Windows 10 boxes do not have this issue.

I've disabled the Windows firewall on all server machines as a test and don't have any other firewall software running. Windows Defender is disabled for the tests as well with no change in results. As a quick test, I created a new install Server 2016 VM and tried to do the telnet test:

"telnet mx1.comcast.net 25"

I get the same result.

Since it's pass-through, I don't have NAT setup, and on my Firewall > NAT > Outbound, I have it set to "Disabled" on the far right.

For the life of me, I cannot figure out why server machines are rejected with this to JUST Comcast mail servers and not desktop machines.

I was hoping someone could maybe clue me in on where to look. The only two settings that make a difference are "Disable packet filtering" under Advanced Settings and "net.link.bridge.pfil_bridge = 0" under System Tunables, which I gather does the same thing. bypassing any filtering on the bridged connections.

Thank you in advance. I can post settings and screenshots if needed, but I really have a super basic setup here with only the geo blocking and even uninstalling that module doesn't fix the issue, nor does a fresh install of pfSense 2.5.2.

Dan

SteveITS

mx1.comcast.net has both A and AAAA records, could one be using IPv6?

danhanisch

@steveits

Thank you for the reply. I checked on the server and on the network adapter, I have IPV6 unchecked on the servers as well as the desktops.

Gertjan

@danhanisch said in SMTP doesn't work to Comcast servers through pfSense:

"telnet mx1.comcast.net 25"

Are you sure about that port 25 ?

Mail clients should use port 587 or even better : 465.

Most ISP's told their customers to use port '25' to send mails, far back in the past.
This has been corrected, and comcast should have done so.

@danhanisch said in SMTP doesn't work to Comcast servers through pfSense:

So I have pfSense setup as a pass through firewall between my main router and the switch serving the computers in the network. I have pfBlockerNG setup for geolocated IP blocking

What do you maen with "pass through firewall" ?
From LAN to WAN is mostly pass through by default.

SteveITS

@gertjan said in SMTP doesn't work to Comcast servers through pfSense:

clients should use port 587

I thought about port 25 being blocked by Comcast (bottom of that page) but he says it works on some PCs, and also with pfSense disconnected.

provels

@danhanisch said in SMTP doesn't work to Comcast servers through pfSense:

mx1.comcast.net 25

~~Try smtp.comcast.net 587~~ <- answer to a question not asked!

Does server/server mail transport still use 25? Never thought about it.

I'd make a log everything rule from your server and see what the logs tell you.

Bob.Dig

@provels said in SMTP doesn't work to Comcast servers through pfSense:

Does server/server mail transport still use 25?

Sure they do.

Derelict

@danhanisch There is nothing in a default pfSense configuration that will prevent an outbound connection to a mail server on port 25. Almost all residential classes of service block such connections. They might allow them to a specific set of addresses for their own mail servers.

Port 587 should be used by MUAs to submit messages to MTAs. Authentication will need to be used.

Port 465 is a microsoft "standard" for TLS on initial connection without STARTTLS being issued, but there is no requirement for authentication like there is for port 587 (SMTP Submit).

The best way to submit email is to instruct the client to connect to port 587, fail if STARTTLS is not successful (To prevent an MITM from presenting an SMTP server that does not offer STARTTLS in the EHLO response), and authenticate.

If your mail provider does not allow such a configuration, use another mail provider.

danhanisch

Ok, so I appreciate everyone's replies, but I found what the problem was.

I'm not aware as to why this would only affect some addresses and not others, but when looking at the packets in Wireshark, I noticed the difference in a few things regarding the TCP settings.

First were the flags (such as congenstion....ECN and the like). That wasn't the cause.

Then I noticed on the TCP Window Scaling, the multiplier on the server machines was much higher than that of the Windows 10 machine.

So going into Powershell, I ran the following command:

Set-NetTCPSetting -AutoTuningLevelLocal Disabled

That takes the window multipler back down to the default level, which is where Windows 10 works.

Voila! It works now. Emails go out, telnet tests succeed.

I haven't gone through the pfSense settings to determine where this setting is. Can anyone point me in the right direction on that one?

Anyhow, I appreciate the responses. I checked the IPV6 scenario and that wasn't it. My communications were going out over IPV4, so I had to look at the raw packets and thankfully came to a resolution.

Dan

danhanisch

@gertjan

I misspoke. I meant to say "transparent", not pass-through.

Gertjan

@provels said in SMTP doesn't work to Comcast servers through pfSense:

Does server/server mail transport still use 25? Never thought about it.

They - server to server - only communicate over 25.
If not, 'your mail server' would never receive the mail others send to you.

edit :
See it like this : you publish a web server, but instead of have it listening to 443, you say : I use port 444.
Your site will work just fine, but many visitors will have issues with it ;)

@derelict said in SMTP doesn't work to Comcast servers through pfSense:

Port 587 should be used by MUAs to submit messages to MTAs. Authentication will need to be used.

Added for clarity :
MUA = in this case, the pfSense 'mail scripts' that deliver a mail to a server for dispatching. = typically your mail server at comcast, gmail, etc.
MTA = the (your) mail server - the one you're allowed to 'drop' mails for dispatching.

@derelict said in SMTP doesn't work to Comcast servers through pfSense:

Port 465 is a microsoft "standard" for TLS on initial connection without STARTTLS being issued, but there is no requirement for authentication like there is for port 587 (SMTP Submit).

Yeah, but activating '465' as a TLS only mail-drop entry implies

smtpd_tls_security_level=encrypt

and in that case you "have-to" :

smtpd_tls_auth_only=yes

See : http://www.postfix.org/postconf.5.html#smtpd_tls_security_level

Btw : 465 with 'auth' will be an an open relay. Having one in the past was funny. Today, it would be a complete disaster.

@danhanisch
Great that you found a solution

Keep in mind :
On a default, out of the box Windows PC with a freshly installed 'Thunderbird' mail client or Outlook Office 365, both MUA's
and
a default, pfSense install with no modification what so ever,
mail traffic, or actually any traffic passes just fine.

danhanisch

For reference, here's the MS document that assisted me:

https://docs.microsoft.com/en-us/windows-server/networking/technologies/network-subsystem/net-sub-performance-tuning-nics

Dan