SMTP over WANB? (Multi-WAN config)

pfnewbe

This screenshot om my rules

![Screenshot-fw1.lan - Firewall: Rules - Mozilla Firefox.png](/public/imported_attachments/1/Screenshot-fw1.lan - Firewall: Rules - Mozilla Firefox.png)
![Screenshot-fw1.lan - Firewall: Rules - Mozilla Firefox.png_thumb](/public/imported_attachments/1/Screenshot-fw1.lan - Firewall: Rules - Mozilla Firefox.png_thumb)

danswartz

Hmmm, looks okay. Are you sure the inbound smtp server has a default gateway pointing back to the pfsense? If so, can you do a packet capture on the LAN interface while you try to connect from outside?

pfnewbe

Yup. Looks OK.

0.0.0.0 192.168.2.254 0.0.0.0 UG 0 0 0 eth0

pfnewbe

Hmmmmm… This looks interesting! I've put all logging on and see this.

block
Aug 23 22:34:51 LAN 192.168.2.16:25 65.55.34.215:43338 TCP:SA
[…]
pass
Aug 23 22:33:17 WANB 65.55.34.215:43338 192.168.2.16:25 TCP:S

Look to the difference between the two timestamps.
What can be the cause of this?

[update]
My rules:
ID Proto Source Port Destination Port Gateway Queue Schedule Description

192.168.2.14 * * * WANB none mail route via WANB
192.168.2.16 * * * WANB none mailgw route via WANB
LAN net * * * * none Default allow LAN to any rule

danswartz

that is odd for sure. i am surprised you only see one SYN packet - if mailhost is not replying within a couple of seconds, we should have seen another. instead of logging on the pfsense, please do a packet capture as i asked.

pfnewbe

LAN or WANB?

danswartz

LAN for starters.

pfnewbe

lol….. wasn't able to upload here. I've send it to you mail.

pfnewbe

with my old firewall works it okay!
So, I cannot imagine that it is a problem on the 192.168.2.16

danswartz

please don't email me things like that. i didn't want an entire packet capture - tracing only inbound SMTP requests should have created a more manageable file.

pfnewbe

Sorry. It was a capture of only port 25.
What do I need to look for?

danswartz

Can you do a numeric one instead? This was on the LAN?

pfnewbe

Yup. This was on LAN.

Here's another one in numeric.

23:43:18.825387 IP 65.55.34.203.19470 > 192.168.2.16.smtp: S 3691429999:3691429999(0) win 65535 <mss 1452,nop,nop,sackok="">23:43:18.825445 IP 192.168.2.16.smtp > 65.55.34.203.19470: S 3797698790:3797698790(0) ack 3691430000 win 5840 <mss 1460,nop,nop,sackok="">23:43:21.747190 IP 65.55.34.203.19470 > 192.168.2.16.smtp: S 3691429999:3691429999(0) win 65535 <mss 1452,nop,nop,sackok="">23:43:21.747222 IP 192.168.2.16.smtp > 65.55.34.203.19470: S 3797698790:3797698790(0) ack 3691430000 win 5840 <mss 1460,nop,nop,sackok="">23:43:23.358357 IP 192.168.2.16.smtp > 65.55.34.203.19470: S 3797698790:3797698790(0) ack 3691430000 win 5840 <mss 1460,nop,nop,sackok="">23:43:27.765613 IP 65.55.34.203.19470 > 192.168.2.16.smtp: S 3691429999:3691429999(0) win 65535 <mss 1452,nop,nop,sackok="">23:43:27.765646 IP 192.168.2.16.smtp > 65.55.34.203.19470: S 3797698790:3797698790(0) ack 3691430000 win 5840 <mss 1460,nop,nop,sackok="">23:43:29.358662 IP 192.168.2.16.smtp > 65.55.34.203.19470: S 3797698790:3797698790(0) ack 3691430000 win 5840 <mss 1460,nop,nop,sackok="">23:43:41.359231 IP 192.168.2.16.smtp > 65.55.34.203.19470: S 3797698790:3797698790(0) ack 3691430000 win 5840 <mss 1460,nop,nop,sackok="">23:43:43.759388 IP 192.168.2.16.smtp > 212.61.26.38.sdo-tls: S 3698667821:3698667821(0) ack 1312522795 win 5792 <mss 6="" 25655941="" 1460,nop,nop,timestamp="" 1212769185,nop,wscale="">23:44:05.570420 IP 192.168.2.16.smtp > 65.55.34.203.19470: S 3797698790:3797698790(0) ack 3691430000 win 5840 <mss 1460,nop,nop,sackok="">23:44:53.782771 IP 192.168.2.16.smtp > 65.55.34.203.19470: S 3797698790:3797698790(0) ack 3691430000 win 5840 <mss 1460,nop,nop,sackok="">Firewall log says:

block
Aug 23 23:42:47 LAN 192.168.2.16:25 65.55.34.203:19470 TCP:SA

pass
Aug 23 23:41:11 WANB 65.55.34.203:19470 192.168.2.16:25 TCP:S</mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss>

danswartz

Okay, I am seeing the inbound SYN and the server is sending back SYN/ACK, and the sender is retrying with backoff which all looks good. The question is why the SYN/ACK is not getting to the remote host. Looking at your NAT and Rules, I note they are for the WAN side only. Can you post your LAN rules and outbound (if any) NAT?

pfnewbe

I've only 1 NAT-rule:
If Proto Src. addr Src. ports Dest. addr Dest. ports NAT IP NAT Ports Description

[Firewall rule ID is managed with this rule] WANB TCP * * WANB address 25 (SMTP) 192.168.2.16 25 (SMTP) NAT SMTP

and my LAN-rules are as mentioned in my post of Reply #20

danswartz

That is the inbound NAT rule - you have no outbound one? Can you post /tmp/rules.debug?

pfnewbe

Nope. This is the only NAT-rule!
When i'm back home I'll post the /tmp/rules.debug

Is the LAN-rule not enough? Everything is allowed to go outside. ???

* LAN net * * * * none Default allow LAN to any rule

danswartz

There are rules that can be added invisibly to what you see in the GUI.

pfnewbe

$ cat /tmp/rules.debug
#System aliases

loopback = "{ lo0 }"
WANA = "{ em2 }"
LAN = "{ em1 }"
WANB = "{ em0 }"
WIFI = "{ em3 }"
DMZ = "{ em4 }"
OpenVPN = "{ openvpn }"

#SSH Lockout Table
table <sshlockout>persist
#Snort2C table
table <snort2c>table <virusprot># User Aliases
table <easyruleblockhostswan>{ 211.154.135.19/32 }
EasyRuleBlockHostsWAN = "<easyruleblockhostswan>"

Gateways

GWWANB = " route-to ( em0 192.168.1.254 ) "
GWGW_WANA = " route-to ( em2 94.209.232.1 ) "
GWGW_OPT1 = " "

set loginterface em2
set loginterface em1
set loginterface em0
set loginterface em3
set loginterface em4
set optimization normal
set limit states 96000

set skip on pfsync0

scrub in on $WANA all fragment reassemble
scrub in on $LAN all fragment reassemble
scrub in on $WANB all fragment reassemble
scrub in on $WIFI all fragment reassemble
scrub in on $DMZ all fragment reassemble

altq on em2 hfsc bandwidth 80Mb queue { qACK, qDefault, qP2P, qVoIP, qGames, qOthersHigh, qOthersLow }
queue qACK on em2 bandwidth 19.792% hfsc ( ecn , linkshare (0b, 100, 19.792%) )
queue qDefault on em2 bandwidth 9.896% hfsc ( ecn , default )
queue qP2P on em2 bandwidth 4.948% hfsc ( ecn , linkshare (4.948%, 300, 4.948%) , upperlimit 4.948% )
queue qVoIP on em2 bandwidth 32Kb hfsc ( ecn , realtime (0b, 10, 512Kb) )
queue qGames on em2 bandwidth 19.792% hfsc ( ecn , linkshare (0b, 50, 19.792%) )
queue qOthersHigh on em2 bandwidth 9.896% hfsc ( ecn , linkshare (0b, 200, 9.896%) )
queue qOthersLow on em2 bandwidth 1% hfsc ( ecn , linkshare (1%, 500, 1%) )

altq on em0 hfsc bandwidth 16Mb queue { qACK, qDefault, qP2P, qVoIP, qGames, qOthersHigh, qOthersLow }
queue qACK on em0 bandwidth 19.76% hfsc ( ecn , linkshare (0b, 100, 19.76%) )
queue qDefault on em0 bandwidth 9.88% hfsc ( ecn , default )
queue qP2P on em0 bandwidth 4.94% hfsc ( ecn , linkshare (4.94%, 300, 4.94%) , upperlimit 4.94% )
queue qVoIP on em0 bandwidth 32Kb hfsc ( ecn , realtime (0b, 10, 512Kb) )
queue qGames on em0 bandwidth 19.76% hfsc ( ecn , linkshare (0b, 50, 19.76%) )
queue qOthersHigh on em0 bandwidth 9.88% hfsc ( ecn , linkshare (0b, 200, 9.88%) )
queue qOthersLow on em0 bandwidth 1% hfsc ( ecn , linkshare (1%, 500, 1%) )

altq on em1 hfsc bandwidth 11000Kb queue { qInternet }
queue qInternet on em1 bandwidth 11000Kb hfsc ( ecn , linkshare (11000Kb, 100, 11000Kb) , upperlimit 11000Kb ) { qACK, qDefault, qP2P, qVoIP, qGames, qOthersHigh, qOthersLow }
queue qACK on em1 bandwidth 19.742% hfsc ( ecn , linkshare (0b, 100, 19.742%) )
queue qDefault on em1 bandwidth 9.871% hfsc ( ecn , default )
queue qP2P on em1 bandwidth 4.9355% hfsc ( ecn , linkshare (4.9355%, 300, 4.9355%) , upperlimit 4.9355% )
queue qVoIP on em1 bandwidth 32Kb hfsc ( ecn , realtime (0b, 10, 512Kb) )
queue qGames on em1 bandwidth 19.742% hfsc ( ecn , linkshare (0b, 50, 19.742%) )
queue qOthersHigh on em1 bandwidth 9.871% hfsc ( ecn , linkshare (0b, 200, 9.871%) )
queue qOthersLow on em1 bandwidth 1% hfsc ( ecn , linkshare (1%, 500, 1%) )

nat-anchor "natearly/"
nat-anchor "natrules/"

Outbound NAT rules

Subnets to NAT

tonatsubnets = "{ 192.168.2.0/24 192.168.20.0/24 192.168.30.0/24 10.0.1.0/24 }"
nat on $WANA from $tonatsubnets port 500 to any port 500 -> 94.209.233.165/32 port 500
nat on $WANA from $tonatsubnets to any -> 94.209.233.165/32 port 1024:65535

nat on $WANB from $tonatsubnets port 500 to any port 500 -> 80.126.204.124/32 port 500
nat on $WANB from $tonatsubnets to any -> 80.126.204.124/32 port 1024:65535

Load balancing anchor

rdr-anchor "relayd/*"

TFTP proxy

rdr-anchor "tftp-proxy/*"
table <direct_networks>{ 94.209.232.0/23 192.168.2.0/24 80.0.0.0/8 192.168.20.0/24 192.168.30.0/24 }

NAT Inbound Redirects

rdr on em0 proto tcp from any to 80.126.204.124 port 25 -> 192.168.2.16

Reflection redirects

rdr on { em1 em3 em4 openvpn } proto tcp from any to 80.126.204.124 port 25 tag PFREFLECT -> 127.0.0.1 port 19000

UPnPd rdr anchor

rdr-anchor "miniupnpd"

anchor "relayd/*"
anchor "firewallrules"
#–-------------------------------------------------------------------------

default deny rules

#---------------------------------------------------------------------------
block in log all label "Default deny rule"
block out log all label "Default deny rule"

We use the mighty pf, we cannot be fooled.

block quick proto { tcp, udp } from any port = 0 to any
block quick proto { tcp, udp } from any to any port = 0

Block all IPv6

block in quick inet6 all
block out quick inet6 all

snort2c

block quick from <snort2c>to any label "Block snort2c hosts"
block quick from any to <snort2c>label "Block snort2c hosts"

package manager early specific hook

anchor "packageearly"

carp

anchor "carp"

SSH lockout

block in log quick proto tcp from <sshlockout>to any port 22 label "sshlockout"
block in quick from <virusprot>to any label "virusprot overload table"
antispoof for em2

allow our DHCP client out to the WANA

anchor "wandhcp"
pass in on $WANA proto udp from any port = 67 to any port = 68 label "allow dhcp client out WANA"
pass out on $WANA proto udp from any port = 68 to any port = 67 label "allow dhcp client out WANA"

Not installing DHCP server firewall rules for WANA which is configured for DHCP.

antispoof for em1

allow access to DHCP server on LAN

anchor "dhcpserverLAN"
pass in on $LAN proto udp from any port = 68 to 255.255.255.255 port = 67 label "allow access to DHCP server"
pass in on $LAN proto udp from any port = 68 to 192.168.2.254 port = 67 label "allow access to DHCP server"
pass out on $LAN proto udp from 192.168.2.254 port = 67 to any port = 68 label "allow access to DHCP server"
antispoof for em0

allow our DHCP client out to the WANB

anchor "opt1dhcp"
pass in on $WANB proto udp from any port = 67 to any port = 68 label "allow dhcp client out WANB"
pass out on $WANB proto udp from any port = 68 to any port = 67 label "allow dhcp client out WANB"

Not installing DHCP server firewall rules for WANB which is configured for DHCP.

antispoof for em3
antispoof for em4
anchor "spoofing"

loopback

anchor "loopback"
pass in on $loopback all label "pass loopback"
pass out on $loopback all label "pass loopback"

anchor "firewallout"

let out anything from the firewall host itself and decrypted IPsec traffic

pass out all keep state allow-opts label "let out anything from firewall host itself"
pass out route-to ( em2 94.209.232.1 ) from 94.209.233.165 to !94.209.232.0/23 keep state allow-opts label "let out anything from firewall host itself"

make sure the user cannot lock himself out of the webConfigurator or SSH

anchor "anti-lockout"
pass in quick on em1 from any to (em1) keep state label "anti-lockout rule"

NAT Reflection rules

pass in inet tagged PFREFLECT keep state label "NAT REFLECT: Allow traffic to localhost"

User-defined rules follow

pass log on { em0 } proto tcp from any to any port 25 flags S/SA keep state queue (qOthersHigh,qACK) label "USER_RULE: m_Other SMTP outbound"
pass out from any to any queue (qOthersLow) label "USER_RULE: Penalty Box"
pass out proto udp from any to any queue (qVoIP) label "USER_RULE: DiffServ/Lowdelay/Upload"
pass out proto tcp from any to any port 6880 >< 7000 queue (qP2P) label "USER_RULE: m_P2P BitTorrent outbound"
pass out proto udp from any to any port 6880 >< 7000 queue (qP2P) label "USER_RULE: m_P2P BitTorrent outbound"
pass out proto tcp from any to any port 4660 >< 4666 queue (qP2P) label "USER_RULE: m_P2P EDonkey2000 outbound"
pass out proto tcp from any to any port 6346 queue (qP2P) label "USER_RULE: m_P2P Gnutella-TCP outbound"
pass out proto udp from any to any port 6346 queue (qP2P) label "USER_RULE: m_P2P Gnutella-UDP outbound"
pass out proto tcp from any to any port 6698 >< 6702 queue (qP2P) label "USER_RULE: m_P2P Napster outbound"
pass out proto tcp from any to any port 8887 >< 8890 queue (qP2P) label "USER_RULE: m_P2P OpenNap outbound"
pass out proto udp from any to any port 17477 >< 17489 queue (qGames) label "USER_RULE: m_Game Delta1 outbound"
pass out proto tcp from any to any port 49000 >< 49003 queue (qGames,qACK) label "USER_RULE: m_Game FarCry-1 outbound"
pass out proto udp from any to any port 49000 >< 49003 queue (qGames) label "USER_RULE: m_Game FarCry-2 outbound"
pass out proto tcp from any to any port 27015 queue (qGames,qACK) label "USER_RULE: m_Game HL-1 outbound"
pass out proto udp from any to any port 27650 queue (qGames) label "USER_RULE: m_Game HL-2 outbound"
pass out proto udp from any to any port 27666 queue (qGames) label "USER_RULE: m_Game HL-3 outbound"
pass out proto udp from any to any port 7776 >< 7788 queue (qGames) label "USER_RULE: m_Game ur1 outbound"
pass out proto tcp from any to any port 7776 >< 7788 queue (qGames,qACK) label "USER_RULE: m_Game ur2 outbound"
pass out proto udp from any to any port 88 queue (qGames) label "USER_RULE: m_Game xbox360-1 outbound"
pass out proto udp from any to any port 3074 queue (qGames) label "USER_RULE: m_Game xbox360-2 outbound"
pass out proto tcp from any to any port 3074 queue (qGames,qACK) label "USER_RULE: m_Game xbox360-3 outbound"
pass in log quick on $WANA reply-to ( em2 94.209.232.1 ) proto tcp from any to 94.209.233.165 port 1194 flags S/SA keep state label "USER_RULE: OpenVPN wizard rules."
pass in log quick on $WANB proto udp from any to 80.126.204.124 port 1194 keep state label "USER_RULE"
pass in log quick on $WANB proto tcp from any to 192.168.2.16 port 25 flags S/SA keep state label "USER_RULE: NAT NAT SMTP"
pass in quick on $WANB proto igmp from 192.168.1.254 to 224.0.0.1 keep state label "USER_RULE: Easy Rule: Passed from Firewall Log View"
pass in quick on $OpenVPN from any to any keep state label "USER_RULE: OpenVPN wizard rules."
pass in log quick on $LAN from 192.168.2.14 to <vpns>keep state label "NEGATE_ROUTE: Negate policy route for vpn(s)"
pass in log quick on $LAN $GWWANB from 192.168.2.14 to any keep state label "USER_RULE: mail route via WANB"
pass in log quick on $LAN from 192.168.2.16 to <vpns>keep state label "NEGATE_ROUTE: Negate policy route for vpn(s)"
pass in log quick on $LAN $GWWANB from 192.168.2.16 to any keep state label "USER_RULE: mailgw route via WANB"
pass in log quick on $LAN from 192.168.2.0/24 to any keep state label "USER_RULE: Default allow LAN to any rule"

VPN Rules

package manager late specific hook

anchor "packagelate"

anchor "tftp-proxy/*"

anchor "limitingesr"

uPnPd

anchor "miniupnpd"

havp proxy ifaces rules</vpns></vpns></virusprot></sshlockout></snort2c></snort2c></direct_networks></easyruleblockhostswan></easyruleblockhostswan></virusprot></snort2c></sshlockout>

danswartz

I think this is the problem?

# Gateways
GWWANB = " route-to ( em0 192.168.1.254 ) " <==== this looks bogus!
GWGW_WANA = " route-to ( em2 94.209.232.1 ) "
GWGW_OPT1 = "  "

You seem to have defined the WANB gateway with the internal IP address. The WANA GW is correct, I think.