Load Balancer, Direct Routing and Default deny rule problems

[alessice]

Hi,

we are an Hosting Provider and we have switched from Fortigate firewall to pfSense.

After the switch some users have problems, like timeout, with their client email (Outlook, Thunderbird, …) when try to sent email via SMTP or during download/upload email via IMAP.

We are using Dovecot and Postfix for IMAP and SMTP and after the switch we have seen an increased number of errors related to connections closed.

Our traffic is around 400-600 Mbits and pfSense (2.4.4-p3 in HA) run on Supermicro server with 2 x Intel Xeon 3104 6 core, 32GB of RAM and 4 x 10Gbit Intel LAN.

We are using a load balancer (CentOS 6 LVS ipvsadm/piranha) configured in Direct Routing, so on LVS is active the VIP IP for POP, IMAP, SMTP. Remote clients start connection with the VIP and after is the backend server to talk directly with remote clients.

This configurations as worked fine from many years until we switched to pfSense.

In the pfSense firewall logs we see many errors like:

Action: block
Interface: > WAN
Rule: Default deny rule IPv4 (1000000104)
Source: VIP-IP or Backend-IP:993 or 143
Destination: remote-client-IP:26018 (or other high ports)
Protocol: TCP:PA or TCP:FPA or TCP:FA

But also if we update the DNS directly with backend server public IP instead of load balancer VIP, the problem is still present, probably for less users but still present.

On pfSense in order to mitigate the problem we have:

• enabled Bypass firewall rules for traffic on the same interface
• add a Rules on IP-PUBLIC interfaces with Pass, Protocol TCP, Source IP-PUBLIC Subnets, Destination Any, TCP Flags Any flags and State type Sloppy
• add a Rules on Floating on interface IP-PUBLIC , Direction out, Source IP-PUBLIC Subnets, Destination Any, Pass, Protocol TCP, TCP Flags Any flags and State type Sloppy

as suggested here: https://docs.netgate.com/pfsense/en/latest/firewall/troubleshooting-blocked-log-entries-due-to-asymmetric-routing.html

but users still have timeout.

What can we do to solve this issue?

Could be a problem on timeout of TCP state and solved with enable Conservative on “Firewall Optimization Options”?

Disable Firewall Scrub can help?

Should we set State type Sloppy on others rules?

I thinks that pfSense is already tuned for local router but probably for our use case must be manually tuned.

Please help me to investigate the problem.
Thanks