Incorrect policy-based routing

dusan · Mar 20, 2010, 2:01 PM

Most of the cases, pf routes to the correct interface if policy-based routing on the in-interface is in effect. But for certain destination IP, it fails. It simply routes to the default gateway instead.

I have 3 policy-based routing rules:

@108 pass in log quick on em1 route-to (em4 192.168.0.65) inet proto tcp from 192.168.12.23 to any port = smtp flags S/SA keep state label "USER_RULE: mx1.abc.com, ISP4 only"
@110 pass in log quick on em1 route-to (em5 192.168.0.69) inet proto tcp from 192.168.12.3 to any port = smtp flags S/SA keep state label "USER_RULE: mx2.xyz.com, ISP5 only"
@112 pass in log quick on em1 route-to (em3 192.168.0.81) inet proto tcp from any to any port = smtp flags S/SA keep state label "USER_RULE: other SMTP out, ISP3 only"

Which state that the Internet link em4 is reserved for the mail server 192.168.12.23, the Internet link em5 is reserved for the mail server 192.168.12.3, and the Internet link em3 is for other mail servers/clients. (Note: em1 is the LAN interface.)

Most of the cases, including gmail.com and yahoo.com, emails from the mail server 192.168.12.23 are sent via the correct gateway (em4).

But for certain destination(s), pf simply sends e-mails via the default gateway, which is not em4. Strange is, even in such a case, the firewall log still shows that pf applied the correct rule (@108). Only monitoring at the modem devices around the pfsense device can proves the failure of routing.

Any comment?

eri-- · Mar 20, 2010, 3:35 PM

It will not do policy-routing for multicast&broadcast and also it won't do policy-routing for destinations ip that are local to the host(ie ips configured on the pfSense itself).
These are tweaks added by pfSense so strange problems are not seen, otherwise it should just policy route it.

So unless your route-to ip is not a local ip of pfSense or your ip falls in multicast range it should be ok.

Packet traces would help here to reproduce the issue.

dusan · Mar 20, 2010, 4:07 PM

The destination IP in question are of our business partner's mail server which I'm sure not multicast/broadcast IP. We're using Internet service from different ISPs. The route-to IPs .0.65, .0.69 and .0.81 in the rules above are next-hop IPs (in fact, they are of PPPoE modems/routers) around the pfSense machine, not of the machine itself. And, as said, the firewall log shows that the correct rule was selected. It fails to route, however.

dusan · Mar 21, 2010, 4:56 PM

I've also found that pfsense replies via the wrong gateway even when the specific IP in question (namely, 118/8) is a source IP.

Thus if the remote client 118.x.y.z initiates a TCP connection to the mail server mx1.abc.com via (em4, 192.168.0.65) which is not default gateway, pfsense still responds via default gateway, which effectively drops the connection.

eri-- · Mar 21, 2010, 5:54 PM

Are by any case the packets being fragmented when being sent out the wan and not following the policy-route?

dusan · Mar 22, 2010, 5:25 AM

Please tell me how to detect packet fragmentation.
I'll check it out and report back.

dusan · Mar 22, 2010, 1:27 PM

I found today that not only SMTP but also HTTP and maybe all TCP protocols are affected; and, not only the IP 118.x.y.z but also 80.254.71.228 (IP address of the web server of m0n0wall project) and maybe many, many IP addresses are affected.

This serious problem can be reproduced easily. Create a firewall rule that routes every web-browsing traffic to some non-default interface, then visit http://m0n0.ch to see that it actually goes through the default interface.