Policy Based Routing Works Outbound Not Inbound
-
Having a weird issue that I have not come across before. I have an internal PFSense firewall and a couple of edge PFSense firewalls. I set up a policy based route on the interface on the internal firewall to point to a gateway for a specific edge router.
pass in quick on vtnet0.200 route-to (vtnet0.2550 10.0.255.13) inet proto tcp from 10.110.200.12 to ! <INTNetworks> flags S/SA keep state label "USER_RULE" pass in quick on vtnet0.200 route-to (vtnet0.2550 10.0.255.13) inet proto udp from 10.110.200.12 to ! <INTNetworks> keep state label "USER_RULE"When the host initiates a connection outbound it takes the correct path and goes out to 10.0.255.13. When I have a port forward set up on the 10.0.255.13 firewall to go to 10.110.200.12, the traffic comes in vtnet0.2550, the system responds, then the traffic goes out from the internal firewall on a different interface (vtnet0.690) to the default configured gateway (not vtnet0.2550). I have disabled Negate rules and have tried almost everything that I can find in the forums to no avail.
Here is the output of "pfctl -vvsr | grep -A3 10.110.200.12"
@68(1588480034) pass in quick on vtnet0.200 route-to (vtnet0.2550 10.0.255.13) inet from 10.110.200.12 to ! <INTNetworks:2> flags S/SA keep state label "USER_RULE" [ Evaluations: 16328 Packets: 2 Bytes: 152 States: 0 ] [ Inserted: pid 46361 State Creations: 1 ] @69(1588480078) pass out quick on vtnet0.690 route-to (vtnet0.2550 10.0.255.13) inet from 10.110.200.12 to ! <INTNetworks:2> flags S/SA keep state label "USER_RULE" [ Evaluations: 15899 Packets: 0 Bytes: 0 States: 0 ] [ Inserted: pid 46361 State Creations: 0 ] @70(1579387434) pass in quick on enc0 inet all flags S/SA keep state label "USER_RULE" -- @74(1588480663) pass in quick on vtnet0.200 inet from 10.110.200.12 to ! <INTNetworks:2> flags S/SA keep state label "USER_RULE" [ Evaluations: 2189 Packets: 0 Bytes: 0 States: 0 ] [ Inserted: pid 46361 State Creations: 0 ] @75(1588476232) pass in quick on vtnet0.200 inet all flags S/SA keep state label "USER_RULE"I have tried both interface firewall rules and floating rules for this and I have still not come across a solution.
Any help would be greatly appreciated.
Packet capture on vtnet0.2550:
23:03:35.556255 IP 198.199.98.246.44347 > 10.110.200.12.8080: tcp 0 23:03:36.555066 IP 198.199.98.246.44347 > 10.110.200.12.8080: tcp 0 23:03:36.560795 IP 198.199.98.246.44352 > 10.110.200.12.8080: tcp 0 23:03:37.557863 IP 198.199.98.246.44352 > 10.110.200.12.8080: tcp 0 23:03:37.562029 IP 198.199.98.246.44354 > 10.110.200.12.8080: tcp 0 23:03:38.558312 IP 198.199.98.246.44354 > 10.110.200.12.8080: tcp 0Packet capture on vtnet0.200 (the end system gateway interface)
23:04:31.673389 IP 198.199.98.246.44425 > 10.110.200.12.8080: tcp 0 23:04:31.673888 IP 10.110.200.12.8080 > 198.199.98.246.44425: tcp 0 23:04:32.670080 IP 198.199.98.246.44425 > 10.110.200.12.8080: tcp 0 23:04:32.670324 IP 10.110.200.12.8080 > 198.199.98.246.44425: tcp 0 23:04:32.674371 IP 198.199.98.246.44427 > 10.110.200.12.8080: tcp 0 23:04:32.674555 IP 10.110.200.12.8080 > 198.199.98.246.44427: tcp 0 23:04:33.670852 IP 198.199.98.246.44427 > 10.110.200.12.8080: tcp 0 23:04:33.671145 IP 10.110.200.12.8080 > 198.199.98.246.44427: tcp 0 23:04:33.678378 IP 198.199.98.246.44432 > 10.110.200.12.8080: tcp 0 23:04:33.678525 IP 10.110.200.12.8080 > 198.199.98.246.44432: tcp 0 23:04:33.689176 IP 10.110.200.12.8080 > 198.199.98.246.44425: tcp 0 23:04:34.677549 IP 198.199.98.246.44432 > 10.110.200.12.8080: tcp 0 23:04:34.677868 IP 10.110.200.12.8080 > 198.199.98.246.44432: tcp 0 23:04:35.705084 IP 10.110.200.12.8080 > 198.199.98.246.44425: tcp 0Packet capture on vtnet0.690 (default gateway link)
23:05:16.578492 IP 10.110.200.12.8080 > 198.199.98.246.44501: tcp 0 23:05:17.577214 IP 10.110.200.12.8080 > 198.199.98.246.44501: tcp 0 23:05:17.579433 IP 10.110.200.12.8080 > 198.199.98.246.44505: tcp 0 23:05:18.576894 IP 10.110.200.12.8080 > 198.199.98.246.44505: tcp 0 23:05:18.577140 IP 10.110.200.12.8080 > 198.199.98.246.44509: tcp 0 23:05:19.574104 IP 10.110.200.12.8080 > 198.199.98.246.44509: tcp 0For more information, the layout is like below:
FW1 (10.0.0.1) ----- (vtnet0.690:10.0.0.3) FW2 (vtnet0.2550:10.0.255.1) ----- (10.0.255.13) FW3
Where FW2 is also the gateway (vtnet0.200:10.110.200.1) for the host (10.110.200.12).The SYN comes in FW3 to FW2 and sent to the host out vtnet0.200. The SYN/ACK comes in vtnet0.200 and out vtnet0.690 to FW1. After more troubleshooting I have created an inbound permit rule on (vtnet0.2550:10.0.255.1) with sloppy state matching any TCP flags and a floating outbound rule on (vtnet0.2550:10.0.255.1) with sloppy state matching any TCP flags, which has not yielded any success, traffic is still going out to FW1.
-
After more testing I am beginning to suspect that PFSense is just straight up ignoring the state table when handling this traffic. This is the state table for 10.110.200.12 after performing a reset on the system state table and then re-running the test.
States DCLINTRTG2550 tcp 198.199.98.246:54237 -> 10.110.200.12:8080 SYN_SENT:ESTABLISHED 2 / 2 120 B / 120 B DCLINTSTORJ200 tcp 198.199.98.246:54237 -> 10.110.200.12:8080 ESTABLISHED:SYN_SENT 2 / 2 120 B / 120 B DCLINTRTG2550 tcp 198.199.98.246:54240 -> 10.110.200.12:8080 SYN_SENT:ESTABLISHED 2 / 2 120 B / 120 B DCLINTSTORJ200 tcp 198.199.98.246:54240 -> 10.110.200.12:8080 ESTABLISHED:SYN_SENT 2 / 2 120 B / 120 B DCLINTRTG2550 tcp 198.199.98.246:54243 -> 10.110.200.12:8080 SYN_SENT:ESTABLISHED 2 / 2 120 B / 120 B DCLINTSTORJ200 tcp 198.199.98.246:54243 -> 10.110.200.12:8080 ESTABLISHED:SYN_SENT 2 / 2 120 B / 120 BIf I'm not mistaken, a statefull firewall should be returning traffic out the interface it received it on if it is tracking the TCP state but PFSense does not appear to be doing that. Not sure if another rule somewhere is overriding that but all I have for rules outside of the policy based routes but outside of those rules my only other rules are permit any/any until I can get things working on this.