floating rule works, interface rule does not

acaronmd

Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.10.10.10        UGS        vmx0
10.10.10.8/29      link#1             U          vmx0
10.10.10.11        link#1             UHS         lo0
10.10.10.16/29     10.10.10.9         UG1        vmx0
10.10.10.24/29     10.10.10.9         UG1        vmx0
10.10.10.32/29     10.10.10.9         UG1        vmx0
10.10.10.40/29     10.10.10.9         UG1        vmx0
10.10.55.0/24      10.10.10.10        UG1        vmx0
10.10.56.0/22      10.10.10.10        UG1        vmx0
10.15.5.2          10.10.10.10        UGH1       vmx0
10.20.0.0/24       link#8             U      vmx1.200
10.20.0.1          link#8             UHS         lo0
10.20.1.0/25       link#7             U      vmx1.200
10.20.1.1          link#7             UHS         lo0
10.20.2.0/26       10.10.10.10        UG1        vmx0
10.20.2.64/26      link#9             U      vmx1.200
10.20.2.65         link#9             UHS         lo0
10.20.2.128/26     10.10.10.10        UG1        vmx0
10.20.2.192/26     10.10.10.10        UG1        vmx0
10.20.3.0/24       link#14            U      vmx1.103
10.20.3.1          link#14            UHS         lo0
10.20.4.0/25       link#11            U      vmx1.210
10.20.4.1          link#11            UHS         lo0
10.20.5.0/24       10.10.10.10        UG1        vmx0
10.20.6.0/24       10.10.10.10        UGS        vmx0
10.20.7.0/24       10.10.10.10        UGS        vmx0
10.20.8.0/24       10.10.10.10        UG1        vmx0
10.20.10.0/24      10.10.10.10        UG1        vmx0
10.20.11.0/24      10.10.10.12        UG1        vmx0
10.21.0.1          10.10.10.9         UGH1       vmx0
10.21.1.0/24       10.10.10.9         UG1        vmx0
10.21.2.0/24       10.10.10.9         UG1        vmx0
10.21.3.0/24       10.10.10.9         UG1        vmx0
10.21.4.0/24       10.10.10.9         UG1        vmx0
10.21.5.0/24       10.10.10.9         UG1        vmx0
10.21.6.0/24       10.10.10.10        UGS        vmx0
10.21.7.0/24       10.10.10.10        UGS        vmx0
10.22.0.1          10.10.10.9         UGH1       vmx0
10.22.1.0/24       10.10.10.9         UG1        vmx0
10.22.2.0/24       10.10.10.9         UG1        vmx0
10.22.4.0/24       10.10.10.9         UG1        vmx0
10.22.5.0/24       10.10.10.9         UG1        vmx0
10.22.6.0/24       10.10.10.10        UGS        vmx0
10.22.7.0/24       10.10.10.10        UGS        vmx0
10.23.0.1          10.10.10.9         UGH1       vmx0
10.23.1.0/24       10.10.10.9         UG1        vmx0
10.23.2.0/24       10.10.10.9         UG1        vmx0
10.23.3.0/24       10.10.10.9         UG1        vmx0
10.23.4.0/24       10.10.10.9         UG1        vmx0
10.23.5.0/24       10.10.10.9         UG1        vmx0
10.23.6.0/24       10.10.10.10        UGS        vmx0
10.23.7.0/24       10.10.10.10        UGS        vmx0
24.142.24.0/21     10.10.10.10        UG1        vmx0
72.172.127.32/27   10.10.10.10        UG1        vmx0
127.0.0.1          link#4             UH          lo0
192.168.32.0/30    10.10.10.10        UG1        vmx0
192.168.91.2       10.10.10.10        UGH1       vmx0
192.168.92.2       10.10.10.10        UGH1       vmx0
192.168.93.2       10.10.10.10        UGH1       vmx0
192.168.94.0/29    10.10.10.10        UG1        vmx0
192.168.96.2       10.10.10.10        UGH1       vmx0
192.168.97.6       10.10.10.10        UGH1       vmx0
192.168.97.10      10.10.10.10        UGH1       vmx0
192.168.97.14      10.10.10.10        UGH1       vmx0
192.168.98.2       10.10.10.10        UGH1       vmx0

Derelict

Ah. Multiple routers on "WAN".

It doesn't work that way by default. The WAN interface has route-to to 10.10.10.10 for anything not in the WAN network, which in that case would be 10.10.10.8/29.

Looking at /tmp/rules.debug would be where I would go next.

Your diagram shows what looks like multiple interfaces but they are not. They are all just vmx0. That is probably the issue.

You could also remove the gateway from the pfSense 10.10.10.8/29 interface (remove the gateway setting from the interface itself) and add a gateway in System > Routing for 10.10.10.10 on WAN and set it to be the default and see if that doesn't clear things up a little. You will still have a default route that way but you won't have route-to there and won't have automatic outbound NAT for it.

acaronmd

I believe this is it here:

pass out  route-to ( vmx0 10.10.10.10 ) from 10.10.10.11 to !10.10.10.8/29 tracker 1000010111 keep state allow-opts label "let out anything from firewall host itself"

the GW already exists

And here is the interface

When I get a chance, I'll try removing it from the interface itself. I seem to remember having issues trying to remove it when I tried long ago... if all else fails, I could add more interfaces to the VM.

acaronmd

BTW, as you might have noticed already, there's multiple VLANs assigned to the vmx1 interface, while the virtual machine has two interfaces.

Derelict

Yeah that will eliminate it from having route-to. The gateway should still be there and should still be the default gateway but double-check it.

There are ways to bypass that using floating rules but in that situation I would just get rid of the gateway on the WAN interface and see how that goes.

Derelict

@acaronmd said in floating rule works, interface rule does not:

BTW, as you might have noticed already, there's multiple VLANs assigned to the vmx1 interface, while the virtual machine has two interfaces.

VLAN interfaces are discrete as far as pfSense is concerned for routing purposes.

acaronmd

Thanks Derelict, you nailed it. Removed the upstream GW from the WAN, everything still working OK, so then disabled the floating rule, and the interface rule is working:

kiokoman

glad you solved, i was following the progress myself. I find that 3D of more experienced people are always instructive

acaronmd

Thanks kiokoman, you were on the right track.

Derelict

That FIN_WAIT_2:ESTABLISHED is still strange. Maybe something to do with the app itself though. Glad it's working.