floating rule works, interface rule does not



  • I've been trying to find my issue with traffic from one interface to another not getting through. Using an interface rule set to allow all source traffic to server on port 443, states shows SYN_SENT:ESTABLISHED but fails. After trying to add the same rule as a floating rule, it works.

    Just wondering if I missed something really basic, as I'd like to have the rule on the interface the traffic is only coming from. Could anyone give me some hints on where to troubleshoot?



  • it would be easier if you took screenshots of your rules to see what's wrong



  • Interface MO_PFSENSE - rule that doesn't work:

    383bc534-95ab-400d-8f34-abf622bf6566-image.png

    Associated states:

    dcc26a3e-1f99-4c4f-b044-07d317c5ef84-image.png

    Interface Group SVR (Interface MO_SVR is part of this group):

    0eb5388d-e174-4197-8010-339c55fcea72-image.png
    Floating rule that works:

    e651518d-0e3a-428f-bc34-ecd46de519b3-image.png

    Floating rule states:

    56cfd09f-9a7f-48ba-83ad-c044e3b802d7-image.png



  • the two interface have the same gateway? did you do a packet capture ?
    it seems like 3-way handshake process does not complete



  • Here's some more context:

    OSPF is enabled on all my switches and pfSense boxes. Routing tables look good.

    This pfSense VM (rt-2) sits behind a physical pfSense box (rt-1) that connects to our ISPs. NAT is disabled on rt-2, and I've renamed the WAN interface as MO_PFSENSE. rt-2 has a default gw assigned. I think I've tried removing it but failed. I feel like that could be causing me problems.

    8b854533-c245-4e1c-84c2-b6db8e2cbbf7-image.png

    MO_SVR interface has no gw assigned:

    dbc98d85-53b8-4572-9878-8fd8b728fd14-image.png

    Traceroute shows same path in both directions (names have been obfuscated)

    From server:

    traceroute to rpi-2 (10.21.3.5), 30 hops max, 60 byte packets
    1 rt-2 (10.20.3.1) 0.180 ms 0.137 ms 0.162 ms
    2 dsw-1 (10.10.10.9) 0.384 ms 0.427 ms 0.424 ms
    3 asw-1 (10.10.10.22) 0.630 ms 0.753 ms 0.735 ms
    4 rpi-2 (10.21.3.5) 0.977 ms 0.951 ms 0.924 ms

    From client:

    traceroute to librenms (10.20.3.29), 30 hops max, 60 byte packets
    1 asw-1 (10.21.3.1) 0.445 ms 0.550 ms 0.487 ms
    2 dsw-1 (10.10.10.17) 0.714 ms 0.792 ms 0.734 ms
    3 rt-2 (10.10.10.11) 0.677 ms 0.764 ms 0.706 ms
    4 librenms (10.20.3.29) 1.201 ms !X 1.140 ms !X 1.083 ms !X



  • it seems to be asymmetric
    probably you are sending syn but the syn-ack answer is going out from somewhere else


  • LAYER 8 Netgate

    @acaronmd : OSPF is enabled on all my switches and pfSense boxes. Routing tables look good.

    Are you sure the Layer 3 switching infrastructure is not also routing between those subnets?

    That would certainly create a scenario where the firewall does not see all of the traffic - aka asymmetric routing.

    Even the floating rule you say works doesn't look right. there should not be a portion of the state that is ESTABLISHED while the rest are FIN_WAIT_2.



  • Here's a rough diagram:

    bbfd3da3-9148-48f0-89d3-9cead2d0bf8e-image.png

    on rt-2:

    O   0.0.0.0/0 [110/10] via 10.10.10.10, vmx0, 22:34:21
    K>* 0.0.0.0/0 [0/0] via 10.10.10.10, vmx0, 22:34:32
    O   10.10.10.8/29 [110/10] is directly connected, vmx0 inactive, 21:51:11
    C>* 10.10.10.8/29 is directly connected, vmx0, 21:51:21
    C>* 10.20.3.0/24 is directly connected, vmx1.103, 21:19:22
    O>* 10.21.3.0/24 [110/10] via 10.10.10.9, vmx0, 21:51:10
    

    On dsw-1:

    Destination        Gateway         VLAN Type      Sub-Type   Metric     Dist.
    ------------------ --------------- ---- --------- ---------- ---------- -----
    0.0.0.0/0          10.10.10.10     2000 ospf      External2  10         110
    10.10.10.8/29      mo-pfsense      2000 connected            1          0
    10.10.10.16/29     mo-tu           2021 connected            1          0
    10.20.3.0/24       10.10.10.11     2000 ospf      External2  20         110
    10.21.3.0/24       10.10.10.22     2021 ospf      External2  10         110
    

    On asw-1:

    Destination        Gateway         VLAN Type      Sub-Type   Metric     Dist.
    ------------------ --------------- ---- --------- ---------- ---------- -----
    0.0.0.0/0          10.10.10.17     2021 ospf      External2  10         110  
    10.10.10.8/29      10.10.10.17     2021 ospf      IntraArea  2          110
    10.10.10.16/29     mo-tu           2021 connected            1          0
    10.20.3.0/24       10.10.10.17     2021 ospf      External2  20         110
    10.21.3.0/24       tu-svr          103  connected            1          0
    

  • LAYER 8 Netgate

    OK those are the zebra routes.

    What is actually in the rtr-2 routing table?

    netstat -rn4



  • Routing tables
    
    Internet:
    Destination        Gateway            Flags     Netif Expire
    default            10.10.10.10        UGS        vmx0
    10.10.10.8/29      link#1             U          vmx0
    10.10.10.11        link#1             UHS         lo0
    10.10.10.16/29     10.10.10.9         UG1        vmx0
    10.10.10.24/29     10.10.10.9         UG1        vmx0
    10.10.10.32/29     10.10.10.9         UG1        vmx0
    10.10.10.40/29     10.10.10.9         UG1        vmx0
    10.10.55.0/24      10.10.10.10        UG1        vmx0
    10.10.56.0/22      10.10.10.10        UG1        vmx0
    10.15.5.2          10.10.10.10        UGH1       vmx0
    10.20.0.0/24       link#8             U      vmx1.200
    10.20.0.1          link#8             UHS         lo0
    10.20.1.0/25       link#7             U      vmx1.200
    10.20.1.1          link#7             UHS         lo0
    10.20.2.0/26       10.10.10.10        UG1        vmx0
    10.20.2.64/26      link#9             U      vmx1.200
    10.20.2.65         link#9             UHS         lo0
    10.20.2.128/26     10.10.10.10        UG1        vmx0
    10.20.2.192/26     10.10.10.10        UG1        vmx0
    10.20.3.0/24       link#14            U      vmx1.103
    10.20.3.1          link#14            UHS         lo0
    10.20.4.0/25       link#11            U      vmx1.210
    10.20.4.1          link#11            UHS         lo0
    10.20.5.0/24       10.10.10.10        UG1        vmx0
    10.20.6.0/24       10.10.10.10        UGS        vmx0
    10.20.7.0/24       10.10.10.10        UGS        vmx0
    10.20.8.0/24       10.10.10.10        UG1        vmx0
    10.20.10.0/24      10.10.10.10        UG1        vmx0
    10.20.11.0/24      10.10.10.12        UG1        vmx0
    10.21.0.1          10.10.10.9         UGH1       vmx0
    10.21.1.0/24       10.10.10.9         UG1        vmx0
    10.21.2.0/24       10.10.10.9         UG1        vmx0
    10.21.3.0/24       10.10.10.9         UG1        vmx0
    10.21.4.0/24       10.10.10.9         UG1        vmx0
    10.21.5.0/24       10.10.10.9         UG1        vmx0
    10.21.6.0/24       10.10.10.10        UGS        vmx0
    10.21.7.0/24       10.10.10.10        UGS        vmx0
    10.22.0.1          10.10.10.9         UGH1       vmx0
    10.22.1.0/24       10.10.10.9         UG1        vmx0
    10.22.2.0/24       10.10.10.9         UG1        vmx0
    10.22.4.0/24       10.10.10.9         UG1        vmx0
    10.22.5.0/24       10.10.10.9         UG1        vmx0
    10.22.6.0/24       10.10.10.10        UGS        vmx0
    10.22.7.0/24       10.10.10.10        UGS        vmx0
    10.23.0.1          10.10.10.9         UGH1       vmx0
    10.23.1.0/24       10.10.10.9         UG1        vmx0
    10.23.2.0/24       10.10.10.9         UG1        vmx0
    10.23.3.0/24       10.10.10.9         UG1        vmx0
    10.23.4.0/24       10.10.10.9         UG1        vmx0
    10.23.5.0/24       10.10.10.9         UG1        vmx0
    10.23.6.0/24       10.10.10.10        UGS        vmx0
    10.23.7.0/24       10.10.10.10        UGS        vmx0
    24.142.24.0/21     10.10.10.10        UG1        vmx0
    72.172.127.32/27   10.10.10.10        UG1        vmx0
    127.0.0.1          link#4             UH          lo0
    192.168.32.0/30    10.10.10.10        UG1        vmx0
    192.168.91.2       10.10.10.10        UGH1       vmx0
    192.168.92.2       10.10.10.10        UGH1       vmx0
    192.168.93.2       10.10.10.10        UGH1       vmx0
    192.168.94.0/29    10.10.10.10        UG1        vmx0
    192.168.96.2       10.10.10.10        UGH1       vmx0
    192.168.97.6       10.10.10.10        UGH1       vmx0
    192.168.97.10      10.10.10.10        UGH1       vmx0
    192.168.97.14      10.10.10.10        UGH1       vmx0
    192.168.98.2       10.10.10.10        UGH1       vmx0
    

  • LAYER 8 Netgate

    Ah. Multiple routers on "WAN".

    It doesn't work that way by default. The WAN interface has route-to to 10.10.10.10 for anything not in the WAN network, which in that case would be 10.10.10.8/29.

    Looking at /tmp/rules.debug would be where I would go next.

    Your diagram shows what looks like multiple interfaces but they are not. They are all just vmx0. That is probably the issue.

    You could also remove the gateway from the pfSense 10.10.10.8/29 interface (remove the gateway setting from the interface itself) and add a gateway in System > Routing for 10.10.10.10 on WAN and set it to be the default and see if that doesn't clear things up a little. You will still have a default route that way but you won't have route-to there and won't have automatic outbound NAT for it.



  • I believe this is it here:

    pass out  route-to ( vmx0 10.10.10.10 ) from 10.10.10.11 to !10.10.10.8/29 tracker 1000010111 keep state allow-opts label "let out anything from firewall host itself"
    

    the GW already exists
    9d857eee-6678-492c-8126-cbeada81889f-image.png

    And here is the interface

    08c8f60b-070d-4398-9e7f-d88248b48b09-image.png

    When I get a chance, I'll try removing it from the interface itself. I seem to remember having issues trying to remove it when I tried long ago... if all else fails, I could add more interfaces to the VM.



  • BTW, as you might have noticed already, there's multiple VLANs assigned to the vmx1 interface, while the virtual machine has two interfaces.

    873de9ab-d054-4173-a62a-b00ae2a15e84-image.png


  • LAYER 8 Netgate

    Yeah that will eliminate it from having route-to. The gateway should still be there and should still be the default gateway but double-check it.

    There are ways to bypass that using floating rules but in that situation I would just get rid of the gateway on the WAN interface and see how that goes.


  • LAYER 8 Netgate

    @acaronmd said in floating rule works, interface rule does not:

    BTW, as you might have noticed already, there's multiple VLANs assigned to the vmx1 interface, while the virtual machine has two interfaces.

    873de9ab-d054-4173-a62a-b00ae2a15e84-image.png

    VLAN interfaces are discrete as far as pfSense is concerned for routing purposes.



  • Thanks Derelict, you nailed it. Removed the upstream GW from the WAN, everything still working OK, so then disabled the floating rule, and the interface rule is working:

    56e7d006-7c42-415a-86c7-5d644484e199-image.png



  • glad you solved, i was following the progress myself. I find that 3D of more experienced people are always instructive



  • Thanks kiokoman, you were on the right track.


  • LAYER 8 Netgate

    That FIN_WAIT_2:ESTABLISHED is still strange. Maybe something to do with the app itself though. Glad it's working.


Log in to reply