Problem reaching certiain servers on LAN from OPT



  • Hi All,

    Ik hope the following is a bit clear. I tried to be as thorough as possible.

    In short: From my OPT1 interface I cannot reach a couple of servers on my LAN interface on certain ports.
    In the text below port 80 to a server doesn't work but i.e. port 22 and 3000 do work. Ping is possible between interfaces.
    On the server there are no enabled firewalls.

    PfSense version:
    2.4.3-RELEASE-p1 (amd64)
    built on Thu May 10 15:02:52 CDT 2018
    FreeBSD 11.1-RELEASE-p10

    Short network setup:

    LAN 10.130.10.0/24 IP 10.130.10.1 GW: none
    OPT1 10.130.20.0/24 IP 10.130.20.1 GW: none

    testsystem ip: 10.130.20.2 via dhcp GW: 10.130.20.1
    servers have GW: 10.130.10.1

    server:
    10.130.10.2 with website on port 80 and 3000
    10.130.10.3 with website on port 80
    10.130.10.2 has ssh running
    10.130.10.3 has rdp running

    What does work:

    Ping from testsystem to 10.130.10.2
    Ping from testsystem to 10.130.10.3
    ping from 10.130.10.2 to testsystem
    ping from 10.130.10.3 to testsystem
    browse from testsystem to 10.130.10.2:3000
    ssh from testsystem to 10.130.10.2:22
    rdp from testsystem to 10.130.10.3:3389

    What doesn't work:
    browse from testsystem to 10.130.10.2:80 ( get's a timeout after a while, not a message that the port is closed )
    browse from testsystem to 10.130.10.3:80 ( get's a timeout after a while, not a message that the port is closed )

    Firewall rules:

    LAN:
    Protocol: IPv4 Any
    Source: LAN Net ( also tried any )
    Port: *
    Destination: *
    Gateway: *

    OPT1:
    Protocol: IPv4 Any
    Source: OPT1 Net ( also tried any )
    Port: *
    Destination: *
    Gateway: *

    In the firewall I see no block for this traffic.
    not working:
    filterlog: 105,,,1527008499,em1,match,pass,in,4,0x0,,64,61759,0,DF,6,tcp,60,10.130.20.2,10.130.10.2,47360,80,0,S,2755069375,,29200,,mss;sackOK;TS;nop;wscale

    working:
    filterlog: 105,,,1527008499,em1,match,pass,in,4,0x0,,64,21020,0,DF,6,tcp,60,10.130.20.2,10.130.10.2,45260,3000,0,S,35224774,,29200,,mss;sackOK;TS;nop;wscale

    traceroute from 10.130.10.2 to testsytem:
    traceroute to 10.130.20.2 (10.130.20.2), 30 hops max, 60 byte packets
    1  10.130.10.1 (10.130.10.1)  0.197 ms  0.167 ms  0.184 ms
    2  10.130.20.2 (10.130.20.2)  0.434 ms  0.425 ms  0.415 ms

    traceroute from testsystem to 10.130.10.2
    traceroute to 10.130.10.2 (10.130.10.2), 30 hops max, 60 byte packets
    1  10.130.20.1 (10.130.20.1)  0.203 ms  0.200 ms  0.185 ms
    2  10.130.10.2 (10.130.10.2)  0.357 ms  0.354 ms  0.346 ms

    Packet capture on CLIENT interface for testsystem -> 10.130.10.2:80
    20:53:52.231750 IP 10.130.20.2.47366 > 10.130.10.2.80: tcp 0
    20:53:53.234312 IP 10.130.20.2.47366 > 10.130.10.2.80: tcp 0
    20:53:55.250307 IP 10.130.20.2.47366 > 10.130.10.2.80: tcp 0
    20:53:59.474295 IP 10.130.20.2.47366 > 10.130.10.2.80: tcp 0

    Packet capture on CLIENT interface for testsystem -> 10.130.10.2:300
    20:54:02.935843 IP 10.130.20.2.45266 > 10.130.10.2.3000: tcp 0
    20:54:02.936061 IP 10.130.10.2.3000 > 10.130.20.2.45266: tcp 0
    20:54:02.936218 IP 10.130.20.2.45266 > 10.130.10.2.3000: tcp 0
    20:54:02.936341 IP 10.130.20.2.45266 > 10.130.10.2.3000: tcp 81
    20:54:02.936558 IP 10.130.10.2.3000 > 10.130.20.2.45266: tcp 0
    20:54:02.939182 IP 10.130.10.2.3000 > 10.130.20.2.45266: tcp 1448
    20:54:02.939192 IP 10.130.10.2.3000 > 10.130.20.2.45266: tcp 1448
    20:54:02.939202 IP 10.130.10.2.3000 > 10.130.20.2.45266: tcp 1200
    20:54:02.939211 IP 10.130.10.2.3000 > 10.130.20.2.45266: tcp 1448
    20:54:02.939220 IP 10.130.10.2.3000 > 10.130.20.2.45266: tcp 1448
    20:54:02.939229 IP 10.130.10.2.3000 > 10.130.20.2.45266: tcp 1448
    20:54:02.939237 IP 10.130.10.2.3000 > 10.130.20.2.45266: tcp 1448
    20:54:02.939246 IP 10.130.10.2.3000 > 10.130.20.2.45266: tcp 1448
    20:54:02.939257 IP 10.130.10.2.3000 > 10.130.20.2.45266: tcp 1448
    20:54:02.939267 IP 10.130.10.2.3000 > 10.130.20.2.45266: tcp 1448
    20:54:02.939341 IP 10.130.20.2.45266 > 10.130.10.2.3000: tcp 0
    20:54:02.939465 IP 10.130.20.2.45266 > 10.130.10.2.3000: tcp 0
    20:54:02.939474 IP 10.130.20.2.45266 > 10.130.10.2.3000: tcp 0
    20:54:02.939556 IP 10.130.10.2.3000 > 10.130.20.2.45266: tcp 1448
    20:54:02.939565 IP 10.130.10.2.3000 > 10.130.20.2.45266: tcp 1448
    20:54:02.939573 IP 10.130.10.2.3000 > 10.130.20.2.45266: tcp 126
    20:54:02.939743 IP 10.130.20.2.45266 > 10.130.10.2.3000: tcp 0
    20:54:02.939876 IP 10.130.20.2.45266 > 10.130.10.2.3000: tcp 0
    20:54:02.940181 IP 10.130.10.2.3000 > 10.130.20.2.45266: tcp 0
    20:54:02.940339 IP 10.130.20.2.45266 > 10.130.10.2.3000: tcp 0

    Packet capture on LAN interface for testsystem -> 10.130.10.2:80
    20:57:53.512921 IP 10.130.20.2.47370 > 10.130.10.2.80: tcp 0
    20:57:53.513099 IP 10.130.10.2.80 > 10.130.20.2.47370: tcp 0
    20:57:54.511909 IP 10.130.10.2.80 > 10.130.20.2.47370: tcp 0
    20:57:54.514466 IP 10.130.20.2.47370 > 10.130.10.2.80: tcp 0
    20:57:54.514657 IP 10.130.10.2.80 > 10.130.20.2.47370: tcp 0
    20:57:56.511906 IP 10.130.10.2.80 > 10.130.20.2.47370: tcp 0
    20:57:56.530455 IP 10.130.20.2.47370 > 10.130.10.2.80: tcp 0
    20:57:56.530648 IP 10.130.10.2.80 > 10.130.20.2.47370: tcp 0
    20:58:00.527901 IP 10.130.10.2.80 > 10.130.20.2.47370: tcp 0
    20:58:00.626402 IP 10.130.20.2.47370 > 10.130.10.2.80: tcp 0
    20:58:00.626590 IP 10.130.10.2.80 > 10.130.20.2.47370: tcp 0
    20:58:08.623959 IP 10.130.10.2.80 > 10.130.20.2.47370: tcp 0

    Packet capture on LAN interface for testsystem -> 10.130.10.2:3000
    20:58:06.888752 IP 10.130.20.2.45270 > 10.130.10.2.3000: tcp 0
    20:58:06.888936 IP 10.130.10.2.3000 > 10.130.20.2.45270: tcp 0
    20:58:06.889111 IP 10.130.20.2.45270 > 10.130.10.2.3000: tcp 0
    20:58:06.889235 IP 10.130.20.2.45270 > 10.130.10.2.3000: tcp 81
    20:58:06.889434 IP 10.130.10.2.3000 > 10.130.20.2.45270: tcp 0
    20:58:06.892184 IP 10.130.10.2.3000 > 10.130.20.2.45270: tcp 1448
    20:58:06.892195 IP 10.130.10.2.3000 > 10.130.20.2.45270: tcp 1448
    20:58:06.892204 IP 10.130.10.2.3000 > 10.130.20.2.45270: tcp 1200
    20:58:06.892212 IP 10.130.10.2.3000 > 10.130.20.2.45270: tcp 1448
    20:58:06.892222 IP 10.130.10.2.3000 > 10.130.20.2.45270: tcp 1448
    20:58:06.892231 IP 10.130.10.2.3000 > 10.130.20.2.45270: tcp 1448
    20:58:06.892240 IP 10.130.10.2.3000 > 10.130.20.2.45270: tcp 1448
    20:58:06.892249 IP 10.130.10.2.3000 > 10.130.20.2.45270: tcp 1448
    20:58:06.892260 IP 10.130.10.2.3000 > 10.130.20.2.45270: tcp 1448
    20:58:06.892268 IP 10.130.10.2.3000 > 10.130.20.2.45270: tcp 1448
    20:58:06.892483 IP 10.130.20.2.45270 > 10.130.10.2.3000: tcp 0
    20:58:06.892492 IP 10.130.20.2.45270 > 10.130.10.2.3000: tcp 0
    20:58:06.892682 IP 10.130.10.2.3000 > 10.130.20.2.45270: tcp 1448
    20:58:06.892693 IP 10.130.10.2.3000 > 10.130.20.2.45270: tcp 1448
    20:58:06.892702 IP 10.130.10.2.3000 > 10.130.20.2.45270: tcp 126
    20:58:06.893000 IP 10.130.20.2.45270 > 10.130.10.2.3000: tcp 0
    20:58:06.893107 IP 10.130.20.2.45270 > 10.130.10.2.3000: tcp 0
    20:58:06.893336 IP 10.130.10.2.3000 > 10.130.20.2.45270: tcp 0
    20:58:06.893607 IP 10.130.20.2.45270 > 10.130.10.2.3000: tcp 0

    Hope any one has an idea. I've been going at this for a week now and going crazy :-)

    The only thing I didn't mention and that I just thought of is that the server/port combo's that aren't reachable from the OPT1 interface, are being used as backend from HAProxy which is also running on my pfsense server.

    Kind regards,

    Rex de Koning



  • Finally figured out what the problem was... It was indeed the fact that the server/ports that were not working were used as backends in HAProxy. I've added added extra portnumbers to the specific websites and changes the backend portnumbers in HAProxy and now eveything works



  • Sorry only just noticed your question now that youve already found the answer.. The issue happens because of haproxy with transparent-client-ip feature on the backend there is a automatic ipfw rule that catches all reply traffic from the server. even if it didnt originate from haproxy to begin with. Disabling that feature would have probably fixed it. But then you wouldnt have the actual client-ip on the servers anymore. sometimes there are other ways to still do that with forward-for headers, or proxy-protocol.. but add some portnumbers specifically for haproxy, or adding a second ip to the server for this purpose is indeed a good workaround to this issue.

    I haven't found a way to avoid this effect sofar. If anyone knows a better way than the ipfw fwd rule please let me know: )



  • @piba no worries... Thanks for your reply and clear explanation


 

© Copyright 2002 - 2018 Rubicon Communications, LLC | Privacy Policy