Navigation

    Netgate Discussion Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search

    Multi LAN - ICMP and UDP ok, TCP broken

    Firewalling
    2
    8
    1269
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • T
      tuatara last edited by

      I'm having difficulty getting a multi LAN setup working. I'm probably missing something obvious so any help is really appreciated! Apologies for the long post but I've tried to put in all possible information now.

      For one of the LANs, ICMP and UDP are okay but TCP is broken for a client directly attached to pfsense. This sounds like asymmetric routing but I can't confirm this and none of the suggested fixes work.

      Setup:
      pfsense 2.2.6-RELEASE on a C2758 with 4 x 1 Gb and a 10 Gb ConnectX-2 card.

      igb0 - WAN - dhcp - (cable modem)
      igb1 - LAN1 - 10.0.1.1/24 (1 Gb switch)
      igb2 - WIFI - 10.0.2.1/24 (to an wifi router acting as an AP)
      igb3 - <unused>mlxen0 - LAN10 - 10.0.10.1/24 (for testing this is directly connected via DAC to a single client WETA, also with a ConnectX-2)

      WAN 
                |
      WIFI – pfsense -- LAN1 -- switch -- other clients
                |                  |                     
              LAN10 --  client ----/

      Clients on LAN1 and WIFI can reach the Internet fine.

      Cient WETA is connected to LAN1 through the switch and to LAN10 directly. It can reach the Internet fine through LAN1. I do not expect to reach the Internet via LAN10. The problem is the 10 Gb link between pfsense and client WETA. Stateless protocols like ICMP and UDP are fine but TCP is broken and cannot establish a connection. I've dumped the packets on mlxen0 for both pfsense and WETA and can't find all the packets for the 3-way handshake, e.g., on attempting a connection from pfsense to WETA I observe pfsense receiving the SYN-ACK but did not observe this being sent by WETA!

      • I've verified the cards in another machine with a static network (no routing) on a direct connection and everything was normal.
      • No interfaces have gateways defined.
      • I've tried the "Bypass firewall rules for traffic on the same interface" option.
      • I've tried manually creating the rules + floating rule as suggested.
      • I've tried listening on all interfaces but did not see packets for the connection handshake.

      Screenshots for rules on all interfaces are attached, plus a screenshot of the broken TCP handshake. Note, pfsense and WETA were synchronized to approximately 30ms so ordering between servers is not perfect.

      ![Screen Shot 2016-03-22 at 22.43.12.png](/public/imported_attachments/1/Screen Shot 2016-03-22 at 22.43.12.png)
      ![Screen Shot 2016-03-22 at 22.43.12.png_thumb](/public/imported_attachments/1/Screen Shot 2016-03-22 at 22.43.12.png_thumb)
      ![Screen Shot 2016-03-22 at 22.43.22.png](/public/imported_attachments/1/Screen Shot 2016-03-22 at 22.43.22.png)
      ![Screen Shot 2016-03-22 at 22.43.22.png_thumb](/public/imported_attachments/1/Screen Shot 2016-03-22 at 22.43.22.png_thumb)
      ![Screen Shot 2016-03-22 at 22.43.30.png](/public/imported_attachments/1/Screen Shot 2016-03-22 at 22.43.30.png)
      ![Screen Shot 2016-03-22 at 22.43.30.png_thumb](/public/imported_attachments/1/Screen Shot 2016-03-22 at 22.43.30.png_thumb)
      ![Screen Shot 2016-03-22 at 22.43.40.png](/public/imported_attachments/1/Screen Shot 2016-03-22 at 22.43.40.png)
      ![Screen Shot 2016-03-22 at 22.43.40.png_thumb](/public/imported_attachments/1/Screen Shot 2016-03-22 at 22.43.40.png_thumb)
      ![Screen Shot 2016-03-22 at 22.43.47.png](/public/imported_attachments/1/Screen Shot 2016-03-22 at 22.43.47.png)
      ![Screen Shot 2016-03-22 at 22.43.47.png_thumb](/public/imported_attachments/1/Screen Shot 2016-03-22 at 22.43.47.png_thumb)
      ![Screen Shot 2016-03-22 at 22.44.04.png](/public/imported_attachments/1/Screen Shot 2016-03-22 at 22.44.04.png)
      ![Screen Shot 2016-03-22 at 22.44.04.png_thumb](/public/imported_attachments/1/Screen Shot 2016-03-22 at 22.44.04.png_thumb)
      ![Screen Shot 2016-03-22 at 22.49.47.png](/public/imported_attachments/1/Screen Shot 2016-03-22 at 22.49.47.png)
      ![Screen Shot 2016-03-22 at 22.49.47.png_thumb](/public/imported_attachments/1/Screen Shot 2016-03-22 at 22.49.47.png_thumb)
      ![Screen Shot 2016-03-22 at 23.01.32.png](/public/imported_attachments/1/Screen Shot 2016-03-22 at 23.01.32.png)
      ![Screen Shot 2016-03-22 at 23.01.32.png_thumb](/public/imported_attachments/1/Screen Shot 2016-03-22 at 23.01.32.png_thumb)</unused>

      1 Reply Last reply Reply Quote 0
      • johnpoz
        johnpoz LAYER 8 Global Moderator last edited by

        "Cient WETA is connected to LAN1 through the switch and to LAN10 directly"

        For what possible reason???

        An intelligent man is sometimes forced to be drunk to spend time with his fools
        If you get confused: Listen to the Music Play
        Please don't Chat/PM me for help, unless mod related
        2440 2.4.5p1 | 2x 3100 2.4.4p3 | 2x 3100 22.01 | 4860 22.01

        1 Reply Last reply Reply Quote 0
        • T
          tuatara last edited by

          "For what possible reason???"

          A couple of reasons. For example, LAN10 will have a number of clients, including my storage node, on a 10 Gb switch. A noisy switch that sometimes I need to turn off but I still want access to the clients on LAN10, so they're also connected to the 1 Gb network.

          The reasons could probably be worked around but is there something fundamentally wrong with having a client on two networks, both routed by the same router? I thought that suitable routing tables and firewall rules would support this?

          1 Reply Last reply Reply Quote 0
          • johnpoz
            johnpoz LAYER 8 Global Moderator last edited by

            This has NOTHING to do with pfsense at all… You could connect your client to every single interface pfsense has and it doesn't care.  It routes/firewalls traffic as it sees it..

            If you client does not have a gateway setup on the lan10 network then it would never even send traffic to pfsense unless you were actually going to a pfsense IP on that network.

            You say you see the syn-ack on pfsense when you try and connect to lan10..  Well there you, since you state you did not see it sent on your client.. Seems like you have a client issue huh... Because if pfsense saw it clearly the client sent it.

            An intelligent man is sometimes forced to be drunk to spend time with his fools
            If you get confused: Listen to the Music Play
            Please don't Chat/PM me for help, unless mod related
            2440 2.4.5p1 | 2x 3100 2.4.4p3 | 2x 3100 22.01 | 4860 22.01

            1 Reply Last reply Reply Quote 0
            • T
              tuatara last edited by

              The client routing table has the 10.0.10.0/24 subnet routed through the correct interface, mlxen0. I expected that to be correct and sufficient?

              I'm trying to make a connection from the pfsense host (10.0.10.1) to the client (10.0.10.102). The pfsense host routing table also has the 10.0.10.0/24 subnet on the correct interface, directly connected to the client.

              Well yes, the client did send the syn-ack somehow :-) but I can't observe it. The client should send the syn-ack back to 10.0.10.1 and the routing table says to use mlxen0. I had parallel tcpdumps on all interfaces of the client but didn't see the syn-ack on any of them. Which is why I'm so confused… I'm obviously doing something stupid, both in my configuration and in my debugging, but I can't figure it out.

              BTW, thanks for taking the time to read over this!

              1 Reply Last reply Reply Quote 0
              • johnpoz
                johnpoz LAYER 8 Global Moderator last edited by

                why are you worried about routing???  Once you put an IP on an interface the OS will correctly on its own show that network connected to that interface, and use that interface to talk to that specific network.

                If you manually created routes be it on pfsense or your client, that is where I would say you messed up.

                Since your not going to use lan10 as path to get to any other network other than the lan10 network, the only route/gateway you should have on your client is talking to pfsense lan IP.

                An intelligent man is sometimes forced to be drunk to spend time with his fools
                If you get confused: Listen to the Music Play
                Please don't Chat/PM me for help, unless mod related
                2440 2.4.5p1 | 2x 3100 2.4.4p3 | 2x 3100 22.01 | 4860 22.01

                1 Reply Last reply Reply Quote 0
                • T
                  tuatara last edited by

                  I didn't create any routes manually, either on pfsense or the client; the routing tables are derived from the IP/network on each interface.

                  Correct, the client has a single default gateway, which is pfsense on the 1 Gb network.

                  1 Reply Last reply Reply Quote 0
                  • T
                    tuatara last edited by

                    I disconnected the second path through the 1 Gb network and tested just the 10 Gb directly connected network. Same problems with TCP handshake: on pfsense tcpdump reports receiving the syn-ack from the client before reporting the syn that it must send. I'm not sure why tcpdump is reporting packets out of order, any thoughts? I've tried en- and dis-abling all the offload options but no difference.

                    1 Reply Last reply Reply Quote 0
                    • First post
                      Last post