Multi LAN - ICMP and UDP ok, TCP broken
-
I'm having difficulty getting a multi LAN setup working. I'm probably missing something obvious so any help is really appreciated! Apologies for the long post but I've tried to put in all possible information now.
For one of the LANs, ICMP and UDP are okay but TCP is broken for a client directly attached to pfsense. This sounds like asymmetric routing but I can't confirm this and none of the suggested fixes work.
Setup:
pfsense 2.2.6-RELEASE on a C2758 with 4 x 1 Gb and a 10 Gb ConnectX-2 card.igb0 - WAN - dhcp - (cable modem)
igb1 - LAN1 - 10.0.1.1/24 (1 Gb switch)
igb2 - WIFI - 10.0.2.1/24 (to an wifi router acting as an AP)
igb3 - <unused>mlxen0 - LAN10 - 10.0.10.1/24 (for testing this is directly connected via DAC to a single client WETA, also with a ConnectX-2)WAN
|
WIFI – pfsense -- LAN1 -- switch -- other clients
| |
LAN10 -- client ----/Clients on LAN1 and WIFI can reach the Internet fine.
Cient WETA is connected to LAN1 through the switch and to LAN10 directly. It can reach the Internet fine through LAN1. I do not expect to reach the Internet via LAN10. The problem is the 10 Gb link between pfsense and client WETA. Stateless protocols like ICMP and UDP are fine but TCP is broken and cannot establish a connection. I've dumped the packets on mlxen0 for both pfsense and WETA and can't find all the packets for the 3-way handshake, e.g., on attempting a connection from pfsense to WETA I observe pfsense receiving the SYN-ACK but did not observe this being sent by WETA!
- I've verified the cards in another machine with a static network (no routing) on a direct connection and everything was normal.
- No interfaces have gateways defined.
- I've tried the "Bypass firewall rules for traffic on the same interface" option.
- I've tried manually creating the rules + floating rule as suggested.
- I've tried listening on all interfaces but did not see packets for the connection handshake.
Screenshots for rules on all interfaces are attached, plus a screenshot of the broken TCP handshake. Note, pfsense and WETA were synchronized to approximately 30ms so ordering between servers is not perfect.
</unused> -
"Cient WETA is connected to LAN1 through the switch and to LAN10 directly"
For what possible reason???
-
"For what possible reason???"
A couple of reasons. For example, LAN10 will have a number of clients, including my storage node, on a 10 Gb switch. A noisy switch that sometimes I need to turn off but I still want access to the clients on LAN10, so they're also connected to the 1 Gb network.
The reasons could probably be worked around but is there something fundamentally wrong with having a client on two networks, both routed by the same router? I thought that suitable routing tables and firewall rules would support this?
-
This has NOTHING to do with pfsense at all… You could connect your client to every single interface pfsense has and it doesn't care. It routes/firewalls traffic as it sees it..
If you client does not have a gateway setup on the lan10 network then it would never even send traffic to pfsense unless you were actually going to a pfsense IP on that network.
You say you see the syn-ack on pfsense when you try and connect to lan10.. Well there you, since you state you did not see it sent on your client.. Seems like you have a client issue huh... Because if pfsense saw it clearly the client sent it.
-
The client routing table has the 10.0.10.0/24 subnet routed through the correct interface, mlxen0. I expected that to be correct and sufficient?
I'm trying to make a connection from the pfsense host (10.0.10.1) to the client (10.0.10.102). The pfsense host routing table also has the 10.0.10.0/24 subnet on the correct interface, directly connected to the client.
Well yes, the client did send the syn-ack somehow :-) but I can't observe it. The client should send the syn-ack back to 10.0.10.1 and the routing table says to use mlxen0. I had parallel tcpdumps on all interfaces of the client but didn't see the syn-ack on any of them. Which is why I'm so confused… I'm obviously doing something stupid, both in my configuration and in my debugging, but I can't figure it out.
BTW, thanks for taking the time to read over this!
-
why are you worried about routing??? Once you put an IP on an interface the OS will correctly on its own show that network connected to that interface, and use that interface to talk to that specific network.
If you manually created routes be it on pfsense or your client, that is where I would say you messed up.
Since your not going to use lan10 as path to get to any other network other than the lan10 network, the only route/gateway you should have on your client is talking to pfsense lan IP.
-
I didn't create any routes manually, either on pfsense or the client; the routing tables are derived from the IP/network on each interface.
Correct, the client has a single default gateway, which is pfsense on the 1 Gb network.
-
I disconnected the second path through the 1 Gb network and tested just the 10 Gb directly connected network. Same problems with TCP handshake: on pfsense tcpdump reports receiving the syn-ack from the client before reporting the syn that it must send. I'm not sure why tcpdump is reporting packets out of order, any thoughts? I've tried en- and dis-abling all the offload options but no difference.
