Two pfsense boxes walk into a bar...
-
While they're talking at the bar, every minute or so, one of them gets up and wanders away before coming back and resuming the conversation. No one knows where it goes or what it does, or even which one it is, because even it doesn't know.
It's a horrible joke, but I've had to listen to it literally for hours, and I'm going crazy.
The network configuration is shown below. pfsense #2 is a client of #1 and uses it as a backup ISP. pfsense1 has no idea that pfsense2 is anything other than a client. A static route is set for LAN2 devices to reach LAN1.
Communications between LAN2 and LAN1 is interrupted every sixty seconds such that communications sessions from Workstation2 to WS1 must be re-established. Even so, WS2 can still ping WS1.
However, when I disable firewall filtering in System>Advanced>Firewall on pfsense1, everything works fine. The problem is that the only firewall rules on both pfsense boxes allow all communications between the LANs. In fact, on pfsense1, there are no other rules. It's a clean install with a simple configuration.
I have allowed all traffic between the LANs, but only LAN2 is routed to LAN1, not vice versa.
I'm not a Linux or FreeBSD guy, so I'm not sure where to look for better logs than are provided by pfsense (which show nothing. Really. You've never read more boring logs.)
I'm guessing there's a problem with the double-NAT, but I don't know what to do about it. If anyone has any suggestions or other ideas, I'll be really grateful. I'll gladly provide more details.
-
@eveningstarnm Your traffic flow is asymmetrical
You should connect your 2 pfsense via transit network.
If your going to nat on your downstream router, then you would use port forwarding and anything you want to access on the downstream network wouldn't be routed, your 172 devices would access your 192 devices by hitting the 172.16.0.13 address and be port forwarded.
An intelligent man is sometimes forced to be drunk to spend time with his fools
If you get confused: Listen to the Music Play
Please don't Chat/PM me for help, unless mod related
SG-4860 24.11 | Lab VMs 2.8, 24.11 -
This post is deleted! -
@johnpoz Yes, the traffic is asymetrical. LAN1 accepts inbound internet traffic. LAN2 does not. It is a client of LAN1, with access in both directions filtered through LAN2's pfsense. It's a common configuration, which is often described as a LAN with a DMZ and an auxiliary ISP. Our network was designed to fit our use-case. The problem you solved was for a different network, and wasn't related to ours.
The cause of the problem turned out to be an ethernet cable that had one wire with exceptionally high resistance. I hadn't noticed the high latency because I had disabled monitoring of the LAN1 gateway by LAN2. I keep forgetting Step One: Check the cables.
-
@eveningstarnm said in Two pfsense boxes walk into a bar...:
The cause of the problem turned out to be an ethernet cable that had one wire with exceptionally high resistance. I hadn't noticed the high latency because I had disabled monitoring of the LAN1 gateway by LAN2. I keep forgetting Step One: Check the cables.
can you please elaborate on that ? i'm curious about how did you came to this conclusion and what does "exceptionally high resistance" mean
thank you! -
@eveningstarnm said in Two pfsense boxes walk into a bar...:
Yes, the traffic is asymetrical. It's a common configuration,
No its not - not if you want stuff to work..
-
Yes, I would expect to see problems with an asymmetric network like that. You would need, at a minimum, a bunch of work-around rules to allow the traffic to pass.
Using a transit subnet between the two routers is much nicer configuration.Steve
-
@stephenw10 said in Two pfsense boxes walk into a bar...:
a bunch of work-around rules
Exactly.. Sure you could run a downstream router/firewall like your doing. But you would either need to host route on every device in the transit network 172.16/16 in your case so your not bouncing downstream destined traffic off pfsense 1 and creating asymmetrical flow with a state that will never see any return traffic.
Or you need to nat at the downstream and port forward at pfsense2 and access resources in the downstream network via hitting 172.16.0.13.. So now pfsense 1 never sees the traffic and is not involved in those conversations.
You might be able to get away with doing some sort of source nat at pfsense1 and sending the traffic on to pfsense2 looking like it came from pfsense 1 IP..
There is also the problem with traffic initiated by the downstream network to a 172.16.x.x host where pfsense would never see the syn, so not state and when the device answers back sending traffic to pfsense (syn,ack) this would be denied.
When you connect routers, they should be connected with a transit network (no hosts on this network).. Now no host routing, no work a rounds.. Just simple basic routing and firewall rules and no concerns of asymmetrical flow.
I get that such a configuration might be "common" - but its borked..
-
@aduzsardi During, my tests, as I was reconfiguring parts of the ethernet network, I noticed the insulation on one cable attached to the pfsense box had pulled out of the CAT-5 connector. I replaced the cable, and everything started working. Immediately. A connection should have been lost a few seconds later, then reconnected, but there were no problems at all, even though nothing else I did had made a dent in the problems. So, delighted to have a semi-legitimate reason to fire up my Fluke 8808A, I tested each wire in the cable. Blue tested something in megaohms. It was an old cable.
-
@stephenw10 Have you ever seen a private network connected as a client to another?
I really don't understand why this configuration appears to be so strange to some. Perhaps if I'd re-arranged the figures, it would have been more recognizable as an exercise in some CS200-level class.
-
@eveningstarnm said in Two pfsense boxes walk into a bar...:
really don't understand why this configuration appears to be so strange to some.
I showed you in the picture! It is asymmetrical!
An intelligent man is sometimes forced to be drunk to spend time with his fools
If you get confused: Listen to the Music Play
Please don't Chat/PM me for help, unless mod related
SG-4860 24.11 | Lab VMs 2.8, 24.11 -
@eveningstarnm said in Two pfsense boxes walk into a bar...:
@stephenw10 Have you ever seen a private network connected as a client to another?
Many times and quite often using a similar setup to this. And usually it's when people open a ticket with us to determine why they are seeing connection issues. It almost always because of asymmetric traffic being blocked in 'pfSense 1' in your diagram because it only sees half the TCP conversation.
You would find it connectivity comes in and out as the first firewall sends an ICMP redirect the host allowing it to work for a short time until it expires. Typically ~30s. It fits what you initially described perfectly.
The only we to avoid that without using a transport subnet (which is the correct way to do it) is to add specially configured firewall rules:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/asymmetric-routing.html#manual-fixSteve
-
@johnpoz Yes, it's asymetrical. One network is a client of another. IT'S SUPPOSED TO BE THAT WAY. But I won't explore here the myriad use cases in which such a design is not only employed but expected.
-
@stephenw10 They're different networks with different purposes. They are not peers. In fact, not every device from the two networks are allowed to talk to each other. In that regard, it's kind of like the entire internet.
One network is a client of another. If you want to solve a different problem, you should start your own thread. The one you know how to solve isn't related to this one.
-
@eveningstarnm said in Two pfsense boxes walk into a bar...:
IT'S SUPPOSED TO BE THAT WAY
Says who - you? It is NOT suppose to be that way.. Already went over all the hoops you have to go through to use it like that..
If you want to run such setup - have fun with it. Couple of minutes to actually create a valid transit network and you wouldn't have to.
You know who runs networks like that - people that don't know any better..
-
@johnpoz Dude. Please avoid my threads. I see no indication that you could be at all helpful to me. You insist on telling me how our networks should be designed when you don't even know how they're related or what they're used for. Those networks don't even have the same TLD. They're different networks with different purposes, and they are not related the way you want them to be. If they were, they would not suit our needs. One is simply using a service that the other provides.
I described the problem. I even found the solution. Your comments have been irrelevant and a total waste of time. I'm blocking you again. The years I spent here not seeing your comments were good ones.
Update: As it turns out, I can't re-block you. I should never have unblocked you.
-
@eveningstarnm good luck! I don't care how you want to use or or intend to use it. Using a network with hosts on it as transit, which is any network that connects routers together.. Is not the proper way to do it.. Sorry but its not..
Can you do it sure - but if you plan on talking to these hosts on this transit - its going to be problematic without the work arounds given, natting or host routing. You were even given a pretty picture showing how traffic will be asymmetrical. But clearly you know better..
-
@johnpoz I don't need luck. I solved the problem, and it's working great. Nothing that you said was helpful.
-
Reviewing this I assume that, effectively, pfSense 2 in your diagram is using pfSense 1 as second WAN?
If it's only using it for that it will work fine. Hosts behind pfSense 2 would be able to access hosts on the pfSense 1 LAN as long as pfSense 2 is outbound NATing the traffic, which removes the route asymmetry. But that obviously obscures the source IP.
Using a transport subnet between the two firewalls is a far more flexible setup and I would certainly recommend doing that if configuring a network where it's possible. But if you know you will never need to route the other way it will work without.
Steve
-
@stephenw10 You are safe in assuming that, since that's what I said in my original post.
