Admin GUI/SSH access disappears in bridging config with pfSense 2.0.1 [long]
-
Hi all, long-time pfSense and FreeBSD user here, with a mystifying problem. I have been trying to figure this problem out for a number of days, over the course of at least a month, and for a long time I've been assuming it was all my problem due to a configuration error (which it still might be). However, I'm starting to think it may be an actual bug in pfSense 2.0.
The setup:
The configuration I have set up is using pfSense as a transparent bridging firewall between the upstream and the router for the company network; no NAT, no real routing. In the current configuration, pfSense is between the telco's DSL modem and a Cisco router which is serving the internal network. The gateway address (for pfSense and for the Cisco) is the .1 address of the /24, and we have an allocation of 5 public IP addresses which are considered "internal" from the standpoint of pfSense. I've put this in place as part of the company's move to a new office.
This is pretty close in concept to a configuration I've been using at home for years with pfSense 1.2.3, so I was confident that it should work. (Also, I've been using FreeBSD since 1996 or thereabouts, so I have faith that the underlying networking should be solid.)
Hardware is a tower case with an i7 870 (quad core) with 4GB RAM, a built-in Intel NIC (em0) and a second Intel NIC (em1) installed in a PCI slot; from my back-of-envelope figures, in principle it should be able to handle moderate filtering on a completely saturated GigE without trouble.
It's configured as LAN = em0, WAN = em1, BRIDGE (aka OPT1) = bridge0;
The problem:
The GUI becomes either inaccessible from the LAN - even with anti-lockout enabled, and manual pass rules as the first thing - or accessible from the WAN interface when the WAN filter rules should be completely blocking it, or both.
Initially I had the management IP address assigned to the LAN side; I switched it to the bridge interface when I started having trouble, then switched it back to the LAN interface when that wouldn't work at all for me. Today I have spent almost the whole day on reconfiguring pfSense, and have seen the GUI become available from inside the LAN, unavailable from inside the LAN but available from outside the LAN, unavailable from anywhere, back to working properly from inside the LAN, and back to unavailable from anywhere again. Once or twice I even saw it working as it was supposed to (unavailable from the WAN but available from the LAN.)
I've successively turned off more and more blocking rules until at this point there is virtually no block rule enabled accept the default deny.
Initially I was trying to relate this to the content of changes I was making, but at this point it seems more as though applying any configuration update has a chance of disabling the GUI independently of the content. For example, when it went offline most recently, I was only adding the "Log" qualifier to some of the explicit rules to permit access to the management IP address, so I could try to figure out which rules were making it work. At various points earlier today, it went away when I was updating the default gateway (note that it doesn't need a gateway to reach our internal LAN by way of the Cisco) or when I was adding new "Pass" rules (which should not have been able to cause something new to be blocked.) Alternatively, it's possible that it just runs for a while and then randomly falls off the network; I can't really tell the difference between those cases.
Right now I'm back at the "can't get there from anywhere" stage and will have to go hang out in the server room getting a console on it again, and see if I can get it visible again.
The one definite clue I have observed when using tcpdump from the shell on pfSense is that at least sometimes when this problem is going on, it appears that pfSense is accepting the TCP connection to SSH or HTTP from inside on em0 (LAN) and replying to it on em1 (WAN); the reply is also visible via tcpdump on bridge0, but never appears on em0. Swapping the physical cables or assignments around did nothing in most cases, or made the GUI "reappear" but only temporarily, only to go away again after a while.
It also seemed like it was working OK every time I tried it in my test configuration, when I had it on the 10.0.0 network, with my PC at 10.0.0.100 directly plugged into em0 (LAN) and the company LAN (10.0.0.1/24) on em1 (WAN). I don't know why it might have worked in that config but not when switched to public addresses, or if I just didn't test and wait long enough to discover the problem in that arrangement.
BTW, another problem I've observed repeatedly during this was that the default gateway would keep going away every time I changed something about the interface assignments; however that isn't related to this problem, because I shouldn't even need a gateway to reach it from the LAN side, as the Cisco sits in the same netblock with it.
Monday a.m. I am supposed to have this fully operational so that everybody can get back to work productively at the new office, but I am resigned that this is very unlikely to happen. Right now it is operational in the sense that traffic is passing through it and everybody's happy, but 1) I've turned off so much of the blocking rules it's not really protecting anything, and 2) I can't change any of the rules on it or even look at them without pulling it offline and moving it back to the internal network, at which point all the address-related settings are wrong, I can't test the validity of the rules, and it's hard to know if something I change will work when it's put back in line.
Although I know pfSense is a great system, at this point I've got some egg on my face for taking so long trying to get it to work, and I'm starting to feel like a bozo for recommending it to my employer.
Any spare clues anybody could spare me would be greatly and humbly appreciated.
I'll attach the most recently saved config as .txt once I've looked over it to make sure there's nothing confidential included; this should be fairly close to what's on it now although not identical.
– Clifton
-
Attached is the last saved configuration. I've only modified the admin password, the domain name, and the high-order octets of the IP address; everything else is as close as I've got to current.
-
I removed the pfBlocker package, thinking maybe that was causing the problem - perhaps bad interaction of the automatic rules with the bridged configuration - and that seemed to have it working for a while. I was hopeful when I could still reach it this morning from my whitelisted external address, but nope.
-
The problem is now moot, as I switched to an entirely different configuration - I split the connection from pfSense to the router into VLAN trunks, so that the management IP address on "LAN" is now in its own netblock and not part of the bridged traffic.
LAN = em0_vlan6 (LAN IP in private IP space), WAN = em1, INTERNAL= em0, BRIDGE = bridge0 = em0 + em1.
The firewall has no IP address on the bridged network and no public IP address. This works.
I still think there's some kind of bug with having the management address on a bridged interface, but it doesn't matter as much now that I can administer the box. (As a bonus, the netblock for the LAN IP is not routed or NATted, so I don't have to worry so much about filtering external access to it.)
– Clifton
-
Normally on a bridged setup you would give the bridge an IP address in the bridged subnet.
But the way you ended up doing it with a LAN address is the way I do mine. I call it the maintenance port.
-
Normally on a bridged setup you would give the bridge an IP address in the bridged subnet.
I tried that configuration at least 3 times. (Not kidding about that; I did a "factory reset" or tore the config down and reconfigured a number of times Thursday and Friday of last week, and a couple more on Saturday.) IP address on the bridge appeared to have the identical problem, both in terms of visible symptoms and according to tcpdump. (I spent a lot of time at the root shell looking at the state of things.)
Normally IP address on the bridge would be my first inclination, although setting it up from the GUI made it a lot easier to configure it on the LAN i/f.
Anyway, thanks for replying. Hearing that somebody else is using the same approach I ended up with gives me more confidence that I went the right way.