pfSense stopped recognising cable modem in bridged mode
-
I have a pfSense install at a remote site on ESXi VM on a Dell R210 which has been running fine for a year. It has two WANs which are brought in from separate ISP modems connected through a VLAN switch to bring them into the box on one cable.
WAN1 is a cable modem running in modem mode (basically bridge mode) and the public IP etc is assigned to pfSense.
WAN2 is a DSL modem running in normal router mode with pfSense in the DMZ (unfortunately there is nothing I can do about this).I got a notification at 0215 on Tuesday 27/11 that WAN1 was down. I checked pfSense and under interface status it showed "Pending" and the interface did not show any gateway or public IP. I tried to ping the cable modem from pfSense and it failed. I called someone on-site to power cycle the cable modem but it did not solve the problem.
Today I visited the site and could not figure out what was going on. I connected the cable modem directly to my laptop and power cycled it and it came up fine, giving my laptop the public IP. So I connected it back to the pfSense and power cycled it again and pfSense never gets an IP. I removed the cable modems LAN IP from the "reject leases box" to see if it would even get given a 192.168.100.x address but that doesn't even happen. It just gets stuck on "Pending". All of the cables are fine and the switch is fine (WAN2 still working perfectly).
Eventually I put the cable modem back in to router mode and connected it to pfSense. It assigned pf a local IP and I can get internet through it no problem. So it seems as though pfSense suddenly stopped recognising the modem in bridge mode.
Notable things from the log:
Nov 27 16:42:00 pfsense kernel: arpresolve: can't allocate llinfo for 92.238.190.1 on em1
^ There are a LOT of these starting from when the WAN1 first went down on 27/11. 92.238.190.1 is an IP on the WAN1 ISP network, I think it's the gateway.Nov 27 20:19:49 pfsense php-fpm: /status_interfaces.php: The command '/usr/local/sbin/dhclient {$ipv} -d -r -lf '/var/db/dhclient.leases.em1' -cf '/var/etc/dhclient_wan.conf' -sf '/usr/local/sbin/pfSense-dhclient-script'' returned exit code '1', the output was 'Internet Systems Consortium DHCP Client 4.3.6-P1 Copyright 2004-2018 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Listening on BPF/em1/00:0c:29:39:5b:b2 Sending on BPF/em1/00:0c:29:39:5b:b2 Can't attach interface {} to bpf device /dev/bpf0: Device not configured If you think you have received this message due to a bug rather than a configuration issue please read the section on submitting bugs on either our web page at www.isc.org or in the README file before submitting a bug. These pages explain the proper process and the information we find helpful for debugging. exiting.'
Nov 27 20:32:01 pfsense php-fpm: /interfaces.php: The command '/sbin/dhclient -c /var/etc/dhclient_wan.conf em1 > /tmp/em1_output 2> /tmp/em1_error_output' returned exit code '15', the output was ''
-
Something upstream blocking the MAC address maybe? Try spoofing it to something else on the WAN setup page.
Steve
-
@stephenw10 said in pfSense stopped recognising cable modem in bridged mode:
Something upstream blocking the MAC address maybe? Try spoofing it to something else on the WAN setup page.
Steve
As part of the troubleshooting I created a new virtual interface on the ESXi, with a different MAC address, and new VLAN etc through the switch. It still didn't work.
-
OK, something really weird is going on here. I can't seem to hit the firewall through the WAN1 link even though it's up and working and pfSense is in the DMZ.
When I try to VPN in through WAN1, I see this:
Fri Nov 30 01:15:52 2018 TCP/UDP: Incoming packet rejected from [AF_INET]1.2.3.4:1194[2], expected peer address: [AF_INET]5.6.7.8:1194 (allow this incoming source address/port by removing --remote or adding --float)
Where 1.2.3.4 is the WAN2 public IP, and 5.6.7.8 is the WAN1 public IP. I have no idea that would happen.
-
Seems like your two WAN adapters have been switched for some reason.
Steve
-
@stephenw10 said in pfSense stopped recognising cable modem in bridged mode:
Seems like your two WAN adapters have been switched for some reason.
Steve
How and why would that happen? WAN2 still works perfectly. It's only WAN1 which has stopped working
-
@jpns said in pfSense stopped recognising cable modem in bridged mode:
So it seems as though pfSense suddenly stopped recognising the modem in bridge mode.
A modem in bridge mode is supposed to be transparent, so there's nothing to recognize. What's supposed to happen is the firewall is supposed to get it's address, etc. from the ISP.
-
@jknott said in pfSense stopped recognising cable modem in bridged mode:
@jpns said in pfSense stopped recognising cable modem in bridged mode:
So it seems as though pfSense suddenly stopped recognising the modem in bridge mode.
A modem in bridge mode is supposed to be transparent, so there's nothing to recognize. What's supposed to happen is the firewall is supposed to get it's address, etc. from the ISP.
Yes, you're right. But that has stopped happening now, and I assume the dhclient errors in the syslog are something to do with it, but I can't understand what they mean.
I'm going to spin up a new VM with a fresh install of pfSense, and get it configured ready to drop in next time I'm on site. I suddenly remembered after I had left that I had a VM snapshot of the broken install from 2 weeks before the failure, which I'm going to try restoring first, but if that doesn't work I'll just delete it and bring the new one online. It would be really nice to figure out what the problem is, though.
-
Both WANs come into pfSense on one cable via two vLANs? Were there any changes on the switch?
Were there any hardware changes/failures on the Dell host?
What kind of NICs are on the host?
I’ve had VMs go weird on me with USB adapters that have created situations like this. That’s why I am asking about the hardware.
What version of ESXi are you running?
-
@tim-mcmanus said in pfSense stopped recognising cable modem in bridged mode:
Both WANs come into pfSense on one cable via two vLANs? Were there any changes on the switch?
Were there any hardware changes/failures on the Dell host?
What kind of NICs are on the host?
I’ve had VMs go weird on me with USB adapters that have created situations like this. That’s why I am asking about the hardware.
What version of ESXi are you running?
Yes, the WANs come into the VM host on one cable via separate VLANs. The pfSense box only has two network cards, and I like to keep the LANs and the WANs on separate physical interfaces. WAN1 comes from the switch on VLAN20 and WAN2 comes from the switch on VLAN30 on the same cable. The VLANs are configured in VMware as the VLAN tags are stripped at the host unless you use virtual guest tagging which I was not aware of when I initially installed the box. It appears to pfSense as two separate physical interfaces. This configuration has always worked for me until now and there were no changes before it stopped working.
There were no changes on the switch and no hardware failures on the switch or server that I can tell. As soon as I switched the WAN1 modem to router mode, it worked. It just won't work with the modem in bridged mode.
I am using the onboard network cards in the host which I believe are Broadcom BCM5722's.
Running ESXi 6.5.
-
Run a packet capture on the WAN whilst trying to pull a lease. Do you see outgoing requests? Any replies at all?
The fact you were seeing incoming traffic from the WAN2 IP when connecting to WAN1 is suspect. Are you somehow outbound NATing traffic from the firewall itself? You should not have any outbound NAT rules with source 'any'.
Steve
-
You are rebooting the cable modem with each change of an interface MAC address.. right??
Depending on how many MAC addresses your ISP allows you have to reboot to release.
-
@chpalmer said in pfSense stopped recognising cable modem in bridged mode:
You are rebooting the cable modem with each change of an interface MAC address.. right??
Depending on how many MAC addresses your ISP allows you have to reboot to release.
Yes I am.
-
Silly question: Do both cable modems go to the same ISP router as their first hop? Are they using the same ISP gateway?
-
@tim-mcmanus said in pfSense stopped recognising cable modem in bridged mode:
Silly question: Do both cable modems go to the same ISP router as their first hop? Are they using the same ISP gateway?
No they are completely separate ISPs. WAN1 is an Arris TG2492LG-VM cable modem/router which I originally had in bridge mode. WAN2 is a Huawei HG633 VDSL modem/router which unfortunately doesn't have a working bridge mode.
-
Just as an update to this. I span up a new VM with a fresh install of pfSense. Copied most of the settings across so it was ready to 'drop in'. I visited the site two weeks ago, shut down the broken pfSense, and booted the new one. Power cycled the cable modem and immediately everything worked. Two weeks in and the cable modem is still recognised and working correctly. So I assume the problem was caused by some sort of corruption in the config file.
