New pfSense box looses Fios connection every 1-2 hours [Solved]

[pfWarrior]

This is a new install, my first pfSense appliance which I'm happy to have after wanting one for years. However, I'm having trouble getting it working reliably with Verizon Fios. I loose internet connectivity from the WAN after about 90-120 minutes of uptime. Rebooting consistently re-establishes the connection under a new leased IP, only to fail again about an hour and a half later. After a week of troubleshooting, I've learned the problem is evidently a connection solicited to the WAN shown below in the screenshot. This is from the firewall log. No blockers or IDS/IPS are installed and only the default pfSense rules are in use.

Here's my equipment and setup:
Alcatel I-211M-L optical network terminal
Supermicro board with Intel NICs running pfSense 2.4.4-RELEASE-p2
Asus wifi AP

ONT -> pfSense WAN (DHCP). pfSense LAN (static)-> Wifi AP

I've read that a double NAT scenario is sometimes encountered with Verizon hardware but my pfSense WAN has never been in a private range. Deselecting the WAN config for blocking bogon and private networks doesn't make a difference, nor does adding a firewall rule allowing SRC 0.0.0.0:68 DST: 255.255.255.255:67. I set this rule to be logged when passed and it appears in the FW log, but the outage still occurs.

Power cycling the ONT off/on has the same effect as rebooting pfSense-connection comes back up but then eventually fails again. Even setting the WAN to statically use an IP and gateway assigned by DHCP doesn't stop the disconnect. PfSense's DHCP log shows the lease time is only 3600 seconds (1hr) which is low, but I read is not uncommon for Fios. The problem seems to be upstream and it's extremely frustrating that my cheap consumer routers can deal with this yet a $600 appliance can not. I'm thinking that a small unmanaged switch between the ONT and pfSense could solve the problem which would look like:

ONT -> switch (DHCP). Switch -> pfSense WAN (static assignment if possible). pfsense LAN (static)-> Wifi AP

I want to check in here for additional thoughts before I spend the money. I've done a lot of searching on the matter and while not much is available matching my situation, what I have found offers no conclusion. It'd be a shame to have to go back to a regular home router at this point to continue using Fios, but a little Netgear switch or maybe a new ONT are seemingly the only remaining possibilities.

Any help would be greatly appreciated.

[dtruesdale]

Don't feel bad, I have a customer that we are begging them to get of Verizon. Sounds like our same issue. We tried bridging the setup but the way Verizon does it, it usually stops in a few days. It has something to do with the modem and the software running it. Verizon has been of no help other than they want to sell us their approved equipment.

[Derelict]

I don't have any insight into what Verizon FIOS needs but those logs are a DHCP client looking for a DHCP server out on the WAN network and getting no response. What is interesting is it is a DHCP CLIENT looking for a server so what is out on WAN doing that? DHCP from pfSense would not be logged there. If it was, it would have the outbound flag showing.

You will need to packet capture on WAN to get the MAC addresses in the packets to know more about who is sending what where.

You can reboot then start a packet capture - Diagnostics > Packet Capture WAN, UDP, Port 67, Count 1000000 and let it run. In this case you might want to set the promiscuous checkbox too. When WAN has stopped working for an amount of time, you can stop the capture and download it.

You might want to disconnect the WAN, reboot, start the packet capture, then connect WAN so the capture will start with a successful lease.

Wireshark does a great job at decoding the DHCP protocol for you.

I'm thinking that a small unmanaged switch between the ONT and pfSense could solve the problem which would look like:
ONT -> switch (DHCP). Switch -> pfSense WAN (static assignment if possible)

It would not look like that. It would look like this:

ONT -> Switch -> pfSense WAN (DHCP). pfSense LAN (static)-> Wifi AP

An unmanaged switch will not get a DHCP lease. It will transparently pass Layer 2 ethernet frames. It has no concept of IP addresses (Layer 3).

[pfWarrior]

Welp turns out this whole ordeal had nothing to do with Verizon or my ONT. I did the packet capture and the mystery device sending DHCP signals to the WAN was the pfSense box's own baseboard management controller. My board's BIOS has an option to disable the IPMI function which is supposed to disable BMC networking along with it, but evidently that doesn't work as explained, or is broken. And even though I'd never connected that network interface to anything, the BMC wants a DHCP lease. I logged into the IPMI GUI, set a static config, and I'm now nearing 24 hours of uninterrupted uptime. The 0.0.0.0:67 - 255.255.255.255:68 entries haven't shown up again in packet captures or the firewall logs.

I'm very happy this was smoothed out and thank you @Derelict for the tip to look at the MAC addresses.

@dtruesdale For anyone else in this situation, a few more tips:

1. You can just set an IPMI address, netmask, and gateway in the BIOS. This is all that's really necessary so you don't actually need expose the BMC to the network.

2. If you fully configure IPMI and intend to leave it network-accessible, you'll of course want to change the default ADMIN/ADMIN username and password. Through significantly more trial and error than it should have taken, I found that even the latest version of my board's IPMI firmware is so old that it doesn't allow special characters in the user passwords. Despite Supermicro support pages saying that the max pw length is 20 characters, I wasn't able to use more than 16. There's also a handful of service ports that are enabled by default so check those out.

3. This site has easy instructions to reset the admin pw for if (when) you lock yourself out: http://tcpip.me/2018/06/23/how-to-recover-forgotten-ipmi-credentials-on-pfsense/