@sn0cr4sh said in FIOS - WAN DHCP Setup for G1100 (FiOS Quantum Router) with pfSense (no bridging):

Duuuude, right on!  I got bit by the IPMI overlap as well. My Super Micro C2758 was using the same port for IPMI that I had configured for WAN. I never realized it and managed to get away with it for several months, but suddenly couldn't hold a WAN IP for more than an hour before getting booted off FIOS completely.

I kept getting these weird errors in my log that said a mac address was using the WAN IP. The mac address was the same as the WAN port, so I couldn't make any sense of it. As soon as I shut down and booted into the BIOS, sure enough, my WAN IP was assigned to the IPMI port.  I disabled IPMI on the board and have been running perfectly ever since.

I would have never figured that out if you hadn't posted, so thank you!

This forum is a godsend. I would also never have figured this out were it not for this post. I have a Netgate with a C2758 (and a broken BMC). When I plugged the dedicated IPMI port into my switch, the problems with the DHCP lease expiring every 2 hours went away.

Apparently even with the BMC broken, it defaults to using igb0 (first Intel ethernet port) as a share/failover port for IPMI. Because my BMC is broken (no firmware update for the BMC nor the mobo fixes it) then I can't even disable this default behavior. So my two options are to either connect this extra cable full-time, or to move the WAN interface to a different ethernet port on the box.

Either one seems to work, and thank you for shedding light on this incredibly hard-to-guess-at issue.

@seeking-sense If I have never set up a "route/tunnel VPS IPs to homelab," I wouldn't be able to help or guide you, but if I know sources were you could find answer, I surely share such as here after a web search: https://blog.fuzzymistborn.com/vps-reverse-proxy-tunnel/

@bgksdfol whatever works - sometimes starting clean is a faster solution. While its more satisfying to know the actual root cause off the issue.

Something was messed up that is clear, out of the box pfsense would hand out its IP on any interface as the gateway to dhcp clients, and also dns.. Unless you edit the dhcp server settings to do something different.

@hannibalking not sure what your asking.. You created a lag, yes the interface should be enabled.. But NO you wouldn't put any config on the interface.. It is now part of the lag.. The lag is the interface..

@tsmalmbe So the public IPs on WAN have no Internet access? That just seems a bit odd and hence my misunderstanding. In that situation if only the one IP has Internet, then there’s not a solution here. You’d have to enter maintenance mode on the primary to move the IP, to update the backup.

Otherwise aliases can work fine if aliased to/on the shared IP, and the ISP/data center routes traffic to the shared IP.

Is there no one with any ideas on how to solve this? it is pretty annoying...

See attachment for recent PPP log.
PPP2.txt

Just closing this one out. The answer to this was posted here: https://forum.netgate.com/topic/171690/comcast-bridge-mode-sudden-packet-dropping-dhcp-release-renew-solves-it

Just returning to say 1.5 months later, no issues. Bridge mode has been 100% solid. Thanks again, @darp.

Did it work fine previously ?

Is this for home or buisness, if it's the latter I'd go back to O2 and ask them to change the second router address to 192.168.1.1.

Having dealt a lot with O2 previously, they are a bit clueless.

Otherwise go with a different ISP for the second link.

@marl_scot
The networks on different interfaces must not overlapping.
And I don't know any router which is capable to route with that settings.
Maybe the ISP can give some recommendations.

Two IPs within the same subnet with the same gateway is not a real failover set up for my understanding.

If the ISP refuses to change one of the subnets your only one option might be to put a router between the ISP and pfSense and nat the traffic to a different subnet.

@steveits
Thank you!
Yes, that's exactly what you need!
Are you a pfsense guru)

VLAN write in PortGroup Switch (Esxi)?

@jarhead thank you, I properly setup routing, gateway and everything else.
Now I have a different problem but probably I'll make another dedicated post.

@jimp
I read on this very forum, and elsewhere as well that many users have the same problem, that is, it doesn't switch back to the main WAN when it gets back to work again and is available. Of course, it is not always the case, but unfortunately I was one of those who strumble upon it. As you said, there are always edge/corner cases that don't work as well as others.
For the time being, I run pfSense in a virtual environment , and as a test, I also installed and run OPNsense in the same scenario, then I set failover up on it and it worked smoothly. Anyway, I want to continue using pfsense since I like it most, but this issue can be a drawback.
Thanks

@rcoleman-netgate
That would make sense. Tried disconnecting WAN from igc0 and status was "no carrier" when I think it should be "down".

@rcoleman-netgate

I tried Google (8.8.4.4) with the same results. The one in the screenshot is OpenDNS.

Same problem led me here. Hard to believe this is still a hack!

Just wanted to update my post now that I figured out the problem and solution.
Turns out the reason I could not access the cable modem has to do with my recent change to a WAN failover gateway setup. My fiber provider is tier1 and comcast is tier2. Once I thought through, why can't I get packets to my cable modem, I realized, doh, It is because the cable modem is backup and rarely is the active gateway. I created a firewall lan rule allowing 192.168.100.1 with a gateway override to the comcast cable modem gateway and it now allows this page to load. It is amazing how many problems we can induce when we do not realize it.

@mikael-0 Skimming that, is 192.168.1.2 the IP set on OPT1? Then it seems like the gateway of that interface should be the IP of the 4G router. You wrote you set no gateway...?

Also a /32 mask is only that IP address, usually the mask is /24. a /32 can't talk to any other IP on the network.

doc on isolating a port:
https://docs.netgate.com/pfsense/en/latest/solutions/netgate-2100/configuring-the-switch-ports.html