Comcast Bridge Mode Sudden Packet Dropping (DHCP release renew solves it)

planedrop

Been digging on this one for maybe 3 hours now, and it very well might be a comcast issue (they have a ticket for me now finally), but wanted to post here because I think I've seen a similar situation with pfSense mentioned before (but no luck Googling).

Long story short, I have a dynamic IP with Comcast Business, modem is in bridge mode, pfSense is plugged into modem. pfSense gets a public IP via DHCP on the WAN side just fine, internet works great for about 5 minutes, then suddenly 100% of packets start dropping. The only fix for this now is to trigger a DHCP release/renew on the WAN interface of pfSense, it will then get the same public IP as before and work for about 5 minutes.

Seems to me like Comcast is dropping the packets after a set time for no reason.

I've done setups like this many times and far more complex (multi WAN 20 IP CARP HA setups) and never ran into issues like this that I couldn't resolve. And I've tried all the typicals, restarting pfSense, the modem, factory resetting and re-enabling bridge mode on the modem, etc... with no luck.

ptyork

@planedrop did you find an answer to this? Same thing is happening to me. I posted a new topic on it before I decided to search for Comcast in addition to Xfinity. :)

planedrop

@ptyork Hey sorry, just now seeing this lol.

The end of this was that the "modem" was dead, was a VERY frustrating situation actually lol.

Basically, if you want more details, this site was a site about a 5 hour drive from where I live, after troubleshooting as much as I could possibly do I ended up leaving thinking it was the modem.

Called Comcast later and told them to send a tech to the site and swap the modem, they said OK, claimed they did it, and it was still having major issues.

So at that point I assumed it was a bad firewall or pfSense install, so I drove back out there and swapped the firewalls from the Protecli I was using to a 4100, and still nothing lol.

Did a little more digging, and guess what, you'll never guess since Comcast has such great service, they NEVER replaced the modem despite the tech himself saying he did. Had another tech come out that day and they actually swapped it and it's been rock solid ever since.

Wasted like 15 hours worth of a trip thanks to Comcast lying lol.

Anyway, not sure if that helps your setup at all but that's my story.

darp

Had the same problem. It only happened with their modem, only when in bridge mode and only with pfsense (opensense too). As someone else figure out, toggling on/off block private networks was making the connection temporarily OK. I was able to figure our that the issue is due to their modem starting to drop packets if it doesn't get ARP request for the modem's IP in the preceding 5 min. Turns out that freebsd is confgured by default to cache ARP entries much longer than linux/windows. Something like 20 min vs 1 min. So the fix was fairly trivial for me. I added the net.link.ether.inet.max_age tunable set to 120 (seconds) and the modem was happy. No more packet loss.

michmoor

@darp Interesting. Bookmarking your fix in case i run into this in the future. But what made you think ARP cache was the problem?

ptyork

@darp thanks! That's some amazing sleuthing! Giving it a shot. Been solid for an hour or so. Fingers crossed.

Oddly, it seems Comcast isn't allow pinging their gateway, though. Having to ping my downstream DNS instead. Had to do this in router mode, obviously, but I thought the default gateway ping worked last time I tried bridge mode. Oh well. Not a actual concern...

darp

@michmoor said in Comcast Bridge Mode Sudden Packet Dropping (DHCP release renew solves it):

@darp Interesting. Bookmarking your fix in case i run into this in the future. But what made you think ARP cache was the problem?

It drove me mad that Linux just worked and pfSense didn't. So I had just pfSense with nothing else connected on the LAN besides me SSH-ing. Then ran tcpdump for quite a while in both cases. Dhcp seemed perfectly fine. I started a ping from the firewall (either to 10.0.0.1 or 8.8.8.8 ... don't recall) and saw the ICMP replies dissappearing. I tried the same with Linux and they kept coming. Tried toggling the private networks. Tried bringing the interface up and down. I was suspecting for a bit that Xfinity does something weird with the IP headers etc. Then at some point I saw that Linux does a lot more ARP requests than pfSense. That seemed weird so I started looking at the ARP cache and that eventually led me to figuring out that FreeBSD has a much bigger TTL than Linux/Windows. I lowered the TTL and had everything working ever since.

michmoor

@darp very nice....good troubleshooting.

ptyork

Just returning to say 1.5 months later, no issues. Bridge mode has been 100% solid. Thanks again, @darp.