runaway delay average and std. dev. on WAN

stephenw10

I mean something upstream of the AT&T router/gateway. Those usually only hand out private IPs themselves when they can't connect to the upstream server.

papaMURKS

@stephenw10 thanks, sorry do you consider upstream in the direction of the ONT or in the direction of my pfsense firewall

stephenw10

Yes sorry in the direction of the ONT.

papaMURKS

@stephenw10 below is my gateway monitoring and i'm definitely experiencing client-side performance issues as of today (random stuff like Amazon loading, Twitch loading, Youtube thumbnails delayed load, etc.)

pings to google and cloudflare as well as facebook, google, twitter all idle around 20+ms (higher than normal) but often spike to 60+ ms.

Going to pull the ethernet from 3100 -> RG, monitor, and update the thread.

papaMURKS

@stephenw10

pulling ethernet from:

3100 -> RG: no effect
RG -> ONT: no effect

pulling power from ONT: no effect
restart RG via the web UI: appears to reset the issue

also definitely had gateway monitoring alarms leading up to this morning.

stephenw10

Hmm, well that definitely seems like an issue in the RG then.

papaMURKS

@stephenw10 seems that way, however i replaced the RG earlier this year and that did not solve the issue.

stephenw10

Software issue in the RG firmware then maybe.

Just to confirm you said rebooting the 3100 made no difference? Only rebooting the RG fixes the issue?

papaMURKS

@stephenw10 i spoke too soon.

restarting the RG did not reset the issue this time. but...

I previously set the system tunable net.isr.dispatch to 'deferred' and removed that 40min ago. it appears that this change may have improved the issue? i'll monitor and report back...

papaMURKS

An update for anyone who may be experiencing this issue.

This issue is caused by ATT's RG firmware. The latency spikes and jitter are resolved on the BGW320-505 as of firmware 6.30.5.

This issue was somewhat widely discussed at /r/ATTFiber. Shame on ATT for taking 8+ months to release a firmware which fixed it. And I was only able to get the firmware update by working with a redditor who had a high-level engineering contact at ATT, who was able to MANUALLY push the firmware update to my device. Who knows when it would have rolled out to me...

Thank you to @stephenw10 for the help along the way.