Strange Packet Duplication On PPPoe WAN
-
Hey All --
I'm experiencing a weird packet duplication issue when sending data on the WAN1 Interface. (Century Link, PPPoe)
I first detected this issue because I am receiving DUP replies when pinging internet bound IP addresses, however after doing some tcpdumps it appears to be happening on all traffic.
This is only happening to the the ingress packets, not egress packets. This has the effect of halving my useful TCP download speed (~450mbps instead of ~900) -- because the duplicate TCP packets are discarded by the TCP algo, but they still occupy bandwidth on the 1000gbps ethernet line coming from the Century Link ONT. My upload speed is mostly unchanged, because the duplicate TCP ACKs don't occupy a meaningful amount of bandwidth.
This does not happen on VLAN to VLAN communication, it only happens on NAT with WAN1. (Century Link, PPPoe)
This also does not happen on my backup WAN2 interface (Cable Modem) -- So this is clearly something with the WAN1 (Century Link, PPPoe) interface, and not pfsense NAT in general.
I setup port mirroring on the switch so I could watch the raw traffic coming from the Century Link ONT, and these packets are clearly being duplicated on the ONT side. In the case of ICMP packets, The port mirror + Wireshark capture shows only a single ICMP packet going out, and 2 packets coming back. The duplicated packet is identical in every way. Using a server hosted on the internet, I verified the external server is only receiving a single ICMP packet and responding with a single ICMP packet.
Strangely -- if I reboot the ONT, everything works great for about ~15 minutes. No DUP packets, and full ingress bandwidth during speed tests. (~900mbps) At some point (10-20 mins) , the duplication starts. Disabling and restarting the PPPoe connection on pfsense also has this effect -- everything works great for about the first 15 mins.
My next troubleshooting step is to place the Century Link router between the ONT and PFsense, and see if I can reproduce the issue. My hope is that I can reproduce with the CL Router, because then it would point to a problem with the ONT and I can summon Century Link to fix/replace. My fear is I won't be able to reproduce with the CL router, in which case Century Link will tell me to go pound sand.
Does anyone have any ideas? Assuming I cannot reproduce the issue when using the CL Router -- what should I try next?
-
I was finally able to get Century Link to come on-site. As it turns out, the PON tap is doing this to my entire neighborhood -- so thankfully this has nothing to do with pfsense. Though, Century Link has no idea what's happening, so I'm not sure If I should be relieved?
The NSA wiretap is probably just malfunctioning and instead of sending copies of our packets back to Fort Meade, they are being sent back down the line to the ONT. Nothing to see here.