ATT Uverse RG Bypass (0.2 BTC)



  • @aus Everything working well. However, when I reboot/lose power, I have to set my interfaces again and ngeth0 is NOT available at the console. I just set igb3 or whatever as wan, set my lan. No internet. I log into webgui and I can assign ngeth0 as WAN there and everything is good again. Not sure what's happening and read someone else was having similar issue. I did find if I reboot from console (Option "5") with 'Reroot' (R) option, everything is great. Also, if someone has compiled the 11.2 FreeBSD netgraph module, please upload and share link (would love to see if that's an issue at all). Thanks all!



  • @t41k2m3 I haven't been able to get to the bottom of some of the speed impact issues. It's hard to compare apples to apples in these benchmarks. What kind of hardware are you using for pfSense?

    The kernel module shouldn't make a difference, since there were no changes changes in ng_etf.c from 11.1 to 11.2. You're welcome to try though. Instructions on how to build are on the README. It's pretty straight forward to do in a FreeBSD VM.

    I made a PR for disabling Promiscuous Mode, which doesn't need to be enabled. In theory, if your NIC is trying to route/drop a lot of traffic that is not intended for your NIC (example: broadcast), then disabling promisc mode could help. Unfortunately, I think that's probably unlikely in a standard setup. You might give it a shot:

    https://github.com/aus/pfatt/pull/7

    @JJB Can you confirm that pfatt.sh is getting executed at <earlyshellcmd> ? if pfSense loads before pfatt.sh has created the ngeth0 netgraph interface, it will not know which interface to assign to WAN/LAN.



  • @t41k2m3 I haven't been able to get to the bottom of some of the speed impact issues. It's hard to compare apples to apples in these benchmarks. What kind of hardware are you using for pfSense?

    Thanks for your reply @aus . Hardware tested on is Netgate quad Atom C2558, 8 GB RAM, bare metal pfS install (not virtualized). Would seem hardware should not cause performance issues (unless some obscure NIC hardware - Intel I350 - or driver issue?).

    The kernel module shouldn't make a difference, since there were no changes changes in ng_etf.c from 11.1 to 11.2. You're welcome to try though. Instructions on how to build are on the README. It's pretty straight forward to do in a FreeBSD VM.
    I made a PR for disabling Promiscuous Mode, which doesn't need to be enabled. In theory, if your NIC is trying to route/drop a lot of traffic that is not intended for your NIC (example: broadcast), then disabling promisc mode could help. Unfortunately, I think that's probably unlikely in a standard setup. You might give it a shot:
    https://github.com/aus/pfatt/pull/7

    Tried new PR while at the same time swapping the kernel module to the 11.1 version (like yours, compiled in a VM). Not sure if it was the script changes (i.e. spoofing MAC to RG MAC, disabling promiscuous mode) or the kernel module change, but it did not connect at all (no IP via DHCP). Should probably try the module and script changes separately to isolate any potential issues. I assume the PR worked in your testing (able to connect, speed comparable to before)?



  • I just got at&t fiber installed yesterday but unfortunately, I'm experiencing speed degradation like @t41k2m3. I have the SG-2440 which runs an Atom C2358. Something tells me you can't do any bridging or use promisc mode without a massive hit on these barely enough boxes.



  • You should be able to confirm whether your hardware is limiting you via various performance tools. Check top CPU usage, systat -vmstat, etc.

    Once the bypass is established, you could also potentially simplify the netgraph to only the necessary nodes. I'm already doing this for the 5268AC issue. This script just removes the EAP bridge to solve Issue #5.

    However, you could potentially further strip down the netgraph to maintain only vlan tagging with ngeth0, vlan0, and $ONT_IF nodes after EAP authentication is complete. But if you loose your data link to the ONT or if it wants you to reauthenticate for any reason, you'll need to re-establish the full netgraph.



  • @aus said in ATT Uverse RG Bypass (0.2 BTC):

    However, you could potentially further strip down the netgraph to maintain only vlan tagging with ngeth0, vlan0, and $ONT_IF nodes after EAP authentication is complete. But if you loose your data link to the ONT or if it wants you to reauthenticate for any reason, you'll need to re-establish the full netgraph.

    Would you mind providing some code for taking the netgraph down to the bare minimum necessary?

    Wasn't sure if you meant waneapfiler, laneapfilter, o2m could be deleted/shut down and then connect ONT_IF directly to vlan0? Or o2m needs to be kept as is (with ONT_IF connected to vlan0 via o2m)?

    Regarding speed degradation, don't believe hardware is at fault in my test, however, still not sure what may be causing it. As someone else indicated on the GitHub repo, the non-promiscuous solution does not connect at all (no DHCP IP). As an aside, it appears that ng_etf.ko compiled on FreeBSD 11.1 and used with pfSense 2.4.4 may be causing the connection to drop and reconnect at random times (error before disconnect and new DHCP request seems to be arpresolve: can't allocate llinfo for <WAN ISP GW IP> on ngeth0). Issue seemed to go away with ng_etf.ko from FreeBSD 11.2.



  • @t41k2m3 I would love to run some tests with ng_etf.ko from 11.2 FreeBSD. Any chance you have a compiled ng_etf.ko from FreeBSD 11.2 which you could link to for download? Thanks either way.



  • @t41k2m3 I haven't had a chance to test this out yet, but you should be able to strip out your netgraph to the bare necessities by using the following commands AFTER pfatt.sh has established authentication and DHCP (ie you can ping external hosts). You can just run these manually from the shell.

    /usr/sbin/ngctl shutdown waneapfilter:
    /usr/sbin/ngctl shutdown laneapfilter:
    /usr/sbin/ngctl shutdown o2m: 
    /usr/sbin/ngctl connect vlan0: $ONT_IF: downstream lower
    

    Again, this is untested, but it might work. Don't forget to swap out $ONT_IF with your ONT interface name. You should have a brief interruption of network connectivity until the last connect command.

    Check the README on how to debug netgraph.

    @JJB I compiled a 11.2 ng_etf.ko and added it to the github repo. I really don't think it will make a difference, but you are welcome to give it a shot.



  • Hello all,

    So I just got the wonderful service off ATT GB fiber installed over the weekend, and I have great speeds (900+/900+) ... when using their piece of sh, I mean, amazing RG ... my problem comes into play when I use either a sg-1000, or my VM of pfsense 2.4.4 running on esxi 6.5 with GB all the way around, I see speeds of no more than 60 down and 150 up. Before I spat out technical jargon, I would like to know if anyone thinks the provided solution above may resolve my problem.

    Thank you for your time and patience, for I am nearly out.

    sincerly,
    Raging IT guy



  • Aha! I am not the only one.

    Have had ATT gigabit fiber service at my house for the last two years and been running PfSense for 5.

    Since the latest update to PfSense I have also has slow throughput around 50 down/120 up. My modem was always on bridge mode DMZ before with Around 940 up and down.

    I noticed if I take the ATT modem out of bridge/DMZ mode I get around 250 down / 300 up but still nowhere near where I was at before.

    No idea what is going on. When I have time I plan to bypass the modem.



  • I may have found an answer for the people experiencing slowness on the Pace modems, it's a firmware update released by att, version 11.*. I have them sending over an Arris box and will be able to test it tonight, if it arrives.



  • @sumguu out of curiosity, did you at any point reboot the RG? I'm thinking the reboot it may have updated.



  • I have rebooted the RG multiple times, including a factory reset when trying to figure out the slower speeds. My Pace RG is currently on 11.1.0.531418-att



  • @jjb said in ATT Uverse RG Bypass (0.2 BTC):

    @t41k2m3 I would love to run some tests with ng_etf.ko from 11.2 FreeBSD. Any chance you have a compiled ng_etf.ko from FreeBSD 11.2 which you could link to for download? Thanks either way.

    sorry for late reply, @aus may have uploaded this to his Github, but in case others want to download from the forum, see attached compiled on 11.2: 0_1544734306835_ng_etf.ko.gz



  • @sumguu said ... My Pace RG is currently on 11.1.0.531418-att

    Yeah yours updated to the throttling one it looks. My new to me one should be here today, I'll let you know how it goes.

    Was anyone experiencing slowness and then setup the RG bypass and it resolved there speed issues?



  • @aus et al - just a quick update, tried trimming down the graph (to just ngeh0, vlan0, ONT_IF, RG_IF) and am seeing same issues:

    1. Some speed degradation (though much more manageable relative to other examples here - 100-200 Mbps loss on Gig WAN);
    2. Connection drops 1-2 times per 24-hour period, which did not use to happen with quite the same frequency when connected via ATT RG in passthrough mode. When it drops, WAN/ngeth0 keeps public IP, however gateway becomes unreachable which triggers disconnect/reconnect bringing whole network down). Sometimes (not always) when disconnects happen, they are preceded by an error like "arpresolve: can't allocate llinfo for <GW IP> on ngeth0". Unable so far to pinpoint a root cause or possible fix (tried suggestions from @aus ). May try disabling gateway monitoring or action, though not sure that's ideal long term in case of more serious connectivity disruptions.

    Anyone else experience this?



  • I just got a new Xeon E3-1230v6 thanks to work and replaced my C2758 firewall last night. Right now I am using IP-Passthrough but will swap back to the method in this thread and see if the performance is still an issue.

    If I can't get good performance with a semi modern xeon then I am afraid it is just never going to happen.



  • @t41k2m3 said in ATT Uverse RG Bypass (0.2 BTC):

    @aus et al - just a quick update, tried trimming down the graph (to just ngeh0, vlan0, ONT_IF, RG_IF) and am seeing same issues:

    Some speed degradation (though much more manageable relative to other examples here - 100-200 Mbps loss on Gig WAN);
    Connection drops 1-2 times per 24-hour period, which did not use to happen with quite the same frequency when connected via ATT RG in passthrough mode. When it drops, WAN/ngeth0 keeps public IP, however gateway becomes unreachable which triggers disconnect/reconnect bringing whole network down). Sometimes (not always) when disconnects happen, they are preceded by an error like "arpresolve: can't allocate llinfo for <GW IP> on ngeth0". Unable so far to pinpoint a root cause or possible fix (tried suggestions from @aus ). May try disabling gateway monitoring or action, though not sure that's ideal long term in case of more serious connectivity disruptions.

    Anyone else experience this?

    That's too bad the netgraph alterations didn't work. At least, we have ruled out that most of the netgraph does not cause speed degradation for you. The "stripped" netgraph is pretty simple, leaving pretty much only ng_vlan (for VLAN0 tagging) and ng_eiface (for creating a NIC). Neither would be very CPU intensive, in theory.

    Regarding your connection drops, the best way to debug this is to catch the problem with tcpdumps running on $RG_IF and $ONT_IF. Maybe you could filter to just EAP and DHCP traffic to reduce load if you can't reproduce. It kind of sounds like something is confusing your DHCP lease on your WAN.

    @pyrodex said in ATT Uverse RG Bypass (0.2 BTC):

    I just got a new Xeon E3-1230v6 thanks to work and replaced my C2758 firewall last night. Right now I am using IP-Passthrough but will swap back to the method in this thread and see if the performance is still an issue.
    If I can't get good performance with a semi modern xeon then I am afraid it is just never going to happen.

    Interested to see how the Xeon shakes out for you.



  • @aus looks like I was able to pull nearly full line speed with the bridge on a E3-1230V6, I saw ~112+ MB/s when pulling a public linux distro torrent.

                   /0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
         Load Average   ||
    
          Interface           Traffic               Peak                Total
             ngeth0  in     30.936 KB/s        114.354 MB/s           27.332 GB
                     out   467.484 KB/s          1.479 MB/s          979.165 MB
    
               igb3  in      0.000 KB/s          0.082 KB/s           14.170 KB
                     out     0.000 KB/s          0.000 KB/s            4.050 KB
    
               igb0  in     33.320 KB/s        114.846 MB/s           27.479 GB
                     out   472.396 KB/s          1.619 MB/s          999.016 MB
    


  • Thanks for the update! Glad you were able sort the speed issues out.



  • I wanted to come here and post a giant "Thank You" for all the work aus and others helping him put forth.

    I was finally able to get my SuperMicro C2758 pfsense box working directly with the fiber ONT.

    I have 4 Gig ports configured as follows:
    igb0 - AT&T Fiber ONT
    igb1 - L3 Switch
    igb2 - UVERSE DVR - VIP2250
    igb3 - RG - BGW210

    One thing I've noticed, is the RG only ever has a single GREEN LED lit. I've powered cycled the RG and the fiber ONT and everything still worked, so.... //shrug//

    My throughput is fantastic (I have Gig service) and my latency dropped ever so slightly as well. Now, my task is getting the DVR to work. Coincidentally, my old unit died, so I just received my replacement.
    One thing I noticed, is the UVERSE LAN needs the AT&T DNS Servers in order to work. I can't filter them through Cloudfare, or other.
    I had IGMP Proxy setup just fine with the old box, but now it doesn't seem to be working. Any thoughts?


  • LAYER 8 Netgate

    @misterbaz said in ATT Uverse RG Bypass (0.2 BTC):

    I had IGMP Proxy setup just fine with the old box, but now it doesn't seem to be working. Any thoughts?

    You should probably start another thread.



  • @misterbaz said in ATT Uverse RG Bypass (0.2 BTC):

    I wanted to come here and post a giant "Thank You" for all the work aus and others helping him put forth.
    Glad to help out!

    One thing I've noticed, is the RG only ever has a single GREEN LED lit. I've powered cycled the RG and the fiber ONT and everything still worked, so.... //shrug//

    This is normal and expected. The RG never reaches full green status because it is expecting to negotiate a DHCP lease. However, netgraph drops that traffic because pfSense is handling the DHCP. You can actually keep the RG disconnected after the 802.1X EAP-TLS authentication completes. However, if your igb0 looses its link (due to power outage, unplug, reboot, or whatever), you will loose connectivity until the RG is reconnected and can authenticate you.

    One thing I noticed, is the UVERSE LAN needs the AT&T DNS Servers in order to work. I can't filter them through Cloudfare, or other.

    This might be true for set top boxes or DVRs, but your entire LAN does not need to use AT&T DNS servers.

    I had IGMP Proxy setup just fine with the old box, but now it doesn't seem to be working. Any thoughts?

    There's been a few threads about configuring the IGMP proxy for AT&T. Basically, it involved adding some of AT&Ts IP ranges. I had it working a while ago, but no longer have TV service to test. You might continue the conversation here:

    https://github.com/aus/pfatt/issues/3

    Definitely accepting PRs if you figure it out.



  • @aus said in ATT Uverse RG Bypass (0.2 BTC):

    @misterbaz said in ATT Uverse RG Bypass (0.2 BTC):

    I wanted to come here and post a giant "Thank You" for all the work aus and others helping him put forth.
    Glad to help out!

    One thing I've noticed, is the RG only ever has a single GREEN LED lit. I've powered cycled the RG and the fiber ONT and everything still worked, so.... //shrug//

    This is normal and expected. The RG never reaches full green status because it is expecting to negotiate a DHCP lease. However, netgraph drops that traffic because pfSense is handling the DHCP. You can actually keep the RG disconnected after the 802.1X EAP-TLS authentication completes. However, if your igb0 looses its link (due to power outage, unplug, reboot, or whatever), you will loose connectivity until the RG is reconnected and can authenticate you.

    One thing I noticed, is the UVERSE LAN needs the AT&T DNS Servers in order to work. I can't filter them through Cloudfare, or other.

    This might be true for set top boxes or DVRs, but your entire LAN does not need to use AT&T DNS servers.

    Correct. The rest of my LAN has Cloudfare DNS servers assigned. The UVERSE DVR is the only thing that needed to see the AT&T DNS Servers.

    I had IGMP Proxy setup just fine with the old box, but now it doesn't seem to be working. Any thoughts?

    There's been a few threads about configuring the IGMP proxy for AT&T. Basically, it involved adding some of AT&Ts IP ranges. I had it working a while ago, but no longer have TV service to test. You might continue the conversation here:

    https://github.com/aus/pfatt/issues/3

    Definitely accepting PRs if you figure it out.

    I figured it out, sort of. I had been trying to nail down every multicast server I could see through pfTop, but it was still hanging up on a lot of channels. So, I instead made a blanket 0.0.0.0/1 (pfSense won't let you use /0) statement in the Upstream setting and every channel came through. This might seem terrible, but remember my UVERSE DVR is on its own separate LAN independent from my normal LAN. Also, this still won't work until you setup an allow rule through your firewall. I believe I only had to setup an alow rule to my UVERSE LAN for 224.0.0.0/8. There might have also been an allow rule for 239.0.0.0/8. I'll have to check it out when I get back home.