ATT Uverse RG Bypass (0.2 BTC)



  • I wanted to come here and post a giant "Thank You" for all the work aus and others helping him put forth.

    I was finally able to get my SuperMicro C2758 pfsense box working directly with the fiber ONT.

    I have 4 Gig ports configured as follows:
    igb0 - AT&T Fiber ONT
    igb1 - L3 Switch
    igb2 - UVERSE DVR - VIP2250
    igb3 - RG - BGW210

    One thing I've noticed, is the RG only ever has a single GREEN LED lit. I've powered cycled the RG and the fiber ONT and everything still worked, so.... //shrug//

    My throughput is fantastic (I have Gig service) and my latency dropped ever so slightly as well. Now, my task is getting the DVR to work. Coincidentally, my old unit died, so I just received my replacement.
    One thing I noticed, is the UVERSE LAN needs the AT&T DNS Servers in order to work. I can't filter them through Cloudfare, or other.
    I had IGMP Proxy setup just fine with the old box, but now it doesn't seem to be working. Any thoughts?


  • LAYER 8 Netgate

    @misterbaz said in ATT Uverse RG Bypass (0.2 BTC):

    I had IGMP Proxy setup just fine with the old box, but now it doesn't seem to be working. Any thoughts?

    You should probably start another thread.



  • @misterbaz said in ATT Uverse RG Bypass (0.2 BTC):

    I wanted to come here and post a giant "Thank You" for all the work aus and others helping him put forth.
    Glad to help out!

    One thing I've noticed, is the RG only ever has a single GREEN LED lit. I've powered cycled the RG and the fiber ONT and everything still worked, so.... //shrug//

    This is normal and expected. The RG never reaches full green status because it is expecting to negotiate a DHCP lease. However, netgraph drops that traffic because pfSense is handling the DHCP. You can actually keep the RG disconnected after the 802.1X EAP-TLS authentication completes. However, if your igb0 looses its link (due to power outage, unplug, reboot, or whatever), you will loose connectivity until the RG is reconnected and can authenticate you.

    One thing I noticed, is the UVERSE LAN needs the AT&T DNS Servers in order to work. I can't filter them through Cloudfare, or other.

    This might be true for set top boxes or DVRs, but your entire LAN does not need to use AT&T DNS servers.

    I had IGMP Proxy setup just fine with the old box, but now it doesn't seem to be working. Any thoughts?

    There's been a few threads about configuring the IGMP proxy for AT&T. Basically, it involved adding some of AT&Ts IP ranges. I had it working a while ago, but no longer have TV service to test. You might continue the conversation here:

    https://github.com/aus/pfatt/issues/3

    Definitely accepting PRs if you figure it out.



  • @aus said in ATT Uverse RG Bypass (0.2 BTC):

    @misterbaz said in ATT Uverse RG Bypass (0.2 BTC):

    I wanted to come here and post a giant "Thank You" for all the work aus and others helping him put forth.
    Glad to help out!

    One thing I've noticed, is the RG only ever has a single GREEN LED lit. I've powered cycled the RG and the fiber ONT and everything still worked, so.... //shrug//

    This is normal and expected. The RG never reaches full green status because it is expecting to negotiate a DHCP lease. However, netgraph drops that traffic because pfSense is handling the DHCP. You can actually keep the RG disconnected after the 802.1X EAP-TLS authentication completes. However, if your igb0 looses its link (due to power outage, unplug, reboot, or whatever), you will loose connectivity until the RG is reconnected and can authenticate you.

    One thing I noticed, is the UVERSE LAN needs the AT&T DNS Servers in order to work. I can't filter them through Cloudfare, or other.

    This might be true for set top boxes or DVRs, but your entire LAN does not need to use AT&T DNS servers.

    Correct. The rest of my LAN has Cloudfare DNS servers assigned. The UVERSE DVR is the only thing that needed to see the AT&T DNS Servers.

    I had IGMP Proxy setup just fine with the old box, but now it doesn't seem to be working. Any thoughts?

    There's been a few threads about configuring the IGMP proxy for AT&T. Basically, it involved adding some of AT&Ts IP ranges. I had it working a while ago, but no longer have TV service to test. You might continue the conversation here:

    https://github.com/aus/pfatt/issues/3

    Definitely accepting PRs if you figure it out.

    I figured it out, sort of. I had been trying to nail down every multicast server I could see through pfTop, but it was still hanging up on a lot of channels. So, I instead made a blanket 0.0.0.0/1 (pfSense won't let you use /0) statement in the Upstream setting and every channel came through. This might seem terrible, but remember my UVERSE DVR is on its own separate LAN independent from my normal LAN. Also, this still won't work until you setup an allow rule through your firewall. I believe I only had to setup an alow rule to my UVERSE LAN for 224.0.0.0/8. There might have also been an allow rule for 239.0.0.0/8. I'll have to check it out when I get back home.



  • Just wanted to chime in and say that this worked great for me. I also have some static IPs from ATT (10$/mo for 5 extra) and I was able to utilize them no issues without the gateway.. Not sure if it was mentioned how to before.

    I did so by creating an another lan interface tagged to vlan 99 dubbed Public Vlan. I assigned the "Gateway IP" given to me by ATT to this interface. after some proper firewall rules to allow traffic from outside to this network and disabling NAT on those addresses. I can confirm the public ip subnet is able to get out and traffic does return as desired. If you need more information just PM me





  • Thank you! That’s very generous of you and much appreciated. I’m glad this solution worked for you. Cheers!



  • Thank you @aus. I'm actually using the supplicant version and it works like a charm!!! Great work. Now I have to figure your static ips, @Dade, I looking at you!



  • @Dade I figured out the static IP, I just setup 1:1 nat, it was pretty straightforward.



  • I set this up a month ago and worked great for 2 week so. Then for the past 2 weeks my connection will drop randomly every day or few days. My logs are linked below. But another question I have is the link to the gateway suppose to work also? When I connect to it it says there is no link and to contact ATT but my PFsense internet works.

    Logs



  • @Makaveli6103 said in ATT Uverse RG Bypass (0.2 BTC):

    I set this up a month ago and worked great for 2 week so. Then for the past 2 weeks my connection will drop randomly every day or few days. My logs are linked below. But another question I have is the link to the gateway suppose to work also? When I connect to it it says there is no link and to contact ATT but my PFsense internet works.

    Logs

    I believe @aus stated that (the gateway not having link) is expected since all traffic from the gateway other than authentication is blocked.



  • Has anyone here using this bypass method noticed the WAN lease time obtained from ATT DHCP is only 1 hour long? Apparently people using other bypass methods have noticed it as well (see here) It only seems to happen when bypassing the gateway. The lease obtained when using ip-passthrough mode is much longer. I can see that in /var/db/dhclient.leases.ngeth0 the leases are 3600 seconds and going a grep "renewal in" /var/log/dhcpd.log shows the lease renewing every 1800 seconds which is the 1/2way point of a 3600 second lease. I'm curious if anyone can come up with a way to increase that lease time. I tried using a "send" dhcp-lease-time option to increase it, but it didn't help; the lease obtained was still 3600 seconds.



  • @gfeiner ok thanks. But any idea why my connection drops?



  • @Makaveli6103 said in ATT Uverse RG Bypass (0.2 BTC):

    @gfeiner ok thanks. But any idea why my connection drops?

    What is connected to igb2? The ONT or the gateway? Your logs are showing link going up and down on that port. Bad cable or faulty device connected to that port.



  • @gfeiner igb2 is the gateway. I will change the cable. I did also turn of gateway monitoring to see if that does anything.



  • @gfeiner I think I know what was wrong. I have the Pace 5268AC gateway and I did add the script to help with the EAP-Logoff issue. But when i added the script to the /rc.d folder I forgot to add .sh. I added it and now see it running in the logs. Hopefully this was the issue.



  • @Makaveli6103 said in ATT Uverse RG Bypass (0.2 BTC):

    @gfeiner I think I know what was wrong. I have the Pace 5268AC gateway and I did add the script to help with the EAP-Logoff issue. But when i added the script to the /rc.d folder I forgot to add .sh. I added it and now see it running in the logs. Hopefully this was the issue.

    Good to know.



  • @gfeiner
    I just got the ATT 1g service, internet only, no TV or VOIP. The RG is BGW210. I changed it's IP addr to 192.168.100.1 because as delivered it was the same as my pfSense box. Passthrough gives me a 5 minute lease although the setup screen is has a different lease time.

    I have an SG-2440 and behind that an unmanaged 1g switch. My speedtests run around 550 Mbps, the RG Diagnostic menu has a speed test built in which shows that it is doing ~950Mbps.

    I realize my setup is double NAT, I am reading here to find out if I can get rid of the double NAT and if that will increase my throughput.

    I'm still seeking more info on the pfatt patch.



  • @JonH did you read the instructions on the GitHub? There is a little learninf curve but isn't too hard.


  • LAYER 8 Netgate

    Just a quick note that the etf kernel module is now available as a command-line-installable package from the Netgate repos.

    [2.4.4-RELEASE][root@pfSense]/root: pkg search etf
    ng_etf-kmod-0.1                ng_etf kernel module
    [2.4.4-RELEASE][root@pfSense]/root: pkg install ng_etf-kmod
    Updating pfSense-core repository catalogue...
    pfSense-core repository is up to date.
    Updating pfSense repository catalogue...
    pfSense repository is up to date.
    All repositories are up to date.
    The following 1 package(s) will be affected (of 0 checked):
    
    New packages to be INSTALLED:
    	ng_etf-kmod: 0.1 [pfSense]
    
    Number of packages to be installed: 1
    
    3 KiB to be downloaded.
    
    Proceed with this action? [y/N]:
    

    No need to scp it from another FreeBSD node and it should track updates by FreeBSD.



  • @Derelict said in ATT Uverse RG Bypass (0.2 BTC):

    etf kernel module is now available

    Nice. Thanks for this info



  • @JonH I've installed pfatt 2 days ago, running w/o problems except my speed tests are still ~550 (~950 if wired directly as per AT&T). I'm not running Snort or Suricata. My cpu generally runs < 15%.

    pfatt.sh contains (in addition to RG MAC addr:
    ONT_IF='igb0'
    RG_IF='igb3'

    /usr/sbin/ngctl list
    There are 13 total nodes:
    Name: igb0 Type: ether ID: 00000001 Num hooks: 1
    Name: <unnamed> Type: socket ID: 00000007 Num hooks: 0
    Name: <unnamed> Type: socket ID: 0000006a Num hooks: 0
    Name: <unnamed> Type: socket ID: 0000006b Num hooks: 0
    Name: <unnamed> Type: socket ID: 0000006c Num hooks: 0
    Name: <unnamed> Type: socket ID: 0000006d Num hooks: 0
    Name: o2m Type: one2many ID: 0000000d Num hooks: 3
    Name: vlan0 Type: vlan ID: 00000010 Num hooks: 2
    Name: ngctl25207 Type: socket ID: 000000d3 Num hooks: 0
    Name: ngeth0 Type: eiface ID: 00000013 Num hooks: 1
    Name: waneapfilter Type: etf ID: 00000017 Num hooks: 2
    Name: laneapfilter Type: etf ID: 0000001b Num hooks: 1
    Name: igb3 Type: ether ID: 0000005d Num hooks: 0

    One question is my interface assignments in the pfSense web configurator: The pfatt readme says "pfSense will detect new interfaces on bootup. Follow the prompts on the console to configure ngeth0 as your pfSense WAN."
    In my case I didn't get any prompts so I read this to mean I should have ngeth0 as my WAN interface. Thus, I changed the WAN from igb0 to ngeth0 (and spoofing RG MAC). This leaves igb0 as "available".

    Is this correct or am I misreading the readme? Should WAN remain igb0?

    There was one comment earlier in this thread to make sure pfatt was being executed at <earlyshellcmd>. How would I determine that? And the etf filters have less hooks than an example posted earlier in this thread. Is that important?


  • LAYER 8 Netgate

    I would not edit the configuration to add the shell command. I would use the Shell Command package. There is an option there to select early.



  • @Derelict said in ATT Uverse RG Bypass (0.2 BTC):

    I would use the Shell Command package.

    Thank you. I was not aware of that package.
    I'll give it a shot.



  • re: which interface, your WAN should be ‘ngeth0’. If pfSense doesn’t prompt you to configure, you should manually set it.

    re: performance, early shell cmd won’t improve that. Unfortunately, Netgraph configured as such does add a bit of CPU overhead at high network utilization. If your total CPU does not exceed ~15% under high network utilization, I would double check your single core performance. It may be maxed on a single core.

    I’ve tested pfatt on a couple different boxes. Some performed better than others. My current CPU can mostly saturate (900+) my 1000/1000 plan:

    AMD GX-420CA SOC
    Current: 800 MHz, Max: 2000 MHz
    4 CPUs: 1 package(s) x 4 core(s)
    AES-NI CPU Crypto: Yes (active)

    Supplicant mode has a little less overhead since the Netgraph is simpler. You might get more out of your hardware with that.



  • @aus: Thanks for feedback.

    ngeth0 is on WAN. In the Interface Assignments menu that leaves igb0 down.
    My CPU at ~15% is just average network usage. I don't run web servers. I have minimal streaming.
    According to top, running in the shell, my largest cpu load is ntopng, I have disabled that and there is no noticeable improvement.

    pfSense is running on a SG-2440 appliance (pre-Netgate appliance). It has 2 Atom C2358 1.7 GHz cpu's. I don't know how to check the individual cpu performance.
    For crypto I think my setting is default, I don't recall setting it. It is set to BSD cryptodev but I will try no crypto to see if there is a noticeable difference.
    I'm using a dumb switch.
    I'm have a BGW210-700 & not using the AT&T wifi.
    Is Supplicant Mode a function of compiling the etf.ko? If not, how do I remove it? I'm using Derelict's Build.

    For kicks, I unplugged my LAN cable (igb1) and plugged a linux box directly into it (leaving a single NAS on igb2 & the RG on igb3). Same ~500 speedtest.net results. That linux box plugged into AT&T default setup is ~800-900.

    You are at 4 cores, I'm at 2 cores. Maybe my throughput is the best I can expect with my SG-2440?



  • FYI. I'm doing this bypass on my netgate SG5100 and I can get in the 900-940Mb range with ATT UVERSE gigabit plan. So maybe it is your CPU.



  • @gfeiner The pfSense CPU? I'm starting to think that.


  • LAYER 8 Netgate

    Be sure powerd is enabled and set to Hiadaptive or Maximum in System > Advanced, Miscellaneous


  • LAYER 8 Netgate

    @JonH said in ATT Uverse RG Bypass (0.2 BTC):

    I'm using Derelict's Build.

    To be clear, it is not my build I'm just the messenger. The main developers at Netgate built it.



  • @Derelict Thank You, it (powerd)was previously set that way.
    I'm going to disable pfBlockerNG to see if that is making a substantial hit on throughput.



  • @gfeiner said in ATT Uverse RG Bypass (0.2 BTC):

    So maybe it is your CPU

    Link below show results of shell command 'systat load' while doing speed test. If I understand the output correctly it looks like my CPU are doing ok.
    ![ScreenShot](<a href="https://imgur.com/oW4yqgC"><img src="https://i.imgur.com/oW4yqgC.png" title="source: imgur.com" /></a>)

    I also did a speed test with pfBlockerNG disabled and there was negligible improvement.



  • Has anyone tried using this netgraph method along with the certificate extraction from gateway method? I have the wpa_supplicant method working, but still have to use the 5port netgear switch in the middle of my ONT and PFsense WAN because of VLAN0. Wondering how i could use netgraph to deal with VLAN 0 issue.



  • So I got things working by not using any netgraph scripts on my ESXi 6.7u2 virtualized pfSense instance. If you follow the instructions below, you should get things working.

    1. Set up a new VSWITCH, port group with VLAN(0) and uplink on a dedicated network uplink (Allow mac address spoofing and the other two just incase)
    2. Connect the ONT to this uplink
    3. Create a new e1000e interface that resides in the port group from 1) in pFsense (em0 for me). I tried vmxnet3 and didn't seem to work
    4. I just took the portion of the script below to start wpa_supplicant. Find all em0 below and change with your adapter.
      /usr/bin/logger -st "pfatt" "starting wpa_supplicant..."
    
      WPA_PARAMS="\
        set eapol_version 2,\
        set fast_reauth 1,\
        ap_scan 0,\
        add_network,\
        set_network 0 ca_cert \\\"/conf/pfatt/wpa/ca.pem\\\",\
        set_network 0 client_cert \\\"/conf/pfatt/wpa/client.pem\\\",\
        set_network 0 eap TLS,\
        set_network 0 eapol_flags 0,\
        set_network 0 identity \\\"$EAP_SUPPLICANT_IDENTITY\\\",\
        set_network 0 key_mgmt IEEE8021X,\
        set_network 0 phase1 \\\"allow_canned_success=1\\\",\
        set_network 0 private_key \\\"/conf/pfatt/wpa/private.pem\\\",\
        enable_network 0\
      "
    
      WPA_DAEMON_CMD="/usr/sbin/wpa_supplicant -Dwired -iem0 -B -C /var/run/wpa_supplicant"
    
      # kill any existing wpa_supplicant process
      PID=$(pgrep -f "wpa_supplicant.*em0")
      if [ ${PID} > 0 ];
      then
        /usr/bin/logger -st "pfatt" "terminating existing wpa_supplicant on PID ${PID}..."
        RES=$(kill ${PID})
      fi
    
      # start wpa_supplicant daemon
      RES=$(${WPA_DAEMON_CMD})
      PID=$(pgrep -f "wpa_supplicant.*em0")
      /usr/bin/logger -st "pfatt" "wpa_supplicant running on PID ${PID}..."
    
      # Set WPA configuration parameters.
      /usr/bin/logger -st "pfatt" "setting wpa_supplicant network configuration..."
      IFS=","
      for STR in ${WPA_PARAMS};
      do
        STR="$(echo -e "${STR}" | sed -e 's/^[[:space:]]*//')"
        RES=$(eval wpa_cli ${STR})
      done
    
      # wait until wpa_cli has authenticated.
      WPA_STATUS_CMD="wpa_cli status | grep 'suppPortStatus' | cut -d= -f2"
    
      /usr/bin/logger -st "pfatt" "waiting EAP for authorization..."
    
      # TODO: blocking for bootup
      while true;
      do
        WPA_STATUS=$(eval ${WPA_STATUS_CMD})
        if [ X${WPA_STATUS} = X"Authorized" ];
        then
          /usr/bin/logger -st "pfatt" "EAP authorization completed..."
          break
        else
          sleep 1
        fi
      done
      /usr/bin/logger -st "pfatt" "em0 should now be available to configure as your WAN..."
      /usr/bin/logger -st "pfatt" "done!"
    else
      /usr/bin/logger -st "pfatt" "error: unknown EAP_MODE. '$EAP_MODE' is not valid. exiting..."
      exit 1
    fi
    
    1. Set em0 as your wan, DHCP, mac spoof (RG of cert MAC address)
    2. Voila!

    I think this works because ESXI will strip and add VLAN0 tags on the port group so no need netgraph business. I don't think this would work by plugging into my Cisco SG500x because I can't define VLAN0 and so the switch would just drop everything. Too bad! Let me know if anyone has any ideas to improve on things.



  • Would @GoldServe (or others) know how to work a similar scenario (without netgraph) with a physical switch (e.g. cisco), instead of an ESXi virtual switch (ONT --> Switch --> pfSense WAN)? Switch should do VLAN0 tagging via dot1p. Is that possible and what (affordable) switches could do that?



  • @bulldog5 sounds like you are looking for the supplicant branch:
    https://github.com/aus/pfatt/tree/supplicant
    Edit pfatt.sh to use EAP_MODE="supplicant" - that should create a simpler netgraph and call wpa_supplicant.
    What netgear switch are you using and does it do outgoing VLAN0 tagging?



  • @t41k2m3
    I'm using a GS105Ev2 switch currently. This switch handles the VLAN0 fine, which is why the esxi method also works. But i'm running my pfsense on baremetal, so that option doesn't really apply to me. It would be "nice" to eliminate the GS105E in the middle of my ONT and pfsense WAN. I'll give the link you sent a shot and see how it goes. Thanks



  • @bulldog5 curious about the config for both pfS and GS105ev2 if you don't mind. Could not get it to work with latest firmware on GS108ev3 and bare metal pfS (should be pretty close to your GS105ev2 setup) - EAP would not go through and no DHCP or anything after. Is your pfS going out on WAN NIC or do you use VLAN (if so what #/priority)? Any special settings on the switch - not much seemed to be configurable beyond 802.1q tagging and (802.1p) CoS of 0?



  • @t41k2m3

    GS105Ev2 settings

    VLAN > 802.1Q TAB

    Basic 802.1Q VLAN Status:

    PORT 1 and 3 are both in VLAN ID 1. Those are my ONT and WAN ports.

    Make sure you're not using Port Based and have that Disabled.



  • @bulldog5 is it possible that pfS is doing the tagging (how is pfS setup?) or that no tagging is required at your location? It's a bit odd because it looks like the switch is just passing traffic through on native VLAN without tagging (802.1q or p).


Log in to reply