Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    ATT Uverse RG Bypass (0.2 BTC)

    Bounties
    80
    555
    1.2m
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      aus
      last edited by

      I just wanted to report back that I got this working, but probably not in the sense you were hoping for.

      I’m running pfSense in a VM on Proxmox (KVM/QEMU). For now, I’ve opted to let the hypervisor (Linux) do the EAP and VLAN work. (Same method basically) Here is my setup:

      Nothing too special configuration was required for my pfSense VM. Here is my config:

      
      balloon: 0
      bootdisk: ide0
      cores: 2
      cpu: host
      ide0: ssd0:vm-100-disk-1,size=32G
      memory: 512
      name: pfSense
      net0: virtio=XX:XX:XX:XX:XX:XX,bridge=vmbr0
      net1: virtio=XX:XX:XX:XX:XX:XX,bridge=vmbr1
      numa: 0
      ostype: other
      serial0: socket
      sockets: 1
      tablet: 0
      
      

      The net0 (LAN) interface bridges to vmbr0. vmbr0 bridges to physical eth0, which is connected to my switch.

      The net1 (WAN) interface bridges to vmbr1. vmbr1 bridges to vlan0. The vlan0 interface is configured off physical eth1, which is connected to the ONT. net1 MAC address also matches my ATT Gateway. Change it in your pfSense WAN interface setting.

      /etc/network/interfaces:

      
      # LAN / eth0                                                                                                                                                                                         
      # Connect to switch
      iface eth0 inet manual
      
      # ONT / eth1
      # Connect to ONT box outside
      iface eth1 inet manual
      
      # RG / eth2
      # Connect to ATT Gateway on ONT port
      iface eth2 inet manual
      
      # LAN Bridge / br0
      # Bridge main switch to pfSense
      # IP is Proxmox host
      auto vmbr0
      iface vmbr0 inet static
              address  192.168.1.2
              netmask  255.255.255.0
              gateway  192.168.1.1
              bridge_ports eth0
              bridge_stp off
              bridge_fd 0
      
      # VLAN0 Bridge / br1
      # Bridge vlan tagged WAN to pfSense
      auto vmbr1
      iface vmbr1 inet manual
              bridge_ports vlan0
              bridge_stp off
              bridge_fd 0
      
      # EAP Bridge / br2
      # Bridge ATT Gateway + ONT so EAP/802.1X auth can complete
      # group_fwd_mask makes sure 802.1X traffic is bridged
      auto vmbr2
      iface vmbr2 inet manual
              bridge_ports eth1 eth2
              bridge_stp off
              bridge_fd 0
              post-up echo 8 > /sys/class/net/vmbr2/bridge/group_fwd_mask
      
      

      Unfortunately, Proxmox conflicts with the vlan debian package, so you have to configure the vlan interface with the ip command instead of the interface file:

      
      ip link add link eth1 name vlan0 type vlan id 0
      
      

      And that’s pretty much it. I haven’t nailed down the timings yet from cold boot to online for a fully automated solution. For some reason, the EAP only takes under certain conditions. I have the best luck with the following:

      1. Cold boot hypervisor
      2. Wait for EAP to authenticate
      3. Start vlan0
      4. Start pfSense VM

      It’s not perfect right now and it will take some more experimenting. But, it feels good to be off their RG!

      I’d still be interested in a pure BSD solution though.

      1 Reply Last reply Reply Quote 0
      • R
        rajl
        last edited by

        Glad to see someone else exploring this.

        I haven't had a chance to test it yet (my wife and toddler will kill me if I cut the internet tinkering on the weekend – toddler cartoons are sacred).

        I hadn't thought of using an ng_tee node, but it sounds like a good idea.  Basically, you would connect the ng_tee left hook to the ng_eth downstream hook.  You would connect the ng_tee right hook to an ng_vlan (for example), which can filter the vlan 0 tagged traffic for you before passing it up the protocol stack.  The ng_etf could be connect to the left2right hook of the ng_tee node using the commands I wrote above to filter the eap traffic.

        A modified script would look something like this:

        
            ngctl mkpeer em0: tee lower left
            ngctl name em0:lower T
        
            #Connect vlan to virtual interface for vlan0 traffic, ignore all untagged traffic
            #by failing to connect the vlan to the ether upper hook (reserved for eap filtering)
        
            ngctl mkpeer T: vlan right downstream
            ngctl name T:right vlan
            ngctl mkpeer vlan: eiface vlan0 ether
            ngctl msg vlan0: 'addfilter { vlan=0 hook="vlan0" }'
        
            #Connect other hook of T node to ng_etf node for eap filtering/proxying
            #Leave "lan filter" the same because we only care about eap traffic
        
            ngctl mkpeer T: etf left2right downstream
            ngctl name T:left2right waneapfilter
            ngctl connect waneapfilter: em0: nomatch upper
        
            ngctl mkpeer em1: etf lower downstream
            ngctl name em1:lower laneapfilter
            ngctl connect laneapfilter: em1: nomatch upper
        
            ngctl connect waneapfilter: laneapfilter: eapout eapout
        
            ngctl msg waneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }'
            ngctl msg laneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }'
        
        

        This is an untested script and may throw errors or error message, but the underlying principles should work for any brave souls willing to try! ;)  Just make sure that you have the ng_etf, ng_vlan, ng_eth, ng_eiface and ng_tee modules loaded.

        1 Reply Last reply Reply Quote 0
        • R
          rajl
          last edited by

          Just occurred to me as an alternative that you could use a NetGraph multiplexer node instead of the ng_tee node.

          1 Reply Last reply Reply Quote 0
          • A
            aus
            last edited by

            I did a bit more testing, but no success just yet.  (I suspect I need to first try to get this to work on phsyical hardware. Currently trying using a pfSense VM and I'm not seeing packet carry over from Linux hypervisor to the pfSense VM)

            Regarding the ng_ether "bug", I did some digging on this. It turns out that this is not a bug necessarily. pfSense actually does a NGM_ETHER_DETACH against interfaces under some circumstanaces.

            https://github.com/pfsense/pfsense/blob/9a18ac7af8ae4a4fde8998c18cc7ba7802056477/src/etc/inc/interfaces.inc#L180

            I think this was for performance reasons back when netgraph had performance overhead.

            Anyways, you think you'd be able to just do a control message of  NGM_ETHER_ATTACH, but that doesn't exist in vanilla FreeBSD. Luckily, pfSense integrates some patches to enable  NGM_ETHER_ATTACH, but you have to call it from PHP.

            https://github.com/pfsense/FreeBSD-ports/blob/e178a5cf520e928efb3c7d896e3d9fcfb41ac7e5/devel/php56-pfSense-module/files/pfSense.c#L3094

            This will re-enable the interface as a node in netgraph:

            php -r 'pfSense_ngctl_attach(".", "em0");'

            Also, for ng_one2many (assuming that's what you mean by multiplexer) I don't think that will work. I initially looked at this too, but it distributes packets in a round-robin fashion so the many's would only see some packets.  At least, that's how I interpret the man page.

            1 Reply Last reply Reply Quote 0
            • R
              rajl
              last edited by

              ng_one2many is what I was referring to.  The man page states that it has several transmission modes, including round-robin and transmit-all.  The man page says that the round robin mode is the default, but my experience when playing with it is that transmit-all was the default.  In either case, you could easily set the transmission mode to transmit-all to ensure that you get the desired behavior.  So it will work and is simpler to work with than ng_tee.

              That's great research on the ng_ether issue.  It's been holding me up for awhile, forcing me to do my testing on other distributions (e.g., vanilla FreeBSD and OPNSense) and then curse when I couldn't get it working on PFSense.  I'll have to see if it works with my scripts on a VM or PFSense.  It should, but Murphy's law always strikes me down!

              1 Reply Last reply Reply Quote 0
              • A
                aus
                last edited by

                Doh! Good catch on the ng_one2many transmit-all algorithm. I was looking at an old man page from an earlier version of FreeBSD, which it didnt support transmit-all yet. That's what I get for googling the man pages, instead of reading them in terminal! May give this a shot later… I'll report back if I have any success.

                Cheers!

                1 Reply Last reply Reply Quote 0
                • A
                  aus
                  last edited by

                  It worked!! True U-verse bridge mode on pfSense!

                  
                  [2.4.2-RELEASE][root@pfsense.knox.lan]/root: ngctl list
                  There are 9 total nodes:
                    Name: T               Type: tee             ID: 00000021   Num hooks: 3
                    Name: ue0             Type: ether           ID: 00000003   Num hooks: 2
                    Name: vlan0           Type: vlan            ID: 00000024   Num hooks: 2
                    Name: <unnamed>Type: socket          ID: 00000006   Num hooks: 0
                    Name: ngctl96372      Type: socket          ID: 00000047   Num hooks: 0
                    Name: ngeth0          Type: eiface          ID: 00000027   Num hooks: 1
                    Name: waneapfilter    Type: etf             ID: 0000002a   Num hooks: 3
                    Name: laneapfilter    Type: etf             ID: 00000031   Num hooks: 3
                    Name: em0             Type: ether           ID: 00000019   Num hooks: 2
                  [2.4.2-RELEASE][root@pfsense.knox.lan]/root: ifconfig em0
                  em0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
                  	options=40098 <vlan_mtu,vlan_hwtagging,vlan_hwcsum,vlan_hwtso>ether xx:xx:xx:xx:xx:xx
                  	hwaddr xx:xx:xx:xx:xx:xx
                  	media: Ethernet autoselect (1000baseT <full-duplex>)
                  	status: active
                  [2.4.2-RELEASE][root@pfsense.knox.lan]/root: ifconfig ue0
                  ue0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
                  	options=8000b <rxcsum,txcsum,vlan_mtu,linkstate>ether xx:xx:xx:xx:xx:xx
                  	hwaddr xx:xx:xx:xx:xx:xx
                  	media: Ethernet autoselect (100baseTX <full-duplex>)
                  	status: active
                  [2.4.2-RELEASE][root@pfsense.knox.lan]/root: ifconfig ngeth0
                  ngeth0: flags=8a43 <up,broadcast,running,allmulti,simplex,multicast>metric 0 mtu 1500
                  	options=28 <vlan_mtu,jumbo_mtu>ether xx:xx:xx:xx:xx:xx
                  	inet xx.xx.xx.xx netmask 0xfffffc00 broadcast xx.xx.xx.xx
                  	media: Ethernet autoselect (1000baseT <full-duplex>)
                  	status: active</full-duplex></vlan_mtu,jumbo_mtu></up,broadcast,running,allmulti,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,linkstate></up,broadcast,running,simplex,multicast></full-duplex></vlan_mtu,vlan_hwtagging,vlan_hwcsum,vlan_hwtso></up,broadcast,running,simplex,multicast></unnamed> 
                  

                  For reference…

                  em0 is connected to my ONT.
                  em1 is connected to my LAN
                  ue0 is connected to my RG (via USB ethernet)
                  ngeth0 is the VLANed device which is configured as my WAN in pfSense

                  Commands to get it running (thanks for the help on ng_tee rajl!)  ...

                  
                  # copy and load ng_etf kernel module
                  
                  kldload /boot/kernel/ng_etf.ko
                  
                  #
                  # setup netgraph nodes
                  #
                  
                  # list out netgraph nodes
                  
                  ngctl list
                  
                  # pfSense for some reason detaches ether devices. reattach any missing devices.
                  
                  php -r 'pfSense_ngctl_attach(".", "em0");'
                  
                  # create tee node to split em0 traffic (one for EAP, one for VLAN0)
                  ngctl mkpeer em0: tee lower left # may get a warning
                  ngctl name em0:lower T
                  
                  # create vlan node + eiface
                  ngctl mkpeer T: vlan right downstream
                  ngctl name T:right vlan0
                  ngctl mkpeer vlan0: eiface vlan0 ether
                  ngctl msg vlan0: 'addfilter { vlan=0 hook="vlan0" }'
                  
                  # create etf and connect to em0 (ONT)
                  ngctl mkpeer T: etf left2right downstream
                  ngctl name T:left2right waneapfilter
                  ngctl connect waneapfilter: em0: nomatch upper
                  
                  # create etf and connect to em1 (RG) 
                  ngctl mkpeer ue0: etf lower downstream
                  ngctl name ue0:lower laneapfilter
                  ngctl connect laneapfilter: ue0: nomatch upper
                  
                  # define filters for EAP traffic
                  ngctl msg waneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }'
                  ngctl msg laneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }'
                  
                  # use filters to bridge EAP traffic
                  ngctl connect waneapfilter: laneapfilter: eapout eapout
                  
                  # change MAC address to match RG (also can be done in pfSense)
                  ifconfig ngeth0 ether xx:xx:xx:xx:xx:xx
                  
                  

                  There is still worked to be done though to make this perfect though…

                  1. Explore using ng_one2many to see if that simplifies the netgraph a bit
                  2. Automate /  Harden change so its persistant across reboots (rajl already documented this earlier)
                  3. Document!

                  And for what it's worth, I'm running this pfSense in a virtual machine via Proxmox (QEMU/KVM). I couldnt get the VLAN0 traffic to bridge across the interface into pfSense, so I ended up doing a PCI passthrough of the NIC device.

                  1 Reply Last reply Reply Quote 0
                  • R
                    rajl
                    last edited by

                    That's awesome!!!  My suspicion is that this would run on baremetal just fine (have to test though).  So let's say there's a 4th todo - test this to run on baremetal for those of use that don't virtualize! :-)  Hopefully, it won't take too much modification.

                    This should be pretty easy to automate so that it executes across reboots.  Just save your commands as a shell script (don't forget the #!/bin/sh at the beginning of the file) and follow the PFSense instructions for executing shell scripts at the end of the boot process.

                    https://doc.pfsense.org/index.php/Executing_commands_at_boot_time

                    I read somewhere that ATT will occassionally push firmware updates to the RG, which this setup may have problems with because the RG is being isolated from the ATT network.  But that's a bridge to cross when we get there.

                    1 Reply Last reply Reply Quote 0
                    • T
                      Tantamount
                      last edited by

                      Would it be possible to have ONT go to port A of a switch set to vlan 20
                      and have port B of that switch also on vlan 20 connect to the RG's ONT port?

                      Would a switch normally process/filter those 802.1x packets in such a setup?

                      My pfsense vm is in a different area of the house on a different switch and I'm curious if I'll be able to get this working.

                      Also, is there any practical benefit to doing this?  For instance, would it open outgoing tcp port 25 traffic?

                      1 Reply Last reply Reply Quote 0
                      • A
                        aus
                        last edited by

                        @Tantamount:

                        Would it be possible to have ONT go to port A of a switch set to vlan 20
                        and have port B of that switch also on vlan 20 connect to the RG's ONT port?

                        Would a switch normally process/filter those 802.1x packets in such a setup?

                        My pfsense vm is in a different area of the house on a different switch and I'm curious if I'll be able to get this working.

                        Also, is there any practical benefit to doing this?  For instance, would it open outgoing tcp port 25 traffic?

                        I don't know the answer to your question, but I suspect that won't work. The problem is that ONT traffic comes in on VLAN0 and needs to egress on VLAN0. I'm not sure your switch would tag VLAN0 <-> VLAN20 accordingly.

                        Also, I'm having some duplicate packets in my previous setup. Hoping one2many might solve that. More to come…

                        1 Reply Last reply Reply Quote 0
                        • R
                          rajl
                          last edited by

                          @Tantamount:

                          Would it be possible to have ONT go to port A of a switch set to vlan 20
                          and have port B of that switch also on vlan 20 connect to the RG's ONT port?

                          Would a switch normally process/filter those 802.1x packets in such a setup?

                          My pfsense vm is in a different area of the house on a different switch and I'm curious if I'll be able to get this working.

                          Also, is there any practical benefit to doing this?  For instance, would it open outgoing tcp port 25 traffic?

                          I'll try to answer your questions in detail.

                          First, the switch setup you're describing won't work because your switch would block the traffic for several reasons.  First, if the switch would drop the ethernet frames because ATT tags them as vlan0, but you're setting your ports for vlan20.  Second, your switch would probably drop all the authentication frames (802.1X) because most (but not all) switches are fully compliant with 802.1D, which requires that switches and bridges not forward 802.1X frames.  However, some switches are not standard compliant and will forward the frames anyway.

                          That said, you could always run a long cable from one of the house to the other to solve the problem.

                          Regarding your question about practical benefits, the main practical benefit is performance.  The RGs tend to have (1) a small state table with a limited number of entries and (2) middling (at best) performance ARM processors that start to choke under load when you start to do "real routing."  As an example, get a few good bit-torrents going on a 1-Gig connection and they try to browser the web.  Your performance will crawl because the RG's state table is too small to keep track of all of the connections and the RG's processor is unable to process all the connections at line-speed.  Bypassing the RG to use your own PFSense box solves both of these problems.

                          Some older (but still relevant) articles on why you would want to replace consumer grade routers with an x86 router (such as one using PFSense) are below:

                          https://arstechnica.com/gadgets/2016/01/numbers-dont-lie-its-time-to-build-your-own-router/
                          https://arstechnica.com/gadgets/2016/09/the-router-rumble-ars-diy-build-faces-better-tests-tougher-competition/

                          1 Reply Last reply Reply Quote 0
                          • T
                            Tantamount
                            last edited by

                            I was afraid you were going to say that – I tried it last night and was unsuccessful.

                            I've got one of those Netgate gs-2440's that I'll use instead of the vm.  I'll be able to put it right next to the RG and ONT.

                            1 Reply Last reply Reply Quote 0
                            • P
                              pyrodex
                              last edited by

                              Going to try this today with my 4 port SuperMicro physical and report back. Had to install FreeBSD 11.1 in a VM to get the kernel module since the link seems to be dead.

                              UPDATE:

                              No luck.

                              I use the crappy switch trick today and swap VLANs and my igb0 is the MAC of the RG. My igb0 (WAN) is connected into my bypass switch on a VLAN with the ONT and the RG is on another VLAN that gets flipped and flopped if the internet goes down.

                              I tried the script and connected igb0 straight into the ONT and igb3 to the RG removing my bypass switch out of line. I had no luck and the RG would attempt to AUTH the port on the ONT but never went past that.

                              Here is the script I used:

                              
                              #igb2 is connected to the ONT
                              #lagg0 is connected to the LAN
                              #igb3 is connected to the RG
                              #ngeth0 is the VLANed device which is configured as my WAN in pfSense
                              # copy and load ng_etf kernel module
                              
                              /sbin/kldload /boot/kernel/ng_etf.ko
                              
                              #
                              # setup netgraph nodes
                              #
                              
                              # list out netgraph nodes
                              
                              /usr/sbin/ngctl list
                              
                              # pfSense for some reason detaches ether devices. reattach any missing devices.
                              
                              php -r 'pfSense_ngctl_attach(".", "igb0");'
                              
                              # create tee node to split ONT traffic (one for EAP, one for VLAN0)
                              /usr/sbin/ngctl mkpeer igb0: tee lower left # may get a warning
                              /usr/sbin/ngctl name igb0:lower T
                              
                              # create vlan node + eiface
                              /usr/sbin/ngctl mkpeer T: vlan right downstream
                              /usr/sbin/ngctl name T:right vlan0
                              /usr/sbin/ngctl mkpeer vlan0: eiface vlan0 ether
                              /usr/sbin/ngctl msg vlan0: 'addfilter { vlan=0 hook="vlan0" }'
                              
                              # create etf and connect to ONT
                              /usr/sbin/ngctl mkpeer T: etf left2right downstream
                              /usr/sbin/ngctl name T:left2right waneapfilter
                              /usr/sbin/ngctl connect waneapfilter: igb0: nomatch upper
                              
                              # create etf and connect to RG
                              /usr/sbin/ngctl mkpeer igb3: etf lower downstream
                              /usr/sbin/ngctl name igb3:lower laneapfilter
                              /usr/sbin/ngctl connect laneapfilter: igb3: nomatch upper
                              
                              # define filters for EAP traffic
                              /usr/sbin/ngctl msg waneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }'
                              /usr/sbin/ngctl msg laneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }'
                              
                              # use filters to bridge EAP traffic
                              /usr/sbin/ngctl connect waneapfilter: laneapfilter: eapout eapout
                              
                              # change MAC address to match RG (also can be done in pfSense)
                              ifconfig ngeth0 ether <mac></mac> 
                              
                              1 Reply Last reply Reply Quote 0
                              • A
                                aus
                                last edited by

                                It's hard to say what your exact issue is without more information.

                                However, the first thing I would do is run some tcpdumps to see what's going on.

                                You should run tcpdumps on the ONT interface and the RG interface:

                                tcpdump -ei em0
                                tcpdump -ei em1
                                

                                From the RG interface, you should see some EAPOL starts:

                                MAC (oui Unknown) > MAC (oui Unknown), ethertype EAPOL (0x888e), length 60: EAPOL start (1) v2, len 0
                                
                                

                                These packets come every so often. I think the RG does some backoff /delay if it doesnt immediately auth correctly. You can always reboot your RG to initiate.

                                If your netgraph is setup correctly, this EAP start packet from the RG will be bridged onto your ONT interface. Then you should see some more EAP packets from the ONT interface and RG interface as they negotiate 802.1/X EAP authentication.

                                Once that completes, you should start seeing 802.1Q (tagged as vlan0) traffic on your ONT interface.

                                I start another tcpdump on my VLAN0 netgraph device to see if netgraph is bridging over the VLAN0 to ngeth0:

                                tcpdump -ei ngeth0
                                

                                If I dont see traffic being bridged between ngeth0 and the ONT interface, then netgraph is not setup correctly. At this point, ngeth0 needs to DHCP using the authorized MAC address. You should see an untagged DCHP request on ngeth0 carry over to the ONT interface tagged as VLAN0. Then you should get a DHCP response and you're in business.

                                1 Reply Last reply Reply Quote 0
                                • T
                                  Tantamount
                                  last edited by

                                  Hey guys,

                                  I'm trying to wrap my head around all of this and how it works.

                                  I believe the network diagram I've created is how things should be wired up.

                                  igb0 (WAN/ONT) needs to have the mac address of the ATT Residential Gateway (RG).
                                  igb2 is connected to the RG and somehow the ONT<->RG authentication magic happens (EAP Proxy?).
                                  igb1 (LAN) goes to switch.

                                  Somehow pfsense is not confused by the mac address on igb0 and the mac address of the RG connected to igb2 being the same.

                                  Are ethernet aliases used? (I think pfsense calls them virtual ips?)  I.E. igb0 has an alias for vlan0 traffic which I think is just the ONT<->RG traffic, while the regular igb0 has DHCP (or static) internet address assigned?

                                  network-diagram1.jpg
                                  network-diagram1.jpg_thumb

                                  1 Reply Last reply Reply Quote 0
                                  • T
                                    Tantamount
                                    last edited by

                                    From what I've read from the original DSL Reports thread (http://www.dslreports.com/forum/r29903721-AT-T-Residential-Gateway-Bypass-True-bridge-mode)
                                    There are two possible solutions to get the ONT to talk to the RG through something.
                                    One is to set up a bridge between the two interfaces (igb0 and igb2).  However, 801.D compliance means that 801.x packets won't pass across, and pfsense's drivers are compliant.  One would need to custom compile the drivers to break this compliance.
                                    Two is to use a proxy.  However, it seems that the proxy solutions mentioned are incompatible?
                                    I found two proxy solutions.  One is written in python:
                                    https://github.com/jaysoffian/eap_proxy
                                    However, Pyrodex mentioned in this thread that it has a linux dependency (PFRING).
                                    As was mentioned by variance in this thread, the other needs to be compiled –
                                    https://github.com/kuwerty/eapolproxy
                                    However, from the DSL thread:

                                    I checked out the eapolproxy, and successfully compiled it on my freebsd dev box.  After getting some dependencies (libstdc++) onto pfsense, it does start and appears to be passing the EAP traffic from the RG on OPT1 up to the WAN interface where the ONT is, but nothing ever comes back - it just keeps spamming the EAPOL start and logoffs.  I will have to keep playing with it, but I feel like the solution is close.  It would be good to have some others try this…

                                    Indeed, that person created an issue in github about this, but it looks like the code has been long abandoned.

                                    Rajl then came up with the netgraph solution which, apparently, should be able to bridge the two interfaces in such a way that the 801.x traffic passes across.  However pfsense's oddities breaks this a bit.

                                    aus got this working, but must rely on the linux vm host to do part of the work – something the rest of us cannot rely on.

                                    Pyrodex is testing on bare metal.  Since pfsense doesn't come with the ng_etf.ko file, he pulled it from a fresh copy of reebsd 11.1.

                                    1 Reply Last reply Reply Quote 0
                                    • T
                                      Tantamount
                                      last edited by

                                      @pyrodex:

                                      /sbin/kldload /boot/kernel/ng_etf.ko

                                      Pyrodex,

                                      In your script, does this have to execute in addition to loading the module at boot?  For security reasons, I thought kernel modules could only load at boot in pfsense?

                                      1 Reply Last reply Reply Quote 0
                                      • P
                                        pyrodex
                                        last edited by

                                        I was able to see the interfaces in the ngctl list command so I know it loaded and they got inserted. I haven't had a chance to do TCPdumps yet and will try again this Sunday while the wife is at work.

                                        
                                        kldstat
                                        Id Refs Address            Size     Name
                                         1   24 0xffffffff80200000 2c2da38  kernel
                                         2    1 0xffffffff82e2f000 316ae8   zfs.ko
                                         3    2 0xffffffff83146000 cae8     opensolaris.ko
                                         4    1 0xffffffff83221000 32ce     cpuctl.ko
                                         5    1 0xffffffff83225000 8191     aesni.ko
                                         6    1 0xffffffff8322e000 4700     cryptodev.ko
                                         7    1 0xffffffff83233000 2c63     coretemp.ko
                                         8    1 0xffffffff83236000 191f     ng_etf.ko
                                        
                                        
                                        1 Reply Last reply Reply Quote 0
                                        • R
                                          rajl
                                          last edited by

                                          Tantamount,

                                          Your network diagram is correct and you're definitely heading in the right direction as far as "wrapping your head around this" goes.

                                          If you're remotely familiar with the OSI networking model this makes a lot more sense.  Basically, we are creating a script using Netgraph to do layer 2 (Ethernet Frame) processing to relay authentication frames between the ONT and the RG while diverting all other Ethernet Frames from the ONT up to layers 3-7 of the PFSense network stack.

                                          To go into more detail, ATT's RG uses embedded certificates and the EAPOL protocol to authenticate the RG with the ONT and ATT's wider network.  However, the EAPOL is susceptible to what's known as a "man-in-the-middle" attack where a third-party can intercept the traffic.  What we are (attempting) to do is two things:
                                          (1) Relay all EAPOL ethernet frames between the RG and the ONT.  This requires using MAC address spoofing so that the WAN interface on the PFSense router uses the MAC address of the RG instead of its own MAC address.  This makes the ONT think it is communicating with the RG instead of an unauthorized PFSense device.  Virtual IPs are not used (they are something completely different as they are operating at layers 3-7 of the network stack).
                                          (2) Intercept all other ethernet frames and send them up to layers 3-7 of the PFSense network stack for further firewall processing.

                                          Step 1 can be solved two ways:
                                          (1) Bridge the traffic.  This requires a non-standards compliant bridge device or driver (a trivial modification to the FreeBSD bridge kernel module if you know C programming).
                                          (2) Write a program that intercepts and relays the EAPOL frames (i.e., performs a man-in-the-middle attack).  The problem with the EAP_Proxy script is that it relies on a number of Python libraries that provide linux specific operating system calls.  However, the algorithm that it uses is really simple, so I created a short NetGraph script to duplicate the functionality of the eap_proxy script because Netgraph is part of the FreeBSD base.

                                          The final issue that is causing problems on bare-metal is that ATT is tagging all internet traffic as being assigned to VLAN 0.  Technically, VLAN 0 means the frame is not assigned to a VLAN.  However, ATT uses it as an actual VLAN identifier.  FreeBSD, being overly standards compliant, therefore refuses to create a virtual interface on VLAN 0 and bind services to VLAN 0, allowing for traffic to be read from and written to VLAN 0.  Solve that problem, and this will work on bare-metal.  AUS has already solved it by using PCI pass-through to work around the issue for virtualized PFSense (which will help a lot of people).

                                          1 Reply Last reply Reply Quote 0
                                          • T
                                            Tantamount
                                            last edited by

                                            @rajl:

                                            The final issue that is causing problems on bare-metal is that ATT is tagging all internet traffic as being assigned to VLAN 0.  Technically, VLAN 0 means the frame is not assigned to a VLAN.  However, ATT uses it as an actual VLAN identifier.  FreeBSD, being overly standards compliant, therefore refuses to create a virtual interface on VLAN 0 and bind services to VLAN 0, allowing for traffic to be read from and written to VLAN 0.  Solve that problem, and this will work on bare-metal.  AUS has already solved it by using PCI pass-through to work around the issue for virtualized PFSense (which will help a lot of people).

                                            Thanks Rajl.

                                            I thought PCI pass-through makes virtual environments more "bare metal" because it makes the hardware exclusive to one virtual machine.  I'm not clear how this would solve the VLAN 0 tagging issue unless his virtual environment was the cause of the problem?

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.