ATT Uverse RG Bypass (0.2 BTC)
-
$200 for a set-and-forget solution.
I may consider other payments methods besides bitcoin if preferred.
-
There's been a surprising amount of progress, although I've encountered some issues with PFSense specifically (vanilla FreeBSD and OPNSense work more smoothly).
Here's a summary so far of what I've done, and the issues I am encountering.
For background, here are links for Linux specific solutions:
http://blog.0xpebbles.org/Bypassing-At-t-U-verse-hardware-NAT-table-limits
https://strscrm.io/bypassing-gigapowers-provided-modem.htmlTo summarize, there are two Linux approaches:
(1a) Create a Bridge and enable the Bridge to Forward EAPOL traffic - Use ebtables (Linux layer 2 firewall) to filter what is forwarded across the bridge.
(1b) Run a Python script called EAP-Proxy that proxies EAPOL traffic in order to avoid creating the bridge
(2) Create a Virtual Interface for your WAN assigned VLAN 0 and assign your services to it.However, FreeBSD is not Linux and there are some issues due to FreeBSD's undying commitment to standard's compliance:
(1) The FreeBSD bridge code does not allow for EAPOL traffic to be forwarded across the bridge because this violates the IEEE 802.1D standard.
(2) The EAP-Proxy Script that is circulating is pretty specific to Linux and requires a lot of massaging to work with FreeBSD. Also, it requires installing python on your firewall, which you may not want to do for security reasons.
(3) FreeBSD's vlan code does not allow you to create a virtual interface assigned to VLAN 0 because IEEE 802.1Q specifies that vlan 0 is reserved.So there are two technical issues that need to be resolved:
(1) Make FreeBSD forward EAPOL traffic between the RG and the ONT.
(2) Allow the FreeBSD WAN interface to communicate on vlan 0 so that it can communicate with the ONT.I have solved the first problem and can demonstrate two working solutions on FreeBSD and OPNSense. However, the preferred solution does not work on PFSense yet. To briefly summarize my work:
(1) The universally working (but not preferred) solution is to patch the kernel's bridge code to forward EAPOL traffic. I've created a kernel patch for the bridge code that allows you to enable or disable forwarding of EAPOL traffic on a bridge using a sysctl variable. However, it's definitely not a set and forget solution as you have to manually patch your kernel and then potentially repatch the kernel with every update. Long-term, it can be a set and forget solution once I submit the patch to -CURRENT and it migrates to -STABLE and it then makes it to -RELEASE. But that's a long migration path.
(2) I've created a set and forget solution for forwarding EAPOL traffic using FreeBSD's builtin Netgraph framework. Basically, I use Netgraph to create a layer 2 firewall that forwards all EAPOL frames between interfaces and forwards all other Ethernet frames on up the network stack per normal. I've tested it successfully on vanilla FreeBSD and OPNSense. However, it's not currently working on PFSense for two reasons.-
The first reason is a show stopper - there's a bug in the PFSense netgraph module ng_eth. When ng_eth is loaded, it is supposed to automatically create a Netgraph node for each ethernet interface on your system. Unfortunately, PFSense's ng_eth module only creates a netgraph node for some, but not all of the ethernet interfaces on my test system. It also deletes nodes it has created when I try to switch the LAN and WAN interfaces to the existing ng_eth nodes. Curiously, the ng_eth module works on FreeBSD 11.1 and OPNSense as expected.
-
The second reason is rather trivial to solve, but still annoying. PFSense doesn't load all of the necessary netgraph modules by default and doesn't allow you to load kernel modules at run time. While this is a good security behavior, it means you have to manually edit your boot configuration to load the necessary netgraph modules
I am still attempting to solve the second problem (communicating on VLAN 0). Looking at TCPDump, I can see the DHCP request and response between my WAN and the ONT, but I am not getting an IP address assigned to my WAN interface. I suspect this is because the traffic is tagged as VLAN 0 in a non-standard compliant manner, which is causing FreeBSD to disregard the DHCP response as invalid. I may be able to solve this problem as well using Netgraph, but I need to investigate further when I have some more time and I can test my solution (which is when my family is not home streaming Netflix).
If anyone else is a networking ninja, I would be glad to partner with them to finish this project more quickly.
-
-
Do you have a bitcoin address? I'd like to give you $100 worth right now to keep the progress going. I don't mind a kernel patch.
-
I see that ATT uses IP-DSL to do the log in. I wonder if there is a way to get Pfsense to do the log in directly.
-
Do you have a bitcoin address? I'd like to give you $100 worth right now to keep the progress going. I don't mind a kernel patch.
I'm doing this for the love and the technical challenge, but tips are always welcome! ;)
My bitcoin wallet address for any donations towards this project (from you or any other generous souls) is:
1H8CaLNXembfzYGDNq1NykWU3gaKAjm8K5
If anyone is going to tip me for my efforts, I figure I should at least give them something in return.
Here’s the Netgraph based solution I have so far (which I have tested more thoroughly than the kernel patch):
Step 1: Copy the ng_etf.ko module from a FreeBSD 11.1 system to /boot/kernel. If you are security conscious, you will copy this module yourself from a FreeBSD 11.1 system you already own/control. For convenience, I have uploaded a copy of the module here which trusting souls may download at their convenience.
Step 2: Add the following line to your /boot/loader.conf file – ng_etf_load=”YES” (include quotation marks)
Step 3: Reboot so that the kernel modules are loaded
Step 4: Clone the MAC address of the RG to your “WAN” port.
Step 5: Use the following Netgraph commands to create the EAP Netgraph Bridge. As of right now you need to enter these commands at the console or include them in a startup/boot script. Because you are disconnecting and reconnecting your Ethernet interfaces to create the necessary graph, you will lock yourself out of your own box if you are doing this over SSH. Hence, you need physical access to your machine when entering these commands or they need to be executed automatically on boot.## Replace “em0” and “em1” with your WAN and ONT Ethernet interfaces ## as appropriate for your machine ngctl mkpeer em0: etf lower downstream ngctl name em0:lower waneapfilter ngctl connect waneapfilter: em0: nomatch upper ngctl mkpeer em1: etf lower downstream ngctl name em1:lower laneapfilter ngctl connect laneapfilter: em1: nomatch upper ngctl connect waneapfilter: laneapfilter: eapout eapout ngctl msg waneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }' ngctl msg laneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }'
If these commands throw a cryptic error message, one of three things happened.
-
Most likely, you omitted or added a colon (“:”) somewhere it didn’t belong (I speak from experience).
-
Less likely, but possible, is that I accidentally omitted or added a colon while copying and pasting these commands into this writeup. :)
-
Possibly, PFSense’s ng_eth module is not recognizing one or more of your interfaces. This happens on some, but not all computers. I’m not sure if this is a software or hardware bug. For example, on my ZOTAC CI323, all of my interfaces are recognized. However, on my QOTOM machine, only 2 of the 4 are recognized (OPNSense and vanilla FreeBSD recognize all four, which leads me to believe there is a subtle PFSense bug rather than an hardware issue).
Step 6: Connect the ONT to your PFSense Box and the RG to your PFSense Box (connecting from PFSense to the ONT port on the RG)
Step 7: Power cycle the RG in order to force authentication with ATT
Step 8: Confirm authentication. After 1-2 minutes, you will see the “Broadband” light on your RG flash green and then go to solid green for a short period of time. This means that the 802.1X port authentication has completed successfully. However, your Broadband light will then start flashing read and then go blank. This is because the RG is not receiving an IP address from the ATT network via DHCP (your PFSense Box is attempting request and receive the IP address).
At this point, I can see the DCHP requests and responses between the PFSense box and the ATT network using tcpdump. However, the PFSense box is currently unable to use the IP address provided by ATT. I assume this is because ATT is tagging all responses as being on VLAN 0 (you can see this in TCPDump). With Linux based solutions, you can solve the problem by assigning vlan 0 to your WAN and then moving all your services over to the virtual interface created for vlan0. However, FreeBSD doesn't handle frames explicitly tagged as vlan 0 very well.
I submitted a bug report to FreeBSD for to see if I could get this resolved. The short answer is "no" but they suggested that the Netgraph VLAN code might be able to convert frames tagged as vlan 0 into untagged frames prior to forwarding them up the network stack. I have not had time to investigate this, but it seems promising. If it works, I'll update my netgraph script above to incorporate the appropriate ng_vlan nodes.
EDIT - UPDATED Link for Kernel Module
-
-
I see that ATT uses IP-DSL to do the log in. I wonder if there is a way to get Pfsense to do the log in directly.
Unfortunately, no. ATT uses cryptographic certificates installed in the ROM of the RG to authenticate the RG with the ATT network. This allows them to prevent "unauthorized" equipment from being attached to the network. Unless you feel like dumping the contents of the ROM, identifying the cryptographic certificates, uploading them to your PFSense box, and creating a custom authorization script, it's easier to just perform a man-in-the-middle attack on the 802.1X authentication.
-
Thanks for your work on this.
https://tradeblock.com/bitcoin/tx/5be26573726e21c9f70d18af1223fb8e307cae6194a656ca294cc4afa99ae767
-
Thanks for your work on this.
https://tradeblock.com/bitcoin/tx/5be26573726e21c9f70d18af1223fb8e307cae6194a656ca294cc4afa99ae767
Received. Thanks!
Digging into this a little more, the VLAN netgraph node may provide the "missing link." I'm hoping to be able to test it in the reasonably near future and see if it actually works.
-
Hi rajl.
For the past few weeks, I've independently gone down similar avenues to explore a solution to this problem. I considered porting the EAP Proxy script, but that seemed painful. I opted to patch the kernal (if_bridge.c to be specific) to remove the drop on the 802.1X MAC 01:80:c2:00:00:03. That worked, but only for the EAP problem.
After discovering this post (a great start btw!) and reading more about netgraph, I agree that is probably the best approach. I hoping we can get this working using a combination of: ng_ether, ng_etf, ng_vlan, ng_tee and ng_eiface
For the VLAN0 problem, I expect the netgraph to look like something like this:
# em0 - ATT RG # em1 - ONT # em2 - LAN # ngeth0 - "WAN" netgraph creates interface, removes VLAN0 tag from ONT traffic # make eth devices addressable in netgraph # (kernel module may already be loaded for you) kldload ng_ether # from em1, create a vlan peer # connect em1's lower hook to vlan's downstream hook ngctl mkpeer em1: vlan lower downstream # name peer vlan ngctl name em1:lower vlan # connect em1's upper hook to vlan's nomatch hook ngctl connect em1: vlan: upper nomatch # from vlan, create eiface peer (ngeth0) # connect vlan's untagged hook to eiface's ether hook ngctl mkpeer vlan: eiface untagged ether # instruct vlan: to send vlan0 traffic to untagged hook # which gets sent to the eiface ether hook (ngeth0) ngctl msg vlan: addfilter '{ vlan=0 hook="untagged" }'
I've tested in locally in a VM and I think this part is working. However, the problem I'm struggling with now is combining the EAP netgraph solution with the VLAN netgraph solution. I think this is where ng_tee comes into play, but I'm still trying to wrap my head around it.
I think we need to use ng_tee to split out the ng_ether-em1 interface. Then hook up the EAP graph to one side and the VLAN graph to the other. But my head spins trying to keep left, right, right2left and left2right straight. :) The lacking documentation about netgraph doesnt help either. It seems no one talks about netgraph much.
Have you had any progress or success?
-
I just wanted to report back that I got this working, but probably not in the sense you were hoping for.
I’m running pfSense in a VM on Proxmox (KVM/QEMU). For now, I’ve opted to let the hypervisor (Linux) do the EAP and VLAN work. (Same method basically) Here is my setup:
Nothing too special configuration was required for my pfSense VM. Here is my config:
balloon: 0 bootdisk: ide0 cores: 2 cpu: host ide0: ssd0:vm-100-disk-1,size=32G memory: 512 name: pfSense net0: virtio=XX:XX:XX:XX:XX:XX,bridge=vmbr0 net1: virtio=XX:XX:XX:XX:XX:XX,bridge=vmbr1 numa: 0 ostype: other serial0: socket sockets: 1 tablet: 0
The net0 (LAN) interface bridges to vmbr0. vmbr0 bridges to physical eth0, which is connected to my switch.
The net1 (WAN) interface bridges to vmbr1. vmbr1 bridges to vlan0. The vlan0 interface is configured off physical eth1, which is connected to the ONT. net1 MAC address also matches my ATT Gateway. Change it in your pfSense WAN interface setting.
/etc/network/interfaces:
# LAN / eth0 # Connect to switch iface eth0 inet manual # ONT / eth1 # Connect to ONT box outside iface eth1 inet manual # RG / eth2 # Connect to ATT Gateway on ONT port iface eth2 inet manual # LAN Bridge / br0 # Bridge main switch to pfSense # IP is Proxmox host auto vmbr0 iface vmbr0 inet static address 192.168.1.2 netmask 255.255.255.0 gateway 192.168.1.1 bridge_ports eth0 bridge_stp off bridge_fd 0 # VLAN0 Bridge / br1 # Bridge vlan tagged WAN to pfSense auto vmbr1 iface vmbr1 inet manual bridge_ports vlan0 bridge_stp off bridge_fd 0 # EAP Bridge / br2 # Bridge ATT Gateway + ONT so EAP/802.1X auth can complete # group_fwd_mask makes sure 802.1X traffic is bridged auto vmbr2 iface vmbr2 inet manual bridge_ports eth1 eth2 bridge_stp off bridge_fd 0 post-up echo 8 > /sys/class/net/vmbr2/bridge/group_fwd_mask
Unfortunately, Proxmox conflicts with the vlan debian package, so you have to configure the vlan interface with the ip command instead of the interface file:
ip link add link eth1 name vlan0 type vlan id 0
And that’s pretty much it. I haven’t nailed down the timings yet from cold boot to online for a fully automated solution. For some reason, the EAP only takes under certain conditions. I have the best luck with the following:
1. Cold boot hypervisor
2. Wait for EAP to authenticate
3. Start vlan0
4. Start pfSense VMIt’s not perfect right now and it will take some more experimenting. But, it feels good to be off their RG!
I’d still be interested in a pure BSD solution though.
-
Glad to see someone else exploring this.
I haven't had a chance to test it yet (my wife and toddler will kill me if I cut the internet tinkering on the weekend – toddler cartoons are sacred).
I hadn't thought of using an ng_tee node, but it sounds like a good idea. Basically, you would connect the ng_tee left hook to the ng_eth downstream hook. You would connect the ng_tee right hook to an ng_vlan (for example), which can filter the vlan 0 tagged traffic for you before passing it up the protocol stack. The ng_etf could be connect to the left2right hook of the ng_tee node using the commands I wrote above to filter the eap traffic.
A modified script would look something like this:
ngctl mkpeer em0: tee lower left ngctl name em0:lower T #Connect vlan to virtual interface for vlan0 traffic, ignore all untagged traffic #by failing to connect the vlan to the ether upper hook (reserved for eap filtering) ngctl mkpeer T: vlan right downstream ngctl name T:right vlan ngctl mkpeer vlan: eiface vlan0 ether ngctl msg vlan0: 'addfilter { vlan=0 hook="vlan0" }' #Connect other hook of T node to ng_etf node for eap filtering/proxying #Leave "lan filter" the same because we only care about eap traffic ngctl mkpeer T: etf left2right downstream ngctl name T:left2right waneapfilter ngctl connect waneapfilter: em0: nomatch upper ngctl mkpeer em1: etf lower downstream ngctl name em1:lower laneapfilter ngctl connect laneapfilter: em1: nomatch upper ngctl connect waneapfilter: laneapfilter: eapout eapout ngctl msg waneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }' ngctl msg laneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }'
This is an untested script and may throw errors or error message, but the underlying principles should work for any brave souls willing to try! ;) Just make sure that you have the ng_etf, ng_vlan, ng_eth, ng_eiface and ng_tee modules loaded.
-
Just occurred to me as an alternative that you could use a NetGraph multiplexer node instead of the ng_tee node.
-
I did a bit more testing, but no success just yet. (I suspect I need to first try to get this to work on phsyical hardware. Currently trying using a pfSense VM and I'm not seeing packet carry over from Linux hypervisor to the pfSense VM)
Regarding the ng_ether "bug", I did some digging on this. It turns out that this is not a bug necessarily. pfSense actually does a NGM_ETHER_DETACH against interfaces under some circumstanaces.
https://github.com/pfsense/pfsense/blob/9a18ac7af8ae4a4fde8998c18cc7ba7802056477/src/etc/inc/interfaces.inc#L180
I think this was for performance reasons back when netgraph had performance overhead.
Anyways, you think you'd be able to just do a control message of NGM_ETHER_ATTACH, but that doesn't exist in vanilla FreeBSD. Luckily, pfSense integrates some patches to enable NGM_ETHER_ATTACH, but you have to call it from PHP.
https://github.com/pfsense/FreeBSD-ports/blob/e178a5cf520e928efb3c7d896e3d9fcfb41ac7e5/devel/php56-pfSense-module/files/pfSense.c#L3094
This will re-enable the interface as a node in netgraph:
php -r 'pfSense_ngctl_attach(".", "em0");'
Also, for ng_one2many (assuming that's what you mean by multiplexer) I don't think that will work. I initially looked at this too, but it distributes packets in a round-robin fashion so the many's would only see some packets. At least, that's how I interpret the man page.
-
ng_one2many is what I was referring to. The man page states that it has several transmission modes, including round-robin and transmit-all. The man page says that the round robin mode is the default, but my experience when playing with it is that transmit-all was the default. In either case, you could easily set the transmission mode to transmit-all to ensure that you get the desired behavior. So it will work and is simpler to work with than ng_tee.
That's great research on the ng_ether issue. It's been holding me up for awhile, forcing me to do my testing on other distributions (e.g., vanilla FreeBSD and OPNSense) and then curse when I couldn't get it working on PFSense. I'll have to see if it works with my scripts on a VM or PFSense. It should, but Murphy's law always strikes me down!
-
Doh! Good catch on the ng_one2many transmit-all algorithm. I was looking at an old man page from an earlier version of FreeBSD, which it didnt support transmit-all yet. That's what I get for googling the man pages, instead of reading them in terminal! May give this a shot later… I'll report back if I have any success.
Cheers!
-
It worked!! True U-verse bridge mode on pfSense!
[2.4.2-RELEASE][root@pfsense.knox.lan]/root: ngctl list There are 9 total nodes: Name: T Type: tee ID: 00000021 Num hooks: 3 Name: ue0 Type: ether ID: 00000003 Num hooks: 2 Name: vlan0 Type: vlan ID: 00000024 Num hooks: 2 Name: <unnamed>Type: socket ID: 00000006 Num hooks: 0 Name: ngctl96372 Type: socket ID: 00000047 Num hooks: 0 Name: ngeth0 Type: eiface ID: 00000027 Num hooks: 1 Name: waneapfilter Type: etf ID: 0000002a Num hooks: 3 Name: laneapfilter Type: etf ID: 00000031 Num hooks: 3 Name: em0 Type: ether ID: 00000019 Num hooks: 2 [2.4.2-RELEASE][root@pfsense.knox.lan]/root: ifconfig em0 em0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=40098 <vlan_mtu,vlan_hwtagging,vlan_hwcsum,vlan_hwtso>ether xx:xx:xx:xx:xx:xx hwaddr xx:xx:xx:xx:xx:xx media: Ethernet autoselect (1000baseT <full-duplex>) status: active [2.4.2-RELEASE][root@pfsense.knox.lan]/root: ifconfig ue0 ue0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=8000b <rxcsum,txcsum,vlan_mtu,linkstate>ether xx:xx:xx:xx:xx:xx hwaddr xx:xx:xx:xx:xx:xx media: Ethernet autoselect (100baseTX <full-duplex>) status: active [2.4.2-RELEASE][root@pfsense.knox.lan]/root: ifconfig ngeth0 ngeth0: flags=8a43 <up,broadcast,running,allmulti,simplex,multicast>metric 0 mtu 1500 options=28 <vlan_mtu,jumbo_mtu>ether xx:xx:xx:xx:xx:xx inet xx.xx.xx.xx netmask 0xfffffc00 broadcast xx.xx.xx.xx media: Ethernet autoselect (1000baseT <full-duplex>) status: active</full-duplex></vlan_mtu,jumbo_mtu></up,broadcast,running,allmulti,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,linkstate></up,broadcast,running,simplex,multicast></full-duplex></vlan_mtu,vlan_hwtagging,vlan_hwcsum,vlan_hwtso></up,broadcast,running,simplex,multicast></unnamed>
For reference…
em0 is connected to my ONT.
em1 is connected to my LAN
ue0 is connected to my RG (via USB ethernet)
ngeth0 is the VLANed device which is configured as my WAN in pfSenseCommands to get it running (thanks for the help on ng_tee rajl!) ...
# copy and load ng_etf kernel module kldload /boot/kernel/ng_etf.ko # # setup netgraph nodes # # list out netgraph nodes ngctl list # pfSense for some reason detaches ether devices. reattach any missing devices. php -r 'pfSense_ngctl_attach(".", "em0");' # create tee node to split em0 traffic (one for EAP, one for VLAN0) ngctl mkpeer em0: tee lower left # may get a warning ngctl name em0:lower T # create vlan node + eiface ngctl mkpeer T: vlan right downstream ngctl name T:right vlan0 ngctl mkpeer vlan0: eiface vlan0 ether ngctl msg vlan0: 'addfilter { vlan=0 hook="vlan0" }' # create etf and connect to em0 (ONT) ngctl mkpeer T: etf left2right downstream ngctl name T:left2right waneapfilter ngctl connect waneapfilter: em0: nomatch upper # create etf and connect to em1 (RG) ngctl mkpeer ue0: etf lower downstream ngctl name ue0:lower laneapfilter ngctl connect laneapfilter: ue0: nomatch upper # define filters for EAP traffic ngctl msg waneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }' ngctl msg laneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }' # use filters to bridge EAP traffic ngctl connect waneapfilter: laneapfilter: eapout eapout # change MAC address to match RG (also can be done in pfSense) ifconfig ngeth0 ether xx:xx:xx:xx:xx:xx
There is still worked to be done though to make this perfect though…
1. Explore using ng_one2many to see if that simplifies the netgraph a bit
2. Automate / Harden change so its persistant across reboots (rajl already documented this earlier)
3. Document!And for what it's worth, I'm running this pfSense in a virtual machine via Proxmox (QEMU/KVM). I couldnt get the VLAN0 traffic to bridge across the interface into pfSense, so I ended up doing a PCI passthrough of the NIC device.
-
That's awesome!!! My suspicion is that this would run on baremetal just fine (have to test though). So let's say there's a 4th todo - test this to run on baremetal for those of use that don't virtualize! :-) Hopefully, it won't take too much modification.
This should be pretty easy to automate so that it executes across reboots. Just save your commands as a shell script (don't forget the #!/bin/sh at the beginning of the file) and follow the PFSense instructions for executing shell scripts at the end of the boot process.
https://doc.pfsense.org/index.php/Executing_commands_at_boot_time
I read somewhere that ATT will occassionally push firmware updates to the RG, which this setup may have problems with because the RG is being isolated from the ATT network. But that's a bridge to cross when we get there.
-
Would it be possible to have ONT go to port A of a switch set to vlan 20
and have port B of that switch also on vlan 20 connect to the RG's ONT port?Would a switch normally process/filter those 802.1x packets in such a setup?
My pfsense vm is in a different area of the house on a different switch and I'm curious if I'll be able to get this working.
Also, is there any practical benefit to doing this? For instance, would it open outgoing tcp port 25 traffic?
-
Would it be possible to have ONT go to port A of a switch set to vlan 20
and have port B of that switch also on vlan 20 connect to the RG's ONT port?Would a switch normally process/filter those 802.1x packets in such a setup?
My pfsense vm is in a different area of the house on a different switch and I'm curious if I'll be able to get this working.
Also, is there any practical benefit to doing this? For instance, would it open outgoing tcp port 25 traffic?
I don't know the answer to your question, but I suspect that won't work. The problem is that ONT traffic comes in on VLAN0 and needs to egress on VLAN0. I'm not sure your switch would tag VLAN0 <-> VLAN20 accordingly.
Also, I'm having some duplicate packets in my previous setup. Hoping one2many might solve that. More to come…
-
Would it be possible to have ONT go to port A of a switch set to vlan 20
and have port B of that switch also on vlan 20 connect to the RG's ONT port?Would a switch normally process/filter those 802.1x packets in such a setup?
My pfsense vm is in a different area of the house on a different switch and I'm curious if I'll be able to get this working.
Also, is there any practical benefit to doing this? For instance, would it open outgoing tcp port 25 traffic?
I'll try to answer your questions in detail.
First, the switch setup you're describing won't work because your switch would block the traffic for several reasons. First, if the switch would drop the ethernet frames because ATT tags them as vlan0, but you're setting your ports for vlan20. Second, your switch would probably drop all the authentication frames (802.1X) because most (but not all) switches are fully compliant with 802.1D, which requires that switches and bridges not forward 802.1X frames. However, some switches are not standard compliant and will forward the frames anyway.
That said, you could always run a long cable from one of the house to the other to solve the problem.
Regarding your question about practical benefits, the main practical benefit is performance. The RGs tend to have (1) a small state table with a limited number of entries and (2) middling (at best) performance ARM processors that start to choke under load when you start to do "real routing." As an example, get a few good bit-torrents going on a 1-Gig connection and they try to browser the web. Your performance will crawl because the RG's state table is too small to keep track of all of the connections and the RG's processor is unable to process all the connections at line-speed. Bypassing the RG to use your own PFSense box solves both of these problems.
Some older (but still relevant) articles on why you would want to replace consumer grade routers with an x86 router (such as one using PFSense) are below:
https://arstechnica.com/gadgets/2016/01/numbers-dont-lie-its-time-to-build-your-own-router/
https://arstechnica.com/gadgets/2016/09/the-router-rumble-ars-diy-build-faces-better-tests-tougher-competition/ -
I was afraid you were going to say that – I tried it last night and was unsuccessful.
I've got one of those Netgate gs-2440's that I'll use instead of the vm. I'll be able to put it right next to the RG and ONT.
-
Going to try this today with my 4 port SuperMicro physical and report back. Had to install FreeBSD 11.1 in a VM to get the kernel module since the link seems to be dead.
UPDATE:
No luck.
I use the crappy switch trick today and swap VLANs and my igb0 is the MAC of the RG. My igb0 (WAN) is connected into my bypass switch on a VLAN with the ONT and the RG is on another VLAN that gets flipped and flopped if the internet goes down.
I tried the script and connected igb0 straight into the ONT and igb3 to the RG removing my bypass switch out of line. I had no luck and the RG would attempt to AUTH the port on the ONT but never went past that.
Here is the script I used:
#igb2 is connected to the ONT #lagg0 is connected to the LAN #igb3 is connected to the RG #ngeth0 is the VLANed device which is configured as my WAN in pfSense # copy and load ng_etf kernel module /sbin/kldload /boot/kernel/ng_etf.ko # # setup netgraph nodes # # list out netgraph nodes /usr/sbin/ngctl list # pfSense for some reason detaches ether devices. reattach any missing devices. php -r 'pfSense_ngctl_attach(".", "igb0");' # create tee node to split ONT traffic (one for EAP, one for VLAN0) /usr/sbin/ngctl mkpeer igb0: tee lower left # may get a warning /usr/sbin/ngctl name igb0:lower T # create vlan node + eiface /usr/sbin/ngctl mkpeer T: vlan right downstream /usr/sbin/ngctl name T:right vlan0 /usr/sbin/ngctl mkpeer vlan0: eiface vlan0 ether /usr/sbin/ngctl msg vlan0: 'addfilter { vlan=0 hook="vlan0" }' # create etf and connect to ONT /usr/sbin/ngctl mkpeer T: etf left2right downstream /usr/sbin/ngctl name T:left2right waneapfilter /usr/sbin/ngctl connect waneapfilter: igb0: nomatch upper # create etf and connect to RG /usr/sbin/ngctl mkpeer igb3: etf lower downstream /usr/sbin/ngctl name igb3:lower laneapfilter /usr/sbin/ngctl connect laneapfilter: igb3: nomatch upper # define filters for EAP traffic /usr/sbin/ngctl msg waneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }' /usr/sbin/ngctl msg laneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }' # use filters to bridge EAP traffic /usr/sbin/ngctl connect waneapfilter: laneapfilter: eapout eapout # change MAC address to match RG (also can be done in pfSense) ifconfig ngeth0 ether <mac></mac>
-
It's hard to say what your exact issue is without more information.
However, the first thing I would do is run some tcpdumps to see what's going on.
You should run tcpdumps on the ONT interface and the RG interface:
tcpdump -ei em0 tcpdump -ei em1
From the RG interface, you should see some EAPOL starts:
MAC (oui Unknown) > MAC (oui Unknown), ethertype EAPOL (0x888e), length 60: EAPOL start (1) v2, len 0
These packets come every so often. I think the RG does some backoff /delay if it doesnt immediately auth correctly. You can always reboot your RG to initiate.
If your netgraph is setup correctly, this EAP start packet from the RG will be bridged onto your ONT interface. Then you should see some more EAP packets from the ONT interface and RG interface as they negotiate 802.1/X EAP authentication.
Once that completes, you should start seeing 802.1Q (tagged as vlan0) traffic on your ONT interface.
I start another tcpdump on my VLAN0 netgraph device to see if netgraph is bridging over the VLAN0 to ngeth0:
tcpdump -ei ngeth0
If I dont see traffic being bridged between ngeth0 and the ONT interface, then netgraph is not setup correctly. At this point, ngeth0 needs to DHCP using the authorized MAC address. You should see an untagged DCHP request on ngeth0 carry over to the ONT interface tagged as VLAN0. Then you should get a DHCP response and you're in business.
-
Hey guys,
I'm trying to wrap my head around all of this and how it works.
I believe the network diagram I've created is how things should be wired up.
igb0 (WAN/ONT) needs to have the mac address of the ATT Residential Gateway (RG).
igb2 is connected to the RG and somehow the ONT<->RG authentication magic happens (EAP Proxy?).
igb1 (LAN) goes to switch.Somehow pfsense is not confused by the mac address on igb0 and the mac address of the RG connected to igb2 being the same.
Are ethernet aliases used? (I think pfsense calls them virtual ips?) I.E. igb0 has an alias for vlan0 traffic which I think is just the ONT<->RG traffic, while the regular igb0 has DHCP (or static) internet address assigned?
-
From what I've read from the original DSL Reports thread (http://www.dslreports.com/forum/r29903721-AT-T-Residential-Gateway-Bypass-True-bridge-mode)
There are two possible solutions to get the ONT to talk to the RG through something.
One is to set up a bridge between the two interfaces (igb0 and igb2). However, 801.D compliance means that 801.x packets won't pass across, and pfsense's drivers are compliant. One would need to custom compile the drivers to break this compliance.
Two is to use a proxy. However, it seems that the proxy solutions mentioned are incompatible?
I found two proxy solutions. One is written in python:
https://github.com/jaysoffian/eap_proxy
However, Pyrodex mentioned in this thread that it has a linux dependency (PFRING).
As was mentioned by variance in this thread, the other needs to be compiled –
https://github.com/kuwerty/eapolproxy
However, from the DSL thread:I checked out the eapolproxy, and successfully compiled it on my freebsd dev box. After getting some dependencies (libstdc++) onto pfsense, it does start and appears to be passing the EAP traffic from the RG on OPT1 up to the WAN interface where the ONT is, but nothing ever comes back - it just keeps spamming the EAPOL start and logoffs. I will have to keep playing with it, but I feel like the solution is close. It would be good to have some others try this…
Indeed, that person created an issue in github about this, but it looks like the code has been long abandoned.
Rajl then came up with the netgraph solution which, apparently, should be able to bridge the two interfaces in such a way that the 801.x traffic passes across. However pfsense's oddities breaks this a bit.
aus got this working, but must rely on the linux vm host to do part of the work – something the rest of us cannot rely on.
Pyrodex is testing on bare metal. Since pfsense doesn't come with the ng_etf.ko file, he pulled it from a fresh copy of reebsd 11.1.
-
/sbin/kldload /boot/kernel/ng_etf.ko
Pyrodex,
In your script, does this have to execute in addition to loading the module at boot? For security reasons, I thought kernel modules could only load at boot in pfsense?
-
I was able to see the interfaces in the ngctl list command so I know it loaded and they got inserted. I haven't had a chance to do TCPdumps yet and will try again this Sunday while the wife is at work.
kldstat Id Refs Address Size Name 1 24 0xffffffff80200000 2c2da38 kernel 2 1 0xffffffff82e2f000 316ae8 zfs.ko 3 2 0xffffffff83146000 cae8 opensolaris.ko 4 1 0xffffffff83221000 32ce cpuctl.ko 5 1 0xffffffff83225000 8191 aesni.ko 6 1 0xffffffff8322e000 4700 cryptodev.ko 7 1 0xffffffff83233000 2c63 coretemp.ko 8 1 0xffffffff83236000 191f ng_etf.ko
-
Tantamount,
Your network diagram is correct and you're definitely heading in the right direction as far as "wrapping your head around this" goes.
If you're remotely familiar with the OSI networking model this makes a lot more sense. Basically, we are creating a script using Netgraph to do layer 2 (Ethernet Frame) processing to relay authentication frames between the ONT and the RG while diverting all other Ethernet Frames from the ONT up to layers 3-7 of the PFSense network stack.
To go into more detail, ATT's RG uses embedded certificates and the EAPOL protocol to authenticate the RG with the ONT and ATT's wider network. However, the EAPOL is susceptible to what's known as a "man-in-the-middle" attack where a third-party can intercept the traffic. What we are (attempting) to do is two things:
(1) Relay all EAPOL ethernet frames between the RG and the ONT. This requires using MAC address spoofing so that the WAN interface on the PFSense router uses the MAC address of the RG instead of its own MAC address. This makes the ONT think it is communicating with the RG instead of an unauthorized PFSense device. Virtual IPs are not used (they are something completely different as they are operating at layers 3-7 of the network stack).
(2) Intercept all other ethernet frames and send them up to layers 3-7 of the PFSense network stack for further firewall processing.Step 1 can be solved two ways:
(1) Bridge the traffic. This requires a non-standards compliant bridge device or driver (a trivial modification to the FreeBSD bridge kernel module if you know C programming).
(2) Write a program that intercepts and relays the EAPOL frames (i.e., performs a man-in-the-middle attack). The problem with the EAP_Proxy script is that it relies on a number of Python libraries that provide linux specific operating system calls. However, the algorithm that it uses is really simple, so I created a short NetGraph script to duplicate the functionality of the eap_proxy script because Netgraph is part of the FreeBSD base.The final issue that is causing problems on bare-metal is that ATT is tagging all internet traffic as being assigned to VLAN 0. Technically, VLAN 0 means the frame is not assigned to a VLAN. However, ATT uses it as an actual VLAN identifier. FreeBSD, being overly standards compliant, therefore refuses to create a virtual interface on VLAN 0 and bind services to VLAN 0, allowing for traffic to be read from and written to VLAN 0. Solve that problem, and this will work on bare-metal. AUS has already solved it by using PCI pass-through to work around the issue for virtualized PFSense (which will help a lot of people).
-
The final issue that is causing problems on bare-metal is that ATT is tagging all internet traffic as being assigned to VLAN 0. Technically, VLAN 0 means the frame is not assigned to a VLAN. However, ATT uses it as an actual VLAN identifier. FreeBSD, being overly standards compliant, therefore refuses to create a virtual interface on VLAN 0 and bind services to VLAN 0, allowing for traffic to be read from and written to VLAN 0. Solve that problem, and this will work on bare-metal. AUS has already solved it by using PCI pass-through to work around the issue for virtualized PFSense (which will help a lot of people).
Thanks Rajl.
I thought PCI pass-through makes virtual environments more "bare metal" because it makes the hardware exclusive to one virtual machine. I'm not clear how this would solve the VLAN 0 tagging issue unless his virtual environment was the cause of the problem?
-
Any more progress on this?
I'm testing using a spare machine while the 4 port nic arrives.
For the moment pfsense is installed baremetal. Running into issues at the following
# create etf and connect to em1 (RG) ngctl mkpeer ue0: etf lower downstream ngctl name ue0:lower laneapfilter ngctl connect laneapfilter: ue0: nomatch upper
All the previous command execute correctly. Interface names have been adjusted as needed based on how my system identified them.
em0 - intel pro 1000 (builtin) - ONT (WAN) xl0 - 3com - gateway ue0 - usb - LAN
So the original snippet is changed to
# create etf and connect to em1 (RG) ngctl mkpeer xl0: etf lower downstream ngctl name xl0:lower laneapfilter ngctl connect laneapfilter: xl0: nomatch upper
I end up getting this error (including entire input/output to this point).
[2.4.3-RELEASE][root@pfSense.localdomain]/root: kldload /boot/kernel/ng_etf.ko
[2.4.3-RELEASE][root@pfSense.localdomain]/root: ngctl list
There are 2 total nodes:
Name: ngctl57043 Type: socket ID: 00000012 Num hooks: 0
Name: <unnamed>Type: socket ID: 00000006 Num hooks: 0
[2.4.3-RELEASE][root@pfSense.localdomain]/root: php -r 'pfSense_ngctl_attach(".", "em0");'
[2.4.3-RELEASE][root@pfSense.localdomain]/root: ngctl mkpeer em0: tee lower left
[2.4.3-RELEASE][root@pfSense.localdomain]/root: ngctl name em0:lower T
[2.4.3-RELEASE][root@pfSense.localdomain]/root: ngctl mkpeer T: vlan right downstream
[2.4.3-RELEASE][root@pfSense.localdomain]/root: ngctl name T:right vlan0
[2.4.3-RELEASE][root@pfSense.localdomain]/root: ngctl mkpeer vlan0: eiface vlan0 ether
[2.4.3-RELEASE][root@pfSense.localdomain]/root: ngctl msg vlan0: 'addfilter { vlan=0 hook="vlan0" }'
[2.4.3-RELEASE][root@pfSense.localdomain]/root: ngctl mkpeer T: etf left2right downstream
[2.4.3-RELEASE][root@pfSense.localdomain]/root: ngctl name T:left2right waneapfilter
[2.4.3-RELEASE][root@pfSense.localdomain]/root: ngctl connect waneapfilter: em0: nomatch upper
[2.4.3-RELEASE][root@pfSense.localdomain]/root: ngctl mkpeer xl0: etf lower downstream
ngctl: send msg: No such file or directory
[2.4.3-RELEASE][root@pfSense.localdomain]/root:</unnamed>Not sure how to move forward from this.
Also, I'm a bit confused on how a pci-passthrough (when virtualized) is any different than bare metal, as far as vlan0 tagging is concerned. When the nic is defined as pci-pass through, does the underlying vm have direct access to the nic? This would be similar as a baremetal implementation, no?
Thanks!
-
hi I've just been reading this thread.
I have a pfsense router but unfortunately Uverse is not here in Australia, so I can't help practically..however i see you have got my ng-etf pair set up so I see we have somehow communicated before :-)
let me see if I understand the full picture..
You need to link the ONT (whatever that stands for .. Optical Network translator?) via vlan0 to the RG, but you only want protocol 8xxx to go to the RG, and all other protocols/ethertypes to go to the pfsense IP stack.. is that right?
-
this MAY work but may not.. it depends if the packets coming out of the interface have already been stripped of vlan tags.
ONT]–--[em0]lower–-downstream[eapfilter:]nomatch–--vlan0[VLAN]downstream–--upper[em0…
eapout
|
|
|
RG]–----[em1]lower–--------------/ngctl mkpeer igb0: etf lower downstream
ngctl name igb0:lower eapfilter
ngctl mkpeer igb0: vlan upper downstream
ngctl name igb0:upper vlanheader
ngctl msg vlanheader: addfilter '{ vlan=0 hook="vlan0" }'
ngctl connect vlanheader: eapfilter: vlan0 nomatch
ngctl connect eapfilter: igb1: eapout lower
ngctl msg waneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }'if they haven't then we may need to put the vlan nodes (two) on the other side of the etf node, so that it doesn't see the vlan tags.
Maybe I should add an option to make etf nodes take vlan tags into account.
If you can attach the nghook program to an inserted ngtee node (inserted somewhere in your current graph) and see what comes out. (with -a ).
then we can see what the packets look like.
-
hi I've just been reading this thread.
I have a pfsense router but unfortunately Uverse is not here in Australia, so I can't help practically..however i see you have got my ng-etf pair set up so I see we have somehow communicated before :-)
let me see if I understand the full picture..
You need to link the ONT (whatever that stands for .. Optical Network translator?) via vlan0 to the RG, but you only want protocol 8xxx to go to the RG, and all other protocols/ethertypes to go to the pfsense IP stack.. is that right?
You've actually been talking to me on the FreeBSD mailing lists, but I posted the final results to share with the world and a number of people have been running with it. :)
To help you with the terminology and define the problem:
ONT = Optical Network Terminal. It's the box in your house where the fiber-optic cable is terminated and the signal is converted into an electrical signal. It has an RJ-45 1000BaseT jack that you use to plug the RG into.
RG = Residential Gateway. It's one of those all-in-one wireless routers that is provided by the ISP (AT&T). It's low-end consumer gear that has a number of issues. Many of us have outgrown it's limitations and want to get rid of it. However, we can't because the RG is required for authentication with the ISP.
EAP-OL = Encapsulated Authentication Protocol Over LAN. It works at layer 2 and can be easily man-in-the-middled. The RG uses this protocol to authenticate with the ISP. My suspicion is that the RG is using EAP-OL to authenticate with the ONT so that the ONT can ensure that only an approved, authenticated device is connected to it/the network. However, where the actual end-point for the EAP-OL frames is located is immaterial.
-
this MAY work but may not.. it depends if the packets coming out of the interface have already been stripped of vlan tags.
ONT]–--[em0]lower–-downstream[eapfilter:]nomatch–--vlan0[VLAN]downstream–--upper[em0…
eapout
|
|
|
RG]–----[em1]lower–--------------/ngctl mkpeer igb0: etf lower downstream
ngctl name igb0:lower eapfilter
ngctl mkpeer igb0: vlan upper downstream
ngctl name igb0:upper vlanheader
ngctl msg vlanheader: addfilter '{ vlan=0 hook="vlan0" }'
ngctl connect vlanheader: eapfilter: vlan0 nomatch
ngctl connect eapfilter: igb1: eapout lower
ngctl msg waneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }'if they haven't then we may need to put the vlan nodes (two) on the other side of the etf node, so that it doesn't see the vlan tags.
Maybe I should add an option to make etf nodes take vlan tags into account.
If you can attach the nghook program to an inserted ngtee node (inserted somewhere in your current graph) and see what comes out. (with -a ).
then we can see what the packets look like.
I'm not sure that will work, because I think it may be stripping VLAN tags at the wrong point. You can correct me if I am wrong.
Here's a rough sketch of how the frames are tagged.
Before:
[ONT] –---[Tagged VLAN 0]–------------ [RG] –---[No VLAN Tags]–----------[LAN]
After:
[ONT] –----[Tagged VLAN 0] –----------[PFSense] –-----[No VLAN Tags]–---[LAN]
|
|
|–------------[Tagged VLAN 0]–-[RG]Hope this helps. Someone on this thread has gotten this to work with virtualized PFSense on ProxMox using PCI Passthrough. No one, however, has yet reported getting this to work on bare metal.
It's been the fun challenge.
-
@rajl As I understand it, the whole reason for doing the pci passthrough is because there's issue with passing vlan0 through the hypervisor? By configuring the nic as pci passthrough, the guest os (pfsense) has full access to the nic. Vlan0 is then passed through the graphing definition.
I've been testing further on the spare machine (bare metal pfsense install). I realized the reason for my errors above - I did not reattach the missing interfaces. Once I did that pfsense ssh accepted all the commands except those below.
I'm using the code from this post: https://forum.pfsense.org/index.php?topic=111043.msg793292#msg793292
# define filters for EAP traffic ngctl msg waneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }' ngctl msg laneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }' # use filters to bridge EAP traffic ngctl connect waneapfilter: laneapfilter: eapout eapout
I resolved this by issuing the connect command first - to establish eapout definition? Once done there were no further errors.
However, monitoring the RG interface with tcpdump, all I see are these. Mac c0:a0:0d:xx:xx:xx is that of the RG.
08:13:56.104742 c0:a0:0d:xx:xx:xx (oui Unknown) > 01:80:c2:00:00:03 (oui Unknown), ethertype EAPOL (0x888e), length 60: EAPOL start (1) v2, len 0
On the WAN interface (ngeth0), I see this. RG mac is spoofed to c0:a0:0d:xx:xx:xx for this interface.
09:06:56.718637 c0:a0:0d:xx:xx:xx (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 346: vlan 0, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from c0:a0:0d:xx:xx:xx (oui Unknown), length 300 09:08:24.668681 00:90:d0:63:ff:01 (oui Unknown) > 01:80:c2:00:00:03 (oui Unknown), ethertype 802.1Q (0x8100), length 68: vlan 0, p 7, ethertype EAPOL, EAP packet (0) v1, len 15 09:09:25.261889 00:90:d0:63:ff:01 (oui Unknown) > 01:80:c2:00:00:03 (oui Unknown), ethertype 802.1Q (0x8100), length 68: vlan 0, p 7, ethertype EAPOL, EAP packet (0) v1, len 4 09:09:25.261895 00:90:d0:63:ff:01 (oui Unknown) > 01:80:c2:00:00:03 (oui Unknown), ethertype 802.1Q (0x8100), length 68: vlan 0, p 7, ethertype EAPOL, EAP packet (0) v1, len 15 09:09:28.585333 c0:a0:0d:xx:xx:xx (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 346: vlan 0, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from c0:a0:0d:xx:xx:xx (oui Unknown), length 300 09:09:39.433119 c0:a0:0d:xx:xx:xx (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 346: vlan 0, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from c0:a0:0d:xx:xx:xx (oui Unknown), length 300 09:09:47.419730 18:4a:6f:11:62:fd (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 64: vlan 0, p 7, ethertype ARP, Request who-has 107.x.x.x (c0:a0:0d:xx:xx:xx (oui Unknown)) tell 0.0.0.0, length 46 09:09:50.440866 c0:a0:0d:xx:xx:xx (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 346: vlan 0, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from c0:a0:0d:xx:xx:xx (oui Unknown), length 300 09:09:52.419034 18:4a:6f:11:62:fd (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 64: vlan 0, p 7, ethertype ARP, Request who-has 107.x.x.x (c0:a0:0d:xx:xx:xx (oui Unknown)) tell 0.0.0.0, length 46 09:09:55.558504 00:90:d0:63:ff:01 (oui Unknown) > 01:80:c2:00:00:03 (oui Unknown), ethertype 802.1Q (0x8100), length 68: vlan 0, p 7, ethertype EAPOL, EAP packet (0) v1, len 15 09:09:57.418368 18:4a:6f:11:62:fd (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 64: vlan 0, p 7, ethertype ARP, Request who-has 107.x.x.x (c0:a0:0d:xx:xx:xx (oui Unknown)) tell 0.0.0.0, length 46 09:09:58.448730 c0:a0:0d:xx:xx:xx (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 346: vlan 0, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from c0:a0:0d:xx:xx:xx (oui Unknown), length 300
So it would seem that eapol requests are going out, but not coming back in? On the RG tcpdump there should be additional traffic showing the response, right?
On the em0: interface (ONT) here's some tcpdump output:
09:29:37.126874 00:90:d0:63:ff:01 (oui Unknown) > 01:80:c2:00:00:03 (oui Unknown), ethertype 802.1Q (0x8100), length 68: vlan 0, p 7, ethertype EAPOL, EAP packet (0) v1, len 15 09:30:02.284280 18:4a:6f:11:62:fd (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 64: vlan 0, p 7, ethertype ARP, Request who-has 107.x.x.x (c0:a0:0d:xx:xx:xx (oui Unknown)) tell 0.0.0.0, length 46 09:30:02.284291 c0:a0:0d:xx:xx:xx (oui Unknown) > 18:4a:6f:11:62:fd (oui Unknown), ethertype 802.1Q (0x8100), length 46: vlan 0, p 0, ethertype ARP, Reply 107.x.x.x is-at c0:a0:0d:77:xx:xx (oui Unknown), length 28 09:33:39.499764 00:90:d0:63:ff:01 (oui Unknown) > 01:80:c2:00:00:03 (oui Unknown), ethertype 802.1Q (0x8100), length 68: vlan 0, p 7, ethertype EAPOL, EAP packet (0) v1, len 4 09:33:39.499769 00:90:d0:63:ff:01 (oui Unknown) > 01:80:c2:00:00:03 (oui Unknown), ethertype 802.1Q (0x8100), length 68: vlan 0, p 7, ethertype EAPOL, EAP packet (0) v1, len 15
-
Quad port nic arrived today. Ready for more testing. While my understanding of netgraph is vague at best, I am able to follow directions well.
Several other questions;
-
In the course of testing this, what is the correct way to determine that EAP authentication is taking place and is successful? The gateway has a broadband light which when connected directly to the ONT, flashes initially then goes solid green. If there's an error it flashes red. Given that we're only passing EAP packets the gateway does not get a dhcp requests honored and won't be getting any ip's. There should be some other means to confirm eap packets are flowing in both directions.
-
Should the pfsense WAN port (em0, not the bridge) have its mac spoofed? The instructions on page 2 indicate to spoof the mac on the bridge to that of the gateway.
I think ideally we want to have vlan0 tags dropped for packets destined for the lan interface and added to those going to wan (ngeth0)?
Thanks!
-
-
Quad port nic arrived today. Ready for more testing. While my understanding of netgraph is vague at best, I am able to follow directions well.
Several other questions;
-
In the course of testing this, what is the correct way to determine that EAP authentication is taking place and is successful? The gateway has a broadband light which when connected directly to the ONT, flashes initially then goes solid green. If there's an error it flashes red. Given that we're only passing EAP packets the gateway does not get a dhcp requests honored and won't be getting any ip's. There should be some other means to confirm eap packets are flowing in both directions.
-
Should the pfsense WAN port (em0, not the bridge) have its mac spoofed? The instructions on page 2 indicate to spoof the mac on the bridge to that of the gateway.
I think ideally we want to have vlan0 tags dropped for packets destined for the lan interface and added to those going to wan (ngeth0)?
Thanks!
The first one is easy 8):
Step 8: Confirm authentication. After 1-2 minutes, you will see the “Broadband” light on your RG flash green and then go to solid green for a short period of time. This means that the 802.1X port authentication has completed successfully. However, your Broadband light will then start flashing read and then go blank. This is because the RG is not receiving an IP address from the ATT network via DHCP (your PFSense Box is attempting request and receive the IP address).
If you don't see it go solid green for a short period of time, then EAP-OL authentication failed. If you go to the status page for the RG, it will say something like status authenticated, but also indicate that the internet is "down" because you were unable to get an IP address via DHCP.
-
The only spoofing that should be done is the RG's WAN port. Specifically, the PFSense WAN port will be spoofing the address of the RG's WAN port. Failure to do this will result in a failure to authenticate, as the MAC address of the RG is verified by AT&T.
-
Ideally, the netgraph framework should be stripping the VID from all Ethernet frames on ingress and after EAP-OL filtering is done. The issue is that if the VLAN is not set to 0, the RG might not recognize the EAP response frames from the ONT. However, because FreeBSD cannot create an interface on VLAN 0, you will need to untag all of the frames before they leave the Netgraph and get passed back up to the kernel stack.
In other words, the Netgraph should do the following on frames coming from the ONT:
(1) Determine whether frame is EAP-OL.
(2) If frame is EAP-OL, then relay/forward unmodified frame to RG.
(3) If frame is not EAP-OL, strip VID from frame and sent modified frame to kernel stack for further processing by systemFor frames being sent to the ONT, we are essentially doing the reverse:
(1) If frame is EAP-OL frame from RG, forward unmodified frame to ONT
(2) If frame is any other frame from RG, drop
(3) If frame is from PFSense kernel, tag it as VLAN 0 and forward to ONT.Hope this makes sense.
-
-
Yes, makes perfect sense. I think #3 clearly defines netgraph inputs and desired outcomes.
For #2, the only spoofing then should be for the ngeth0 interface, not em0 (physical interface ONT is connected to)?
Rereading the thread a few more times I think AUS got around the vlan tags on the lan issue by defining a particular filter in proxmox. It seems like netgraph is fairly powerful tool for low level packet routing and this type of function may very well be possible. I wish I had a better understanding of netgraph fundamentals. It would make a whole lot more sense.
Your ascii pic in post https://forum.pfsense.org/index.php?topic=111043.msg798687#msg798687 paints a good rough draft. In fact, starting with a visual is probably the best way to attack this problem. Just need to fill in the details.
-
For #2, the only spoofing then should be for the ngeth0 interface, not em0 (physical interface ONT is connected to)?
Not quite. You're over-thinking this part. In PFSense, go to the settings for your WAN interface and enter in the MAC address you want to use (the RG's WAN interface). NG_eth read and writes packets using the parameters specified for the operating system. So if you override the MAC address of the WAN interface with the MAC address for the RG's WAN interface, ng_eth will pick up these changes and write frames using the spoofed MAC address. Does that make sense?
Rereading the thread a few more times I think AUS got around the vlan tags on the lan issue by defining a particular filter in proxmox.
That maybe. Unfortunately, I am completely unfamiliar with ProxMox. I was hoping that if he got this working with ProxMox that the solution could be easily ported to bare-metal. But we seem to be missing something.
It seems like netgraph is fairly powerful tool for low level packet routing and this type of function may very well be possible. I wish I had a better understanding of netgraph fundamentals. It would make a whole lot more sense.
I feel like I am staring over the edge of the abyss. NetGraph is so powerful that it is awe inspiring. Unfortunately, it is so infrequently used that the documentation is quite sparse. Thank God that we have jeli on this thread to guide us when we go astray. Seriously, he's one of the creators of Netgraph and if he doesn't know the answer, probably no one does.
Your ascii pic in post https://forum.pfsense.org/index.php?topic=111043.msg798687#msg798687 paints a good rough draft. In fact, starting with a visual is probably the best way to attack this problem. Just need to fill in the details.
:D
-
For #2, the only spoofing then should be for the ngeth0 interface, not em0 (physical interface ONT is connected to)?
Not quite. You're over-thinking this part. In PFSense, go to the settings for your WAN interface and enter in the MAC address you want to use (the RG's WAN interface). NG_eth read and writes packets using the parameters specified for the operating system. So if you override the MAC address of the WAN interface with the MAC address for the RG's WAN interface, ng_eth will pick up these changes and write frames using the spoofed MAC address. Does that make sense?
Rereading the thread a few more times I think AUS got around the vlan tags on the lan issue by defining a particular filter in proxmox.
That maybe. Unfortunately, I am completely unfamiliar with ProxMox. I was hoping that if he got this working with ProxMox that the solution could be easily ported to bare-metal. But we seem to be missing something.
It seems like netgraph is fairly powerful tool for low level packet routing and this type of function may very well be possible. I wish I had a better understanding of netgraph fundamentals. It would make a whole lot more sense.
I feel like I am staring over the edge of the abyss. NetGraph is so powerful that it is awe inspiring. Unfortunately, it is so infrequently used that the documentation is quite sparse. Thank God that we have jeli on this thread to guide us when we go astray. Seriously, he's one of the creators of Netgraph and if he doesn't know the answer, probably no one does.
Your ascii pic in post https://forum.pfsense.org/index.php?topic=111043.msg798687#msg798687 paints a good rough draft. In fact, starting with a visual is probably the best way to attack this problem. Just need to fill in the details.
:D
So if I am reading you right in the GUI we don’t change the WAN interface to ngeth and don’t change the MAC address to the RG since this is handled by net graph?