ATT Uverse RG Bypass (0.2 BTC)

GPz1100

@rajl As I understand it, the whole reason for doing the pci passthrough is because there's issue with passing vlan0 through the hypervisor? By configuring the nic as pci passthrough, the guest os (pfsense) has full access to the nic. Vlan0 is then passed through the graphing definition.

I've been testing further on the spare machine (bare metal pfsense install). I realized the reason for my errors above - I did not reattach the missing interfaces. Once I did that pfsense ssh accepted all the commands except those below.

I'm using the code from this post: https://forum.pfsense.org/index.php?topic=111043.msg793292#msg793292

# define filters for EAP traffic
ngctl msg waneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }'
ngctl msg laneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }'

# use filters to bridge EAP traffic
ngctl connect waneapfilter: laneapfilter: eapout eapout

I resolved this by issuing the connect command first - to establish eapout definition? Once done there were no further errors.

However, monitoring the RG interface with tcpdump, all I see are these. Mac c0:a0:0d:xx:xx:xx is that of the RG.


08:13:56.104742 c0:a0:0d:xx:xx:xx (oui Unknown) > 01:80:c2:00:00:03 (oui Unknown), ethertype EAPOL (0x888e), length 60: EAPOL start (1) v2, len 0

On the WAN interface (ngeth0), I see this. RG mac is spoofed to c0:a0:0d:xx:xx:xx for this interface.


09:06:56.718637 c0:a0:0d:xx:xx:xx (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 346: vlan 0, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from c0:a0:0d:xx:xx:xx (oui Unknown), length 300
09:08:24.668681 00:90:d0:63:ff:01 (oui Unknown) > 01:80:c2:00:00:03 (oui Unknown), ethertype 802.1Q (0x8100), length 68: vlan 0, p 7, ethertype EAPOL, EAP packet (0) v1, len 15
09:09:25.261889 00:90:d0:63:ff:01 (oui Unknown) > 01:80:c2:00:00:03 (oui Unknown), ethertype 802.1Q (0x8100), length 68: vlan 0, p 7, ethertype EAPOL, EAP packet (0) v1, len 4
09:09:25.261895 00:90:d0:63:ff:01 (oui Unknown) > 01:80:c2:00:00:03 (oui Unknown), ethertype 802.1Q (0x8100), length 68: vlan 0, p 7, ethertype EAPOL, EAP packet (0) v1, len 15
09:09:28.585333 c0:a0:0d:xx:xx:xx (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 346: vlan 0, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from c0:a0:0d:xx:xx:xx (oui Unknown), length 300
09:09:39.433119 c0:a0:0d:xx:xx:xx (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 346: vlan 0, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from c0:a0:0d:xx:xx:xx (oui Unknown), length 300
09:09:47.419730 18:4a:6f:11:62:fd (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 64: vlan 0, p 7, ethertype ARP, Request who-has 107.x.x.x (c0:a0:0d:xx:xx:xx (oui Unknown)) tell 0.0.0.0, length 46
09:09:50.440866 c0:a0:0d:xx:xx:xx (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 346: vlan 0, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from c0:a0:0d:xx:xx:xx (oui Unknown), length 300
09:09:52.419034 18:4a:6f:11:62:fd (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 64: vlan 0, p 7, ethertype ARP, Request who-has 107.x.x.x (c0:a0:0d:xx:xx:xx (oui Unknown)) tell 0.0.0.0, length 46
09:09:55.558504 00:90:d0:63:ff:01 (oui Unknown) > 01:80:c2:00:00:03 (oui Unknown), ethertype 802.1Q (0x8100), length 68: vlan 0, p 7, ethertype EAPOL, EAP packet (0) v1, len 15
09:09:57.418368 18:4a:6f:11:62:fd (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 64: vlan 0, p 7, ethertype ARP, Request who-has 107.x.x.x (c0:a0:0d:xx:xx:xx (oui Unknown)) tell 0.0.0.0, length 46
09:09:58.448730 c0:a0:0d:xx:xx:xx (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 346: vlan 0, p 0, ethertype IPv4, 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from c0:a0:0d:xx:xx:xx (oui Unknown), length 300

So it would seem that eapol requests are going out, but not coming back in? On the RG tcpdump there should be additional traffic showing the response, right?

On the em0: interface (ONT) here's some tcpdump output:


09:29:37.126874 00:90:d0:63:ff:01 (oui Unknown) > 01:80:c2:00:00:03 (oui Unknown), ethertype 802.1Q (0x8100), length 68: vlan 0, p 7, ethertype EAPOL, EAP packet (0) v1, len 15

09:30:02.284280 18:4a:6f:11:62:fd (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 64: vlan 0, p 7, ethertype ARP, Request who-has 107.x.x.x (c0:a0:0d:xx:xx:xx (oui Unknown)) tell 0.0.0.0, length 46
09:30:02.284291 c0:a0:0d:xx:xx:xx (oui Unknown) > 18:4a:6f:11:62:fd (oui Unknown), ethertype 802.1Q (0x8100), length 46: vlan 0, p 0, ethertype ARP, Reply 107.x.x.x is-at c0:a0:0d:77:xx:xx (oui Unknown), length 28
09:33:39.499764 00:90:d0:63:ff:01 (oui Unknown) > 01:80:c2:00:00:03 (oui Unknown), ethertype 802.1Q (0x8100), length 68: vlan 0, p 7, ethertype EAPOL, EAP packet (0) v1, len 4
09:33:39.499769 00:90:d0:63:ff:01 (oui Unknown) > 01:80:c2:00:00:03 (oui Unknown), ethertype 802.1Q (0x8100), length 68: vlan 0, p 7, ethertype EAPOL, EAP packet (0) v1, len 15

GPz1100

Quad port nic arrived today. Ready for more testing. While my understanding of netgraph is vague at best, I am able to follow directions well.

Several other questions;

In the course of testing this, what is the correct way to determine that EAP authentication is taking place and is successful? The gateway has a broadband light which when connected directly to the ONT, flashes initially then goes solid green. If there's an error it flashes red. Given that we're only passing EAP packets the gateway does not get a dhcp requests honored and won't be getting any ip's. There should be some other means to confirm eap packets are flowing in both directions.
Should the pfsense WAN port (em0, not the bridge) have its mac spoofed? The instructions on page 2 indicate to spoof the mac on the bridge to that of the gateway.

I think ideally we want to have vlan0 tags dropped for packets destined for the lan interface and added to those going to wan (ngeth0)?

Thanks!

rajl

@GPz1100:

Quad port nic arrived today. Ready for more testing. While my understanding of netgraph is vague at best, I am able to follow directions well.

Several other questions;

In the course of testing this, what is the correct way to determine that EAP authentication is taking place and is successful? The gateway has a broadband light which when connected directly to the ONT, flashes initially then goes solid green. If there's an error it flashes red. Given that we're only passing EAP packets the gateway does not get a dhcp requests honored and won't be getting any ip's. There should be some other means to confirm eap packets are flowing in both directions.

Should the pfsense WAN port (em0, not the bridge) have its mac spoofed? The instructions on page 2 indicate to spoof the mac on the bridge to that of the gateway.

I think ideally we want to have vlan0 tags dropped for packets destined for the lan interface and added to those going to wan (ngeth0)?

Thanks!

The first one is easy 8):

@rajl:

Step 8: Confirm authentication. After 1-2 minutes, you will see the “Broadband” light on your RG flash green and then go to solid green for a short period of time. This means that the 802.1X port authentication has completed successfully. However, your Broadband light will then start flashing read and then go blank. This is because the RG is not receiving an IP address from the ATT network via DHCP (your PFSense Box is attempting request and receive the IP address).

If you don't see it go solid green for a short period of time, then EAP-OL authentication failed. If you go to the status page for the RG, it will say something like status authenticated, but also indicate that the internet is "down" because you were unable to get an IP address via DHCP.

The only spoofing that should be done is the RG's WAN port. Specifically, the PFSense WAN port will be spoofing the address of the RG's WAN port. Failure to do this will result in a failure to authenticate, as the MAC address of the RG is verified by AT&T.
Ideally, the netgraph framework should be stripping the VID from all Ethernet frames on ingress and after EAP-OL filtering is done. The issue is that if the VLAN is not set to 0, the RG might not recognize the EAP response frames from the ONT. However, because FreeBSD cannot create an interface on VLAN 0, you will need to untag all of the frames before they leave the Netgraph and get passed back up to the kernel stack.

In other words, the Netgraph should do the following on frames coming from the ONT:
(1) Determine whether frame is EAP-OL.
(2) If frame is EAP-OL, then relay/forward unmodified frame to RG.
(3) If frame is not EAP-OL, strip VID from frame and sent modified frame to kernel stack for further processing by system

For frames being sent to the ONT, we are essentially doing the reverse:
(1) If frame is EAP-OL frame from RG, forward unmodified frame to ONT
(2) If frame is any other frame from RG, drop
(3) If frame is from PFSense kernel, tag it as VLAN 0 and forward to ONT.

Hope this makes sense.

GPz1100

Yes, makes perfect sense. I think #3 clearly defines netgraph inputs and desired outcomes.

For #2, the only spoofing then should be for the ngeth0 interface, not em0 (physical interface ONT is connected to)?

Rereading the thread a few more times I think AUS got around the vlan tags on the lan issue by defining a particular filter in proxmox. It seems like netgraph is fairly powerful tool for low level packet routing and this type of function may very well be possible. I wish I had a better understanding of netgraph fundamentals. It would make a whole lot more sense.

Your ascii pic in post https://forum.pfsense.org/index.php?topic=111043.msg798687#msg798687 paints a good rough draft. In fact, starting with a visual is probably the best way to attack this problem. Just need to fill in the details.

rajl

@GPz1100:

For #2, the only spoofing then should be for the ngeth0 interface, not em0 (physical interface ONT is connected to)?

Not quite. You're over-thinking this part. In PFSense, go to the settings for your WAN interface and enter in the MAC address you want to use (the RG's WAN interface). NG_eth read and writes packets using the parameters specified for the operating system. So if you override the MAC address of the WAN interface with the MAC address for the RG's WAN interface, ng_eth will pick up these changes and write frames using the spoofed MAC address. Does that make sense?

@GPz1100:

Rereading the thread a few more times I think AUS got around the vlan tags on the lan issue by defining a particular filter in proxmox.

That maybe. Unfortunately, I am completely unfamiliar with ProxMox. I was hoping that if he got this working with ProxMox that the solution could be easily ported to bare-metal. But we seem to be missing something.

@GPz1100:

It seems like netgraph is fairly powerful tool for low level packet routing and this type of function may very well be possible. I wish I had a better understanding of netgraph fundamentals. It would make a whole lot more sense.

I feel like I am staring over the edge of the abyss. NetGraph is so powerful that it is awe inspiring. Unfortunately, it is so infrequently used that the documentation is quite sparse. Thank God that we have jeli on this thread to guide us when we go astray. Seriously, he's one of the creators of Netgraph and if he doesn't know the answer, probably no one does.

@GPz1100:

Your ascii pic in post https://forum.pfsense.org/index.php?topic=111043.msg798687#msg798687 paints a good rough draft. In fact, starting with a visual is probably the best way to attack this problem. Just need to fill in the details.

:D

pyrodex

@rajl:

@GPz1100:

For #2, the only spoofing then should be for the ngeth0 interface, not em0 (physical interface ONT is connected to)?

Not quite. You're over-thinking this part. In PFSense, go to the settings for your WAN interface and enter in the MAC address you want to use (the RG's WAN interface). NG_eth read and writes packets using the parameters specified for the operating system. So if you override the MAC address of the WAN interface with the MAC address for the RG's WAN interface, ng_eth will pick up these changes and write frames using the spoofed MAC address. Does that make sense?

@GPz1100:

Rereading the thread a few more times I think AUS got around the vlan tags on the lan issue by defining a particular filter in proxmox.

That maybe. Unfortunately, I am completely unfamiliar with ProxMox. I was hoping that if he got this working with ProxMox that the solution could be easily ported to bare-metal. But we seem to be missing something.

@GPz1100:

It seems like netgraph is fairly powerful tool for low level packet routing and this type of function may very well be possible. I wish I had a better understanding of netgraph fundamentals. It would make a whole lot more sense.

I feel like I am staring over the edge of the abyss. NetGraph is so powerful that it is awe inspiring. Unfortunately, it is so infrequently used that the documentation is quite sparse. Thank God that we have jeli on this thread to guide us when we go astray. Seriously, he's one of the creators of Netgraph and if he doesn't know the answer, probably no one does.

@GPz1100:

Your ascii pic in post https://forum.pfsense.org/index.php?topic=111043.msg798687#msg798687 paints a good rough draft. In fact, starting with a visual is probably the best way to attack this problem. Just need to fill in the details.

:D

So if I am reading you right in the GUI we don’t change the WAN interface to ngeth and don’t change the MAC address to the RG since this is handled by net graph?

rajl

@pyrodex:

So if I am reading you right in the GUI we don’t change the WAN interface to ngeth and don’t change the MAC address to the RG since this is handled by net graph?

Half correct. In the PFSense GUI, you set the WAN interface to whatever you want (e.g., WAN = em0, LAN = em1, RG=em2). In the netgraph script, you set the ng_eth node representing the WAN to whatever interface you specified in the PFSense GUI. In the PFSense GUI, you set the value for the MAC address equal to the MAC address of the RG's WAN interface (i.e., you're having the PFSense WAN spoof the MAC address of the RG instead of using its own MAC address).

Everything else is handled by Netgraph at layer 2 (Ethernet frames). Since PFSense mostly operates at layers 3 and 4, it will be oblivious to what Netgraph is doing. As far as PFSense is concerned, it writes packets to and receives packets from the network interface. Once the network interface receives the packets, Netgraph sorts and processes them.

pyrodex

@rajl:

@pyrodex:

So if I am reading you right in the GUI we don’t change the WAN interface to ngeth and don’t change the MAC address to the RG since this is handled by net graph?

Half correct. In the PFSense GUI, you set the WAN interface to whatever you want (e.g., WAN = em0, LAN = em1, RG=em2). In the netgraph script, you set the ng_eth node representing the WAN to whatever interface you specified in the PFSense GUI. In the PFSense GUI, you set the value for the MAC address equal to the MAC address of the RG's WAN interface (i.e., you're having the PFSense WAN spoof the MAC address of the RG instead of using its own MAC address).

Everything else is handled by Netgraph at layer 2 (Ethernet frames). Since PFSense mostly operates at layers 3 and 4, it will be oblivious to what Netgraph is doing. As far as PFSense is concerned, it writes packets to and receives packets from the network interface. Once the network interface receives the packets, Netgraph sorts and processes them.

This could have been the issue I had when I tried on my physical box. I changed the WAN interface to ng_eth and never got access fully… I can try again on Sunday while the wife is at work and I'll do the following.


#igb2 is connected to the ONT
#lagg0 is connected to the LAN
#igb3 is connected to the RG
#ngeth0 is the VLANed device which is configured as my WAN in pfSense
# copy and load ng_etf kernel module

/sbin/kldload /boot/kernel/ng_etf.ko

#
# setup netgraph nodes
#

# list out netgraph nodes

/usr/sbin/ngctl list

# pfSense for some reason detaches ether devices. reattach any missing devices.

php -r 'pfSense_ngctl_attach(".", "igb0");'

# create tee node to split ONT traffic (one for EAP, one for VLAN0)
/usr/sbin/ngctl mkpeer igb0: tee lower left # may get a warning
/usr/sbin/ngctl name igb0:lower T

# create vlan node + eiface
/usr/sbin/ngctl mkpeer T: vlan right downstream
/usr/sbin/ngctl name T:right vlan0
/usr/sbin/ngctl mkpeer vlan0: eiface vlan0 ether
/usr/sbin/ngctl msg vlan0: 'addfilter { vlan=0 hook="vlan0" }'

# create etf and connect to ONT
/usr/sbin/ngctl mkpeer T: etf left2right downstream
/usr/sbin/ngctl name T:left2right waneapfilter
/usr/sbin/ngctl connect waneapfilter: igb0: nomatch upper

# create etf and connect to RG
/usr/sbin/ngctl mkpeer igb3: etf lower downstream
/usr/sbin/ngctl name igb3:lower laneapfilter
/usr/sbin/ngctl connect laneapfilter: igb3: nomatch upper

# define filters for EAP traffic
/usr/sbin/ngctl msg waneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }'
/usr/sbin/ngctl msg laneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }'

# use filters to bridge EAP traffic
/usr/sbin/ngctl connect waneapfilter: laneapfilter: eapout eapout

# change MAC address to match RG (also can be done in pfSense)
ifconfig ngeth0 ether <mac></mac>

Does this syntax look ok?

GPz1100

Unless I'm misunderstanding, there's still a puzzle piece (or 2) missing.

What connects the wan to the lan? Ok, maybe masquerading in pfsense.
More importantly wan frames are still tagged with vlan0. The missing piece (netgraph filter) strips these tags when traffic is routed anywhere else but the gateway.

GPz1100

Using some obscure search terms I can't even remember at this point I came across this. It's greek to me but might make more sense to others. Look for the posts by toast0.

https://news.ycombinator.com/item?id=16741660 which links to

http://ruka.org/~toast/steal/

Some missing .h files ( pcap-int.h & portability.h ) found here :
https://github.com/freebsd/freebsd/tree/master/contrib/libpcap

jeli

Ok, so firstly here is a pretty good introduction to using netgraph:

https://people.freebsd.org/~julian/netgraph.html

Since I don't have one of these I can't experiment, but it seems we have a number of things we need to do, and a number of things to keep in mind.

traffic between the RG and ONT is all on vlan 0, including real user traffic.
since vlan 0 is a reserved value, it is not certain that the hardware interfaces (NIC) can generate them, and in any case different people have different interfaces so we can't assume anything.
see this https://en.wikipedia.org/wiki/IEEE_802.1Q
the etf (ether type filter) node was not written to take vlan tags into account. Since vlan tags include an ether type of 0x8100, it will stop the etf node from seeing the eap etherype.

so maybe this is what we need, with HW vlan disabled and the interfaces in promiscuous mode. Also with the MAC of the main interface set to spoof that of the RG.

ONT)–-[ngiface]lower–--vlan0[ngvlan]downstream–--downstream[ngetf]nomatch–-----upper[ngiface]
|
RG)–- [ngiface]lower–--vlan0[ngvlan]downstream–-------------/

jeli

oops in the above the interfaces should be ngether not ngiface.. that's a completely different node type.

pyrodex

Any luck on a physical system? I was swamped this past weekend with yard work and will attempt Sunday since it will be raining.

GPz1100

I was unable to get anywhere on a pfsense system but was semi successful on opnsense with this -

https://news.ycombinator.com/item?id=16740694
http://ruka.org/~toast/steal/

Several notes:

steal_util.c has to be modified to reflect actual interface names. There's a total of 4 places to update, 2 for each interface. As the code comes, em1 refers to the ONT, em0 refers to the RGW
ngeth0 is defined as the wan interface in opnsense
elsewhere in the code update the static ip to your own att public ip. Do not touch the 172.xxx lines
I also had to define the rgw mac and ip address in the ngeth0 interface in opnsense

If you're familiar with freebsd, this can be set up on that too, as a novice, I found opnsense easier to configure.

Main issues I ran into were speed. I wasn't able to pull full bandwidth like I could before. My per connection speed dropped to about 200 mbps where as before I could top it out at ~700-800 mbps. It wasn't a cpu issue. I5 4590, top showed a single core at most at ~30% with enough threads to saturate, 200 mbps was barely 5-6%.

I did not write this code but came across it after finding the above forums posts. Maybe it can be made to work with pfsense too.

Good luck!

P.S. In the end I went a different direction

https://forum.opnsense.org/index.php?topic=7298.msg37970#msg37970

snelly

Hi all. I'm new to this thread but I've been working on this a long time. My goal has been to implement eap_proxy.py on PFSense. If you look at eap_proxy, the problem is that it uses PF_RING to sniff the packets, which is not available on FreeBSD. To get around this, I started working on an implementation using libpcap, which does work on FreeBSD/PFSense.

However, libpcap has very poor performance and wouldn't be a good solution for an embedded router on a fiber AT&T connection. I did a bunch of digging and discovered a project, netmap-libpcap (https://github.com/luigirizzo/netmap-libpcap/) that has integrated the very fast netmap filter into libpcap. This software is capable of sniffing packets on 10 Gbe, so it sounded like a better fit.

I code in Go and so my first effort was to write a simple sniffer that looked for ethernet type 0x888E (EAP) packets and printed them out whenever it found one. This wasn't too hard and I produced a binary that worked on Linux and successfully printed EAP packets coming from the ONT. The next challenge was to get it working on FreeBSD/PFSense. This was a lot tougher but I was ultimately successful. I did this by building netmap-libpcap on a FreeBSD VM, then compiling my Go program on that VM and statically linking it to the netmap-patched libpcap. I then copied this binary to the PFSense box and I'll be damned, it works. Well, sort of. Read on…

I was able to get my statically compiled, super fast sniffer running on PFSense and it can sniff all kinds of packets but I was not able to sniff the EAP packets. This is due to the VLAN ID 0 issues that have been discussed at length in this thread. I'm stuck. I don't see any EAP packets coming through and need to figure out how to strip the VLAN ID. I really don't want a solution that requires a custom kernel for every PFSense release.

If I can solve this problem, I feel confident that I can write a very fast and CPU-friendly eap_proxy clone for PFSense.

Can someone catch me up with where we are on VLAN 0 tagged frames?

GPz1100

@snelly Given my results with the dumb switch, I don't think vlan 0 plays much of a role if any.

Are you doing your testing in a VM or bare metal? If the former, make sure the nics (or vswitches) are in promiscuous mode. On my esxi setup, even with the dumb switch, I had to enable the following to make the mac spoofing work. I don't have promiscuous mode on because there's no eap traffic.

https://i.imgur.com/AactcPF.png

snelly

I'm doing my work on a Netgate 8-port router running PFSense. The dumb switch is an interesting idea but I wonder if they can really handle bidirectional, line-rate gigabit traffic flows.

bulldog5

How long does the dumb switch method work for before it wants a reauth?

pyrodex

@bulldog5:

How long does the dumb switch method work for before it wants a reauth?

Mine work with a TPL2000 switch for about a year and I think my switch is failing personally since I started experiencing packet loss issues. I did the IP passthrough method on my BGW210 and seeing if that resolves the issues before moving back over to another switch or moving the ports around.

nathanbradshaw

I've been following this thread with a lot of interest - I also have AT&T fiber 1gbit/1gbit and would love to bypass the RG unit with pfSense hardware.

I agree we're getting very close, and the remaining issue is VLAN0 support under FreeBSD.

I'm not capable of doing much dev in this area, although I have a Netgate SG-3100 and would be happy to assist with testing.

I also have an older Netgate APU4 pfSense hardware router. If it is useful to anyone working on this solution, I would be happy to mail it to you - just reply or email me.