ATT Uverse RG Bypass (0.2 BTC)

rajl

So if I am reading you right in the GUI we don’t change the WAN interface to ngeth and don’t change the MAC address to the RG since this is handled by net graph?

Half correct. In the PFSense GUI, you set the WAN interface to whatever you want (e.g., WAN = em0, LAN = em1, RG=em2). In the netgraph script, you set the ng_eth node representing the WAN to whatever interface you specified in the PFSense GUI. In the PFSense GUI, you set the value for the MAC address equal to the MAC address of the RG's WAN interface (i.e., you're having the PFSense WAN spoof the MAC address of the RG instead of using its own MAC address).

Everything else is handled by Netgraph at layer 2 (Ethernet frames). Since PFSense mostly operates at layers 3 and 4, it will be oblivious to what Netgraph is doing. As far as PFSense is concerned, it writes packets to and receives packets from the network interface. Once the network interface receives the packets, Netgraph sorts and processes them.

pyrodex

@rajl:

@pyrodex:

So if I am reading you right in the GUI we don’t change the WAN interface to ngeth and don’t change the MAC address to the RG since this is handled by net graph?

Half correct. In the PFSense GUI, you set the WAN interface to whatever you want (e.g., WAN = em0, LAN = em1, RG=em2). In the netgraph script, you set the ng_eth node representing the WAN to whatever interface you specified in the PFSense GUI. In the PFSense GUI, you set the value for the MAC address equal to the MAC address of the RG's WAN interface (i.e., you're having the PFSense WAN spoof the MAC address of the RG instead of using its own MAC address).

Everything else is handled by Netgraph at layer 2 (Ethernet frames). Since PFSense mostly operates at layers 3 and 4, it will be oblivious to what Netgraph is doing. As far as PFSense is concerned, it writes packets to and receives packets from the network interface. Once the network interface receives the packets, Netgraph sorts and processes them.

This could have been the issue I had when I tried on my physical box. I changed the WAN interface to ng_eth and never got access fully… I can try again on Sunday while the wife is at work and I'll do the following.


#igb2 is connected to the ONT
#lagg0 is connected to the LAN
#igb3 is connected to the RG
#ngeth0 is the VLANed device which is configured as my WAN in pfSense
# copy and load ng_etf kernel module

/sbin/kldload /boot/kernel/ng_etf.ko

#
# setup netgraph nodes
#

# list out netgraph nodes

/usr/sbin/ngctl list

# pfSense for some reason detaches ether devices. reattach any missing devices.

php -r 'pfSense_ngctl_attach(".", "igb0");'

# create tee node to split ONT traffic (one for EAP, one for VLAN0)
/usr/sbin/ngctl mkpeer igb0: tee lower left # may get a warning
/usr/sbin/ngctl name igb0:lower T

# create vlan node + eiface
/usr/sbin/ngctl mkpeer T: vlan right downstream
/usr/sbin/ngctl name T:right vlan0
/usr/sbin/ngctl mkpeer vlan0: eiface vlan0 ether
/usr/sbin/ngctl msg vlan0: 'addfilter { vlan=0 hook="vlan0" }'

# create etf and connect to ONT
/usr/sbin/ngctl mkpeer T: etf left2right downstream
/usr/sbin/ngctl name T:left2right waneapfilter
/usr/sbin/ngctl connect waneapfilter: igb0: nomatch upper

# create etf and connect to RG
/usr/sbin/ngctl mkpeer igb3: etf lower downstream
/usr/sbin/ngctl name igb3:lower laneapfilter
/usr/sbin/ngctl connect laneapfilter: igb3: nomatch upper

# define filters for EAP traffic
/usr/sbin/ngctl msg waneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }'
/usr/sbin/ngctl msg laneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }'

# use filters to bridge EAP traffic
/usr/sbin/ngctl connect waneapfilter: laneapfilter: eapout eapout

# change MAC address to match RG (also can be done in pfSense)
ifconfig ngeth0 ether <mac></mac>

Does this syntax look ok?

GPz1100

Unless I'm misunderstanding, there's still a puzzle piece (or 2) missing.

What connects the wan to the lan? Ok, maybe masquerading in pfsense.
More importantly wan frames are still tagged with vlan0. The missing piece (netgraph filter) strips these tags when traffic is routed anywhere else but the gateway.

GPz1100

Using some obscure search terms I can't even remember at this point I came across this. It's greek to me but might make more sense to others. Look for the posts by toast0.

https://news.ycombinator.com/item?id=16741660 which links to

http://ruka.org/~toast/steal/

Some missing .h files ( pcap-int.h & portability.h ) found here :
https://github.com/freebsd/freebsd/tree/master/contrib/libpcap

jeli

Ok, so firstly here is a pretty good introduction to using netgraph:

https://people.freebsd.org/~julian/netgraph.html

Since I don't have one of these I can't experiment, but it seems we have a number of things we need to do, and a number of things to keep in mind.

traffic between the RG and ONT is all on vlan 0, including real user traffic.
since vlan 0 is a reserved value, it is not certain that the hardware interfaces (NIC) can generate them, and in any case different people have different interfaces so we can't assume anything.
see this https://en.wikipedia.org/wiki/IEEE_802.1Q
the etf (ether type filter) node was not written to take vlan tags into account. Since vlan tags include an ether type of 0x8100, it will stop the etf node from seeing the eap etherype.

so maybe this is what we need, with HW vlan disabled and the interfaces in promiscuous mode. Also with the MAC of the main interface set to spoof that of the RG.

ONT)–-[ngiface]lower–--vlan0[ngvlan]downstream–--downstream[ngetf]nomatch–-----upper[ngiface]
|
RG)–- [ngiface]lower–--vlan0[ngvlan]downstream–-------------/

jeli

oops in the above the interfaces should be ngether not ngiface.. that's a completely different node type.

pyrodex

Any luck on a physical system? I was swamped this past weekend with yard work and will attempt Sunday since it will be raining.

GPz1100

I was unable to get anywhere on a pfsense system but was semi successful on opnsense with this -

https://news.ycombinator.com/item?id=16740694
http://ruka.org/~toast/steal/

Several notes:

steal_util.c has to be modified to reflect actual interface names. There's a total of 4 places to update, 2 for each interface. As the code comes, em1 refers to the ONT, em0 refers to the RGW
ngeth0 is defined as the wan interface in opnsense
elsewhere in the code update the static ip to your own att public ip. Do not touch the 172.xxx lines
I also had to define the rgw mac and ip address in the ngeth0 interface in opnsense

If you're familiar with freebsd, this can be set up on that too, as a novice, I found opnsense easier to configure.

Main issues I ran into were speed. I wasn't able to pull full bandwidth like I could before. My per connection speed dropped to about 200 mbps where as before I could top it out at ~700-800 mbps. It wasn't a cpu issue. I5 4590, top showed a single core at most at ~30% with enough threads to saturate, 200 mbps was barely 5-6%.

I did not write this code but came across it after finding the above forums posts. Maybe it can be made to work with pfsense too.

Good luck!

P.S. In the end I went a different direction

https://forum.opnsense.org/index.php?topic=7298.msg37970#msg37970

snelly

Hi all. I'm new to this thread but I've been working on this a long time. My goal has been to implement eap_proxy.py on PFSense. If you look at eap_proxy, the problem is that it uses PF_RING to sniff the packets, which is not available on FreeBSD. To get around this, I started working on an implementation using libpcap, which does work on FreeBSD/PFSense.

However, libpcap has very poor performance and wouldn't be a good solution for an embedded router on a fiber AT&T connection. I did a bunch of digging and discovered a project, netmap-libpcap (https://github.com/luigirizzo/netmap-libpcap/) that has integrated the very fast netmap filter into libpcap. This software is capable of sniffing packets on 10 Gbe, so it sounded like a better fit.

I code in Go and so my first effort was to write a simple sniffer that looked for ethernet type 0x888E (EAP) packets and printed them out whenever it found one. This wasn't too hard and I produced a binary that worked on Linux and successfully printed EAP packets coming from the ONT. The next challenge was to get it working on FreeBSD/PFSense. This was a lot tougher but I was ultimately successful. I did this by building netmap-libpcap on a FreeBSD VM, then compiling my Go program on that VM and statically linking it to the netmap-patched libpcap. I then copied this binary to the PFSense box and I'll be damned, it works. Well, sort of. Read on…

I was able to get my statically compiled, super fast sniffer running on PFSense and it can sniff all kinds of packets but I was not able to sniff the EAP packets. This is due to the VLAN ID 0 issues that have been discussed at length in this thread. I'm stuck. I don't see any EAP packets coming through and need to figure out how to strip the VLAN ID. I really don't want a solution that requires a custom kernel for every PFSense release.

If I can solve this problem, I feel confident that I can write a very fast and CPU-friendly eap_proxy clone for PFSense.

Can someone catch me up with where we are on VLAN 0 tagged frames?

GPz1100

@snelly Given my results with the dumb switch, I don't think vlan 0 plays much of a role if any.

Are you doing your testing in a VM or bare metal? If the former, make sure the nics (or vswitches) are in promiscuous mode. On my esxi setup, even with the dumb switch, I had to enable the following to make the mac spoofing work. I don't have promiscuous mode on because there's no eap traffic.

https://i.imgur.com/AactcPF.png

snelly

I'm doing my work on a Netgate 8-port router running PFSense. The dumb switch is an interesting idea but I wonder if they can really handle bidirectional, line-rate gigabit traffic flows.

bulldog5

How long does the dumb switch method work for before it wants a reauth?

pyrodex

@bulldog5:

How long does the dumb switch method work for before it wants a reauth?

Mine work with a TPL2000 switch for about a year and I think my switch is failing personally since I started experiencing packet loss issues. I did the IP passthrough method on my BGW210 and seeing if that resolves the issues before moving back over to another switch or moving the ports around.

nathanbradshaw

I've been following this thread with a lot of interest - I also have AT&T fiber 1gbit/1gbit and would love to bypass the RG unit with pfSense hardware.

I agree we're getting very close, and the remaining issue is VLAN0 support under FreeBSD.

I'm not capable of doing much dev in this area, although I have a Netgate SG-3100 and would be happy to assist with testing.

I also have an older Netgate APU4 pfSense hardware router. If it is useful to anyone working on this solution, I would be happy to mail it to you - just reply or email me.

bulldog5

@nathanbradshaw:

I've been following this thread with a lot of interest - I also have AT&T fiber 1gbit/1gbit and would love to bypass the RG unit with pfSense hardware.

I agree we're getting very close, and the remaining issue is VLAN0 support under FreeBSD.

I'm not capable of doing much dev in this area, although I have a Netgate SG-3100 and would be happy to assist with testing.

I also have an older Netgate APU4 pfSense hardware router. If it is useful to anyone working on this solution, I would be happy to mail it to you - just reply or email me.

I just tried the dumb switch method this evening and got it working with netgear gs105. However, I had to statically assign my IP to the pfsense WAN, it wouldn't pull anything with DHCP. What am I doing wrong? I'm assuming when the lease expires its going to try to re-auth and i'll lose the connection? My main switch is a procurve 2800 48port. I tried Tagging 3 ports with the same VLAN ID and the modem wouldn't AUTH at all going through there.

GPz1100

^^Try it with a basic dumb switch.

bulldog5

@GPz1100:

^^Try it with a basic dumb switch.

I did? a netgear GS105 is an unmanaged dumb switch.

GPz1100

Strange. I've tested this successfully with a cheap dlink 5 port switch and an asus rt-ac68u. What color is the broadband light flashing on the rgw?

bulldog5

@GPz1100:

Strange. I've tested this successfully with a cheap dlink 5 port switch and an asus rt-ac68u. What color is the broadband light flashing on the rgw?

I set my pfsense wan interface MAC to the same as the rgw and left as DHCP. I plug the rgw and ONT in the netgear switch. Let the broadband light go solid green, then unplug the RGW and plug the pfsense WAN nic in. pfsense WAN stays 0.0.0.0, never pulled an IP. So i tried it a 2nd time but used Static and that worked.

GPz1100

I haven't implemented pfsense yet, but under sophos utm it pulls an ip via dhcp within seconds.

Maybe pfsense has some advance options for dhcp that need to be adjusted?

Is your pfsense running baremetal or as a vm?