CenturyLink, PPPoE, a static /29 subnet, without 1:1 NAT
-
Hello everybody,
I have CenturyLink fiber service and it uses PPPoE like their DSL service. Along with this, I have a /29 static IP block that, using example IP addresses, has a network of 9.20.30.16/29 (so usable range is 17 through 21) and gateway of 9.20.30.22. The .22 address is assigned to the PPPoE link when it is established.
What I would like to do is be able to have three interfaces with pfSense. WAN would be where my PFsense device connects to my ONT; LAN would be where 192.168.1.0/24 clients are NAT'd out through a single IP address from my block; DMZ would have machines that can have public addresses assigned directly to them (so a machine connected to DMZ would show an IP address of, say, 9.20.30.17 when I type ifconfig). 1:1 NAT is not my preferred outcome.
I've gotten WAN and LAN to work. I have my PPPoE credentials and enter them into pfSense and it connects and LAN devices get an IP and can do their thing. But I'm running into a problem with the DMZ interface where I can't get any routing to work. The .22 address, which is also the gateway address for any of the other public IPs, is only assigned to the PPPoE link not the physical WAN interface itself (so pfSense substitutes the PPPoE virtual interface as "WAN" when PPP connects).
My troubleshooting took me into my CenturyLink-provided router and I noticed something very strange: br0 (the LAN bridge on my CL router that connects the 4 physical LAN-labeled ports together) has a secondary IP of 9.20.30.22, which is also assigned to the ppp0 interface. Look:
# ifconfig br0 Link encap:Ethernet HWaddr 12:33:99:00:11:22 inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING ALLMULTI MULTICAST MTU:1500 Metric:1 br0:0 Link encap:Ethernet HWaddr 12:33:99:00:11:22 inet addr:9.20.30.22 Bcast:209.180.203.23 Mask:255.255.255.248 UP BROADCAST RUNNING ALLMULTI MULTICAST MTU:1500 Metric:1 ppp0.1 Link encap:Point-to-Point Protocol inet addr:9.20.30.22 P-t-P:1.2.3.68 Mask:255.255.255.255 UP POINTOPOINT RUNNING NOARP ALLMULTI MULTICAST MTU:1492 Metric:1
(I anonymized the MAC address and deleted the lines about packet statistics.)
Does this make any sense? I've been beating my head against this off and on for a month and haven't figured it out so I'm really hoping one of y'all knows what I am doing wrong. :D
-
I have not seen this work with PPPoE, but you could maybe bridge DMZ to WAN but that's a bit ugly (as all bridging is ugly).
There would not be any way to have both WAN and DMZ on the same subnet with IP addresses in the same subnet assigned to both. Set DMZ to have no IP address, bridge it to WAN, and then set your DMZ hosts to use the upstream gateway.
If you kept the centurylink device doing PPPoE and not pfSense then it may be more likely to work, though you'd lose one IP address out of the block since both pfSense and the centurylink box would both be using an IP address.
-
Apologies for the Necro, but seeing as it was the exact same issue I'm having (down to the ISP and service), I felt it was warranted.
First, to the OP: Did you ever find a solution to this? I really don't want to use the consumer grade Zyxel C2100t modem to provide this functionality.
Second, the relevant info for my connection (ip's/mac's changed to protect the innocent):
routed subnet: 64.101.146.144/29
upstream gw: 52.120.09.57
pfSense WAN: 64.101.146.150It seems that CenturyLink takes the last usable IP in the block and assigns it as the gateway and serves it up as the client side of the PPPoE connection. For example, my block was 64.101.146.144/29, where .144 would be the network, .145-.150 would be the usable IP's, and .151 would be the broadcast. CenturyLink assigned my side of the PPPoE connection 64.101.146.150/32, which as the OP noted pfSense assigned to the WAN interface of the router.
Following jimp's suggestion, I tried bridging my WAN and DMZ port. I created basic allow any traffic rules between them, assigned a static IP of 64.101.146.145/29 and using the 52.120.09.57 gw to a machine connected to the DMZ interface. This machine was unable to ping anything and a traceroute would just show it stalling out on the first hop, unable to reach the 52.120.09.57 gw. (I also tried using the pfsense WAN IP as a gw, with the same result)
Out of curiosity, using a machine connected to my lan segment, I tried pinging the machine connected to the DMZ. I then tried a traceroute, which showed the following:
Tracing route to 64.101.146.145 over a maximum of 30 hops 1 <1 ms <1 ms <1 ms 10.11.12.1 2 1 ms 1 ms 1 ms 52.120.09.57 3 8 ms 1 ms 1 ms 64.101.146.150 4 * * * Request timed out.
This shows the trace going to the LAN side of the pfSense box, out to the upstream gateway, which then correctly routed it back to the pfSense box, where it stopped. Assuming it was a routing issue, I looked at the routing table on the pfSense box and there was no entry for the 64.101.146.144/29 subnet.
Then, like the OP, I found a CenturyLink modem and inserted it between the pfSense box and the ONT. I configured it for PPPoE, selected static IP, and it had a dropdown selection available to select between Single Static IP and Block of Static IP Addresses.
Here's the resultant screen, to which I input 64.101.146.150 as gw, with a 255.255.255.248 subnet, and selected Public LAN for the DHCP. I then connected my pfSense router to a small switch attached the lan port on the modem, manually assigned 64.101.146.149/29 with 64.101.146.150 as it's gw, and everything works as expected. I was also able to connect other hosts to the switch and assign them addresses within the subnet successfully.
Again, like the OP, I too ssh'd into the modem to see what was going on under the hood. Unfortunately, it had a very limited command set and few tools, but I was able to pull up the interfaces and show the routing table:
> ifconfig bcmsw Link encap:Ethernet HWaddr 63:9A:E7:D8:02:3D UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 br0 Link encap:Ethernet HWaddr 63:9A:E7:D8:02:3D inet addr:192.168.0.1 Bcast:192.168.0.255 Mask:255.255.255.0 UP BROADCAST RUNNING ALLMULTI MULTICAST MTU:1500 Metric:1 br0:0 Link encap:Ethernet HWaddr 63:9A:E7:D8:02:3D inet addr:64.101.146.150 Bcast:64.101.146.151 Mask:255.255.255.248 UP BROADCAST RUNNING ALLMULTI MULTICAST MTU:1500 Metric:1 eth0 Link encap:Ethernet HWaddr 63:9A:E7:D8:02:3D inet6 addr: fe80::5a8b:f3ff:fec7:134e/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 eth1 Link encap:Ethernet HWaddr 63:9A:E7:D8:02:3D UP BROADCAST MULTICAST MTU:1500 Metric:1 eth2 Link encap:Ethernet HWaddr 63:9A:E7:D8:02:3D UP BROADCAST MULTICAST MTU:1500 Metric:1 eth3 Link encap:Ethernet HWaddr 63:9A:E7:D8:02:3D UP BROADCAST MULTICAST MTU:1500 Metric:1 eth4 Link encap:Ethernet HWaddr 63:9A:E7:D8:02:3D inet6 addr: fe80::5a8b:f3ff:fec7:134e/64 Scope:Link UP BROADCAST RUNNING ALLMULTI MULTICAST MTU:1500 Metric:1 eth4.1 Link encap:Ethernet HWaddr 63:9A:E7:D8:02:49 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 ppp1.1 Link encap:Point-to-Point Protocol inet addr:64.101.146.150 P-t-P:52.120.09.57 Mask:255.255.255.255 UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1492 Metric:1
> route show Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 52.120.09.57 * 255.255.255.255 UH 0 0 0 ppp1.1 64.101.146.144 * 255.255.255.248 U 0 0 0 br0 192.168.0.0 * 255.255.255.0 U 0 0 0 br0 default * 0.0.0.0 U 0 0 0 ppp1.1
While it works, I would really prefer to remove that consumer grade piece of hardware and additional single point of failure and move this functionality to my router/firewall if at all possible. Also, this modem doesn't seem capable of handling the full speed of my connection. When I have pfSense connected directly to the ONT, I get about 900mb down and 930mb up. With the modem, the best I've seen is 695mb either direction. (First World problems, I know :) )
Based on the output and naming conventions for interfaces, it seems this device is using some form of embedded linux, and as such, it is my hope that the principles used on the modem will translate to pfSense and the underlying FreeBSD.
-
I believe I have been able to reproduce on pfSense the effect of the "Public LAN Subnet" configuration in CenturyLink's firmware.
My static IP block is 67.XXX.YYY.40/29. This network breaks down as follows:
67.XXX.YYY.40 - Reserved Network Address
67.XXX.YYY.41 - 67.XXX.YYY.45 - User-Assignable
67.XXX.YYY.46 - Reserved Gateway
67.XXX.YYY.47 - BroadcastWhen using the CenturyLink modem, the public address is assigned as 67.XXX.YYY.46. This is consistent on pfSense–when initiating the PPPoE connection, the pfSense WAN interface is automatically assigned 67.XXX.YYY.46.
If you ssh into the CenturyLink modem you can obtain the following:
Interfaces:
# ifconfig <--snip--> br0 Link encap:Ethernet HWaddr 9C:1E:95:64:37:40 inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::9e1e:95ff:fe64:3740/64 Scope:Link UP BROADCAST RUNNING ALLMULTI MULTICAST MTU:1500 Metric:1 RX packets:63054895 multicast:1459543 unicast:60717936 broadcast:877416 RX errors:0 dropped:0 overruns:0 frame:0 TX packets:107518214 multicast:0 unicast:107518214 broadcast:0 TX errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2873083199 (2.6 GiB) TX bytes:3653633113 (3.4 GiB) RX multicast bytes:489301437 (466.6 MiB) TX multicast bytes:0 (0.0 B) br0:0 Link encap:Ethernet HWaddr 9C:1E:95:64:37:40 inet addr:67.XXX.YYY.40 Bcast:67.255.255.255 Mask:255.0.0.0 UP BROADCAST RUNNING ALLMULTI MULTICAST MTU:1500 Metric:1 <--snip--> ppp0.1 Link encap:Point-to-Point Protocol inet addr:67.42.252.46 P-t-P:207.108.176.16 Mask:255.255.255.255 UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1492 Metric:1 RX packets:84767648 multicast:0 unicast:84767648 broadcast:0 RX errors:0 dropped:0 overruns:0 frame:0 TX packets:47079448 multicast:0 unicast:47079448 broadcast:0 TX errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:3 RX bytes:13034999 (12.4 MiB) TX bytes:2993581539 (2.7 GiB) RX multicast bytes:0 (0.0 B) TX multicast bytes:0 (0.0 B) <--snip-->
Routing Table:
# route show Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 205.171.3.65 * 255.255.255.255 UH 0 0 0 ppp0.1 207.108.176.16 * 255.255.255.255 UH 0 0 0 ppp0.1 192.168.1.0 * 255.255.255.0 U 0 0 0 br0 67.0.0.0 * 255.0.0.0 U 0 0 0 br0 default * 0.0.0.0 U 0 0 0 ppp0.1
So…the CenturyLink modem is simply adding the network address (67.XXX.YYY.40) to the LAN interface (br0), with a netmask of 255.0.0.0, and a corresponding routing table entry.
I can reproduce this in pfSense by adding my reserved network address a virtual IP (Firewall -> Virtual IPs) on the LAN interface with an 8-bit netmask--i.e., Type: IP Alias, Interface: LAN, Address Type: Single Address; Address(es) 67.XXX.YYY.40/8. pfSense's interfaces and routing tables now look similar to the CenturyLink modem's, and all of my statically-assigned IPs seem to route properly under pfSense.
Interfaces:
: ifconfig <--snip--> em1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500 options=4209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwtso>ether 00:eb:cb:70:03:da hwaddr 00:eb:cb:70:03:da inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255 inet 67.XXX.YYY.40 netmask 0xfffffff0 broadcast 67.XXX.YYY.47 inet6 fe80::1:1%em1 prefixlen 64 scopeid 0x2 nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>) status: active <--snip--> pppoe0: flags=88d1 <up,pointopoint,running,noarp,simplex,multicast>metric 0 mtu 1492 inet 67.XXX.YYY.46 --> 207.108.176.16 netmask 0xffffffff inet6 fe80::2eb:cbff:fe70:3d9%pppoe0 prefixlen 64 scopeid 0x9 nd6 options=21 <performnud,auto_linklocal><--snip--></performnud,auto_linklocal></up,pointopoint,running,noarp,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast>
Routing Table:
: netstat -rn4 Routing tables Internet: Destination Gateway Flags Netif Expire default 207.108.176.16 UGS pppoe0 67.0.0.0/8 link#2 U em1 67.XXX.YYY.40 link#2 UHS lo0 67.XXX.YYY.46 link#9 UHS lo0 127.0.0.1 link#6 UH lo0 192.168.1.0/24 link#2 U em1 192.168.1.1 link#2 UHS lo0 207.108.176.16 link#9 UH pppoe0 208.67.220.220 207.108.176.16 UGHS pppoe0 208.67.222.222 207.108.176.16 UGHS pppoe0
Note that the netmask of 255.0.0.0 (8 bits) seems far more expansive than it needs to be (i.e., 29 bits). This has actually led to problems for me when using CenturyLink's modem–any IP address in the 67.0.0.0/29 network is caught by the routing rule and can be made unreachable to LAN clients. This complicated an MS Office update a couple weeks ago, for example, because the update server's ip address started with 67. Rather than using an 8-bit netmask, I have actually used a 28-bit netmask to limit this impact (you cannot use 29 bits, because pfSense complains that you can't assign the network address as a VIP). Another solution is to simply assign one of the user-assignable static IPs to the LAN interface, rather than the network address (67.XXX.YYY.40), but this ties up a static IP for no good reason.
Hope this helps someone...
-
An addendum to my previous post. This setup seems to create an Asymmetric Routing issue. I fixed this by creating manual firewall rules per https://doc.pfsense.org/index.php/Asymmetric_Routing_and_Firewall_Rules.
In addition, outgoing packets appear to be NAT'ed (they all appear from the outside come from the gateway address). I've experimented exhaustively with the NAT settings and have not been able to resolve this yet. I'll post an update if I figure it out. For now, it hasn't affected any of my services, except that e-mails appear to come from the gateway address, not the mail server address–necessitating some PTR record tweaks.