CenturyLink, PPPoE, a static /29 subnet, without 1:1 NAT



  • Hello everybody,

    I have CenturyLink fiber service and it uses PPPoE like their DSL service.  Along with this, I have a /29 static IP block that, using example IP addresses, has a network of 9.20.30.16/29 (so usable range is 17 through 21) and gateway of 9.20.30.22.  The .22 address is assigned to the PPPoE link when it is established.

    What I would like to do is be able to have three interfaces with pfSense.  WAN would be where my PFsense device connects to my ONT; LAN would be where 192.168.1.0/24 clients are NAT'd out through a single IP address from my block; DMZ would have machines that can have public addresses assigned directly to them (so a machine connected to DMZ would show an IP address of, say, 9.20.30.17 when I type ifconfig).  1:1 NAT is not my preferred outcome.

    I've gotten WAN and LAN to work.  I have my PPPoE credentials and enter them into pfSense and it connects and LAN devices get an IP and can do their thing.  But I'm running into a problem with the DMZ interface where I can't get any routing to work.  The .22 address, which is also the gateway address for any of the other public IPs, is only assigned to the PPPoE link not the physical WAN interface itself (so pfSense substitutes the PPPoE virtual interface as "WAN" when PPP connects).

    My troubleshooting took me into my CenturyLink-provided router and I noticed something very strange: br0 (the LAN bridge on my CL router that connects the 4 physical LAN-labeled ports together) has a secondary IP of 9.20.30.22, which is also assigned to the ppp0 interface.  Look:

    # ifconfig
    br0       Link encap:Ethernet  HWaddr 12:33:99:00:11:22
              inet addr:192.168.1.1  Bcast:192.168.1.255  Mask:255.255.255.0
              UP BROADCAST RUNNING ALLMULTI MULTICAST  MTU:1500  Metric:1
    
    br0:0     Link encap:Ethernet  HWaddr 12:33:99:00:11:22
              inet addr:9.20.30.22  Bcast:209.180.203.23  Mask:255.255.255.248
              UP BROADCAST RUNNING ALLMULTI MULTICAST  MTU:1500  Metric:1
    
    ppp0.1    Link encap:Point-to-Point Protocol
              inet addr:9.20.30.22  P-t-P:1.2.3.68  Mask:255.255.255.255
              UP POINTOPOINT RUNNING NOARP ALLMULTI MULTICAST  MTU:1492  Metric:1
    
    

    (I anonymized the MAC address and deleted the lines about packet statistics.)

    Does this make any sense?  I've been beating my head against this off and on for a month and haven't figured it out so I'm really hoping one of y'all knows what I am doing wrong. :D


  • Rebel Alliance Developer Netgate

    I have not seen this work with PPPoE, but you could maybe bridge DMZ to WAN but that's a bit ugly (as all bridging is ugly).

    There would not be any way to have both WAN and DMZ on the same subnet with IP addresses in the same subnet assigned to both. Set DMZ to have no IP address, bridge it to WAN, and then set your DMZ hosts to use the upstream gateway.

    If you kept the centurylink device doing PPPoE and not pfSense then it may be more likely to work, though you'd lose one IP address out of the block since both pfSense and the centurylink box would both be using an IP address.



  • Apologies for the Necro, but seeing as it was the exact same issue I'm having (down to the ISP and service), I felt it was warranted.

    First, to the OP:  Did you ever find a solution to this?  I really don't want to use the consumer grade Zyxel C2100t modem to provide this functionality.

    Second, the relevant info for my connection (ip's/mac's changed to protect the innocent):

    routed subnet:  64.101.146.144/29
    upstream gw:  52.120.09.57
    pfSense WAN:  64.101.146.150

    It seems that CenturyLink takes the last usable IP in the block and assigns it as the gateway and serves it up as the client side of the PPPoE connection.  For example, my block was 64.101.146.144/29, where .144 would be the network, .145-.150 would be the usable IP's, and .151 would be the broadcast.  CenturyLink assigned my side of the PPPoE connection 64.101.146.150/32, which as the OP noted pfSense assigned to the WAN interface of the router.

    Following jimp's suggestion, I tried bridging my WAN and DMZ port.  I created basic allow any traffic rules between them, assigned a static IP of 64.101.146.145/29 and using the 52.120.09.57 gw to a machine connected to the DMZ interface.  This machine was unable to ping anything and a traceroute would just show it stalling out on the first hop, unable to reach the 52.120.09.57 gw.  (I also tried using the pfsense WAN IP as a gw, with the same result)

    Out of curiosity, using a machine connected to my lan segment, I tried pinging the machine connected to the DMZ.  I then tried a traceroute, which showed the following:

    Tracing route to 64.101.146.145 over a maximum of 30 hops
    
      1    <1 ms    <1 ms    <1 ms  10.11.12.1 
      2     1 ms     1 ms     1 ms  52.120.09.57
      3     8 ms     1 ms     1 ms  64.101.146.150
      4     *        *        *     Request timed out.
    

    This shows the trace going to the LAN side of the pfSense box, out to the upstream gateway, which then correctly routed it back to the pfSense box, where it stopped.  Assuming it was a routing issue, I looked at the routing table on the pfSense box and there was no entry for the 64.101.146.144/29 subnet.

    Then, like the OP, I found a CenturyLink modem and inserted it between the pfSense box and the ONT.  I configured it for PPPoE, selected static IP, and it had a dropdown selection available to select between Single Static IP and Block of Static IP Addresses. 

    Here's the resultant screen, to which I input 64.101.146.150 as gw, with a 255.255.255.248 subnet, and selected Public LAN for the DHCP. I then connected my pfSense router to a small switch attached the lan port on the modem, manually assigned 64.101.146.149/29 with 64.101.146.150 as it's gw, and everything works as expected.  I was also able to connect other hosts to the switch and assign them addresses within the subnet successfully.

    Again, like the OP, I too ssh'd into the modem to see what was going on under the hood.  Unfortunately, it had a very limited command set and few tools, but I was able to pull up the interfaces and show the routing table:

      > ifconfig
    bcmsw     Link encap:Ethernet  HWaddr 63:9A:E7:D8:02:3D  
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
    
    br0       Link encap:Ethernet  HWaddr 63:9A:E7:D8:02:3D  
              inet addr:192.168.0.1  Bcast:192.168.0.255  Mask:255.255.255.0
              UP BROADCAST RUNNING ALLMULTI MULTICAST  MTU:1500  Metric:1
    
    br0:0     Link encap:Ethernet  HWaddr 63:9A:E7:D8:02:3D  
              inet addr:64.101.146.150  Bcast:64.101.146.151  Mask:255.255.255.248
              UP BROADCAST RUNNING ALLMULTI MULTICAST  MTU:1500  Metric:1
    
    eth0      Link encap:Ethernet  HWaddr 63:9A:E7:D8:02:3D  
              inet6 addr: fe80::5a8b:f3ff:fec7:134e/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
    
    eth1      Link encap:Ethernet  HWaddr 63:9A:E7:D8:02:3D  
              UP BROADCAST MULTICAST  MTU:1500  Metric:1
    
    eth2      Link encap:Ethernet  HWaddr 63:9A:E7:D8:02:3D  
              UP BROADCAST MULTICAST  MTU:1500  Metric:1
    
    eth3      Link encap:Ethernet  HWaddr 63:9A:E7:D8:02:3D  
              UP BROADCAST MULTICAST  MTU:1500  Metric:1
    
    eth4      Link encap:Ethernet  HWaddr 63:9A:E7:D8:02:3D  
              inet6 addr: fe80::5a8b:f3ff:fec7:134e/64 Scope:Link
              UP BROADCAST RUNNING ALLMULTI MULTICAST  MTU:1500  Metric:1
    
    eth4.1    Link encap:Ethernet  HWaddr 63:9A:E7:D8:02:49  
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
    
    lo        Link encap:Local Loopback  
              inet addr:127.0.0.1  Mask:255.0.0.0
              inet6 addr: ::1/128 Scope:Host
              UP LOOPBACK RUNNING  MTU:16436  Metric:1
    
    ppp1.1    Link encap:Point-to-Point Protocol  
              inet addr:64.101.146.150  P-t-P:52.120.09.57  Mask:255.255.255.255
              UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1492  Metric:1
    
     > route show
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
    52.120.09.57    *               255.255.255.255 UH    0      0        0 ppp1.1
    64.101.146.144  *               255.255.255.248 U     0      0        0 br0
    192.168.0.0     *               255.255.255.0   U     0      0        0 br0
    default         *               0.0.0.0         U     0      0        0 ppp1.1
    

    While it works, I would really prefer to remove that consumer grade piece of hardware and additional single point of failure and move this functionality to my router/firewall if at all possible.  Also, this modem doesn't seem capable of handling the full speed of my connection.  When I have pfSense connected directly to the ONT, I get about 900mb down and 930mb up.  With the modem, the best I've seen is 695mb either direction.  (First World problems, I know :) )

    Based on the output and naming conventions for interfaces, it seems this device is using some form of embedded linux, and as such, it is my hope that the principles used on the modem will translate to pfSense and the underlying FreeBSD.



  • I believe I have been able to reproduce on pfSense the effect of the "Public LAN Subnet" configuration in CenturyLink's firmware.

    My static IP block is 67.XXX.YYY.40/29. This network breaks down as follows:

    67.XXX.YYY.40 - Reserved Network Address
    67.XXX.YYY.41 - 67.XXX.YYY.45 - User-Assignable
    67.XXX.YYY.46 - Reserved Gateway
    67.XXX.YYY.47 - Broadcast

    When using the CenturyLink modem, the public address is assigned as 67.XXX.YYY.46. This is consistent on pfSense–when initiating the PPPoE connection, the pfSense WAN interface is automatically assigned 67.XXX.YYY.46.

    If you ssh into the CenturyLink modem you can obtain the following:

    Interfaces:

    
    # ifconfig
    <--snip-->
    br0       Link encap:Ethernet  HWaddr 9C:1E:95:64:37:40
              inet addr:192.168.1.1  Bcast:192.168.1.255  Mask:255.255.255.0
              inet6 addr: fe80::9e1e:95ff:fe64:3740/64 Scope:Link
              UP BROADCAST RUNNING ALLMULTI MULTICAST  MTU:1500  Metric:1
              RX packets:63054895 multicast:1459543 unicast:60717936 broadcast:877416
              RX errors:0 dropped:0 overruns:0 frame:0
              TX packets:107518214 multicast:0 unicast:107518214 broadcast:0
              TX errors:0 dropped:0 overruns:0 carrier:0 collisions:0
              txqueuelen:0
              RX bytes:2873083199 (2.6 GiB) TX bytes:3653633113 (3.4 GiB)
              RX multicast bytes:489301437 (466.6 MiB) TX multicast bytes:0 (0.0 B)
    
    br0:0     Link encap:Ethernet  HWaddr 9C:1E:95:64:37:40
             inet addr:67.XXX.YYY.40  Bcast:67.255.255.255  Mask:255.0.0.0
              UP BROADCAST RUNNING ALLMULTI MULTICAST  MTU:1500  Metric:1
    
    <--snip-->
    
    ppp0.1    Link encap:Point-to-Point Protocol
              inet addr:67.42.252.46  P-t-P:207.108.176.16  Mask:255.255.255.255
              UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1492  Metric:1
              RX packets:84767648 multicast:0 unicast:84767648 broadcast:0
              RX errors:0 dropped:0 overruns:0 frame:0
              TX packets:47079448 multicast:0 unicast:47079448 broadcast:0
              TX errors:0 dropped:0 overruns:0 carrier:0 collisions:0
              txqueuelen:3
              RX bytes:13034999 (12.4 MiB) TX bytes:2993581539 (2.7 GiB)
              RX multicast bytes:0 (0.0 B) TX multicast bytes:0 (0.0 B)
    <--snip-->
    
    

    Routing Table:

    
    # route show
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
    205.171.3.65    *               255.255.255.255 UH    0      0        0 ppp0.1
    207.108.176.16  *               255.255.255.255 UH    0      0        0 ppp0.1
    192.168.1.0    *               255.255.255.0   U     0      0        0 br0
    67.0.0.0        *               255.0.0.0       U     0      0        0 br0
    default         *               0.0.0.0         U     0      0        0 ppp0.1
    
    

    So…the CenturyLink modem is simply adding the network address (67.XXX.YYY.40) to the LAN interface (br0), with a netmask of 255.0.0.0, and a corresponding routing table entry.

    I can reproduce this in pfSense by adding my reserved network address a virtual IP (Firewall -> Virtual IPs) on the LAN interface with an 8-bit netmask--i.e., Type: IP Alias, Interface: LAN, Address Type: Single Address; Address(es) 67.XXX.YYY.40/8. pfSense's interfaces and routing tables now look similar to the CenturyLink modem's, and all of my statically-assigned IPs seem to route properly under pfSense.

    Interfaces:

    
    : ifconfig
    <--snip-->
    em1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
    	options=4209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwtso>ether 00:eb:cb:70:03:da
    	hwaddr 00:eb:cb:70:03:da
    	inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
    	inet 67.XXX.YYY.40 netmask 0xfffffff0 broadcast 67.XXX.YYY.47
    	inet6 fe80::1:1%em1 prefixlen 64 scopeid 0x2
    	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
    	status: active
    <--snip-->
    pppoe0: flags=88d1 <up,pointopoint,running,noarp,simplex,multicast>metric 0 mtu 1492
    	inet 67.XXX.YYY.46 --> 207.108.176.16  netmask 0xffffffff
    	inet6 fe80::2eb:cbff:fe70:3d9%pppoe0 prefixlen 64 scopeid 0x9
    	nd6 options=21 <performnud,auto_linklocal><--snip--></performnud,auto_linklocal></up,pointopoint,running,noarp,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast> 
    

    Routing Table:

    
    : netstat -rn4
    Routing tables
    
    Internet:
    Destination        Gateway            Flags     Netif Expire
    default            207.108.176.16     UGS      pppoe0
    67.0.0.0/8         link#2             U           em1
    67.XXX.YYY.40       link#2             UHS         lo0
    67.XXX.YYY.46       link#9             UHS         lo0
    127.0.0.1          link#6             UH          lo0
    192.168.1.0/24    link#2             U           em1
    192.168.1.1       link#2             UHS         lo0
    207.108.176.16     link#9             UH       pppoe0
    208.67.220.220     207.108.176.16     UGHS     pppoe0
    208.67.222.222     207.108.176.16     UGHS     pppoe0
    
    

    Note that the netmask of 255.0.0.0 (8 bits) seems far more expansive than it needs to be (i.e., 29 bits). This has actually led to problems for me when using CenturyLink's modem–any IP address in the 67.0.0.0/29 network is caught by the routing rule and can be made unreachable to LAN clients. This complicated an MS Office update a couple weeks ago, for example, because the update server's ip address started with 67. Rather than using an 8-bit netmask, I have actually used a 28-bit netmask to limit this impact (you cannot use 29 bits, because pfSense complains that you can't assign the network address as a VIP). Another solution is to simply assign one of the user-assignable static IPs to the LAN interface, rather than the network address (67.XXX.YYY.40), but this ties up a static IP for no good reason.

    Hope this helps someone...



  • An addendum to my previous post. This setup seems to create an Asymmetric Routing issue. I fixed this by creating manual firewall rules per https://doc.pfsense.org/index.php/Asymmetric_Routing_and_Firewall_Rules.

    In addition, outgoing packets appear to be NAT'ed (they all appear from the outside come from the gateway address). I've experimented exhaustively with the NAT settings and have not been able to resolve this yet. I'll post an update if I figure it out. For now, it hasn't affected any of my services, except that e-mails appear to come from the gateway address, not the mail server address–necessitating some PTR record tweaks.