Dedicated Business Fiber Internet and Netgate

ltechnology

Dedicated business internet access is a fairly popular product these days. Most vendors use an underlying carrier, mostly ATT for the fiber loops, and the vendor provides service on both ends for access. ATT typically terminates their loop with a NID, providing an ethernet interface for CPE connections - although I have seen instances where a fiber bulkhead was installed. The end provider normally offers and recommends they supply a router to provide the interface from the fiber loop to an ethernet connection with the customer IP(s). These routers are typically Cisco or comparable routers, and take the ATT network assigned address and routes them to a single or multiple IPs assigned for customer use on the output ethernet on the router. The customer then connects their firewall, router and/or network equipment to this connection.

I was surprised to find that there is little or not discussion or documentation to integrate this process within Pfsense/Netgate, especially with a higher end configuration that has some horsepower.

So using a post from a prior post on this forum, the vendor provides the following information for configuration on the intermediate router:
IP Address Block
IP Address: 12.xxx.xxx.128/29

WAN Information
CR Serial IP Address: 32.xxx.xx.226/30
AR Serial IP Address: 32.xxx.xx.225/30
WAN IP Address: 32.xxx.xx.224/30

LAN Information
LAN IP Gateway: 12.xxx.xx.129
First Network Assignable: 12.xxx.xx.130
Usable IP Range: 12.xxx.xx.130 - 12.xxx.xx.134
Broadcast IP Address: 12.xxx.xx.135
Subnet Mask: 255.255.255.248

The intention would then be with this intermediate router in place with the above routing, the pfsense/Netgate unit would take 12.xx.xx.190 as the WAN and then provide a LAN interface, something like 192.168.10.1 with DHCP, etc.

My question is why can't you take that segment services by the intermediate router and incorporate it in the Netgate with the hope of avoiding a double NAT situation, or now that the OS supports multiple virtual instances - create two instances with one to provide the intermediate function and the 2nd to provide the basic firewall configuration. I have seen information on the virtual abilities but have yet to explore them.

Can anyone provide some thoughts, guidelines or suggestions?

Thanks
R

bp81

@ltechnology said in Dedicated Business Fiber Internet and Netgate:

My question is why can't you take that segment services by the intermediate router and incorporate it in the Netgate with the hope of avoiding a double NAT situation...

Just looking at the configuration, I don't see how you could get into a double NAT situation, even if you tried to. We have a very similar configuration (AT&T MIS fiber with intermediate Cisco router terminating the fiber and giving us a cat6 handoff to a pfSense router, exactly as you describe) at our HQ and we don't have a double NAT running.

The set of usable IP addresses that AT&T gives you, they refer to as "LAN" addresses (your listed usable range here is 12.xxx.xx.130 - 12.xxx.xx.134. They gave us something very similar). Just because AT&T calls them "LAN" addresses doesn't mean they're running a NAT there. They aren't. I think they call it LAN because it's the network segment of addresses your company gets to use and they are "private" to your company in the sense that only your company gets to use them.

Effectively, what AT&T calls "LAN" is really a WAN address block from the perspective of your pfSense router.

Simply statically assign the IP address you want to use for the WAN interface on your router from the one of the addresses in the "LAN" block AT&T gave you, and set your gateway correctly and you should be good to go. No double NAT involved.

ltechnology

Thanks so much for the comment and insight.

So - I would assign 12.xx.xx.130 for the Wan port, and use the 32.xxx.xxx.xxx port for the remote gateway?

Not clear how the translation from the Serial IP to 'Lan' IP takes place if I don't do it on the Netgate.

Sorry - it should be pretty clear but every time I think it through and play on the test unit, I end up down the rabbit hole.

Thanks.

bp81

@ltechnology said in Dedicated Business Fiber Internet and Netgate:

Thanks so much for the comment and insight.

So - I would assign 12.xx.xx.130 for the Wan port, and use the 32.xxx.xxx.xxx port for the remote gateway?

Not clear how the translation from the Serial IP to 'Lan' IP takes place if I don't do it on the Netgate.

Sorry - it should be pretty clear but every time I think it through and play on the test unit, I end up down the rabbit hole.

Thanks.

I tried to find one of our old AT&T order confirmations to compare to, but couldn't find one, so I am going on my recollection here. Grain of salt, but I think I'm right.

No, I don't believe you would use 32.xxx.xx.224/30 as your gateway. The "WAN Information" AT&T provides, if I remember correctly, will be applied to the router they'll put on your premise. We have such a setup and we never configured anything on that router other than bolting into our rack, making connections, and powering it on.

In other words, the "WAN Information" is, for your purposes, a red herring. You probably don't even need to know any of it. They gave us that kind of information as well and we had no need for it. It might matter if you wanted to have your own router in place of the AT&T provided router that accepts the fiber optic connection and hands off as cat6 to you, but we never had a need for anything like that here. Personally I was fine with AT&T managing that device for me, as I feel I have enough to manage already.

Given the info that AT&T has provided, it appears they've given you /29 CIDR block of addresses, so your CIDR block is 12.xxx.xx.128/29 and contains addresses ending in .128 through .135. The first and last addresses in the block (.128 and .135) aren't available for use because they fulfill special functions. .129 won't be available because that is going to be your gateway IP address (as a side note: almost always, the gateway IP address of a wan connection will be inside the same subnet / CIDR block as the router's WAN IP address. If you see a configuration where that's not the case, it's a red flag and is a misconfiguration 99% of the time, barring a few special edge cases).

That leaves you .130 through .134, as your information from AT&T specifies in your Usable IP Range information.

With that information in hand, here is the configuration I would attempt on pfSense:

First, create your gateway in System -> Routing -> Gateways.

Click Add to create a new gateway.
Select your WAN interface in the "Interface" dropdown.
Address Family: IPv4
Name: add a reasonable name. ATT or whatever you like.
Gateway: 12.xxx.xx.129 (this is the LAN IP Gateway Address in the info AT&T provided)
Setup Gateway monitoring if you want to use it, skip if you don't.
Enter a Description if you want. Optional field I believe, and has no effect on function.
Save and Apply Changes.

Now configure your WAN interface:

Check the box to enable the inferface
Enter a Description for the interface. Probably WAN, but whatever you like is fine.
IPv4 Configuration Type: Static IPv4
IPv6 Configuration Type: None
Under Static IPv4 Configuration section, set IPv4 Address to: 12.xxx.xx.130/29
IPv4 Upstream Gateway: Select the Gateway you just created (12.xxx.xx.129)
Recommended that you checkmark the boxes to "block private networks and loopback addresses" and "block bogon networks."
Save and apply changes.

If this is as I remembered it, your internet connection should be live now.

If you are very interested in how I think the WAN IP stuff works with AT&T's equipment, read on:

First, let me describe how this stuff is usually physically wired. It will typically look like this:

pfSense WAN ---> AT&T Cisco Ethernet Port
AT&T Cisco Fiber Optic Port ---> Fiber cable to demark box's "inward" facing interface
Fiber Demark Box's "outward" facing interface ---> whatever is out there on the utility pole

AT&T Cisco Ethernet Port is your gateway and is 12.xxx.xx.129/29
I think, maybe, the WAN IP Address is the address of the AT&T Cisco Router's Fiber Optic Port. I think that the AR Serial IP Address is the IP address of the inward facing fiber interface on the demark box that is connected to the Cisco Router's Fiber Optic Port.

I think the CR Serial IP Address is the IP address of the outward facing fiber port on the demark box that is connected to whatever is outside the building on the utility pole.

Both the Cisco router and the fiber demark box will likely have their own routing tables to correctly direct traffic, just as your pfSense router does. That's how you don't have a double NAT here, just ordinary routing tables.

ltechnology

Thanks for the added information.
I think what I did not communicate properly is that I don't want to put a 2nd piece of equipment (or use their provided router) as I have in the past. I have traditionally opted to purchase a Cisco router, configure and put this in place as the intermediate piece instead of giving ATT any extra money.

What I am hoping to do is integrate the functionality and routing of this Cisco piece into the front end of the Negate, eliminating the hardware.

I think that one of two options might work:

Set up the WAN port with the serial IP, and set the LAN IP (from ATT) to a virtual IP. Route the virtual IP to the LAN on configured on the Netgate.
Utilize the virtual layers now available, set up 1 virtual to handle the ATT portion, and then have it hand off to the 2nd virtual which would be my normal Netgate firewall./router configuration, using the 1st virtual as my WAN interface.

I know they are using a Fortinet Fortigate router for the provided equipment now and doing this all in that unit - so it should be workable on the Netgate as well. Strangely noone has done this - or has taken the time to post and share their efforts.

Thanks

bp81

@ltechnology said in Dedicated Business Fiber Internet and Netgate:

Thanks for the added information.
I think what I did not communicate properly is that I don't want to put a 2nd piece of equipment (or use their provided router) as I have in the past. I have traditionally opted to purchase a Cisco router, configure and put this in place as the intermediate piece instead of giving ATT any extra money.

What I am hoping to do is integrate the functionality and routing of this Cisco piece into the front end of the Negate, eliminating the hardware.

I think that one of two options might work:

Set up the WAN port with the serial IP, and set the LAN IP (from ATT) to a virtual IP. Route the virtual IP to the LAN on configured on the Netgate.

Utilize the virtual layers now available, set up 1 virtual to handle the ATT portion, and then have it hand off to the 2nd virtual which would be my normal Netgate firewall./router configuration, using the 1st virtual as my WAN interface.

I know they are using a Fortinet Fortigate router for the provided equipment now and doing this all in that unit - so it should be workable on the Netgate as well. Strangely noone has done this - or has taken the time to post and share their efforts.

Thanks

OK I understand now.

I've never done this. I think the challenge is that the upstream device you will be directly connecting your Netgate WAN port to is in a different subnet than the static addresses you are allowed to use. You'll probably need a Netgate with an SFP port, since you'll be attaching directly to fiber (or a media converter). I can't really test this, so I can only suggest things in a "spitballing it" kind of way.

One should note that this may not be possible without getting AT&T to change some things.

First Option - Use the AT&T "WAN Information" instead of the LAN information.

Set your Netgate's WAN IP to 32.xxx.xx.224/30. Use a gateway address of 32.xxx.xx.225/30 (or possibly 226/30. I'm making an assumption about the IP of the gateway directly upstream here. It could be either one, I merely suspect it's 225/30).
Test the connection.

If AT&T does any vlan tagging or MAC address filtering / authentication, this may fail right out of the box. If this is the case, you'd need AT&T to give you a mac address that is authenticated that you could spoof on your Netgate's WAN port (not impossible you could get the address right off the AT&T Cisco router as well) and what the vlan tagging configuration is, if any.

You state you have previously purchased your own Cisco router and configured this. If that's the case, you may know the vlan tagging configuration already and know how to deal with possible mac authentication issues if they even exist.

AT&T may also expect packets to originate from you LAN block (12.xxx.xx.129/30) rather than from AT&T's own WAN block (32.xxx.xx.224/30). AT&T's equipment might reject this traffic without them changing the configuration of the upstream devices.

Downsides to this method

you may need to get AT&T to change things on their end for this to work
you will need to be able to directly accept a fiber connection (SFP port should work here)
you don't have a guarantee that the SFP port on the Netgate will be compatible with the SFP module connecting fiber to your WAN interface (it's likely it will be, but it's not assured. If it isn't you might need a media converter to convert fiber to ethernet/rj45 and use an ethernet port for WAN).
You only get one public facing IP address instead your full block of 5.

Second Option - Configure your WAN gateway to allow non-local gateway addressing.

This is similar to your previous proposal about using 32.xxx.xx.129/30 as your gateway and using 12.xxx.xx.130/29 as your WAN interface IP address.

Configure a Gateway in System -> Routing. It's IP address is 32.xxx.xx.225/30. Before saving the Gateway, click Display Advanced. At the very bottom, check the box labeled "Use non-local gateway through interface specific route." Save and Apply Changes.
Configure WAN interface to use IP address 12.xxx.xx.130/29. Select the gateway you just created for the IPv4 upstream gateway. Because you checked the box allowing a non-local gateway, this setting will be permitted as valid whereas in a default setup it would not.

This may still fail for similar reasons as the first method, relating to mac address authentication, vlan tagging issues, and source IP address issues. This is also a very non-standard configuration. 99% of the time if the gateway for your WAN interface is in a different subnet than your WAN interface's IP, it's a misconfiguration. There are a few legit instances where it's needed, it could possibly work here.

I'm not sure which I'd attempt first for certain, I might actually lean slightly towards the first method if you are good with a single WAN IP address on your Netgate. If you need to use multiple addresses in the block, it basically pushes you into the second method.

A more ideal solution might be if AT&T makes the device that is typically upstream from their Cisco router to assign 12.xxx.xx.129/30 as the inward facing IP address (instead of 32.xxx.xx.224/30). Then you'd have your full block of 5 IP addresses and a very standard WAN configuration on your Netgate. Since AT&T doesnt do it this way, I presume there are reasons why they don't, so it may not be possible or something they'd be willing to change.