Replacing old router with a new instance of pfsense + BGP protocol to configure a dual link to the ISP

mauro.tridici · Nov 15, 2022, 10:59 AM

Dear users,

During the last few days, the need to connect an existing old router of ours to our ISP via a dual 1Gb link (activation of an additional backup link) has arisen.

Our ISP asked us to set up the environment by replacing the existing old router (which does not have BGP protocol) with a new router capable of handling BGP protocol and two links to the ISP.

Our idea is to replace the old router with a new instance of pfsense installed on a physical server and enable the BGP protocol.

In your opinion, is this a good idea? Can pfsense manage two links (primary + backup) by activating an automatic fault tolerance service? If yes, are there any guides to follow to implement such type of configuration?
In the backend, so behind the router, could everything remain the same? Would the reference IP 3.3.3.1/25 (mentioned in the schema below) always remain valid and unique?

I apologize in advance for the amount of questions, but this is the first time I am faced with this kind of request/problem.

For your convenience, I attach an image with the current situation and the future situation to this one.
I thank you in advance.
Good day,
Mauro

stephenw10 · Nov 15, 2022, 10:30 PM

Yes, you should be able to do that using pfSense and FRR.

Is '3.3.3.1/25' a public subnet they are routing to you? If so then yes it will remain the same. You advertise that subnet to the ISP across both links, to both neighbours, with different weighing/AD so one link is preferred.
I won't pretend to be a BGP expert though.
The details of the configuration should be fairly simple. It's only one subnet.

Cool_Corona · Nov 15, 2022, 10:38 PM

Pls. let me know more since we could be facing the same scenario soon....

stephenw10 · Nov 16, 2022, 7:31 AM

This sums it up pretty well in general terms: https://www.youtube.com/watch?v=R4P40o0yupQ

There are a number of ways you can set the preference between the two links. I don't really have an opinion on which is better. Others probably will!

If I was paying for dual 1G links I would want to load-balance across both of them. If I could.

Steve

mauro.tridici · Nov 16, 2022, 7:31 AM

@stephenw10 many many thanks for your help.

I really appreciated your support. It is helping me to understand a lot of things :)
I will take a look at the link you provided.

Have a great day,
Mauro

bingo600 · Nov 16, 2022, 8:14 PM

@stephenw10 said in Replacing old router with a new instance of pfsense + BGP protocol to configure a dual link to the ISP:

If I was paying for dual 1G links I would want to load-balance across both of them. If I could.

In general BGP is like "The Highlander" ... "There can be only ONE" ... (Active route)

There is some MED "trickery" , but it all depends.

/Bingo

stephenw10 · Nov 16, 2022, 8:21 PM

@bingo600 said in Replacing old router with a new instance of pfsense + BGP protocol to configure a dual link to the ISP:

In general BGP is like "The Highlander"

Yup, it would require something other than BGP.

Cool_Corona · Nov 16, 2022, 10:53 PM

@bingo600 Trying to wrap my head around it....

Dos BGP require end to end BGP between client and ex. a DC??

Or is the routing "internally" at the fiber link provider?

So BGP terminates at the exit of the fiber provider before traffic is sent over the internet?

stephenw10 · Nov 16, 2022, 11:33 PM

The BGP session itself is between router at each end of the link. In this case between the same routers at each end of two links, so two sessions. pfSense with FRR is the local router. Whatever the ISP is running is the remote router.
The ISP would almost certainly have further BGP links to other routers upstream but that doesn't matter to us, we don't see any of that.

mauro.tridici · Nov 18, 2022, 10:23 AM

Hi all,

thank you very much for your reply and feedback.
Are there some particular suggestions (or configurations settings) in order to configure the router as "customer border router"?
The current old router will be replaced by a baremetal pfsense instance and it will be connected to the ISP upstream gateway using a x.x.x.x/30 IP address (on the WAN side) and a y.y.y.y/25 public IP in the LAN side.

I remember that I already tried to do it in the past, but something went wrong (maybe something related to NAT...)

Is there some tech guide for this kind of needs?

Thank you in advance,
Mauro

stephenw10 · Nov 18, 2022, 1:24 PM

We did a hangout on configuring FRR: https://youtu.be/4IlKcB17rWk?t=2389

We also have the docs here: https://docs.netgate.com/pfsense/en/latest/packages/frr/bgp/example.html

It's really quite specific to your setup though. How is the current router setup? What is your ISP telling you to configure?

Steve

mauro.tridici · Nov 21, 2022, 7:45 AM

@stephenw10

Wow! Thanks, the hangout is very interesting, thank you.
Also the official netgate docs are so useful.

My ISP gave me one IP address on a /30 subnet for the point to point link with the upstream gateway and a range of public IPs on a /25 subnet.
The IPS provided also the ASN for the BGP configuration.

I already simulated the future scenario in GNS3 and it works, but before going in the "real world" I would like to know if, in order to configure pfSense as a simple router with BGP, I should activate (or deactivate) some particular pfSense feature.

I read that, to reach my target configuration, I should do something like that:

https://blog.matrixpost.net/convert-pfsense-into-a-routing-only-platform/

(adding some other changes for the BGP).

In you opinion, is it really the right procedure to run pfSense as a router only?
I'm doing this kind of question because it is not an official netgate page.

Thank you in advance.
Have a great day,
Mauro

stephenw10 · Nov 21, 2022, 6:16 PM

You would want to disable outbound NAT for that /25 subnet on the WAN since it's a public subnet.

You probably don't want to disable pf entirely because that disables the firewall and you would still want to be able to filter traffic.

Steve

mauro.tridici · Nov 22, 2022, 7:38 AM

@stephenw10

Thank you, Steve :) your suggestions are always useful.

Have a great day.
Mauro

mauro.tridici · Nov 22, 2022, 12:07 PM

Hello Steve,

I was able to do the work mentioned above in a real scenario.
Outbound NAT has been disabled.

In this pfsense/router instance I have 4 interfaces:

WAN1 (/30 primary link to ISP)
WAN2 (/30 backup link to ISP)
PUBLIC (/25 public subnet behind the router
LAN (for the management of the pfsense instance itself).

Do you think that I should enable the NAT only for the LAN?
And if yes, how can I do it?

Thank you,
Mauro

stephenw10 · Nov 22, 2022, 9:15 PM

Yes, you will need outbound NAT for the LAN.

You can do that do that either by setting outbound NAT in manual mode and only adding rules for the LAN subnet via each WAN. And possibly localhost too.
Or by setting oubound NAT in hybrid mode and then adding do-not-NAT rules for the public subnet.

You will probably want to setup both WANs in a gateway group and use that to provide failover or load-balancing for connections from LAN.

Steve

mauro.tridici · Nov 23, 2022, 8:43 PM

@stephenw10

Hello Steve,

a short description of the current pfsense router configuration running in GNS3:

WAN1 (192.168.1.2/30 primary link to ISP)
WAN2 (192.168.2.2/30 backup link to ISP)
PUBLIC (192.168.99.1/25 public subnet behind the router)
LAN (for the management of the pfsense instance itself). 192.168.10.1/24

I decided to use HYBRID OUTBOUND NAT adding the do-not-NAT rules for the public subnet.
Could you please check if the NAT rules in the OUTBOUND NAT section are the right ones?

So, if everything is ok, LAN should be NATted and PUBLIC should not be NATted. Both the hosts on LAN subnet and the hosts on PUBLIC subnet are able to reach internet.

But how can I verify that NAT is actually disabled between PUBLIC and WAN1/WAN2 interface? Is there a way to do it using "packet capture" feature of GNS3?

The following lines have been added after reading carefully the official documentation

I think that I made a mistake in the previous configuration. I'm not sure, but I think that the INTERFACE field in the MAPPING section should be WAN1 and WAN2. Please correct me if I'm wrong again.

In this case, the hosts in the LAN subnet are still able to reach internet (thanks to the enabled NAT), but the hosts in the PUBLIC subnet are not.

What should I do to fix this problem?

Thank you in advance,
Mauro

Cool_Corona · Nov 23, 2022, 7:37 PM

@mauro-tridici How do you map a NAT rule from WAN to LAN?

Its like everything is backwards....?

mauro.tridici · Nov 23, 2022, 8:21 PM

@cool_corona sorry, but I'm still a newbie, I'm at beginning. Could you please explain where is my error and the right solution to my problem?

Is It the first one I proposed (first screenshot) or the second one?

In addition, how can I check that NAT is working as expected ( maybe using the pf sense packet capture)?

Thank you.

Cool_Corona · Nov 23, 2022, 8:32 PM

@mauro-tridici No I an a newbie too in BGP but I dont get the NAT in regards to the public /25 that imho is the one that gets routed from WAN to LAN