Use FreeBSDs quagga package for use of OSPF and BGP
-
Hi,
I'd like to use OSPF and BGP on a single machine. In an other xxx-sense forum I read about just to install the pkg from FreeBSD in order to use both. Is this possible with pfSense if I only use the CLI instead of WebGUI stuff?
Are there any issues which could break things with other packages or dependencies?
Thanks!
-
No, it's not possible. CLI or WebGUI does not matter.
https://github.com/pfsense/FreeBSD-ports/blob/devel/net/openbgpd/Makefile#L17
https://github.com/pfsense/FreeBSD-ports/blob/devel/net/quagga/Makefile#L17 -
Is this even the case if I manually modify the Quagga package/config like this old pull request?
https://github.com/pfsense/pfsense-packages/pull/1258/files
I'm aware of current troubles with openbgpd and ipsec tunnels and am wondering if Quagga will work to provide BGP.
-
Irrelevant, the pkg package manager shockingly just won't let you install the conflicting packages.
-
I mean modify the Quagga package to provide both OSPF and BGP, it looks like all the binary's are there for this, the BGP ones are just set not to run.
The OpenBGPd package wouldn't be installed at all.
-
Which seems to work
quagga 35252 0.0 0.7 27792 3644 - Ss 3:52PM 0:00.00 /usr/local/sbin/zebra -d -f /var/etc/quagga/zebra.conf
quagga 35361 0.0 0.8 32344 3996 - Ss 3:52PM 0:00.00 /usr/local/sbin/ospfd -d -f /var/etc/quagga/ospfd.conf
quagga 35739 0.0 1.1 30560 5296 - Ss 3:52PM 0:00.00 /usr/local/sbin/bgpd -d -f /var/etc/quagga/bgpd.confpfSense.localdomain# sh ip bgp
BGP table version is 0, local router ID is 10.101.1.2
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Origin codes: i - IGP, e - EGP, ? - incompleteNetwork Next Hop Metric LocPrf Weight Path
*> 192.168.1.0 0.0.0.0 1 32768 ?I can see that original PR went stale, would this be accepted again if I updated it for 2.3?
-
I've been working on this problem for the last couple of months. I rewrote elements of the package to provide proper integration and BGP configuration saves into the /conf/config.xml file.
That said, everything was working GREAT in my lab until last night when I decided it was time to try MD5 authentication between peers… that's when my little world imploded. I've been spending today chasing down issues outlined https://www.barryodonovan.com/2013/09/22/bird-quagga-with-md5-support-for-ipv46-on-freebsd-linux
So far I've identified that all relevant options are already in the compiled kernel, which was a huge relief. I'm going to try working with the ipsec bits next.
(meanwhile, I'm trying to dig into how openBGPd deals with this issue and whether it uses the OS's crypto components for MD5 or if it's doing something on its own)
-quad
-
I've been working on this problem for the last couple of months. I rewrote elements of the package to provide proper integration and BGP configuration saves into the /conf/config.xml file.
That said, everything was working GREAT in my lab until last night when I decided it was time to try MD5 authentication between peers… that's when my little world imploded. I've been spending today chasing down issues outlined https://www.barryodonovan.com/2013/09/22/bird-quagga-with-md5-support-for-ipv46-on-freebsd-linux
So far I've identified that all relevant options are already in the compiled kernel, which was a huge relief. I'm going to try working with the ipsec bits next.
(meanwhile, I'm trying to dig into how openBGPd deals with this issue and whether it uses the OS's crypto components for MD5 or if it's doing something on its own)
-quad
Good news - I was able to use setkey to specify the MD5 password to be used in peering. I'm considering releasing the code modifications i've made to the plugin to get BGP working neatly alongside OSPF. At this point, i've basically ignored a guided GUI and made a text-based front-end for the GUI that lets you directly modify the config files, but still save all config parameters into the base pfSense config file, which was important to us for backup purposes. It's still a little less than ideal, but it's working.
-quad
-
I've been working on this problem for the last couple of months. I rewrote elements of the package to provide proper integration and BGP configuration saves into the /conf/config.xml file.
That said, everything was working GREAT in my lab until last night when I decided it was time to try MD5 authentication between peers… that's when my little world imploded. I've been spending today chasing down issues outlined https://www.barryodonovan.com/2013/09/22/bird-quagga-with-md5-support-for-ipv46-on-freebsd-linux
So far I've identified that all relevant options are already in the compiled kernel, which was a huge relief. I'm going to try working with the ipsec bits next.
(meanwhile, I'm trying to dig into how openBGPd deals with this issue and whether it uses the OS's crypto components for MD5 or if it's doing something on its own)
-quad
Good news - I was able to use setkey to specify the MD5 password to be used in peering. I'm considering releasing the code modifications i've made to the plugin to get BGP working neatly alongside OSPF. At this point, i've basically ignored a guided GUI and made a text-based front-end for the GUI that lets you directly modify the config files, but still save all config parameters into the base pfSense config file, which was important to us for backup purposes. It's still a little less than ideal, but it's working.
-quad
Further update -
I managed to script the loading of the md5 keys and pretty much have things working how I want. Next step: sorting out some CARP issues after a failover incident. I need to examine the order that components are loaded after the CARP checker runs.
-quad
-
Further update - I've satisfactorily sorted out the CARP issues and have both OSPF and BGP working side by side. I'm using BGP for our MPLS and redistributing routes both ways into BGP from OSPF and vice versa.
I'm hoping to contact the pfSense devs to show them what I've cooked up… this has been a couple weeks of solid work (unfortunately, strung out over several months) and would live to see this brought into production, primarily because i don't want to have to run my own package repository :D
-
Glad you got this all working. So just to confirm, this is the Quagga versions of OSPF and BGP running together? Would love to hear more details about your implementation.
I'm in the same boat considering what my options are going forward and trying to support Quagga BGP on pfsense. Ideally I would like to get the Cumulus version of Quagga on pfsense as that brings several nice new features to the table.
-
Glad you got this all working. So just to confirm, this is the Quagga versions of OSPF and BGP running together? Would love to hear more details about your implementation.
I'm in the same boat considering what my options are going forward and trying to support Quagga BGP on pfsense. Ideally I would like to get the Cumulus version of Quagga on pfsense as that brings several nice new features to the table.
Your original comment about getting the bgpd to run is where I started down this long road back in… November? Eventually I dug into the existing plugin more and modified the code to store the BGP config in config.xml etc. The biggest roadblocks i had were dealing with md5 keys and i managed to script that as well. We're running HA pairs of pfSense firewalls and I've got it working with CARP as well - that, fortunately, didn't require any modifications from the original plugin. The bummer is that when one FW fails, the quagga process has to start cold on the other FW, so you lose routing for 30-60 seconds depending on protocols in use. I have dreams of figuring out how to do state transfers with this someday, but i'm not sure if quagga is the best choice for that.
I'm really curious how Palo Alto has implemented their routing stack, since they do share states and when a PAN firewall dies, the standby unit picks up right away. I'm REALLY hoping they used some open source components for this, as code might be available.
-
Your work is very interresting. We have two test plateforms with 2 pfsense HA with CARP and BGP Peers, and the same with Debian Quagga. We can help you to test and finalize your package ;)
We have chosen to distribute full BGP routes with OSPF because BGP Package on PFSense seems not working as we expected (see https://forum.pfsense.org/index.php?topic=129317.0)Bertrand
-
I'm really happy to hear someone is working on enabling BGP in Quagga. I also find OpenBGPd to have surprising (and unpredictable) behavior, so replacing it with widely-used and well-understood Quagga would be a big step forward.
-
Glad you got this all working. So just to confirm, this is the Quagga versions of OSPF and BGP running together? Would love to hear more details about your implementation.
I'm in the same boat considering what my options are going forward and trying to support Quagga BGP on pfsense. Ideally I would like to get the Cumulus version of Quagga on pfsense as that brings several nice new features to the table.
Your original comment about getting the bgpd to run is where I started down this long road back in… November? Eventually I dug into the existing plugin more and modified the code to store the BGP config in config.xml etc. The biggest roadblocks i had were dealing with md5 keys and i managed to script that as well. We're running HA pairs of pfSense firewalls and I've got it working with CARP as well - that, fortunately, didn't require any modifications from the original plugin. The bummer is that when one FW fails, the quagga process has to start cold on the other FW, so you lose routing for 30-60 seconds depending on protocols in use. I have dreams of figuring out how to do state transfers with this someday, but i'm not sure if quagga is the best choice for that.
I'm really curious how Palo Alto has implemented their routing stack, since they do share states and when a PAN firewall dies, the standby unit picks up right away. I'm REALLY hoping they used some open source components for this, as code might be available.
I would imagine the solution would be to have Quagga running on both boxes all the time, so that the route table is ready to go. There is a patch available which stops Quagga getting restarted with everything else when there is an IP/GW change so maybe this would do the same for CARP failover?
http://files.atx.pfsense.org/jimp/patches/skip_restart_for_routing_packages-2.3.1.patch
-
Some of you may have noticed a new version of QuaggaOSPF out there - Looks like some people got on this and did some of the basic legwork.
https://github.com/pfsense/FreeBSD-ports/pull/356
This, of course, makes me pretty happy as i'd love to stop installing my own custom stuff but there's still a couple of glaring issues:
-
MD5 passwords have not been implemented at the OS level - I managed to solve for this, hopefully I can get them to put my code into place.
-
I made a number of modifications to the raw config page so that changes made via vtysh would be visible and copyable into the overall pfSense config file
I added my code onto the tail end of that thread, so you all can finally see what I came up with. If i get time in the next few days, I may try putting my own code into github and see if we can get it merged into prod!
-
-
You're more than welcome to contribute to the package. You'll have to submit it as a proper github PR for it to be considered, however.
Some light reading:
https://doc.pfsense.org/index.php/Getting_Started_with_pfSense_Developmenthttps://help.github.com/articles/working-with-forks/
https://help.github.com/articles/creating-a-pull-request-from-a-fork/
https://help.github.com/articles/creating-a-pull-request/ -
You're more than welcome to contribute to the package. You'll have to submit it as a proper github PR for it to be considered, however.
Some light reading:
https://doc.pfsense.org/index.php/Getting_Started_with_pfSense_Developmenthttps://help.github.com/articles/working-with-forks/
https://help.github.com/articles/creating-a-pull-request-from-a-fork/
https://help.github.com/articles/creating-a-pull-request/I've contributed and it looks like things are moving along - what happens next?
https://github.com/pfsense/FreeBSD-ports/pull/360
-
I was out on vacation last week. I don't see anything holding that one up, probably just that there wasn't someone with free time to do a final review+merge.
-
I was out on vacation last week. I don't see anything holding that one up, probably just that there wasn't someone with free time to do a final review+merge.
OK, I'll keep an eye on github if any changes are necessary.
-
I was out on vacation last week. I don't see anything holding that one up, probably just that there wasn't someone with free time to do a final review+merge.
Any idea what's holding this up? I've made a number of changes since the last submission and didn't want to start another pull request until the previous one was merged.
Thanks!
-
Mostly waiting on anyone else to test it and make sure it's OK before it gets committed.
-
I'm going to install PFSense on a box with a bunch of SFP+ ports and really want to use it for BGP. I found this thread and it was like a dream come true with BGP and OSPF!!!!
@admins can we push this through :)
Thanks guys!!
-
It was merged two weeks ago. It's already in there now.
-
Fantastic. I'm going to attempt to use this instead of VyOS or RouterOS. I love PFSense to be honest but i never tried to use it as a Core Edge Router before so this is new!
Thanks everyone!!
-
tdale sent me a message directly asking how to use the new BGP functionality in Quagga. I'm pasting the message I sent him here for everyone's future reference.
–-----------
The latest version of the package (0.6.20) has my changes rolled in! The BGP config is entirely manual. We have the customized package running on close to 30 pfSenae FWs in our environment and so far it's working pretty well. I'm going to submit the latest version pretty soon that more gracefully handles high availability situations by keeping both units in the pair active but re standby unit is automatically set to a really high OSPF cost or in the case of BGP, it inserts a pretending route map to make the path longer. I have the OSPF side working in production for the last month now and it's awesome.
As for how to use the plugin, what I do is setup a super basic OSPF config - setup an interface, on the interfaces page, then on the main page set a router id (I create alias on my loopback address and use that as you would with a traditional router - you can do this in the virtual IP section under the firewall dropdown) and enable logging.
Once you've done that, the quagga daemon starts up at which point I switch over to the CLI via SSH. The command to run from the CLI is "vtysh". The first thing I do is set the terminal length to 0 using the "terminal length 0" command due to some hinkyness with the app. At this point, you're dealing with a front end similar to a Cisco IOS 12.x and you can do whatever you want in there.
Do note that due to how md5 encryption is handled for BGP, you must set your source (your firewall) and destination (BGP peer) addresses and password in the quagga -> raw config page. Note that due to a glitch in the pfsense package editor, if you have more than one password, you will have to enter it and save it twice. It's a glitch beyond this pagkage but I hope to fix it soon. .
The important part to know is that running the "write mem" command does NOT save your config to the pfSense /conf/config.xml file however I made a very easy method to save your config permanently - you go to the Quagga OSPF plugin in the GUI and go to the raw config page. There you see 2 text boxes for each daemon. The "saved" config is what is loaded when quagga is started - this is pulled from /conf/config.xml - and a "running"config that is config you saved when you ran "write mem" from within the vtysh app. If a pair of boxes is orange, your saved and running configs do not match. If they are green, they are the same. You can click the "copy" button below the "running" config for each daemon and then go down to the bottom did the page and hit "saved. This reloads quagga and its daemons with your newly saved config!
Due to the way that quagga puts configs in different files I prefer to work from vtysh so that the configs are normalized, but I do occasionally make manual changes.
Hope this helps and I'd love to hear ideas to make this work better!
-
So right after i submitted this last update, i realized that "frr" (in many ways Quagga's successor) was just released by the pfSense team and it incorporates everything that the quagga package has including IS-IS and a GUI front end for all protocols. I'll be running this in my lab by the end of the day and applying my yet-unsubmitted quagga mods to this package now :D
In short, i highly suggest checking out the frr package, i suspect i'll be adopting it widely soon.
quad
-
frr is still in testing but I've been working on it for a few weeks now. It has numerous improvements over the Quagga package. It doesn't do IS-IS (yet?) but it does do BGP, OSPF, and OSPF6 in the GUI without the need for manual configuration. Though you can still manually configure it if you like.
I've still got a few things left to do like input validation
-
frr is still in testing but I've been working on it for a few weeks now. It has numerous improvements over the Quagga package. It doesn't do IS-IS (yet?) but it does do BGP, OSPF, and OSPF6 in the GUI without the need for manual configuration. Though you can still manually configure it if you like.
I've still got a few things left to do like input validation
Good to know. Does FRR support multiple ospf instances/processes? I know Cumulus had made it work with quagga. This would be pretty beneficial in my environment.
-
No, it only supports one instance of OSPF or BGP, FRR may be capable of multiple but setting that up would be a lot more work for very little benefit considering how much it would be used.
-
No, it only supports one instance of OSPF or BGP, FRR may be capable of multiple but setting that up would be a lot more work for very little benefit considering how much it would be used.
Good to know. Sounds like I may have a new challenge for myself ;)