FRR - BGP - Multi-Path/ECMP

[jchinyou]

HI All,

I've got a simple setup that I seem to struggle getting to work correctly.

I have a single PFSense box running FRR. It is connected to OSPF on the LAN side and BGP peered over 2x VPN connections.

I can't seem to find a way to enable BGP Multi-Path/Load-Sharing over my 2 VPN links. Anyone know how to do this? Searching seems to have been fruitless. :(

LAN <-> PFSense <-> 2x VPN Tunnels <-> 2x Remote Routers (Single AS across both Remote Routers)

BGP via FRR only puts a single route in the routing table. No ECMP.

Is there a way to enable ECMP?

[ChrisT]

@jchinyou Not sure if this is related, but in order to run BGP, you will have first to install OpenBGPD. Go to the System/Package Manager and search for this package.

For the rest I cannot help because I am currently struggling to make OpenBGPD work.

[NogBadTheBad]

@christ said in FRR - BGP - Multi-Path/ECMP:

@jchinyou Not sure if this is related, but in order to run BGP, you will have first to install OpenBGPD. Go to the System/Package Manager and search for this package.

For the rest I cannot help because I am currently struggling to make OpenBGPD work.

Nope.

"FRR routing daemon for BGP, OSPF, and OSPF6 Conflicts with Quagga OSPF and OpenBGPD. These packages cannot be installed at the same time."

[topgun_kebab_hero_1000GTR]

Hi,
I can also confirm ecmp doesn't seem to be working. It is probably down to the compile options which I think need to explicitly set the ecmp max number of paths.
Cheers!
Jon.

[donaldsharp]

Hi!

I'm one of the maintainers of FRR and was asked about this, from the output I was given pfsense compiles FRR w/out any ecmp settings so the ecmp defaults to 1. If you want a higher value FRR needs to be recompiled with --enable-multipath=X where X is the desired max ecmp allowed.

[donaldsharp]

To circle back around, I just spoke with a pfsense developer and was informed that pfsense does not support multipath, so this will never work and as such there is no point in changing the way FRR is compiled.

[jwt]

To circle back around again, this is now slated for pfsense 2.5.

https://redmine.pfsense.org/issues/9544
https://redmine.pfsense.org/issues/9545

[tom7]

@jchinyou I have faced same issue and tried to build from source FRR in pfsense. Finally I confirmed ECMP works well. Let me share just example procedure how to build FRR7.0-stable in pfsense2.4.
pfSense FRR part in GUI partially doesn't work after installing new FRR. Please be careful.

#--------------------------------------------------------------------------
# Install FRR6 via pkg
# To generate user, group and /usr/local/etc/rc.d/frr
#--------------------------------------------------------------------------
pkg install -y frr6
 
# rc.conf
cat >> /etc/rc.conf <<EOF
frr_enable="YES"
frr_flags=""
frr_daemons="zebra bgpd"
frr_vtysh_boot="YES"
EOF
 
# generage a file
touch /usr/local/etc/frr/vtysh.conf
touch /usr/local/etc/frr/frr.conf
 
# restart
service frr restart
service frr status
vtysh -c "show version"


#--------------------------------------------------------------------------
# Reference
# http://docs.frrouting.org/projects/dev-guide/en/latest/building-frr-for-freebsd11.html
#--------------------------------------------------------------------------
# Enable FreeBSD repository
cat /etc/pkg/FreeBSD.conf > /usr/local/etc/pkg/repos/FreeBSD.conf
sed -ie "s/{ enabled: no }/{ enabled: yes }/g" /usr/local/share/pfSense/pkg/repos/pfSense-repo.conf
pkg update
 
# Install package
pkg install -y git
pkg install -y autoconf automake libtool gmake
pkg install -y json-c pkgconf bison flex
pkg install -y c-ares texinfo texinfo
pkg install -y py36-pytest py36-sphinx python36
pkg install -y cmake llvm gcc debhelper libyang
# gcc is needed to install build tool like "ld".

#--------------------------------------------------------------------------
# Workaround for build
#--------------------------------------------------------------------------
#/bin/sh: python: not found
#gmake: *** [Makefile:9515: tests/tests.xml] Error 127
ln -s /usr/local/bin/python3.6 /usr/local/bin/python
python --version
# > Python 3.6.8

# To pass gmake check
# Use clang instead of "gcc". PATH should be "cc"
ln -s /usr/local/bin/clang /usr/bin/cc
cc --version
# > clang version 8.0.0 (tags/RELEASE_800/final)
 
# Don't use gcc for compiling. use cc (clang8)
# gmake check will be failed by compiling gcc so use cc(clang8).
# check frr/config.log after compiling
rm /usr/local/bin/gcc

# Copy C-header files from another FreeBSD11 to pfsense
# on FreeBSD11
scp /usr/include/*.h root@PFSENSE:/usr/include
scp -r /usr/include/sys/ root@PFSENSE:/usr/include/
scp -r /usr/include/machine/ root@PFSENSE:/usr/include/
scp -r /usr/include/x86/ root@PFSENSE:/usr/include/
scp -r /usr/include/xlocale root@PFSENSE:/usr/include/
scp -r /usr/include/netinet root@PFSENSE:/usr/include/
scp -r /usr/include/netinet6 root@PFSENSE:/usr/include/
scp -r /usr/include/net root@PFSENSE:/usr/include/
scp -r /usr/include/arpa root@PFSENSE:/usr/include/

#--------------------------------------------------------------------------
# Build FRR
#--------------------------------------------------------------------------
cd /root
git clone https://github.com/frrouting/frr.git frr
cd frr
git branch -a | grep 7.0
git checkout -t origin/stable/7.0
git branch
./bootstrap.sh
setenv MAKE gmake
setenv LDFLAGS -L/usr/local/lib
setenv CPPFLAGS -I/usr/local/include
ln -s /usr/local/bin/sphinx-build-3.6 /usr/local/bin/sphinx-build
./configure \
    --sysconfdir=/usr/local/etc/frr \
    --enable-pkgsrcrcdir=/usr/pkg/share/examples/rc.d \
    --localstatedir=/var/run/frr \
    --prefix=/usr/local \
    --enable-multipath=32 \
    --enable-user=frr \
    --enable-group=frr \
    --enable-vty-group=frrvty \
    --enable-configfile-mask=0640 \
    --enable-logfile-mask=0640 \
    --enable-fpm \
    --with-pkg-git-version \
    --with-pkg-extra-version=-compiled
 
# Compile
gmake -j4
gmake check
 
# Install
gmake install
 
# restart
service frr restart
service frr status
vtysh -c "show version"

# Check
vtysh -c "show ip bgp ipv4 unicast"
#   Network          Next Hop            Metric LocPrf Weight Path
#*= 0.0.0.0/0       xxxx
#*>                       xxxx

[jimp]

That is an awful way to go about things. Installing a compiler environment on the firewall is a massive security no-no. Spin up a FreeBSD VM and just compile it from ports without all the hoop jumping.

[tom7]

@jimp I didn't know that compiler environment has security risk. I will only use compiled binary which is copied from FreeBSD to pfsense.

[MarceloAlm_]

Hi we are building a similar network and in need to decide between netgate or mikrotik router. There is any chance to enable ecmp on current pfsense?

[A Former User]

@marceloalm_

Hi we are building a similar network and in need to decide > between netgate or mikrotik router. There is any chance to > enable ecmp on current pfsense?

I would assume that you will be better go with MikroTik
RouterOS or VyOS.

https://wiki.mikrotik.com/wiki/ECMP_load_balancing_with_masquerade