@nzkiwi68

I see https://forum.netgate.com/topic/162722/frr-doesn-t-follow-carp-after-2-5-0-upgrade

Could it be that I need to apply a system patch?

If yes, is it;
7dbe76cd5756082cbd67db1b93acb606ad84996e
or the later one
99b3a5cb0ef4586222a331045df3cee17bb25d31

from:
https://redmine.pfsense.org/issues/11290#note-12

@viktor_g said in No connection to RPKI cache server:

/var/etc/frr/frr.conf

cat /var/etc/frr/frr.conf ##################### DO NOT EDIT THIS FILE! ###################### ################################################################### # This file was created by an automatic configuration generator. # # The contents of this file will be overwritten without warning! # ################################################################### ! frr defaults traditional hostname password log syslog service integrated-vtysh-config service password-encryption ! ip router-id 10.50.1.254 ! ip route 240e:ff:f000::/36 pppoe0 ip route 240e:bc::/31 pppoe0 ip route 2402:4e00::/32 pppoe0 ip route 2402:4e00:1800::/40 pppoe0 ip route 240e:688::/32 pppoe0 ip route 2401:b180::/32 pppoe0 ip route 240e:96c::/32 pppoe0 ip route 2001:da8:215::/48 pppoe0 ip route 2001:da8::/32 pppoe0 ip route 2402:f000::/32 pppoe1 ip route 2408:8256:681::/48 pppoe0 ip route 2408:8256::/36 pppoe0 ip route 2408:8256::/32 pppoe0 ip route 2001:250:1001::/48 pppoe0 ip route 2001:250::/32 pppoe0 ip route 240e::/24 pppoe0 ip route 240e:358::/29 pppoe0 ip route 2409:8a55:800::/40 pppoe0 ip route 2409:8a55::/32 pppoe0 ip route 2409:8000::/20 pppoe0 ip route 2400:dd01:1032::/48 pppoe0 ip route 2400:dd00::/28 pppoe0 ip route 240d:c040::/44 pppoe0 ip route 2001:df6:f400::/48 pppoe0 ip route 2408:874c::/32 pppoe0 ip route 2408:4000::/22 pppoe0 ip route 2408:4001::/33 pppoe0 ip route 240e:83::/37 pppoe0 ip route 240e:0:9000::/37 pppoe0 ! router bgp 65105 bgp log-neighbor-changes no bgp default ipv4-unicast bgp router-id 10.50.1.254 timers bgp 180 300 bgp default local-preference 100 no bgp fast-external-failover no bgp network import-check bgp deterministic-med bgp always-compare-med bgp bestpath as-path confed bgp bestpath med confed no bgp ebgp-requires-policy neighbor 2a0d:2 remote-as 59753 neighbor 2a0d: description fr bgp neighbor 2a0d: update-source 2a0d:240 neighbor 2602: remote-as 59753 neighbor 2602:f description FMT neighbor 2602:fed2:7020:ca:: update-source 2602:fed2: neighbor 2602:fed2:7020:ca:: capability dynamic neighbor 2602:f remote-as 59753 neighbor 2602:fed description fmt2 zhu neighbor 2602:fed update-source 2602:fe neighbor 2602:f capability dynamic ! address-family ipv6 unicast redistribute static redistribute kernel network 2602:fed2:5021::/48 neighbor 2a0d:2406 activate neighbor 2602:fed2:activate neighbor 2602:feda activate no neighbor 2a0d:240 send-community neighbor 2a0d:2406: next-hop-self neighbor 2a0d:240 soft-reconfiguration inbound neighbor 2a0d:240 prefix-list ipv6in in neighbor 2a0d:240 prefix-list myv6out out no neighbor 2602:fed send-community neighbor 2602:fed2: next-hop-self neighbor 2602:fed2: soft-reconfiguration inbound neighbor 2602:fed2 prefix-list ipv6in in neighbor 2602:fed2 prefix-list myv6out out no neighbor 2602:feda: send-community neighbor 2602:feda: next-hop-self neighbor 2602:feda soft-reconfiguration inbound neighbor 2602:feda: prefix-list ipv6in in neighbor 2602:feda: prefix-list myv6out out exit-address-family ! ! rpki rpki polling_period 600 rpki expire_interval 3600 rpki retry_interval 600 rpki cache 134.195.121.55 3323 preference 1 rpki cache 2602:fed 3323 preference 2 rpki cache rpki-validator.realmv6.org 8282 preference 3 ! ipv6 prefix-list ipv6in seq 200 permit any ipv6 prefix-list myv6out seq 50 permit 2602:fed2:5021::/48 ipv6 prefix-list myv6out seq 999 deny any ipv6 prefix-list myv6out description my ipv6 out ! route-map FR deny 20 match rpki invalid route-map FR permit 30 set metric 5 set local-preference 100 match rpki notfound route-map FR permit 50 set metric 0 set local-preference 110 match rpki valid ! line vty ! end

@beb-consulting I'm having the exact same problem. The service simply won't start. When I try to start it via the terminal here's what I get:

[21.05-RELEASE][admin@xg71001u]/var/etc/frr: /usr/local/etc/rc.d/frr restart all Checking intergrated config... Checking vtysh.conf line 15: % Unknown command[4]: no bgp network import-check line 16: % Unknown command[4]: neighbor 169.254.0.1 remote-as 4200000002 line 17: % Unknown command[4]: neighbor 169.254.0.1 description Google Cloud HA VPN (us-central1) line 18: % Unknown command[4]: neighbor 169.254.0.1 update-source 169.254.0.2 line 20: % Unknown command[4]: address-family ipv4 unicast line 21: % Unknown command[4]: neighbor 169.254.0.1 activate line 22: % Unknown command[4]: no neighbor 169.254.0.1 send-community line 23: % Unknown command[4]: neighbor 169.254.0.1 prefix-list InboundNetworks in line 24: % Unknown command[4]: neighbor 169.254.0.1 prefix-list OutboundNetworks out line 25: % Unknown command[4]: exit-address-family FAILED

I have yet to find a solution and removing/reinstalling doesn't work at all.

UPDATE: Well this is embarrassing... Check the length of your ASN; e.g., If you're using a 4 byte ASN sure it's 10 digits... not 11. 😬

That was the reason BGP wasn't starting in my situation (one too many zeroes).

EDIT: To clarify, I had the BGP session up and running at one point but I needed to change the ASN in addition to a few other settings, and after making all my changes and restarting the service, that's when all hell broke loose. The snippet I posted earlier threw me off the trail since it gives no indication that you may have typo'd something in the GUI (side note to the frr package maintainers: can we get amaxlength="10" on that field and/or a better error message? 😀). You might also notice the ASN is the correct length in the output above; it was my local ASN that was incorrect.

EDIT #2: I'm full of the typos today, apparently.

Update 2:
Added an alias for RFC1918 networks and configured an outbound NAT rule with RFC1918 as source and any destination on all pfSenses.
This solved what seemed like a routing problem but turned out to be a NATing problem.
However I'll probably have issues if/when I have multiple WAN connections.
Still would like to hear if there are any best practices.

Any ideas?

Edit: I forgot to mention the routing works when package is coming from LAN, but doesn't work when it's coming back from WAN.

@defunct78 it looks like this was raised as a 'harmless error'. Presume there'll be a fix along in a future update.

https://redmine.pfsense.org/issues/12084

Hi,
I have you correct your problem?

I migrate to 2.5.1 : I had ipv4 route but no ipv6 route with bgp
I had route maps and prefix lists on neighbors with no success

have you an configuration example ?

Very strange, I use pfSense 2.4.5-p1, 2.5.1-REL, 2.5.2-RC, 2.6.0-DEV with FRR doing BGP and this issue has always been solvable.

Only showstopper I have come across these past few weeks is https://redmine.pfsense.org/issues/11545 - If you assign virtual IPv6 addresses to WAN, FRR in pfSense may decide to use one of them for outbound communication, even to your BGP peers which will usually result in the session stuck in "active" and not getting "established" because the peer expects you to come from a specific address.

As an update, I have done some more troubleshooting on the issue:

Switching to static routes over the VTI tunnel works. Using regular tunnel IPv4 also works Its only when we use FRR via OSPF (have not tested BGP) that traffic does not flow between hub and spoke.

Topology is 1 Hub (virtual) with 3 spokes (2 virtual, 1 physical pfSense). Its the physical pfSense spoke that is having issue

Enable IPsec MSS Clamping with different values, 1400, 1350, 1200, etc. on both hub and spoke and no issue. Also adjusted the VTI MTU value as well with no luck

Both sides are using AES-NI CPU Crypto. Enable/Disabling this has no effect

Both sides are using IPsec Asynchronous Cryptography. Enable/Disabling this has no effect

Tried different P2 encryption options but no luck. Currently using

P1: AES128-GCM (128 bits) AES-XCBC via 14 (2048) DH Group P2: ESP AES128-GCM (128 bits) PFS Group: 14 (2048). NO Hash algorithms

It appears another user on Reddit is facing similar issues: https://www.reddit.com/r/PFSENSE/comments/mzab6v/251_and_ipsec_vti/

Any ideas why FRR and OSPF is not sending traffic over the network? What troubleshooting steps can I take to debug this further?

@ghezzino

This is off topic in the FRR category.

I opened a chat.

@zawi FRR OSPF in 2.5.1 in the overview screen: E.g. I don't know what Interface opt2 or opt3 is, if i do not enter a description manually. It would be very helpful to show the interface description from pfsense in this screen as well. Like:

opt2 ('Description from pfSense') => opt2 (OpenVPNS1)

Unbenannt.PNG

@kom Oh nice. Didn't see that. Thanks.

@jimp

That worked absolutely perfectly. Thanks!

My Fault.
in AS PATH BGP i configure "set" instead of "set prepend".

You need to install the System Patches package: https://docs.netgate.com/pfsense/en/latest/development/system-patches.html
And apply Patch ID 7dbe76cd5756082cbd67db1b93acb606ad84996e

Then you need to reinstall the FRR package.

see https://redmine.pfsense.org/issues/11290#note-12