@sipher
issue solved from the original post, just disable "reply-to" from the FW rule.

https://forum.netgate.com/topic/165849/how-to-enable-asymmetric-routing-on-pfsense-frr?_=1629724281949

@viktor_g

Agreed. I recompiled frr, and replaced zebra and ospfd, and it now correctly advertises routes over the gre tunnels. Thank you very much for the pointer!

@spearless
Under BGP, when I add a neighbor, I have an option for 'update source' that I can set to the carp ip. Do you have a similar setting under OSPF?

redmine issue: https://redmine.pfsense.org/issues/11610

Please create a feature request:
https://docs.netgate.com/pfsense/en/latest/development/feature-requests.html

@wtholmes there is a feature request: https://redmine.pfsense.org/issues/11130

@nzkiwi68

I see https://forum.netgate.com/topic/162722/frr-doesn-t-follow-carp-after-2-5-0-upgrade

Could it be that I need to apply a system patch?

If yes, is it;
7dbe76cd5756082cbd67db1b93acb606ad84996e
or the later one
99b3a5cb0ef4586222a331045df3cee17bb25d31

from:
https://redmine.pfsense.org/issues/11290#note-12

@viktor_g said in No connection to RPKI cache server:

/var/etc/frr/frr.conf

cat /var/etc/frr/frr.conf ##################### DO NOT EDIT THIS FILE! ###################### ################################################################### # This file was created by an automatic configuration generator. # # The contents of this file will be overwritten without warning! # ################################################################### ! frr defaults traditional hostname password log syslog service integrated-vtysh-config service password-encryption ! ip router-id 10.50.1.254 ! ip route 240e:ff:f000::/36 pppoe0 ip route 240e:bc::/31 pppoe0 ip route 2402:4e00::/32 pppoe0 ip route 2402:4e00:1800::/40 pppoe0 ip route 240e:688::/32 pppoe0 ip route 2401:b180::/32 pppoe0 ip route 240e:96c::/32 pppoe0 ip route 2001:da8:215::/48 pppoe0 ip route 2001:da8::/32 pppoe0 ip route 2402:f000::/32 pppoe1 ip route 2408:8256:681::/48 pppoe0 ip route 2408:8256::/36 pppoe0 ip route 2408:8256::/32 pppoe0 ip route 2001:250:1001::/48 pppoe0 ip route 2001:250::/32 pppoe0 ip route 240e::/24 pppoe0 ip route 240e:358::/29 pppoe0 ip route 2409:8a55:800::/40 pppoe0 ip route 2409:8a55::/32 pppoe0 ip route 2409:8000::/20 pppoe0 ip route 2400:dd01:1032::/48 pppoe0 ip route 2400:dd00::/28 pppoe0 ip route 240d:c040::/44 pppoe0 ip route 2001:df6:f400::/48 pppoe0 ip route 2408:874c::/32 pppoe0 ip route 2408:4000::/22 pppoe0 ip route 2408:4001::/33 pppoe0 ip route 240e:83::/37 pppoe0 ip route 240e:0:9000::/37 pppoe0 ! router bgp 65105 bgp log-neighbor-changes no bgp default ipv4-unicast bgp router-id 10.50.1.254 timers bgp 180 300 bgp default local-preference 100 no bgp fast-external-failover no bgp network import-check bgp deterministic-med bgp always-compare-med bgp bestpath as-path confed bgp bestpath med confed no bgp ebgp-requires-policy neighbor 2a0d:2 remote-as 59753 neighbor 2a0d: description fr bgp neighbor 2a0d: update-source 2a0d:240 neighbor 2602: remote-as 59753 neighbor 2602:f description FMT neighbor 2602:fed2:7020:ca:: update-source 2602:fed2: neighbor 2602:fed2:7020:ca:: capability dynamic neighbor 2602:f remote-as 59753 neighbor 2602:fed description fmt2 zhu neighbor 2602:fed update-source 2602:fe neighbor 2602:f capability dynamic ! address-family ipv6 unicast redistribute static redistribute kernel network 2602:fed2:5021::/48 neighbor 2a0d:2406 activate neighbor 2602:fed2:activate neighbor 2602:feda activate no neighbor 2a0d:240 send-community neighbor 2a0d:2406: next-hop-self neighbor 2a0d:240 soft-reconfiguration inbound neighbor 2a0d:240 prefix-list ipv6in in neighbor 2a0d:240 prefix-list myv6out out no neighbor 2602:fed send-community neighbor 2602:fed2: next-hop-self neighbor 2602:fed2: soft-reconfiguration inbound neighbor 2602:fed2 prefix-list ipv6in in neighbor 2602:fed2 prefix-list myv6out out no neighbor 2602:feda: send-community neighbor 2602:feda: next-hop-self neighbor 2602:feda soft-reconfiguration inbound neighbor 2602:feda: prefix-list ipv6in in neighbor 2602:feda: prefix-list myv6out out exit-address-family ! ! rpki rpki polling_period 600 rpki expire_interval 3600 rpki retry_interval 600 rpki cache 134.195.121.55 3323 preference 1 rpki cache 2602:fed 3323 preference 2 rpki cache rpki-validator.realmv6.org 8282 preference 3 ! ipv6 prefix-list ipv6in seq 200 permit any ipv6 prefix-list myv6out seq 50 permit 2602:fed2:5021::/48 ipv6 prefix-list myv6out seq 999 deny any ipv6 prefix-list myv6out description my ipv6 out ! route-map FR deny 20 match rpki invalid route-map FR permit 30 set metric 5 set local-preference 100 match rpki notfound route-map FR permit 50 set metric 0 set local-preference 110 match rpki valid ! line vty ! end

@beb-consulting I'm having the exact same problem. The service simply won't start. When I try to start it via the terminal here's what I get:

[21.05-RELEASE][admin@xg71001u]/var/etc/frr: /usr/local/etc/rc.d/frr restart all Checking intergrated config... Checking vtysh.conf line 15: % Unknown command[4]: no bgp network import-check line 16: % Unknown command[4]: neighbor 169.254.0.1 remote-as 4200000002 line 17: % Unknown command[4]: neighbor 169.254.0.1 description Google Cloud HA VPN (us-central1) line 18: % Unknown command[4]: neighbor 169.254.0.1 update-source 169.254.0.2 line 20: % Unknown command[4]: address-family ipv4 unicast line 21: % Unknown command[4]: neighbor 169.254.0.1 activate line 22: % Unknown command[4]: no neighbor 169.254.0.1 send-community line 23: % Unknown command[4]: neighbor 169.254.0.1 prefix-list InboundNetworks in line 24: % Unknown command[4]: neighbor 169.254.0.1 prefix-list OutboundNetworks out line 25: % Unknown command[4]: exit-address-family FAILED

I have yet to find a solution and removing/reinstalling doesn't work at all.

UPDATE: Well this is embarrassing... Check the length of your ASN; e.g., If you're using a 4 byte ASN sure it's 10 digits... not 11. 😬

That was the reason BGP wasn't starting in my situation (one too many zeroes).

EDIT: To clarify, I had the BGP session up and running at one point but I needed to change the ASN in addition to a few other settings, and after making all my changes and restarting the service, that's when all hell broke loose. The snippet I posted earlier threw me off the trail since it gives no indication that you may have typo'd something in the GUI (side note to the frr package maintainers: can we get amaxlength="10" on that field and/or a better error message? 😀). You might also notice the ASN is the correct length in the output above; it was my local ASN that was incorrect.

EDIT #2: I'm full of the typos today, apparently.

Update 2:
Added an alias for RFC1918 networks and configured an outbound NAT rule with RFC1918 as source and any destination on all pfSenses.
This solved what seemed like a routing problem but turned out to be a NATing problem.
However I'll probably have issues if/when I have multiple WAN connections.
Still would like to hear if there are any best practices.

Any ideas?

Edit: I forgot to mention the routing works when package is coming from LAN, but doesn't work when it's coming back from WAN.

@defunct78 it looks like this was raised as a 'harmless error'. Presume there'll be a fix along in a future update.

https://redmine.pfsense.org/issues/12084