Source based Routing with pfSense
-
The style of routing described on that link won't work since pfSense doesn't enable the options for multiple routing tables.
What you want is actually called Policy Routing and works fine in pfSense using
route-toinpf.You can match traffic in firewall rules, by source or by other other matching criteria you want, and set a gateway to pick which WAN the traffic will exit.
It's well-covered in the docs under Multi-WAN and also in several hangouts.
-
That might work vor TCP or even UDP Traffic, but ICMP does not work.
I got two external public IPv6 Addresses, from different Networks
From the internet, I can only ping one of them, because policy based routing routes the answer from the second IPv6 Address through the 'wrong' interface - because in the policy I can only refer to networks but not to interfaces.
The accomplished scenario would be
ping -> IPv6-Address1 (em0) -> answer ICMP request from em0 as source
ping -> IPv6 Address2 (em1) -> answer ICMP request from em1 as source.Totaly independent of the IP source address of the ping origin, which could be the same.
Currently I have to define one default gw of the two Networks, to make it happen for that interface.
If I define two gateways (for each network one) and no default gateway it does not work at all.I did not find a solution for this for now....
If I miss something in the policy based routing
Cheers,
4920441
-
That isn't policy based or source routing. That would be handled via
reply-toand it also works fine when you have the interfaces properly configured as WAN-type interfaces.If you have an address on each interface, and a gateway set on each interface (setup in System > Routing and selected on Interfaces > WAN1 and Interfaces > WAN2 for example) then it will work. It doesn't matter which one is default, it will respond properly from the appropriate interface/gateway.
And it always works with any protocol, again, assuming your interfaces are configured correctly.
-
But why isn't it working then?
IPv6 I got two seperate IPv6 gateways.
Currently, the Hurricane Electrics is default gw and works fine - including all subnets behind it (I got half a dozen /64 configured on the trusted site)
The IPv6 Address I got from 'WAN_DHCP6' is also usable - I use it for pfsense to connect via IPSec and openvpn to to other gateways.
When I try to ping the WAN_DHCP6 Address from external I dont get any echos a tcpdump shows me that all packets are leaving via the Hurricane Electric Gateway.When I change the default gateway to the WAN_DHCP6 gateway, everything from external is answered correctly, but now only the WAN_DHCP6 gateway is working properly, the Hurricane Electrics has seized working... despite the fact I have some policy based outgoing routes set in the firewall rules....:
Or is it simply because my provider (Deutsche Telekom) submits me a link lokal Address as default gateway which confuses pfsense somehow? - despite it is working fine if it is the only ipv6 gateway...:
Maybe it helps to mention I opened the case a couple of weeks ago in a similar manner but I worked kinda around the problem...:
https://forum.netgate.com/topic/141353/pppoe-ipv6-hurricane-electrics-ipv6-assymmetric-routing-no-ping-from-external-to-pppoe-assigned-ipv6-adddress-possibile-despite-that-everything-is-working
Thanks a lot for your help... I
Cheers
4920441
-
I already outlined the things you need to check.
I have two IPv6 WANs here (both HE.net) and they both work fine, and I can send traffic out either one, and contact either one from remote locations and it all works as expected. Link-local gateways wouldn't confuse it, that's expected with IPv6 in many cases.
-
Works fine. This is my lab WAN that uses part of the HE.NET /48
local_nets_v6 includes the /56 PD from Cox, the /48 from HE.NET, and fc00::/7
-
-
No. That is on the interface I want to go out the HENET tunnel for everything but local IPv6 traffic.
Most interfaces need nothing and use the default gateway (/56 PD from the ISP)
-
Hi,
for me it is not working, despite everything should be fine according to the webinterface...
Since my pfsense router config is grown since 2011, its maybe possible that the config shown in the webinterface is not the actual running config any more...How can I doublecheck which rules are applied on the console? something like iptables-save or iptables -lav
The funny thing is, despite the fact that there is only one single rule for ipv6 icmp packets, it does not match if I send a ipv6 icmp packet from external (counter in webinterface does not get higher).... So maybe my config is screwed over the last 8 years...
Cheers
4920441
-
@4920441-0 said in Source based Routing with pfSense:
despite everything should be fine according to the webinterface...
What I have found over the years as users quite often say its configured like X, but in reality when they post up their rules/config pages its really like Y..
If you want to view the full rules
https://docs.netgate.com/pfsense/en/latest/firewall/viewing-the-full-pf-ruleset.html -
I know, I am confronted with that issue myself every day...
But making lots of screenshot won't help, this config has a dozen vlans multiple ipsec and openvpn connections and posting all firewall rules with pngs would be more confusing than being helpful, I think.I grepped (and anonymized) all pppoe related rules with icmp context (okay, there might be son "any proto" rules, but there are not, I checked it, there is not one single "proto any" rule
The first one I cannot find in the webinterface labled "NEGATE_ROUTE:..." is this some kind of default route? Since it is for ipv4 and ipv6 valid and icmp, it could be my problem here...:
grep pppoe rules-all.txt | grep -i icmp pass in quick on pppoe inet proto icmp from any to <negate_networks> keep state label "NEGATE_ROUTE: Negate policy routing for destination" pass in quick on pppoe route-to (pppoe0 62.123.34.56) inet proto icmp all keep state label "USER_RULE" pass in quick on pppoe0 reply-to (pppoe0 62.123.34.56) inet proto icmp from any to (self) keep state label "USER_RULE" pass in log quick on pppoe0 route-to (pppoe0 fe80::123:ab1:33ab:accc) inet6 proto ipv6-icmp from any to (self) keep state label "USER_RULE" pass in quick on pppoe0 reply-to (pppoe0 62.123.34.56) inet proto icmp from 66.1.2.3 to any keep state label "USER_RULE: HurricaneElectics" pass in quick on pppoe0 reply-to (pppoe0 62.123.34.56) inet proto icmp from 216.66.80.30 to any keep state label "USER_RULE: HurricaneElectics" pppoe0 icmp 217.12.34.56:65361 -> 217.76.54.32:65361 0:0 pppoe0 ipv6-icmp 2003:aa:bbbb:2222:3333:444f:555f:666f[182] -> 2620:fe::9[182] NO_TRAFFIC:NO_TRAFFIC pppoe0 icmp 217.12.34.56:36974 (192.168.64.61:3691) -> 8.8.8.8:36974 0:0Thanks a lot for your patience..
Cheers
4920441
-
@4920441-0 said in Source based Routing with pfSense:
But making lots of screenshot won't help, this config has a dozen vlans multiple ipsec and openvpn connections and posting all firewall rules with pngs would be more confusing than being helpful
Then post what is necessary to display a single example of a single connection that you think should be behaving one way but is behaving a different way.
Negate routes are an attempt to automatically bypass policy routing for VPN routes. If you look in /tmp/rules.debug you can see what is listed in that table. If you know it is causing problems you can disable those in System > Advanced, Firewall & NAT, Disable Negate Rules.
-
@4920441-0 said in Source based Routing with pfSense:
pass in log quick on pppoe0 route-to (pppoe0 fe80::123:ab1:33ab:accc) inet6 proto ipv6-icmp from any to (self) keep state label "USER_RULE"You manually configured a gateway on a WAN interface rule. Don't do that.
-
despite that "USER_RULE" tag I remember the gateway was automatically added when IPv6 was configured for the WAN interface.
But the WAN Interface had had several states of evolution .... first it was only 6to4, after myprovider switched to dual stack it came via DHCP over pppoe - maybe in the transition period something went wrong.
Should I delete the IPv6 Gateway? For testing I disabled it right now, but without it could not work either.
Cheers,
-
...as said, I cannot see any rule which is responsible for that behaviour... neither an obvious routing misconfiguration....:
Is there a way to check/'dump' with pfctl what my tcpdump does on layer3, so I can see which rule is involved?[2.4.4-RELEASE][root@router]/root: tcpdump -nnfi pppoe0 icmp6 and host 2601:183:0:3131:11d2:2128:af93:c6c9 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on pppoe0, link-type NULL (BSD loopback), capture size 262144 bytes 10:12:38.779234 IP6 2601:183:0:3131:11d2:2128:af93:c6c9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo request, seq 17, length 64 10:12:39.803005 IP6 2601:183:0:3131:11d2:2128:af93:c6c9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo request, seq 18, length 64 10:12:40.823277 IP6 2601:183:0:3131:11d2:2128:af93:c6c9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo request, seq 19, length 64 10:12:41.847030 IP6 2601:183:0:3131:11d2:2128:af93:c6c9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo request, seq 20, length 64 10:12:42.871295 IP6 2601:183:0:3131:11d2:2128:af93:c6c9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo request, seq 21, length 64 ^C 5 packets captured 656 packets received by filter 0 packets dropped by kernelIf I Initiate a (outgoing) ping on the pppoe Ipv6 Interface, everything is working fine:
[2.4.4-RELEASE][root@rotorouter]/root: tcpdump -nnfi pppoe0 icmp6 and host 2620:fe::9 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on pppoe0, link-type NULL (BSD loopback), capture size 262144 bytes 10:16:24.688594 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2620:fe::9: ICMP6, echo request, seq 6558, length 8 10:16:24.703858 IP6 2620:fe::9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo reply, seq 6558, length 8 10:16:25.195955 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2620:fe::9: ICMP6, echo request, seq 6559, length 8 10:16:25.210875 IP6 2620:fe::9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo reply, seq 6559, length 8 10:16:25.699812 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2620:fe::9: ICMP6, echo request, seq 6560, length 8 10:16:25.714615 IP6 2620:fe::9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo reply, seq 6560, length 8 10:16:26.210344 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2620:fe::9: ICMP6, echo request, seq 6561, length 8 10:16:26.225103 IP6 2620:fe::9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo reply, seq 6561, length 8 10:16:26.734244 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2620:fe::9: ICMP6, echo request, seq 6562, length 8 10:16:26.749107 IP6 2620:fe::9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo reply, seq 6562, length 8 ^C 10 packets captured 331 packets received by filter 0 packets dropped by kernelOutgoing pings via the WAN Interfae work fine and are answered on the same Network Interface.
But Incoming Pings are reply'ed on the wrong interface, the leave the firewall on the hurricane electrics tunnel, not on the interface they are received in the first place:
[2.4.4-RELEASE][root@router]/root: tcpdump -nnfi gif0 icmp6 and host 2601:183:0:3131:11d2:2128:af93:c6c9 tcpdump: WARNING: foreign (-f) flag used but: gif0: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on gif0, link-type NULL (BSD loopback), capture size 262144 bytes 10:18:31.035483 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2601:183:0:3131:11d2:2128:af93:c6c9: ICMP6, echo reply, seq 361, length 64 10:18:32.055503 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2601:183:0:3131:11d2:2128:af93:c6c9: ICMP6, echo reply, seq 362, length 64 10:18:33.079271 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2601:183:0:3131:11d2:2128:af93:c6c9: ICMP6, echo reply, seq 363, length 64 10:18:34.103536 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2601:183:0:3131:11d2:2128:af93:c6c9: ICMP6, echo reply, seq 364, length 64 10:18:35.127287 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2601:183:0:3131:11d2:2128:af93:c6c9: ICMP6, echo reply, seq 365, length 64 10:18:36.155275 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2601:183:0:3131:11d2:2128:af93:c6c9: ICMP6, echo reply, seq 366, length 64 ^C 6 packets captured 144 packets received by filter 0 packets dropped by kernel -
@4920441-0 Look at the rule set. You should not be explicitly setting a gateway on any WAN rules. Again, post your rules.
-
@4920441-0 said in Source based Routing with pfSense:
@jimp
despite that "USER_RULE" tag I remember the gateway was automatically added when IPv6 was configured for the WAN interface.User rules do not work that way. You had to have manually added that rule and manually picked a gateway on that rule. A gateway on the interface does not automatically get set on a rule, ever.
-
Hi @Derelict
since you kindly asked for all my rules, here they are. I hope I did not miss anything which should be anonymized....
Please take a look adnd tell me what I overlook.... .
Thanks alot for your help, really appreciated.
Since it didn't make sense to put all rules directly in this post, I attached a txt file.
Cheers
4920441
-
Hi,
I have 3 WAN connections, configured the policy routing and working as expected, but incoming connection on WAN1 or WAN2 from IP: a.b.c.192 (when WAN3 ip is a.b.c.62, GW: a.b.c.1 ) responded on WAN3. Can I somehow resolve this? I expect the response to go out through incoming interface.
Thanks
Levi -
@birtalevente
Dude, this thread is almost two years old, so let him rest in peace!@birtalevente said in Source based Routing with pfSense:
but incoming connection on WAN1 or WAN2 from IP: a.b.c.192 (when WAN3 ip is a.b.c.62, GW: a.b.c.1 ) responded on WAN3.
Can you describe your WAN interface configuration a little more detailed?
It's not really clear when you use alphabetic characters and not mention any network mask. Are these connected to different internet providers, are they in different subnets with a gateway on each?
