Source based Routing with pfSense

4920441 0 · Apr 6, 2019, 9:39 AM

Hi,

could anyone give me a hint, who I accomplisch source based routing, 'maybe' without tampering directly in the pf.conf and generating routing tables around the webinterface directly in the shell?

https://www.mmacleod.ca/2011/06/source-based-routing-with-freebsd-using-multiple-routing-table/

Maybe I am missing the forest for the trees.... thanks a lot for your help.

Cheers,

JeGr · Apr 10, 2019, 3:11 PM

What are you wanting to do? That is an 8yo article with nothing more than problems, that don't exist (for me) with pfsense. So unless you tell us what you want to do or what isn't working, just linking things with problems in year old topics won't do anything to help.

jimp · Apr 10, 2019, 3:33 PM

The style of routing described on that link won't work since pfSense doesn't enable the options for multiple routing tables.

What you want is actually called Policy Routing and works fine in pfSense using route-to in pf.

You can match traffic in firewall rules, by source or by other other matching criteria you want, and set a gateway to pick which WAN the traffic will exit.

It's well-covered in the docs under Multi-WAN and also in several hangouts.

4920441 0 · Apr 10, 2019, 5:52 PM

@jimp

That might work vor TCP or even UDP Traffic, but ICMP does not work.

I got two external public IPv6 Addresses, from different Networks

From the internet, I can only ping one of them, because policy based routing routes the answer from the second IPv6 Address through the 'wrong' interface - because in the policy I can only refer to networks but not to interfaces.

The accomplished scenario would be

ping -> IPv6-Address1 (em0) -> answer ICMP request from em0 as source
ping -> IPv6 Address2 (em1) -> answer ICMP request from em1 as source.

Totaly independent of the IP source address of the ping origin, which could be the same.

Currently I have to define one default gw of the two Networks, to make it happen for that interface.
If I define two gateways (for each network one) and no default gateway it does not work at all.

I did not find a solution for this for now....

If I miss something in the policy based routing

Cheers,

4920441

jimp · Apr 10, 2019, 7:25 PM

That isn't policy based or source routing. That would be handled via reply-to and it also works fine when you have the interfaces properly configured as WAN-type interfaces.

If you have an address on each interface, and a gateway set on each interface (setup in System > Routing and selected on Interfaces > WAN1 and Interfaces > WAN2 for example) then it will work. It doesn't matter which one is default, it will respond properly from the appropriate interface/gateway.

And it always works with any protocol, again, assuming your interfaces are configured correctly.

4920441 0 · Apr 11, 2019, 7:10 PM

But why isn't it working then?

IPv6 I got two seperate IPv6 gateways.

Currently, the Hurricane Electrics is default gw and works fine - including all subnets behind it (I got half a dozen /64 configured on the trusted site)

The IPv6 Address I got from 'WAN_DHCP6' is also usable - I use it for pfsense to connect via IPSec and openvpn to to other gateways.
When I try to ping the WAN_DHCP6 Address from external I dont get any echos a tcpdump shows me that all packets are leaving via the Hurricane Electric Gateway.

When I change the default gateway to the WAN_DHCP6 gateway, everything from external is answered correctly, but now only the WAN_DHCP6 gateway is working properly, the Hurricane Electrics has seized working... despite the fact I have some policy based outgoing routes set in the firewall rules....:

Or is it simply because my provider (Deutsche Telekom) submits me a link lokal Address as default gateway which confuses pfsense somehow? - despite it is working fine if it is the only ipv6 gateway...:

login-to-view
login-to-view

Maybe it helps to mention I opened the case a couple of weeks ago in a similar manner but I worked kinda around the problem...:

https://forum.netgate.com/topic/141353/pppoe-ipv6-hurricane-electrics-ipv6-assymmetric-routing-no-ping-from-external-to-pppoe-assigned-ipv6-adddress-possibile-despite-that-everything-is-working

Thanks a lot for your help... I

Cheers

4920441

jimp · Apr 11, 2019, 7:13 PM

I already outlined the things you need to check.

I have two IPv6 WANs here (both HE.net) and they both work fine, and I can send traffic out either one, and contact either one from remote locations and it all works as expected. Link-local gateways wouldn't confuse it, that's expected with IPv6 in many cases.

Derelict · Apr 11, 2019, 7:25 PM

login-to-view

Works fine. This is my lab WAN that uses part of the HE.NET /48

local_nets_v6 includes the /56 PD from Cox, the /48 from HE.NET, and fc00::/7

4920441 0 · Apr 14, 2019, 7:19 AM

@Derelict

Hi in which context did you put these rules? Floating?

Thanks a lot!

Derelict · Apr 14, 2019, 5:10 PM

No. That is on the interface I want to go out the HENET tunnel for everything but local IPv6 traffic.

Most interfaces need nothing and use the default gateway (/56 PD from the ISP)

4920441 0 · Apr 21, 2019, 1:10 PM

@Derelict

Hi,

for me it is not working, despite everything should be fine according to the webinterface...
Since my pfsense router config is grown since 2011, its maybe possible that the config shown in the webinterface is not the actual running config any more...

How can I doublecheck which rules are applied on the console? something like iptables-save or iptables -lav

The funny thing is, despite the fact that there is only one single rule for ipv6 icmp packets, it does not match if I send a ipv6 icmp packet from external (counter in webinterface does not get higher).... So maybe my config is screwed over the last 8 years...

Cheers

4920441

johnpoz · Apr 21, 2019, 1:47 PM

@4920441-0 said in Source based Routing with pfSense:

despite everything should be fine according to the webinterface...

What I have found over the years as users quite often say its configured like X, but in reality when they post up their rules/config pages its really like Y..

If you want to view the full rules
https://docs.netgate.com/pfsense/en/latest/firewall/viewing-the-full-pf-ruleset.html

4920441 0 · Apr 21, 2019, 4:18 PM

@johnpoz

I know, I am confronted with that issue myself every day...
But making lots of screenshot won't help, this config has a dozen vlans multiple ipsec and openvpn connections and posting all firewall rules with pngs would be more confusing than being helpful, I think.

I grepped (and anonymized) all pppoe related rules with icmp context (okay, there might be son "any proto" rules, but there are not, I checked it, there is not one single "proto any" rule

The first one I cannot find in the webinterface labled "NEGATE_ROUTE:..." is this some kind of default route? Since it is for ipv4 and ipv6 valid and icmp, it could be my problem here...:

grep  pppoe  rules-all.txt  | grep -i icmp

pass in quick on pppoe inet proto icmp from any to <negate_networks> keep state label "NEGATE_ROUTE: Negate policy routing for destination"
pass in quick on pppoe route-to (pppoe0 62.123.34.56) inet proto icmp all keep state label "USER_RULE"
pass in quick on pppoe0 reply-to (pppoe0 62.123.34.56) inet proto icmp from any to (self) keep state label "USER_RULE"
pass in log quick on pppoe0 route-to (pppoe0 fe80::123:ab1:33ab:accc) inet6 proto ipv6-icmp from any to (self) keep state label "USER_RULE"
pass in quick on pppoe0 reply-to (pppoe0 62.123.34.56) inet proto icmp from 66.1.2.3 to any keep state label "USER_RULE: HurricaneElectics"
pass in quick on pppoe0 reply-to (pppoe0 62.123.34.56) inet proto icmp from 216.66.80.30 to any keep state label "USER_RULE: HurricaneElectics"
pppoe0 icmp 217.12.34.56:65361 -> 217.76.54.32:65361       0:0
pppoe0 ipv6-icmp 2003:aa:bbbb:2222:3333:444f:555f:666f[182] -> 2620:fe::9[182]       NO_TRAFFIC:NO_TRAFFIC
pppoe0 icmp 217.12.34.56:36974 (192.168.64.61:3691) -> 8.8.8.8:36974       0:0

Thanks a lot for your patience..

Cheers

4920441

Derelict · Apr 21, 2019, 5:16 PM

@4920441-0 said in Source based Routing with pfSense:

But making lots of screenshot won't help, this config has a dozen vlans multiple ipsec and openvpn connections and posting all firewall rules with pngs would be more confusing than being helpful

Then post what is necessary to display a single example of a single connection that you think should be behaving one way but is behaving a different way.

Negate routes are an attempt to automatically bypass policy routing for VPN routes. If you look in /tmp/rules.debug you can see what is listed in that table. If you know it is causing problems you can disable those in System > Advanced, Firewall & NAT, Disable Negate Rules.

jimp · Apr 21, 2019, 5:14 PM

@4920441-0 said in Source based Routing with pfSense:

pass in log quick on pppoe0 route-to (pppoe0 fe80::123:ab1:33ab:accc) inet6 proto ipv6-icmp from any to (self) keep state label "USER_RULE"

You manually configured a gateway on a WAN interface rule. Don't do that.

4920441 0 · Apr 21, 2019, 6:59 PM

@jimp

despite that "USER_RULE" tag I remember the gateway was automatically added when IPv6 was configured for the WAN interface.

But the WAN Interface had had several states of evolution .... first it was only 6to4, after myprovider switched to dual stack it came via DHCP over pppoe - maybe in the transition period something went wrong.

Should I delete the IPv6 Gateway? For testing I disabled it right now, but without it could not work either.

Cheers,

4920441 0 · Apr 21, 2019, 7:03 PM

@Derelict

...as said, I cannot see any rule which is responsible for that behaviour... neither an obvious routing misconfiguration....:
Is there a way to check/'dump' with pfctl what my tcpdump does on layer3, so I can see which rule is involved?

[2.4.4-RELEASE][root@router]/root: tcpdump -nnfi pppoe0 icmp6 and host 2601:183:0:3131:11d2:2128:af93:c6c9
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on pppoe0, link-type NULL (BSD loopback), capture size 262144 bytes
10:12:38.779234 IP6 2601:183:0:3131:11d2:2128:af93:c6c9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo request, seq 17, length 64
10:12:39.803005 IP6 2601:183:0:3131:11d2:2128:af93:c6c9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo request, seq 18, length 64
10:12:40.823277 IP6 2601:183:0:3131:11d2:2128:af93:c6c9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo request, seq 19, length 64
10:12:41.847030 IP6 2601:183:0:3131:11d2:2128:af93:c6c9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo request, seq 20, length 64
10:12:42.871295 IP6 2601:183:0:3131:11d2:2128:af93:c6c9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo request, seq 21, length 64
^C
5 packets captured
656 packets received by filter
0 packets dropped by kernel

If I Initiate a (outgoing) ping on the pppoe Ipv6 Interface, everything is working fine:

[2.4.4-RELEASE][root@rotorouter]/root: tcpdump -nnfi pppoe0 icmp6 and host 2620:fe::9
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on pppoe0, link-type NULL (BSD loopback), capture size 262144 bytes
10:16:24.688594 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2620:fe::9: ICMP6, echo request, seq 6558, length 8
10:16:24.703858 IP6 2620:fe::9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo reply, seq 6558, length 8
10:16:25.195955 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2620:fe::9: ICMP6, echo request, seq 6559, length 8
10:16:25.210875 IP6 2620:fe::9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo reply, seq 6559, length 8
10:16:25.699812 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2620:fe::9: ICMP6, echo request, seq 6560, length 8
10:16:25.714615 IP6 2620:fe::9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo reply, seq 6560, length 8
10:16:26.210344 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2620:fe::9: ICMP6, echo request, seq 6561, length 8
10:16:26.225103 IP6 2620:fe::9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo reply, seq 6561, length 8
10:16:26.734244 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2620:fe::9: ICMP6, echo request, seq 6562, length 8
10:16:26.749107 IP6 2620:fe::9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo reply, seq 6562, length 8
^C
10 packets captured
331 packets received by filter
0 packets dropped by kernel

Outgoing pings via the WAN Interfae work fine and are answered on the same Network Interface.

But Incoming Pings are reply'ed on the wrong interface, the leave the firewall on the hurricane electrics tunnel, not on the interface they are received in the first place:

[2.4.4-RELEASE][root@router]/root: tcpdump -nnfi gif0 icmp6 and host 2601:183:0:3131:11d2:2128:af93:c6c9
tcpdump: WARNING: foreign (-f) flag used but: gif0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on gif0, link-type NULL (BSD loopback), capture size 262144 bytes
10:18:31.035483 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2601:183:0:3131:11d2:2128:af93:c6c9: ICMP6, echo reply, seq 361, length 64
10:18:32.055503 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2601:183:0:3131:11d2:2128:af93:c6c9: ICMP6, echo reply, seq 362, length 64
10:18:33.079271 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2601:183:0:3131:11d2:2128:af93:c6c9: ICMP6, echo reply, seq 363, length 64
10:18:34.103536 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2601:183:0:3131:11d2:2128:af93:c6c9: ICMP6, echo reply, seq 364, length 64
10:18:35.127287 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2601:183:0:3131:11d2:2128:af93:c6c9: ICMP6, echo reply, seq 365, length 64
10:18:36.155275 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2601:183:0:3131:11d2:2128:af93:c6c9: ICMP6, echo reply, seq 366, length 64
^C
6 packets captured
144 packets received by filter
0 packets dropped by kernel

Derelict · Apr 21, 2019, 7:58 PM

@4920441-0 Look at the rule set. You should not be explicitly setting a gateway on any WAN rules. Again, post your rules.

jimp · Apr 22, 2019, 12:37 PM

@4920441-0 said in Source based Routing with pfSense:

@jimp
despite that "USER_RULE" tag I remember the gateway was automatically added when IPv6 was configured for the WAN interface.

User rules do not work that way. You had to have manually added that rule and manually picked a gateway on that rule. A gateway on the interface does not automatically get set on a rule, ever.

4920441 0 · Apr 22, 2019, 1:27 PM

Hi @Derelict

since you kindly asked for all my rules, here they are. I hope I did not miss anything which should be anonymized....

Please take a look adnd tell me what I overlook.... .

Thanks alot for your help, really appreciated.

Since it didn't make sense to put all rules directly in this post, I attached a txt file.

allrules.txt

Cheers

4920441