Source based Routing with pfSense

4920441 0 · Apr 21, 2019, 4:18 PM

I know, I am confronted with that issue myself every day...
But making lots of screenshot won't help, this config has a dozen vlans multiple ipsec and openvpn connections and posting all firewall rules with pngs would be more confusing than being helpful, I think.

I grepped (and anonymized) all pppoe related rules with icmp context (okay, there might be son "any proto" rules, but there are not, I checked it, there is not one single "proto any" rule

The first one I cannot find in the webinterface labled "NEGATE_ROUTE:..." is this some kind of default route? Since it is for ipv4 and ipv6 valid and icmp, it could be my problem here...:

grep  pppoe  rules-all.txt  | grep -i icmp

pass in quick on pppoe inet proto icmp from any to <negate_networks> keep state label "NEGATE_ROUTE: Negate policy routing for destination"
pass in quick on pppoe route-to (pppoe0 62.123.34.56) inet proto icmp all keep state label "USER_RULE"
pass in quick on pppoe0 reply-to (pppoe0 62.123.34.56) inet proto icmp from any to (self) keep state label "USER_RULE"
pass in log quick on pppoe0 route-to (pppoe0 fe80::123:ab1:33ab:accc) inet6 proto ipv6-icmp from any to (self) keep state label "USER_RULE"
pass in quick on pppoe0 reply-to (pppoe0 62.123.34.56) inet proto icmp from 66.1.2.3 to any keep state label "USER_RULE: HurricaneElectics"
pass in quick on pppoe0 reply-to (pppoe0 62.123.34.56) inet proto icmp from 216.66.80.30 to any keep state label "USER_RULE: HurricaneElectics"
pppoe0 icmp 217.12.34.56:65361 -> 217.76.54.32:65361       0:0
pppoe0 ipv6-icmp 2003:aa:bbbb:2222:3333:444f:555f:666f[182] -> 2620:fe::9[182]       NO_TRAFFIC:NO_TRAFFIC
pppoe0 icmp 217.12.34.56:36974 (192.168.64.61:3691) -> 8.8.8.8:36974       0:0

Thanks a lot for your patience..

Cheers

4920441

Derelict · Apr 21, 2019, 5:16 PM

@4920441-0 said in Source based Routing with pfSense:

But making lots of screenshot won't help, this config has a dozen vlans multiple ipsec and openvpn connections and posting all firewall rules with pngs would be more confusing than being helpful

Then post what is necessary to display a single example of a single connection that you think should be behaving one way but is behaving a different way.

Negate routes are an attempt to automatically bypass policy routing for VPN routes. If you look in /tmp/rules.debug you can see what is listed in that table. If you know it is causing problems you can disable those in System > Advanced, Firewall & NAT, Disable Negate Rules.

jimp · Apr 21, 2019, 5:14 PM

@4920441-0 said in Source based Routing with pfSense:

pass in log quick on pppoe0 route-to (pppoe0 fe80::123:ab1:33ab:accc) inet6 proto ipv6-icmp from any to (self) keep state label "USER_RULE"

You manually configured a gateway on a WAN interface rule. Don't do that.

4920441 0 · Apr 21, 2019, 6:59 PM

@jimp

despite that "USER_RULE" tag I remember the gateway was automatically added when IPv6 was configured for the WAN interface.

But the WAN Interface had had several states of evolution .... first it was only 6to4, after myprovider switched to dual stack it came via DHCP over pppoe - maybe in the transition period something went wrong.

Should I delete the IPv6 Gateway? For testing I disabled it right now, but without it could not work either.

Cheers,

4920441 0 · Apr 21, 2019, 7:03 PM

@Derelict

...as said, I cannot see any rule which is responsible for that behaviour... neither an obvious routing misconfiguration....:
Is there a way to check/'dump' with pfctl what my tcpdump does on layer3, so I can see which rule is involved?

[2.4.4-RELEASE][root@router]/root: tcpdump -nnfi pppoe0 icmp6 and host 2601:183:0:3131:11d2:2128:af93:c6c9
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on pppoe0, link-type NULL (BSD loopback), capture size 262144 bytes
10:12:38.779234 IP6 2601:183:0:3131:11d2:2128:af93:c6c9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo request, seq 17, length 64
10:12:39.803005 IP6 2601:183:0:3131:11d2:2128:af93:c6c9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo request, seq 18, length 64
10:12:40.823277 IP6 2601:183:0:3131:11d2:2128:af93:c6c9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo request, seq 19, length 64
10:12:41.847030 IP6 2601:183:0:3131:11d2:2128:af93:c6c9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo request, seq 20, length 64
10:12:42.871295 IP6 2601:183:0:3131:11d2:2128:af93:c6c9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo request, seq 21, length 64
^C
5 packets captured
656 packets received by filter
0 packets dropped by kernel

If I Initiate a (outgoing) ping on the pppoe Ipv6 Interface, everything is working fine:

[2.4.4-RELEASE][root@rotorouter]/root: tcpdump -nnfi pppoe0 icmp6 and host 2620:fe::9
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on pppoe0, link-type NULL (BSD loopback), capture size 262144 bytes
10:16:24.688594 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2620:fe::9: ICMP6, echo request, seq 6558, length 8
10:16:24.703858 IP6 2620:fe::9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo reply, seq 6558, length 8
10:16:25.195955 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2620:fe::9: ICMP6, echo request, seq 6559, length 8
10:16:25.210875 IP6 2620:fe::9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo reply, seq 6559, length 8
10:16:25.699812 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2620:fe::9: ICMP6, echo request, seq 6560, length 8
10:16:25.714615 IP6 2620:fe::9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo reply, seq 6560, length 8
10:16:26.210344 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2620:fe::9: ICMP6, echo request, seq 6561, length 8
10:16:26.225103 IP6 2620:fe::9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo reply, seq 6561, length 8
10:16:26.734244 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2620:fe::9: ICMP6, echo request, seq 6562, length 8
10:16:26.749107 IP6 2620:fe::9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo reply, seq 6562, length 8
^C
10 packets captured
331 packets received by filter
0 packets dropped by kernel

Outgoing pings via the WAN Interfae work fine and are answered on the same Network Interface.

But Incoming Pings are reply'ed on the wrong interface, the leave the firewall on the hurricane electrics tunnel, not on the interface they are received in the first place:

[2.4.4-RELEASE][root@router]/root: tcpdump -nnfi gif0 icmp6 and host 2601:183:0:3131:11d2:2128:af93:c6c9
tcpdump: WARNING: foreign (-f) flag used but: gif0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on gif0, link-type NULL (BSD loopback), capture size 262144 bytes
10:18:31.035483 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2601:183:0:3131:11d2:2128:af93:c6c9: ICMP6, echo reply, seq 361, length 64
10:18:32.055503 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2601:183:0:3131:11d2:2128:af93:c6c9: ICMP6, echo reply, seq 362, length 64
10:18:33.079271 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2601:183:0:3131:11d2:2128:af93:c6c9: ICMP6, echo reply, seq 363, length 64
10:18:34.103536 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2601:183:0:3131:11d2:2128:af93:c6c9: ICMP6, echo reply, seq 364, length 64
10:18:35.127287 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2601:183:0:3131:11d2:2128:af93:c6c9: ICMP6, echo reply, seq 365, length 64
10:18:36.155275 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2601:183:0:3131:11d2:2128:af93:c6c9: ICMP6, echo reply, seq 366, length 64
^C
6 packets captured
144 packets received by filter
0 packets dropped by kernel

Derelict · Apr 21, 2019, 7:58 PM

@4920441-0 Look at the rule set. You should not be explicitly setting a gateway on any WAN rules. Again, post your rules.

jimp · Apr 22, 2019, 12:37 PM

@4920441-0 said in Source based Routing with pfSense:

@jimp
despite that "USER_RULE" tag I remember the gateway was automatically added when IPv6 was configured for the WAN interface.

User rules do not work that way. You had to have manually added that rule and manually picked a gateway on that rule. A gateway on the interface does not automatically get set on a rule, ever.

4920441 0 · Apr 22, 2019, 1:27 PM

Hi @Derelict

since you kindly asked for all my rules, here they are. I hope I did not miss anything which should be anonymized....

Please take a look adnd tell me what I overlook.... .

Thanks alot for your help, really appreciated.

Since it didn't make sense to put all rules directly in this post, I attached a txt file.

allrules.txt

Cheers

4920441

birtalevente · Jan 13, 2021, 7:47 PM

Hi,

I have 3 WAN connections, configured the policy routing and working as expected, but incoming connection on WAN1 or WAN2 from IP: a.b.c.192 (when WAN3 ip is a.b.c.62, GW: a.b.c.1 ) responded on WAN3. Can I somehow resolve this? I expect the response to go out through incoming interface.

Thanks
Levi

viragomann · Jan 13, 2021, 8:53 PM

@birtalevente
Dude, this thread is almost two years old, so let him rest in peace!

@birtalevente said in Source based Routing with pfSense:

but incoming connection on WAN1 or WAN2 from IP: a.b.c.192 (when WAN3 ip is a.b.c.62, GW: a.b.c.1 ) responded on WAN3.

Can you describe your WAN interface configuration a little more detailed?
It's not really clear when you use alphabetic characters and not mention any network mask. Are these connected to different internet providers, are they in different subnets with a gateway on each?

birtalevente · Jan 14, 2021, 10:32 AM

@viragomann
I tried to explain more detailed, but always got the spam message

WAN1 and WAN3 are same ISP, but different media (fiber and radio) and network, both netmask is /24.
WAN2 is other ISP, obviously different network than others, netmask is /25.

So, the WAN3 network let's say is a.b.c.0/24, WAN3 IP is a.b.c.62

There is another location, other company where the ISP assigned the a.b.c.192 IP address. From this another location, other company they have to connect to some services at mine location, where the pfsense router is installed with WAN1,2,3.
From this another location, other company the connection is initiated to the WAN1 and WAN2 IPs, but the responses are routed out through the WAN3 ... which is somehow logic because I have in the routing table a.b.c.0/24 on WAN3.
As I said before, I'd like the whole traffic to going on incoming interface
Thanks!
Levi

viragomann · Jan 14, 2021, 11:13 AM

@birtalevente said in Source based Routing with pfSense:

From this another location, other company the connection is initiated to the WAN1 and WAN2 IPs, but the responses are routed out through the WAN3 ... which is somehow logic because I have in the routing table a.b.c.0/24 on WAN3

No, this is logic, because the destination IP lies within the subnet of WAN3 if I did undersand right your alphabetic variables:

@birtalevente said in Source based Routing with pfSense:

So, the WAN3 network let's say is a.b.c.0/24, WAN3 IP is a.b.c.62
There is another location, other company where the ISP assigned the a.b.c.192 IP address

So if here a.b.c are the same in both variables, your WAN3 IP and that one of the other company are in the same subnet.
If so, the other company should access your router at WAN3 and nothing other.
If they come in on an other WAN, they may have set a wrong mask in the WAN configuration (not /24).

Your router cannot response to an address on another interface if the destionation is in the subnet of WAN3 in the end.

birtalevente · Jan 14, 2021, 12:01 PM

@viragomann said in Source based Routing with pfSense:

@birtalevente said in Source based Routing with pfSense:

From this another location, other company the connection is initiated to the WAN1 and WAN2 IPs, but the responses are routed out through the WAN3 ... which is somehow logic because I have in the routing table a.b.c.0/24 on WAN3

No, this is logic, because the destination IP lies within the subnet of WAN3 if I did undersand right your alphabetic variables:

@birtalevente said in Source based Routing with pfSense:

So, the WAN3 network let's say is a.b.c.0/24, WAN3 IP is a.b.c.62
There is another location, other company where the ISP assigned the a.b.c.192 IP address

So if here a.b.c are the same in both variables, your WAN3 IP and that one of the other company are in the same subnet.
If so, the other company should access your router at WAN3 and nothing other.

This is not possible...WAN3 is low speed and dedicatet to other services.

If they come in on an other WAN, they may have set a wrong mask in the WAN configuration (not /24).

They come in on the right WAN because thats how is set up on they side!

Your router cannot response to an address on another interface if the destionation is in the subnet of WAN3 in the end.

That sucks ...

So I need to reconfigure a little bit

Thanks anyway !

Levi