Source based Routing with pfSense

4920441 0 · Apr 21, 2019, 6:59 PM

despite that "USER_RULE" tag I remember the gateway was automatically added when IPv6 was configured for the WAN interface.

But the WAN Interface had had several states of evolution .... first it was only 6to4, after myprovider switched to dual stack it came via DHCP over pppoe - maybe in the transition period something went wrong.

Should I delete the IPv6 Gateway? For testing I disabled it right now, but without it could not work either.

Cheers,

4920441 0 · Apr 21, 2019, 7:03 PM

@Derelict

...as said, I cannot see any rule which is responsible for that behaviour... neither an obvious routing misconfiguration....:
Is there a way to check/'dump' with pfctl what my tcpdump does on layer3, so I can see which rule is involved?

[2.4.4-RELEASE][root@router]/root: tcpdump -nnfi pppoe0 icmp6 and host 2601:183:0:3131:11d2:2128:af93:c6c9
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on pppoe0, link-type NULL (BSD loopback), capture size 262144 bytes
10:12:38.779234 IP6 2601:183:0:3131:11d2:2128:af93:c6c9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo request, seq 17, length 64
10:12:39.803005 IP6 2601:183:0:3131:11d2:2128:af93:c6c9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo request, seq 18, length 64
10:12:40.823277 IP6 2601:183:0:3131:11d2:2128:af93:c6c9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo request, seq 19, length 64
10:12:41.847030 IP6 2601:183:0:3131:11d2:2128:af93:c6c9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo request, seq 20, length 64
10:12:42.871295 IP6 2601:183:0:3131:11d2:2128:af93:c6c9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo request, seq 21, length 64
^C
5 packets captured
656 packets received by filter
0 packets dropped by kernel

If I Initiate a (outgoing) ping on the pppoe Ipv6 Interface, everything is working fine:

[2.4.4-RELEASE][root@rotorouter]/root: tcpdump -nnfi pppoe0 icmp6 and host 2620:fe::9
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on pppoe0, link-type NULL (BSD loopback), capture size 262144 bytes
10:16:24.688594 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2620:fe::9: ICMP6, echo request, seq 6558, length 8
10:16:24.703858 IP6 2620:fe::9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo reply, seq 6558, length 8
10:16:25.195955 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2620:fe::9: ICMP6, echo request, seq 6559, length 8
10:16:25.210875 IP6 2620:fe::9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo reply, seq 6559, length 8
10:16:25.699812 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2620:fe::9: ICMP6, echo request, seq 6560, length 8
10:16:25.714615 IP6 2620:fe::9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo reply, seq 6560, length 8
10:16:26.210344 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2620:fe::9: ICMP6, echo request, seq 6561, length 8
10:16:26.225103 IP6 2620:fe::9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo reply, seq 6561, length 8
10:16:26.734244 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2620:fe::9: ICMP6, echo request, seq 6562, length 8
10:16:26.749107 IP6 2620:fe::9 > 2003:aaf:d33f:4344:333c:23ff:3221:23f8: ICMP6, echo reply, seq 6562, length 8
^C
10 packets captured
331 packets received by filter
0 packets dropped by kernel

Outgoing pings via the WAN Interfae work fine and are answered on the same Network Interface.

But Incoming Pings are reply'ed on the wrong interface, the leave the firewall on the hurricane electrics tunnel, not on the interface they are received in the first place:

[2.4.4-RELEASE][root@router]/root: tcpdump -nnfi gif0 icmp6 and host 2601:183:0:3131:11d2:2128:af93:c6c9
tcpdump: WARNING: foreign (-f) flag used but: gif0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on gif0, link-type NULL (BSD loopback), capture size 262144 bytes
10:18:31.035483 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2601:183:0:3131:11d2:2128:af93:c6c9: ICMP6, echo reply, seq 361, length 64
10:18:32.055503 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2601:183:0:3131:11d2:2128:af93:c6c9: ICMP6, echo reply, seq 362, length 64
10:18:33.079271 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2601:183:0:3131:11d2:2128:af93:c6c9: ICMP6, echo reply, seq 363, length 64
10:18:34.103536 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2601:183:0:3131:11d2:2128:af93:c6c9: ICMP6, echo reply, seq 364, length 64
10:18:35.127287 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2601:183:0:3131:11d2:2128:af93:c6c9: ICMP6, echo reply, seq 365, length 64
10:18:36.155275 IP6 2003:aaf:d33f:4344:333c:23ff:3221:23f8 > 2601:183:0:3131:11d2:2128:af93:c6c9: ICMP6, echo reply, seq 366, length 64
^C
6 packets captured
144 packets received by filter
0 packets dropped by kernel

Derelict · Apr 21, 2019, 7:58 PM

@4920441-0 Look at the rule set. You should not be explicitly setting a gateway on any WAN rules. Again, post your rules.

jimp · Apr 22, 2019, 12:37 PM

@4920441-0 said in Source based Routing with pfSense:

@jimp
despite that "USER_RULE" tag I remember the gateway was automatically added when IPv6 was configured for the WAN interface.

User rules do not work that way. You had to have manually added that rule and manually picked a gateway on that rule. A gateway on the interface does not automatically get set on a rule, ever.

4920441 0 · Apr 22, 2019, 1:27 PM

Hi @Derelict

since you kindly asked for all my rules, here they are. I hope I did not miss anything which should be anonymized....

Please take a look adnd tell me what I overlook.... .

Thanks alot for your help, really appreciated.

Since it didn't make sense to put all rules directly in this post, I attached a txt file.

allrules.txt

Cheers

4920441

birtalevente · Jan 13, 2021, 7:47 PM

Hi,

I have 3 WAN connections, configured the policy routing and working as expected, but incoming connection on WAN1 or WAN2 from IP: a.b.c.192 (when WAN3 ip is a.b.c.62, GW: a.b.c.1 ) responded on WAN3. Can I somehow resolve this? I expect the response to go out through incoming interface.

Thanks
Levi

viragomann · Jan 13, 2021, 8:53 PM

@birtalevente
Dude, this thread is almost two years old, so let him rest in peace!

@birtalevente said in Source based Routing with pfSense:

but incoming connection on WAN1 or WAN2 from IP: a.b.c.192 (when WAN3 ip is a.b.c.62, GW: a.b.c.1 ) responded on WAN3.

Can you describe your WAN interface configuration a little more detailed?
It's not really clear when you use alphabetic characters and not mention any network mask. Are these connected to different internet providers, are they in different subnets with a gateway on each?

birtalevente · Jan 14, 2021, 10:32 AM

@viragomann
I tried to explain more detailed, but always got the spam message

WAN1 and WAN3 are same ISP, but different media (fiber and radio) and network, both netmask is /24.
WAN2 is other ISP, obviously different network than others, netmask is /25.

So, the WAN3 network let's say is a.b.c.0/24, WAN3 IP is a.b.c.62

There is another location, other company where the ISP assigned the a.b.c.192 IP address. From this another location, other company they have to connect to some services at mine location, where the pfsense router is installed with WAN1,2,3.
From this another location, other company the connection is initiated to the WAN1 and WAN2 IPs, but the responses are routed out through the WAN3 ... which is somehow logic because I have in the routing table a.b.c.0/24 on WAN3.
As I said before, I'd like the whole traffic to going on incoming interface
Thanks!
Levi

viragomann · Jan 14, 2021, 11:13 AM

@birtalevente said in Source based Routing with pfSense:

From this another location, other company the connection is initiated to the WAN1 and WAN2 IPs, but the responses are routed out through the WAN3 ... which is somehow logic because I have in the routing table a.b.c.0/24 on WAN3

No, this is logic, because the destination IP lies within the subnet of WAN3 if I did undersand right your alphabetic variables:

@birtalevente said in Source based Routing with pfSense:

So, the WAN3 network let's say is a.b.c.0/24, WAN3 IP is a.b.c.62
There is another location, other company where the ISP assigned the a.b.c.192 IP address

So if here a.b.c are the same in both variables, your WAN3 IP and that one of the other company are in the same subnet.
If so, the other company should access your router at WAN3 and nothing other.
If they come in on an other WAN, they may have set a wrong mask in the WAN configuration (not /24).

Your router cannot response to an address on another interface if the destionation is in the subnet of WAN3 in the end.

birtalevente · Jan 14, 2021, 12:01 PM

@viragomann said in Source based Routing with pfSense:

@birtalevente said in Source based Routing with pfSense:

From this another location, other company the connection is initiated to the WAN1 and WAN2 IPs, but the responses are routed out through the WAN3 ... which is somehow logic because I have in the routing table a.b.c.0/24 on WAN3

No, this is logic, because the destination IP lies within the subnet of WAN3 if I did undersand right your alphabetic variables:

@birtalevente said in Source based Routing with pfSense:

So, the WAN3 network let's say is a.b.c.0/24, WAN3 IP is a.b.c.62
There is another location, other company where the ISP assigned the a.b.c.192 IP address

So if here a.b.c are the same in both variables, your WAN3 IP and that one of the other company are in the same subnet.
If so, the other company should access your router at WAN3 and nothing other.

This is not possible...WAN3 is low speed and dedicatet to other services.

If they come in on an other WAN, they may have set a wrong mask in the WAN configuration (not /24).

They come in on the right WAN because thats how is set up on they side!

Your router cannot response to an address on another interface if the destionation is in the subnet of WAN3 in the end.

That sucks ...

So I need to reconfigure a little bit

Thanks anyway !

Levi