To clarify what does work:

What works is that from either site, a client device with an IP of 10.40.x.x is able to traverse the tailscale tunel to the other site by using the 10.65.x.x addresses. However, no device in this 10.40.x.x subnet can get to a 10.40.x.x IP at the other end.

I realise that I am NATing the outbound connections rather than directly routing them due to the limitations of pfsense, so I am thinking I need to translate the 10.40.x.x subnet on the way out of the site, but nothing I have tried seems to work so far.

@sgw said in splitting a subnet, moving from LAN to WAN:

When I try to ping an IP that has not been used before and in the LAN-part (10.96.25.128/25) from "upstream"/"outside", it gets logged on the pfsense as blocked by the firewall (which is OK in terms of fw-rules).

So the pfsense seems to take over traffic pointed to these IPs.

Could I modify NAT-rules maybe? Is that related to Outbound NAT? Could I somehow exclude

It seems that the lower IP range ist routed to pfSense WAN address. If so, you can do nothing. You cannot use the same IP outside pfSense, because pfSense was not able to route the traffic to the VM, since the subnet is defined on the LAN.

You only option would be to use an IP of the upper /25 on the VM and forward it on pfSense. But note that doing this also requires an outbound NAT rule (masquerading) for the forwarded traffic.

@pwood999 Indeed.

I also run dual fiber links (on different physical fibers, with different building entry points)
for resilience, not speed.

Plain failover is enough for me too.

But issues are issues and should nevertheless be fixed.

@E-I
I'll second that.
I get the same issue.

@cust
You don't need to wait anything.
You can apply this patch via system patches package, just use commit id 62b1bc8b4b2606d3b20a48a853ef373ff1d71e26

Resolved by implementing BGP peering over IPSEC.

Thanks again for the video. It solved my problem.

If anyone bumps into this thread in the future, the static route showed in a screenshot above here was correct, however here's what I did wrong:

On site2 I had set "IPv4 Upstream gateway" in the interface config to the gateway on site1. This makes pfsense NAT the traffic instead of routing it. Here's a timestamped link to the video where this is explained.

I really wish I could do the same here. I get some false positive failovers because the single IP monitor becomes offline, but the gateway is working fine...

Gateway 1 monitor 8.8.8.8
Gateway 1 monitor 1.1.1.1

Gateway 2 monitor 8.8.4.4
Gateway 2 monitor 1.0.0.1

Gateway 3

Then prioritize them and route following this order.

OK, I think I just figured it out!! I didn't have the subnet enabled. Not sure how it happened, but it now seems to be working. I can open the printer's web page!!

Now I can revisit things.

Do you see errors on the parent interface?

You can try the dtrace commands shown in this thread and see if you're hitting some error other than 55 (no buffers).

@4o4rh i am struggling to get policy based routing working on truenas scale.
either i get the situation where non-media vlans can access the jellyfin server on the media vlan (but then the truenas/smb vlan is not accessible, or the truenas/smb vlan is accessible by not the jellyfin vlan (other than from a device on the vlan).

It both cases the non-accessible vlan is appearing on the wrong pfsense interface.

so, it seems truenas is returning via the default route rather than the desired return route