@chris-doldolia yes. Alone my starlink download speed peak was a bit under 300mbps (usually lower, but I'm not motivated enough to run hundreds of runs across a few days and compute real statistics ;>). Download peak was about 150 (again, usually slower). Even the worst case starlink hasn't been observed to be too bad). And mostly our workload is dominated by video streaming (with various package and ISO downloads sometimes providing challenges). If/when I go back to having enough video conferences to justify special rules, I might want to steer those to Comcast (of course, Comcast unreliability is why we got a starlink ;>). Our comcast performance peaks at about 300/300 (paying for business grade service). But typically the Comcast upload is significantly slower ... so sometimes starlink is ahead, but Comcast is usually ahead.

I set the gateway group rule to {packetloss|high latency} which probably favors Comcast (their latency is typically significantly lower than starlink). What those precise values are isn't obvious to me (but no doubt documented somewhere in the FM!)

Rereading your last response, I've never seen the group result being worse than the starlink standalone ... and disabling Comcast and using only the gateway group the results are a wash with connecting directly (except for latency .. the starlink speedtest removes some of my LAN internal hops, or at least that's the most obvious explanation for what I observe).

@Bob-Dig You can set the gateway to any IP you want, but its not a normal, typical setup - and like the warning I showed you - the OS thinks you must of made a typo ;) It might still arp for it since its gateway IP and has to be reachable on the same L2

arp.jpg

But you can not be sure device or OS will do that. Nor can you be sure what your arping for will answer if the IP is not on the gateways network IP range.

@lowbug

Before the rule you can see 'counters'.
Like these :

ec743d53-134a-416d-9504-ad11fa52b0f6-image.png

If it stays at 0/0, the rule wasn't used ... = no matching traffic.

The current routes too
f7e0340e-76a2-4ded-aa95-4eafa084847e-image.png

@Zerejekim said in Routing across PFSense Interfaces or VLANs not working:

Outstanding!!!!!

Can't thank you enough.

Great, I also have a Synology here, it is old, but still works, DS218+ with two Ironwolf 2TB drives in a RAID 1 config.
More than 5 years with this guy and never lost a file, scrub once a month and that is it..

@jagradang
Let us know what you end up with.

What's the work-around that you have with aliases?

I'm currently experimenting trying to see whether I can get the Unbound for pfsense, to send queries down different VPNs - depending on which VLAN sent in the request. ChatGPT says it's possible, but, I'm yet to see it work. My workaround, was spinning up different adguard instances for each VLAN, and they work fine per vlan - per vpn. Prefer to keep things on pfsense if that's even possible. But don't know.

@padrino121
Drop the WAN circuits on a switch will work as far as not having to physically go on site but unfortunately the FRR configuration still requires manual intervention.
8300s is expensive gear. I would follow up with sales and see if they can provide a better solution. The money you paid it’s really unacceptable that there is this level of shortcomings especially with something so basic

This was too disruptive to the client to troubleshoot this further. We sent someone out to do a clean install of 2.7, then load the config back in. No issues since.

@duvel said in Can't ping 8.8.8.8 or google.com:

I would rather select TCP/UDP & ICMP than allow Al

This works just fine, and is the default :

828b4fd3-e69a-4749-9271-5aa975251d92-image.png

because you don't trust the other 252 ?:

@chrisjx Hi,
I also have a location with two ISPs, one is the primary and the second is a Starlink.
So I know how to setup the LAN4 as a OPT and assigned VLAN 40 to it. But how do I make sure the Starlink is on VLAN 40 then?

Did you managed to get this working?

BR
Nick

@manjotsc great job finding the issue, I had this machine that had a line on the monitor once, guys before me replaced the monitor the cable, I got there and took the cpu off it had a bent pin no lies reseated it or got a new one I can’t remember but that fixed the issue, it is amazing I didn’t understand why everything else worked, one pin caused the issue, also over doing heat compound can cause issues when it gets on pins.

@Bob-Dig Because I still saw all the traffic still going out my WAN1 interface and my WAN2 interface is idle.

@0x010C LAN should typically NOT show up as a gateway in that list... You can have a gateway in the LAN segment, like a standalone VPN server or similar. In that case you set up a static route to it though...

Are you saying that C automagically became a default gateway when you created it? Have you tried changing the default, saving and changing back again?
Also, under gateway group you can create like a failover group, using A, B and C, and setting A to Tier 1 and the others at some higher Tier 2 and 3. Then use this group as the default gateway. All normal traffic wil then go through A, unless A is down. All policy routed traffic will go as per the policy... through B or C.

@totalimpact said in Pfsense cannot port forward to Layer 3 switch:

having static routes requires a gateway on the Transit network.

Not on the interface - you create a gateway to the IP on the transit network, but you don't actually put that gateway on the interface of pfsense on the transit.. Or pfsense thinks a wan interface and creates an outbound nat on it.

You create the gateway in the routing gateway section not on the specific interface.

pfsense-layer-3-switch.png

@seanr22a said in IPsec routing problem:

I have three web sites including Nextcloud on the server at siteB and they are behind Cloudflare CDN (free version). I use an Apache reverse proxy at siteA now to get around the port blocking issue (Sending the traffic over the IPsec to the server at siteB). The ping time is around 230ms and I get around 10Mb up and 45Mb down from siteB to siteA. I spend most of my time in Thailand so the speed I get here is most important.

I get the Proxy setup, that's what I use to access my NextCloud server, as well as my Homeassistant and some other stuff. I just happen to use Nginx.

But I'm not sure I understand how Cloudflare CDN fits into this setup that you have?
If you host your server at your home in Thailand, and you access it via Sweden using some DynDNS service to find your Swedish IP, then you go directly via the VPN to site B. Where does Cloudflare come into play?

And I'm curious, which ISP is it, and which ports do they block? And what ports don't they block?

I've seen that many users say nginx is faster and use less resources but in my very small setup I really don't think it matters.

I agree, probably wouldn't make a noticeable difference if you changed. If you are curious however, and use docker, it's actually super simple to set up and has a very intuitive UI...

BUT, what could potentially improve performance quite a bit is if you change VPN to Wireguard. Depends on what HW you run pfsense on of course, but on smaller machines I can see a real difference even at moderate speeds.
I have a site with pfsense running on a tiny PC Engines APU2 and I can saturate the 250 Mbit connection to that site over Wireguard. But on an IPSec connection I can perhaps get 80-90 Mbit when testing with e.g. iperf or openspeedtest.

Looks like this is starting to happen again.

However it is limited, only some traffic is being routed over the backup connections.