10 bucks the real cause is your NIC and not the system load. ;-)
Realtek is just bad and combined with PPPoE it's even more worse.
Are you running the driver shipped with FreeBSD/pfSense? You could try with this one then https://forum.netgate.com/topic/135850/official-realtek-driver-binary-1-95-for-2-4-4-release
Or try with Intel NIC if possible.

-Rico

Well pick and interface on your 3100 and 5100 to use as your transit network.. For your 3100 you will prob have to carve out one of the switch ports to use. Put it on a different vlan... The 5100 has discrete interfaces so a bit easier.

I restored a backup from March which worked. Seems like I configured something wrong and it only showed after the rebooot.

@hopkins said in Routing 169.254 Networks:

shows no blockage for packets in 169.254.0.0/16

To talk to something on another vlan that is using apipa, that device would have to have a gateway.. Or you would have to nat to it, etc.

Ok, I had it set up right then. From some reason, it doesn't come out right of the untangle so that may be my problem. But just wanted to make sure the outbound of the pfsense was setup right.

Thanks

Re-reading all the docs over the weekend and going through forum posts. The following forum posts and doc links explained the intra-VLAN connectivity issue:

Forum post
Firewall Rules and Policy Route Negotiation
By-passing Policy Route Negotiation - Inclues example local segment rule
Mulit-WAN Policy Routing
Another Multi-WAN Policy Routing page

My missing link was my failure to understand that the second you create a gateway group you enter policy routing territory (as soon as you set the gateway on a rule). My simple "allow all" one rule per VLAN then stop working for local segments as of course everything is directed out the gateway. Seems obvious now. So I need to split my rules up as explained by the forum post so that local traffic is passed without a gateway group and the gateway group rule is that last rule in the set.

I am still completely confused about why the firewall itself can't get out externally though. The IPv4 default gateway is the gateway group. This forum post says that is what is needed for the the firewall to maintain external connectivity. It also talks about a setting that flushes state when any gateway goes down. But in my scenario the Tier 1 gateway is down as the firewall comes up so I'm not sure what needs to be reset. I will try this setting anyway and see if anything changes. My understanding is that this setting will cause disconnections for both gateways if either fails but I'll worry about that if the setting fixes the current issue. I found a couple of old posts that talked about the previous feature "Default Gateway Switching" not working with PPPoE gateways. So maybe my Tier 2 gateway isn't taking over because it is a PPPoE gateway.

I will experiment further and update here after. I've posted all links used in case anyone else is struggling.

And the winner was outgoing NAT-rule on the VPN-interface :)

Hello,

thanks for your reply and suggestion. Yes, I supposed to do an experiment with a less important Lan to try if it works; my setup is with a vip (because I have 2 firewalls) and NAT rules on WAN connection.

I supposed that, even without any rule on the opt, I can change the firewall default gateway and for example navigate to internet from a pc inside the network, because on the lan interface I have the default rule that you said and left the default gateway.

Sincerely my worry is for ipsec VPN where the endpoint is with the old ip and I don't know, even if I suppose that it is so, if the traffic is routed correctly.

I'll try.

okay, that is some info i can use for something.

use one IP address as WAN IP address for example 78.X.X.74/29 and GW 78.X.X.73
and assign other IPs as VIP on wan interface

https://docs.netgate.com/pfsense/en/latest/book/firewall/virtual-ip-addresses.html

https://docs.netgate.com/pfsense/en/latest/book/firewall/methods-of-using-additional-public-ip-addresses.html

Rip? Really? You have needs for a dynamic internal routing protocol?

Apart from that, smells like nat configuration and/or routing on the second box.
Post a network diagram with subnet ip's and your routing and nat settings

@Daniel1972 I am trying to implement the same configuration, not because I'm running out of ports but because I'd like to have separate machines handling the WAN and internal LANs as well as having an internal LAN for monitoring. I just set up a(nother) 192.168.x.x/24 LAN between the boxes and used RIP to share the routes. Unfortunately, I'm running into an issue with routing to the Internet:
https://forum.netgate.com/topic/153989/1-pfsense-router-works-2-pfsense-doesn-t

Yeah...I wasn't looking at a router at the time and I hadn't looked at this one in a month. Oops.

Although that did light a bulb for me. Loss Interval says "Time interval in milliseconds before packets are treated as lost. Default is 2000." Do "treated as" packets actually get marked in the percentage lost? With an average of 1300 perhaps a few are taking longer than 2000ms and are considered "lost" although they arrive in, say, 2100ms and thus the 0% loss shown? I think I'll try using 120s for the time interval to see if that "provides smoother results."

Overall the goal was just to not have the connection drop/failover now and again, with 0% loss shown. High latency isn't great but moving the traffic from cable to DSL isn't generally going to improve that if it's due to traffic.

@cobrahead No, you are good to go. Everything else is mainly for carp and high availability setups.

Check out the great hangout done by jimp: https://www.netgate.com/resources/videos/openvpn-as-a-wan-on-pfsense.html
I'm sure this will answer all your questions. :-)

-Rico