Good Evening, I figured out the transmission issue. It had to do with the negotiation between the MetroNode and the Chelsio NIC. I contacted my ISP and they turned off auto negotiation on the MetroNode and it started transmitting. It seems to be something in the driver for the T540-CR that I am using inside of ESXI. Therefore, everything seems to be working now. Thanks for the replies!

@alex-atkin-uk I have tried routing data from the interface's page but also not working specially for TCP connections. also tried the manual fix by adding rules for TCP connections on the interface's page with any flags and sloppy state but not even routing to the determined gateway. but if i made servers of Facebook or twitch static routes on specific gateway and bypassing firewall rules for it's traffic, it works just fine.

@ddbnj said in 21.02p1 policy based routing not passing replies on ipsec-VTI: @jimp Does wireguard also need static routes to exist before creating policy based routing rules? If you place the rules to pass traffic on the assigned wireguard interface rule tabs, then it will work properly in both directions, just like it would work an appropriate OpenVPN setup.

@maddy_in65 You do not mention if you have set that load balance group as your default gateway or defined some firewall rules to use it. Without either of those, it will default to the first WAN.

@larsn-0 Check System, Advanced, Miscellaneous. This should only be happening if "Skip rules when gateway is down" is ticked, maybe that somehow got enabled on upgrade? If not then it looks like a bug. If so then a short term fix would be to add a block rule underneath that rule to deny connectivity for that IP address, it will only ever be used if the rule above it does not match/is missing.

@monaco not sure. From searching google, it seems to be a problem that has existed for a while

@bingo600 I used to work for a telecom that provided Telenet service in Canada. We had PADs that converted plain ASCII via dial up modem to X.25, which then connected to Pr1me computers, which the Telenet network ran on. I relly liked IBM's OS/2 Communications Manager , used it alot back then I also used to work at IBM Canada, providing 3rd level OS/2 support. I never worked with CM, but I did support Personal Communications, which provided 3270, 5250 and telnet terminal emulation over IP & SNA. Back in those days I actually memorized my 5 SNA addresses, 1 for my own computer and 4 for testing in my work. I also had 5 IP addresses. The one for my computer 9.29.146.147.

@snewby said in Issue Routing Between Subnets With Multi-WAN: f you select a gateway group that using a different routing table than when you use the default gateway. Pretty much - when you set a gateway like that, you take the normal routing table that pfsense would use to know where to send traffic X, and just shoves it down that gateway.. Be it can get to where it wants to go or not. Rules are evaluated top down, first rule to trigger wins, no other rules are allowed. If you have a rule before you shove it out a gateway that allows said traffic - pfsense will then route that traffic per its normal routing table. if the traffic is also attached, it knows exactly where to send it. Or if say you had another gateway for a downstream router via typical routing then it would know to send it to that gateway. Glad you got it sorted - here to help, even if just a general sort of networking question. Happy to help when I can, even if not some specific to pfsense.. Just ask it in the off topic section if has zero to do with pfsense.

@sokolum said in many Interfaces assinged to an FIB: I have found a post about how to assign a interface to a FIB, in my that would be a VLAN interface on PFsense. The example is using net.conf, what is not used on the PFsense, what is the proper way to assign interface to a FIB on PFsense? https://forums.freebsd.org/threads/using-same-ip-address-on-different-fibs.52565/ Use case: I want (need) to create 5 VLAN interfaces, al has the same /24 subnet configured (mandatory) and every interface has configured the same IP address on their interface. I believe this is possible with FIB. Example: vmx1 - vlan 10 : fib 1 - all traffic is handled on fib 1 - VM 10.0.0.1/24 vmx1 - vlan 20 : fib 1 - connected to LAN vmx1 - vlan 30 : fib 2 - all traffic is handled on fib 2 - VM 10.0.0.1/24 vmx1 - vlan 40 : fib 2 - connected to LAN vmx1 - vlan 50 : fib 3 - all traffic is handled on fib 3 - VM 10.0.0.1/24 vmx1 - vlan 60 : fib 3 - connected to LAN etc, etc NOTE: on Cisco I would create a new VRF and associate that interface to an VRF. Want to achieve similar on PFsense. Any help very much appreciated! Wasn't able to edit my post. What I actually try to achieve> I have 3 networks, all same LAN Subnets and each uses for NAT a different WAN address: Example for what i want to build: LAN-1: vmx1 - vlan 10 : fib 1 - all traffic is handled on fib 1 - VM 10.0.0.1/24 WAN-1: vmx1 - vlan 20 : fib 1 - WAN: 192.168.0**.11** LAN-2:vmx1 - vlan 30 : fib 2 - all traffic is handled on fib 2 - VM 10.0.0.1/24 WAN-1vmx1 - vlan 40 : fib 2 - WAN: 192.168.0**.12** LAN-3:vmx1 - vlan 50 : fib 3 - all traffic is handled on fib 3 - VM 10.0.0.1/24 WAN-1vmx1 - vlan 60 : fib 3 - WAN: 192.168.0**.13**

@shpokas I found my old post and the fix was to find and remove "ghost" DNS servers in configuration export then reimport exported configuration. https://redmine.pfsense.org/issues/8390

[image: 1615048760807-fd0d13e9-1456-4a43-9261-b77d3a0c1cce-afbeelding.png] uncheck this apparently I had this turned on.

@ttblum see https://forum.netgate.com/topic/160694/frr-7-3-7-5-bgp-not-announcing-routes

@genuine said in "sendto error: 65" after 2.5.0: Try to change the gateway monitoring ip to a dns of your isp see if thats better or take a public one example 4.2.2.1 I added my gw monitoring ip to Rejected Leases => [image: 1614903973338-9efd731f-0e6c-48e3-b46e-95efd937fb48-resized.jpeg] And that error was gone. Trying to understand which is a better solution? Thx for the reply

@alefe thank you for your offer, but I don't want to waste to much of your time trying to schedule a remote session. Let me try explain what is the problem on home lab example: We have following gateways config with default gateway set to failover group preferring GW1 [image: 1614895364891-be01e3f0-9d6c-49a0-ad07-52bd239ca1f6-image.png] [image: 1614895400296-d0ab0bb3-ffef-42af-bbd7-678094b0e21b-image.png] And LAN rules are set to use only GW1 172.16.0.1/24 only, do not use failover. [image: 1614895990189-1d84f43e-ca38-4e1b-bc89-272b36ec45dd-image.png] and when you have GW1 down [image: 1614896373622-40b81554-6f93-42b5-936b-a27aa3a2be3b-image.png] FW makes a failover to WAN2 regardless of the rules setting to use only GW1 [image: 1614896513050-7e980426-2fb3-4609-85a4-c77e96dd657c-image.png] Only if I set default GW to something different than GW group like automatic or ether GW [image: 1614896653818-10ea2306-6833-429b-b52e-65a91ea0a868-image.png] Then the GW settings on FW rules are followed/respected: [image: 1614896992960-dd811aca-a9ee-412d-8d42-a70493c06ffe-image.png] Hope I explained my query clearer now. And my question is: Is this is expected behaviour? Best regards, Piotr Marchewka

@griffo said in Using same gateway monitor IP not allowed: But I don't want to pick some random service provider gateway IP OK.. I was thinking of your own provider (ISP), it's not random... DNS servers are not designed and used to send ICMP responses depending on their workload, the responses received also differ, so they do not provide relevant information so let’s stick with this first ISP GW as a good solution BTW: the forum is full of discussions on this theme the end is always that the DNS server(s) is not a monitor IP alternative

@gwaitsi after further reading, it seems the pfsense device would gain by having all nics with either network and using the isp provided fritz box as a switch in between. They have configured one of the switch ports as a pppoe wan connection. and the other 3 ports to the lan side. So I am also left with a bypass option. I have more of a performance drop from the J1900 than from the fritzbox which in any case.