@jcyr using a /25 vs a /24 is not a "hack" hehehe

But to be honest use of 192.168.1 or even say 192.168.0 can be problematic - these are 2 of the most common networks. So you run into stuff like your seeing with your ATT devices IP address. You can also see issues with remote access via vpn where the remote site is also using the same IP range your using and then have problems accessing stuff via the vpn.

Using the 192.168.1 could also lead to problem if you fire up some new device on your network and it defaults to using say .1 or .254 and that ends up stepping on say pfsense IP..

You were better off just using 192.168.2 network, or if you like the 1 there as the 3rd octet.. Use maybe 172.16.1/24 or 10.1.1/24 or 192.168.10/24

But using a /25 can work.. for your current setup, but could also be problematic because I doubt your ATT device is using a /25 and you could still run into a issue. If it happened to say assign your pfsense wan the same IP in your /25 as your lan side network interface..

Your better off just moving to some network on your pfsense lan that in no way is overlapping with the network your ATT device is using.

@arkoulikosta said in routing to secondary gateway:

@Gblenn said in routing to secondary gateway:

@arkoulikosta What do you mean with "access to" secondary gateway?
Do you simply want to access the upstream router and manage that via it's UI.
Or do you want to route traffic that way?

The first part is as simple as typing in it's IP in your web browser...

If you for example want to have your PC use WAN2 for internet access you can create a simple rule on the LAN interface for that.
Your source will be the IP of the PC and destination is Any. If you expand the Advanced section, there is an item calle Gateway with a dropdown where you will find your WAN2 in the list. That's pretty much it...

Perhaps it can be done in a different way but that's how I did it when playing around and testing this.

i wanted to access the router ui, but could not do so if the wan1 gw was active
and it was indeed as simple as adding a rule with the wan2 gw as default.
thank you!!
Good that it works, but if it is just the UI that you are trying to access, I don't see that you need to create a policy rule for that...

I have a similar setup, where my second WAN (failover) is connected to a 4G router which hands out a 192... IP to pfsense. It looks like this and I access that router UI directly just by typing in the IP on any browser (192.168.3.2 in my case). There is no need to have any policy rule in order to access that subnet.

39863ca2-736f-4ef2-b1dc-2b70b3c68107-image.png

You should be able to reach both upstream routers from your LAN, as long as both are considered UP.

@johnpoz adding a dns redirect as a workaround helps for now. https://docs.netgate.com/pfsense/en/latest/recipes/dns-redirect.html

I just double checked on other pfSense hosts I am managing. On all of them the above dig@ command works without issue. The only real difference is that they are all single WAN.

I'd consider this somewhat solved for now, but I will have to investigate this behavior further, it seems I am missing something more or less obvious.

Anyways thanks for your assistance! ☺

@Sergei_Shablovsky
I totally ditched this solution. Multi-wan load balancing works to poorly on freebsd with and fights you every step of the way.
I moved to a Cisco L3+ switch with enterprise services and gigamon and a specialized load balancing solution. EOL is around the corner in a few years on my cisco switch so I am looking to upgrade now.

@cyberconsultants That too. Though I think at one point in time I had an odd situation where an ISP blocked pings to 1.1.1.1 after a certain amount of time (maybe it was some form of DDoS protection), haven't had it happen since but I know back then I had to disable gateway monitoring action in order to get a gateway to function. It's been a while though so I don't remember all the details.

@mpcjames I'm really outside of my comfort zone here but doesn't those lines indicate some problem with IPv6 gateway?So how have you defined your Default gateway(s) (IPv4 and IPv6) ?

In my setup which has been working flawlessly when testing and in at least one actual failover transition, I have (System / Routing / Gateways) :

IPv4 - WAN_1_FIRST
IPv6 - Automatic

BTW, I have my Trigger Level set to Member Down, but I believe it worked when I tested with it set to Packet Loss as well.

I really dont know what whent on here .... But after rebootinh Both the servers holdiong the VPFSense [ Thisical servers ] ..... Everything is starting to work as expected...... im not going to hold my breath but for now everything functions......

I have ZERO Idea what was going on but the old saying..... Hello I.T Have you tried turning it off and on again......

@afd1219 As I stated LAN is on the switch, WAN is on IX1, most everything else is on IX0.

Screenshot 2024-03-13 131017.png

@dpravd
To ensure that the proper rule is applied, enable logging in the rule and check the filter log after initiating traffic from remote.
Note the logged rule ID, which is passing the traffic and check if it's the rule you added on the VPN interface.

@vsmaldino True. But also you have a gateway set on WAN.
My setup is also different that I don't use WAN for those VPN-Gateways on the same interface and none of those gateways is the default gateway.

Neglected to mention that Failover to Tier 2 Gateway works fine. Just doesn't show the gateway as down.

@Tilburg-013 I guess so as the associated bug https://redmine.pfsense.org/issues/15043 has been closed with 24.03 as the target release. You can also see the release overview here: https://redmine.pfsense.org/projects/pfsense/roadmap#pfsense-plus-24.03

SOLVED! on my test rig I tried a state-killing option that had NOT solved the problem on my live box, but on the test rig it worked. The setting is in System/Routing/Gateways, "State Killing on Gateway Failure". After changing that from the default to "Kill states using this gateway when it is down", subsequent failover events created a few arpresolve errors in the log, but within 1 second they stopped, after an entry in the log showing a state killing action:

/rc.filter_configure_sync: GW States: Killing states for dynamic down gateway: WAN_DHCP, XX.XX.XX.1

After that worked, I had to figure out why this solved the problem with my test rig but not my live box. Eventually I traced it to a setting in System/Advanced/Miscellaneous in the Gateway Monitoring Section, "Skip rules when gateway is down". In my live box, which has some traffic that needs to be routed only through a VPN, I had enabled the setting "Do not create rules when gateway is down" years ago to make sure, if the VPN was down, that pfSense wouldn't route the traffic through the non-VPN WAN. But as soon as I cleared that check box, my failover arpresolve problem went away. So apparently that setting interacts with the failover in a way that prevents the state-killing action from working properly.

Next job is to figure out a different way to kill VPN-bound traffic if the VPN is down... Googling that now.

@authenticx said in Need help with static routing please:

The subnet in DR is the same as production thus I have it isolated in a VLAN.

if both subnet's are the same, then you can not route between them.

*** Just to add i reconfigured the WAN port with the backup ISP. No issues afterwards***

@IT-CPUC

Found a problem with WANs showing a "Pending" status under Gateways when running pfSense version 23.09.1-RELEASE but likely been there for a while.

This appears to be caused by the process dpinger. Tied to using a DHCP from the ISP. Does not happen if ISP is using a static IP for the WAN.

Occurs if the cable (RJ-45) between the modem and router/firewall is unplugged/replugged or if the modem is power reset. The problem happens with a single WAN and is not related to using a failover configuration. Can stay stuck in Pending state and is hard to get out of. Try, disabling the interface, then be sure the modem is fully online and ready to issue the IP by DHCP, and then reactive the interface. I've also tried releasing the IP and reassigning it under the Interfaces page.

If your ISP is Comcast, I also found a bug in their lease time for DHCPs if their default value is changed. The problem I found is when the lease time is changed, the modem will fail at 1/2 the assigned lease time. Logs will show monitoring pings don't respond. Way to fix this was to reissue another DHCP lease time or release the IP and reassign it under the Interfaces page

Another problem I found is modems in bridge mode will randomly disconnect because they stop responding to ARPs. Think the phSense default is 1200. Found it was necessary to reduce under 5 mins. I am using 240 for net.link.ether.inet.max_age found under Advanced/System Tunables. So far, this change appears to help stop the disconnects.