@mikey_s 2GB is a lot of dns requests... But sure could see that happening. Lets do some math for curiosity sake. Does your 2GB count both up and down traffic? Lets say a dns query is 500 Bytes total up and down.. Doubled what I saw in simple sniff just for cushion in our math. You would have to [image: 1715511113523-query.jpg] So that is what 4 million queries? My whole network, lots of clients in last 24 hours have done [image: 1715511253574-queries.jpg] Now keep in mind that I change the min ttl to 1 hour, so this will be skewed.. Many ttls these days are short 60 seconds, 5 minutes.. So sure number of queries will be up.. So if you were doing 10x that or 280k queries a day.. Doesn't take long to get to 4million queries.. A 5 minute ttl if something is being asked for all the time would equate to lot of queries, and if something is banging its head looking for something.. Shoot I have had a single alexa do 2Million queries in 24 hours before. Lets not forget the pings for monitor, default is what 2 a second. Small but there will be some data there. Even with zero byte payload. So yeah I would think it quite possible to use up a 2GB of bandwidth without really even moving any traffic at all. I would suggest you do a sniff for say an hour of traffic out your lte interface.. With no clients really even using it.. Then do some math to how long it would take to eat that 2GB up. With such a low amount data limit to work with.. I would prob make that failover a manual process.. And I wouldn't let it do dns queries out it until such time that is your only connection. And I would for sure limit the min ttl to something less than many sites use these days of those insanely low ttls.. And look to see how much data just monitoring is using.. So just adding up the pings, and have payload set to 0... [image: 1715512834601-500m.jpg] 30 seconds is 6KB, so what is that like 17MB a day just in pings, or 30 days like 500MB.. which would be 1/4 of your monthly quota just in monitor if the gateway is up. LTE can make for a great backup, but if you have a low data quota - it would be quite easy to suck that up all with just background noise like dns and monitoring to be honest. Depending on what counts against your quota..

Did you get that for free? Its not quiet, and it sucks juice.. Prob about 75W just idle.. No it has no "switch" ports.. That is not a box I would recommend to use in a home.. Unless you were a labber/IT hobbiest, etc. You in the UK, sky broadband is the big one over there so that is what I would guess.. Whats your electric cost like 24p a kwh? So that thing sucking 75 watts idle cost you like a 160 a year.. Your going to spend more than just getting a 200 something box that uses 20 watts (prob way less) in 2 years.. 200 box 42 a year for electric 42 2nd year your at 285 lets call it.. With that box in 2 years you have spent 320 just in electric. If you got it free. This sort of gear doesn't make a lot of sense for home user.. They are normally loud! they suck a lot of juice compared to other options. They are normally way overkill for home use. Unless you lab with it, and its not on very often. Even when you get something like this for free - it can end up costing you more money then if you would of just bought something appropriate for your use case. Whats your internet speed? Something like a sg1100 be better suited for a home use of pfsense. Its small compared to that thing. It sucks like only 4 watts idle..

@Antibiotic I wait 5 minutes and test again, now looks like start using Tier 1

@rikazkhan Your message was just a quote. Did you mean to add something ?

@manjotsc glad you got it sorted, and thanks for explaining what the actual problem was - this for sure helps the next guy!

@klubar well I would just delete the other route as well to see if they come back. The only thing that I recall where routes like that would be set is if had set dns per interface. With the gateway, see my dropdown that says none above.. And I thought that if you allowed dns to be overriden by dhcp it could add a route there too Maybe one of those things is where they got added, but never got deleted.. I would just delete the other route, and make sure next time you reboot that they don't come back. Are they using dns to resolve what they should ping to see if hey have internet? Not a fan of company XYZ using some other service to check if your device you sell has an internet connection.. And you sure and the heck shouldn't hard code IPs.. If you want to check if your device you sell to people can get to the internet it should use what ever dns was provided to your device by either dhcp or set on it and look up some fqdn that you the company controls.. If you then want it to ping that IP, it should be pinging your resource not some other companies IP.. NTP is in line with this as well - if you making a device that will want/use ntp.. And you want to point it to the ntp pool, then get with ntp org and get your own vendor fqdn.. And if your just going to point to ntp in general, don't for F sake point to your country code..When the device is going to be used in different country.. I had one of those smart wifi plug things, still do actually just use it when put out the xmas lights. But clearly the thing is not going to be used in the UK.. We use different power plugs, so there is no freaking way I bought this in the uk and just using it in the us.. But it wants to check its time with the uk.ntp pool.. So I created a host override for what is was looking for.. [image: 1714671290374-ntp.jpg] Just pure laziness if you ask me, or they are hiring the cheapest developers they can hire? Or they have some developer and said hey make this work by tmrw we need to start shipping them out next week.. And failed to even give him any parameters or where it might be used.. Hey have it check ntp time while your at it ;) My other pet peeve is these iot devices that have zero dns cache.. Ok we know you want to talk to fqdn xyz.. But do you really need to ask dns every 2 seconds for it, when you were just given the answer 10 seconds with a ttl of 24 hours ;) I mean it takes really nothing to store the dns record for the next time you want to go there in 2 seconds.. You don't have to store 10k records you need to store the handful you might be wanting to talk too.. Well now I have just gotten off on a rant, sorry ;)

@heliop100 I think you have to give permission - route - to the LAN segment to go out each of the gateways. This is done under firewall, NAT, Outbound. Usually it is recommended before adding rules to select manual then save. Then start adding rules for routing. [image: 1714666796595-untitled.jpg]

@viragomann said in HA, gateway groups and firewall's default route.: Is it this, why you were looking for localhost? Not OpenVPN, but the outbound connections from the firewall itself. In the documentation, localhost should be 'NATted' to interface IP address, which is OK, this part is clear to me. What wasn't clear to me is if I should have created a new gateway group, using Interface address, to use as the default route of the firewall. However, since it doesn't translate anything and everything is working, including IPsec, I'll leave as it is right now, everything using the same gateway group, that has the CARP IP there. Note that without those CARP IPs in the gateway group, IPsec tunnels won't go up.

Gut, denn das sollte einfach so funktionieren, wenn die IP der Fritz im Netz liegt was durch den VPN Tunnel geroutet wird. Also Route in die Fritz mit Ziel pfSense für das gleiche netzt und auf geht es.

@Dobby_ Yes, this is how we are setup. There is not a static IP address, but the cable modem assigns an address to the WAN port on the pfsense router through DHCP. This has been stable for several years before it suddenly stopped working and showed an IP address of 0.0.0.0. Since MAC spoofing solves it, I think the cable modem or ISP is somehow locking the old MAC address and excluding it.

@madbrain I rebooted one more time, and the problem went away - Verizon gateway no longer showed. Very strange. It's an intermittent problem. I decided to remove the USB Realtek 8156B NIC, and replace it with a PCI-E Intel I-225V (B3) that I bought at Central Computers earlier today. This necessitated shutting down the machine to insert the PCI-E card. During the next boot, this is what the interface assignment screen looked like (MAC adresses omitted) : [image: 1714104594603-58150e9e-53b2-46dd-81e9-09f2c62e0d3d-image-resized.png] As you can see, both the Comcast and Verizon interfaces are assigned to the same ix0 network port. Previously, Verizon was assigned to the ue0 network port for the RTL8156B. That port no longer exists since the USB NIC was unplugged. It is a bit disconcerting to see two interfaces on the same NIC, to say the list. Not sure what the right behavior should be, though. Maybe have the interface be unbound (no network port) ? So far, I am not seeing the same issue with all 4 Ethernet ports using Intel NICs (motherboard 1 Gbe, X550-T2 2 x 10 GBe, I225-V (B3) 2.5 Gbe), but it's only been a few minutes.

@viragomann perfect - thank you for the response

@ssppcc You just need to add a static route for the docker subnet. So in System > Routing > Gateway add a new gateway within the server network with the IP 10.0.200.8. Then go to the Static routes tab and and a new static route for 10.31.0.0/16 and state the gateway you've created before.

@Gblenn I teach daily, days and nights so my occurrences are likely more visible. Typically, it fails over to Tier 2 and often is unnoticable. (I find out later looking at logs.) However, on this latest instance, the Tier 1 was flapping and zoom reported my connection was unstable. Students reported poor audio and video. It was only resolved by changing the backup to Tier 1 and restarting zoom. Later that night my main connection stabilized. so naturally I'm thinking it is trying to switch back as soon as Tier 1 is available. I will test to verify.