@aiden21c Good! I still think that some good came out of this whole situation, though.
For one, even if your current setup works well, the ideal setup for your whole company network is still with VLANs
The order of the firewall rules needs to be held in mind (PfSense processes firewall packet rules from top to bottom):
Rule 3 catches all traffic filtered by rules 4, 5, 6. It needs to be last. Rules 5 and 6 have destination address "Any" instead of "LAN Address". A way that helps (me personally) to keep fw rules tidy is to add 4 separators, the top one named "GENERAL BLOCK" (for entire protocols, for example, no need to allow GRE, ESP, AH, OSFP... on a LAN with interconnected servers if there is no explicit need), a second separator named "INCOMING", a third separator named "LOCAL TO FW" and a fourth one named "OUTGOING". I also add separators named "PASS" and "BLOCK", with that order, under each main separator.
Even if no further network changes seem necessary, it is best to avoid NAT. In the future, in order to reduce latencies or enable certain UDP services that cannot be NATed, you can check if the Cisco Router can do PPPoE passthrough for PfSense. Because PPPoE is a separate interface in PfSense, you can have both a PfSense-to-Cisco connection (OFFICE - 192.168.20.40/24, not as a Gateway) and a PPPoE adapter as a direct PfSense Gateway (because PPPoE is a Layer 2 protocol, doesn't use IPs, that is why its Point-To-Point, so it doesn't interfere with the 192.168.20.0/24 subnet at all) with a public IPv4 for PfSense.
At some point, instead of having separate rules for each gateway and traffic type, you might want to implement Multi-WAN Load Balancing and Traffic Shaping to control which traffic type uses what Gateway.
It is best to set static IPs for LAN through the DHCP server (without a dynamic address pool) and set your private IPs as Static Mappings. That way, you can use Host Overrides on Unbound, which would allow you to use hostnames (and no IPs) in your setups, and avoid unnecessary config nightmares in case, say, you want to put everything in Docker. You can just change the IPs in the Static Mappings of PfSense Unbound, add a BIND container to Docker (just to handle the inter-container IPs using the same hostnames) and be done with it.
so upon disabling and re enabling the WAN interface this is when i see the issue occur. the only action that can be taken it seems is to manually select the gateway removing it off the automatic option. restarting the gateway service nor reboot changes its behaviour.
Running on 2.6.0-RELEASE (amd64) wonder if anyone else is getting the same issue?
Well Holy Crap @johnnyf1ve! I have been dealing with this same problem with Metronet for over 2 years. The cron script you initially wanted to implement is here. I've been using it for a long time to work around the issue.
I too have a Protectli appliance right after Metronet's Nokia modem. Bet you got one of those modem's too. I can say it's definitely not a hardware issue on my pfSense appliance because I've used the Protectli and tested with an old Dell with a Dual NIC Intel NIC. Same problems with dropped WAN.
I have long thought this is a unique problem with Metronet. They have cheap Fiber and you gotta cut costs somewhere. I have other Protectli boxes in other environments running with Metronet with the same result of losing the WAN connection. If you have a static IP from Metronet the problem of course goes away and that was the ultimate fix.
I just made the changes that you have found from the Reddit user and I'll see how it goes. I'll probably keep my Cron pintest.sh scheduled just in case. I can check the log files to see if it's actually resetting the interface after your suggestion. Thanks man!
Lastly, here are my last few pingtest.sh logs with the dropped connection from the past few weeks. When the ping test fails it turns the WAN interface off for a few seconds and then right back on which fixes the problem 99% of the time. As you can see, this happens a lot.
20220917.181601 All pings failed. Resetting interface igb0.
20220917.181635 All pings failed. Resetting interface igb0.
20220919.071749 All pings failed. Resetting interface igb0.
20220920.211301 All pings failed. Resetting interface igb0.
20220920.211335 All pings failed. Resetting interface igb0.
20220922.180104 All pings failed. Resetting interface igb0.
20220922.180138 All pings failed. Resetting interface igb0.
20220924.160249 All pings failed. Resetting interface igb0.
20220924.160323 All pings failed. Resetting interface igb0.
20220927.105456 All pings failed. Resetting interface igb0.
20220927.105530 All pings failed. Resetting interface igb0.
@wifi75 non mancava nulla.
Avevo fatto tutto esattamente come hai suggerito tu, ma non c'era login.
Ho chiamato il provider il quale ha inizialmente detto che poteva essere un problema del mio router, così mi sono procurato un altro router ma neanche con questo c'era login.
Di conseguenza hanno aperto un ticket con OpenFiber, e alla fine è venuto fuori che quando hanno fatto l'allacciamento si sono dimenticati di attivare qualcosa, per cui non c'era possibilità di connettersi.
Io avevo dato per scontato che fosse un problema di configurazione perché dopo che OpenFiber ha fatto l'allacciamento ho chiesto espressamente se la linea dovesse essere attivata dal provider, ma mi hanno assicurato che "potevo già navigare!"
In the time it took to fix this critical bug, I was able to:
Set up and thoroughly test out OPNsense in a staging environment
Find viable replacements for all the pfSense plugins and features I was using
Weigh the pros and cons of switching to OPNsense
Realize that open source pfSense has become a second class citizen
Provision a new production firewall with OPNsense
Manually copy the configuration from pfSense to the new OPNsense box
Retire my pfSense box and switch permanently to OPNsense
I figured out the transmission issue. It had to do with the negotiation between the MetroNode and the Chelsio NIC. I contacted my ISP and they turned off auto negotiation on the MetroNode and it started transmitting. It seems to be something in the driver for the T540-CR that I am using inside of ESXI. Therefore, everything seems to be working now.
I introduced a firewall rule on my HENET interface :
I have a DNS record that point's to my WAN IPv4, not my WAN IPv6, so I had to use my IPv6 WAN IP to connect to the GUI.
I had a cert warning from my browser, of course.
But the access worked well :
"Well" means for me : knowing that my IPv6 is using a tunnel to tunnel.ne.net (Huricane IPv6 ISP) the speed was somewhat limited, about 10 Mbytes /sec.
I could browse the entire pfSense GUI very well, no hick-ups ....
edit : I'll leave the IPv6 access open for a while.
PM me, and I can even send you an 'access' so you can test drive yourself.
That is, if you promise not to change something, as this is a "live' environment ;)
Connected to the router via wifi and my phone, got a "this network wants you to sign in" and when I clicked that, it brought up the comcast login
That's your OS / brower playing the captive portal detection mode !
That means your WAN is using a RFC1918 IP, and when you start your bowser it hits the GUI web server of the modem, because it's router part is redirecting the browser requests to it's internal Web GUI, where you have to login.
What about playing with these option on the WAN interface :
I realized that I do not need to add 192.168.0.x since my WAN interface is 192.168.0.1 and /32 was incorrect too. I have removed that. I can see the route in the table but still the ping to google.com or 184.108.40.206 or 192.168.0.1 from a VM(192.168.1.100) connected to pfsense is very random. how can I troubleshoot that?
edit: do I have to reboot each time I save anything? that seems to do the trick
With IPAliases you can usually use either /32 or the correct subnet size. The important thing is you have at least one IP defined on the interface with the correct subnet in order to add the correct routing.