pfSense drops Internet ?
-
So this has happened 3 or 4 times now ... and it's driving me batty because I can't find any rhythm or reason to why or what is causing it.
Everything will be fine. Internet will be working, no issues ... then it drops off. In the past it's happened at night, but today it was in the middle of the day, so I decided to try and diagnose it.
My cable modem had not dropped .. it was still synced. I could still access pfsense, so I ssh'ed in. I tried to ping 8.8.8.8 and could not. Ping would return "Permission Denied". As shown in the follwing
So I tried to power cycle the modem, figured why not. It synced back up with no issues, but I had no WAN ip. I tried to release/renew the interface, still could not get a WAN ip. In the past, I had tried to disable and re-enable the WAN interface, but that doesn't work either.
I was forced to reboot. Reboot always seems to fix it.
I hate rebooting =( ... I'd like to figure out what is causing it ... but I don't know where to start. It's also random. It could go a month or more now and be fine --- it could go a day or so. I don't know.
Below is my NIC info;
em0: <Intel(R) PRO/1000 Network Connection 7.6.1-k> port 0xe020-0xe03f mem 0xfea80000-0xfea9ffff,0xfea60000-0xfea7ffff irq 24 at device 0.0 on pci1 em0: Using an MSI interrupt em0: Ethernet address: 00:15:17:f1:72:f0 em0: netmap queues/slots: TX 1/1024, RX 1/1024 em1: <Intel(R) PRO/1000 Network Connection 7.6.1-k> port 0xe000-0xe01f mem 0xfea20000-0xfea3ffff,0xfea00000-0xfea1ffff irq 25 at device 0.1 on pci1 em1: Using an MSI interrupt em1: Ethernet address: 00:15:17:f1:72:f1 em1: netmap queues/slots: TX 1/1024, RX 1/1024I'm wondering if it's a tweakable or something I need to adjust in the advanced settings? Some table or memory that is running out ??
This is running on an hp thinclient, 8gb of RAM ...
Thanks for the help,
wangel -
Ping would return "Permission Denied"
I know how help you. Lol -
@wangel Answer simplest question:
Did you use 8.8.8..8 as monitoring IP for WAN?
If answer is: "YES", then: change it. Google ban icmp from time to time. I have such experience before. You can use 1.1.1.1, at least I don't have issue until with it. If you use tunnel broker you can ping your tunnel endpoint (good practice too) -
@dragoangel
Thanks for that info ... but I tried pinging other ips too. I tried 1 I know I can always ping, 192.107.41.3.I also tried 8.8.4.4.
The entire Internet drops out --- not just pinging google =(
Thanks,
wangel -
@wangel Really strange that only reboot help. You tried reboot modem and not pfsense?
-
@dragoangel
Yes sir --- rebooting the modem does no good, I can't get a DHCP address =( -
@wangel said in pfSense drops Internet ?:
@dragoangel
Yes sir --- rebooting the modem does no good, I can't get a DHCP address =(If you connect pc straight to modem this not reproduce?
P.s. hate modems)) -
@dragoangel
Correct, if I connect PC straight to modem it works. -
@wangel said in pfSense drops Internet ?:
Correct, if I connect PC straight to modem it works.
Have you tried spoofing the MAC address of your PC under Interfaces > WAN > General > MAC Address?
-
What is your ISP and modem?
-Rico
-
and if i can ask.. do you have suricata/snort running inline mode?
-
@kiokoman Suricata is running, but it is running in Legacy mode.
I thought it might have something to do with Suricata also.... but I didn't see anything in the logs showing that....
-
@Rico ISP is Spectrum ... modem is a Surfboard 6183 or 6180, I forget which. But it's a Surfboard.
-
@biggsy
No, I have not tried that. I can just to see what would happen tho, heh. -
@wangel said in pfSense drops Internet ?:
Can you please show a screenshot of this page- SystemRoutingGatewaysEdit
-
-
Try clicking the "Disable Gateway Monitoring Action" box and see if it reoccurs. Re-enable the gateway monitoring.
You should let the monitoring happen so you can look and see later if your ISP is dropping out or not by watching the logging graphs. StatusMonitoring
-
@chpalmer Done.
Will monitor/report back if it happens anymore. Thank you sir!
-
I have similar issue (Internet stop working on all interfaces) described here: https://forum.netgate.com/topic/143661/one-interface-loses-internet-access-and-i-could-get-it-back-only-after-reboot-the-pfsense
but still no solution found -
@ady2 said in pfSense drops Internet ?:
I have similar issue
Probably not.
@wangel didn't post back, so, fingers crossed : case closed.
The other thread is also a case closed.Btw : problems described might match, and if so the answers in these threads contain the solutions.
If your problem is identical, the proposed solutions would also work for you.
So, what is it : the problem looks identical, but you did not apply proposed solution ? Why not ? -
@Gertjan You jinxed it!!!!
Not really ... I hadn't posted back because things had been working, until about 4pm EST yesterday, when it died again.
I was able to access resources from the outside (ie: My Camera system etc), but internet out would not work. I also could connect to the vpn, type in my password, but the vpn would never establish.
When I got home, I had no internet. I could get on the pfsense box, but if I tried to ping the outside world, I would get "permission denied". Now was the time I needed to figure this out.
Over the past couple of days, I had been working on setting up Graylog and sending my logs from Pfsense to it. It's working, so I figured I'd go digging.
Sometime yesterday, about 3:35pm EST, Unbound, or Quad9, or something lost their minds. It started flooding my firewall and pfsense blocked my own public ip. That explains "permission denied". I tried running /etc/rc.filter_configure. That fixed it so I could ping OUT from the pfsense box, but nothing from the LAN side could still ping. At this point, I rebooted.
Whatever it was, was Quad9. Everything was on port 853, and to and from ip 149.112.112.112.
That being said, DNS is setup on pfsense going to cloudflare first (1.1.1.1, 1.0.0.1) and then I had 9.9.9.9 and 149.112.112.112 as the 3rd and 4th dns entries.
I've removed Quad9 from the equation, so we will see if it happens again. I don't know if it was unbound doing something ... or something on my network that started making a ton of calls to 149.112.112.112, but whatever it was cause the PF to end up blocking my WAN Interface.
I will continue to monitor and report back.
Thanks all!
-
@Gertjan said in pfSense drops Internet ?:
didn't post back, so, fingers crossed : case closed
you brought bad luck
-
I wonder about cases where a ton of traffic is going out of the network like that. Does the ISP possibly flag it as malicious and block it?
-
Try disabling the hardware checksum.
I had a problem where my internet would randomly drop after moving to a new box. I made the change to disable hardware checksum and haven't had a problem since.
-
Permission denied like that is from pfSense itself. It's not the other end refusing the echo requests.
As you found it's usually the traffic being blocked by Squid/Suricata. The WAN IP itself would be inside the default Homenet alias and therefore excluded from blocking. I assume it was blocking the ping target therefore? If it was blocking the actual WAN IP then something is misconfigured.
Steve
-
@stephenw10 said in pfSense drops Internet ?:
Permission denied like that is from pfSense itself. It's not the other end refusing the echo requests.
As you found it's usually the traffic being blocked by Squid/Suricata. The WAN IP itself would be inside the default Homenet alias and therefore excluded from blocking. I assume it was blocking the ping target therefore? If it was blocking the actual WAN IP then something is misconfigured.
Steve
@stephenw10 It's not Suricata.... as I had it turned off, because I thought it was. Nothing shows up in the suricata logs either. So I know that's not it.
I don't run squid.
Not sure what could be misconfigured --- I am using unbound over SSL for dns (853) ... I had 4 dns servers configured, cloudflare (1.1.1.1) and Quad9.
Like I said, according to the logs ... something happened with a ton of traffic from Quad9's 2nd dns server, and it PF ended up blocking my WAN IP. (or Interface).
I was able to run /etc/rc.filter_configure, and I was able to ping out from the console again, but routing was not working for the LAN, so a reboot obviously fixed that.
filterlog: 18,,,1000000110,em0,match,block,in,4,0x0,,53,51099,0,DF,6,tcp,83,149.112.112.112,publiciphere,853,58393,31,FPA,1666564533:1666564564,3751497853,118,,nop;nop;TSWhich is odd to me .... Why would Quad9's server try connecting to me? It's only this server that's done it, I tried searching the logs for 9.9.9.9 and nothing. Strange honestly.
Anyways, I took Quad9's info out of PFsense, only running off Cloudflare for now, so we will see what happens :)
-
That is their reply packets being dropped because the TCP state outbound to them had already been closed. That's quite common and usually nothing to be concerned about:
https://docs.netgate.com/pfsense/en/latest/firewall/troubleshooting-blocked-log-entries-for-legitimate-connection-packets.htmlSteve
