A little support for a home user.

behemyth

Does your internet connectivity come back automatically when it stops working, or do you have to power cycle either pfsense or the cable modem?

I have a suspicion that what's actually happening is your cable modem isn't responding to dhcp requests from pfsense, and because its not your wan interface is dropping because the lease time is expiring. Does this happen right around 8 hours or so?

badfrogg

Its is random throughout the day and it comes back on its own after a minute or two.

behemyth

Can you log into the cable modem? If you can make sure they disabled everything the firewall is doing on it. If your passing through the public IP, you dont need the firewall running on that at all anymore.

I know when I had ATT, i could pass through the public IP, but I still had to disable all the firewall services. I wonder if thats part of your problem. There shouldn't be an issue with the gateway (ARP) unless the ISP has something wrong, and I highly doubt that. My guess is your lease time is expiring, its requesting a new one (different packet types) and your firewall is responding and renewing your lease, thus your internet connectivity comes back.

I wonder if that error is because your lease time expires, pfsense ARPs for the gateway, but because the lease time expired its not technically on a network, and then when it gets or renews the IP you had its back to working again.

behemyth

I would personally go buy a new cable modem, that is certified to work on Spectrum's network. I bet that would solve all your issues. Here is a list of Approved Modems for Spectrum:

https://www.spectrum.net/support/internet/compliant-modems-charter-network/

I use the Motorola MB8600, it has been rock solid for me for years.

badfrogg

I dont think I can log into the Modem. If I plug my computer directly into it, it will give me the public IP. I haven't tried to force the private IP and try to browse the GUI. Do you think that would work?

My Gateway Log if it helps...

May 31 15:08:07	dpinger		WAN_DHCP 173.xxx.xxx.xxx: sendto error: 65
May 31 15:08:06	dpinger		WAN_DHCP 173.xxx.xxx.xxx: sendto error: 65
May 31 15:08:05	dpinger		WAN_DHCP 173.xxx.xxx.xxx: sendto error: 65
May 31 15:08:05	dpinger		WAN_DHCP 173.xxx.xxx.xxx: sendto error: 65
May 31 15:08:04	dpinger		WAN_DHCP 173.xxx.xxx.xxx: sendto error: 65
May 31 15:08:04	dpinger		WAN_DHCP 173.xxx.xxx.xxx: sendto error: 65
May 31 15:04:41	dpinger		WAN_DHCP 173.xxx.xxx.xxx: Clear latency 350437us stddev 826073us loss 0%
May 31 15:04:32	dpinger		WAN_DHCP 173.xxx.xxx.xxx: Alarm latency 450853us stddev 885261us loss 0%
May 31 15:04:27	dpinger		WAN_DHCP 173.xxx.xxx.xxx: Alarm latency 521943us stddev 967372us loss 0%
May 31 15:04:22	dpinger		WAN_DHCP 173.xxx.xxx.xxx: Alarm latency 63587us stddev 66054us loss 25%
May 31 15:04:18	dpinger		send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 173.xxx.xxx.xxx bind_addr 173.168.225.119 identifier "WAN_DHCP "
May 31 15:04:05	dpinger		send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 173.xxx.xxx.xxx bind_addr 173.168.225.119 identifier "WAN_DHCP "
100%

behemyth

Ok, so your latency spikes and your actually having packet loss. That would explain why your losing your internet connectivity. I would replace that modem, preferably with a non rented one from the list in my previous post, and i bet all your problems would go away.

If you cant currently afford to buy your own, have spectrum replace that one. There is definitely something weird going on with it.

It's also complaining about your WAN DHCP IP, so I think I'm on the right track with that too.

badfrogg

@behemyth Thank you. I will look at a new modem.

behemyth

It is also possible there is a line problem causing the latency, maybe if the wind blows the wrong way its moving around, but that's not something they will check until you have done everything else.

Once you replace the modem, if you still have problems, remove pfsense from the loop and just run your network off the all-in-one and see if your problem goes away. If it does, then we know for sure its between the modem and pfsense.

badfrogg

I was using an old DD-WRT router for years with this modem without any problems (other than it was just a 10/100 speed). I also tested with a cheap D-link router and that is fine as well.

DaddyGo

@behemyth
I would first test that without pfSense, another router (SOHO) or desktop one is experiencing this issue
because only then is your modem bad category...

it's easy to tell - BAD MODEM - and must be replaced immediately, first prove it's bad what you're throwing away

behemyth

If you used a cheap router already and it worked than it’s your hardware pfsense is running on.

Not using intel NICs is asking for problems. eBay a cheap 2 port server nic for like 20 bucks and I bet your problems will resolve themselves. Someone said this above and their probably right.

I ran pfsense virtualized in VMware for a long time using intel network cards and it ran great. Your using barebones hardware, should have the same experience.

badfrogg

You missed the part where I built a whole new box with intel pro/1000 nic card. Still the exact same problem on two different pfsens boxes.

DaddyGo

yes, I saw that too, @badfrogg "You missed the part where I built a whole new box with intel pro/1000 nic card. "

How localized is the time interval when the issue occurs?
since, then one should pay attention in the logs to what is actually happening

the phenomenon is random or predictable, that’s what I wanted to ask

behemyth

The issue is definitely the latency and packet loss your seeing. Pfsense might be more sensitive to this since it has a check that runs all the time making sure that it’s working correctly than another router probably is.

It’s possible the new network card you bought is causing the problems, but the driver that’s used would be in the version of free-bsd that their building pfsense in. You would have to check your model against supposed versions for that version of free-bsd.

It’s also possible the modem is the problem, and has been a problem but the previous routers you used were less sensitive to the issue.

There is no denying the packet loss and latency spikes in the logs you posted above though.

badfrogg

It is random. Cant find any pattern, Sometimes it is back to back. can happen three times in one hour. I have also seen it go three days without happening at all. No change on my end as a user. I work remote while the TV is streaming with Roku, some usual phone activity. Pretty normal use.

This definitely dose not happen with a cheap walmart router. I know because the TV is streaming and My VPN to work will warn me when I lose internet.

DaddyGo

well, so i'm right, you don't need to throw out the modem

can you possibly monitor (for a longer period of time) and present a log of when this happens ?

badfrogg

So... For no particular reason I decided to spoof the MAC address. I simply typed the MAC address in from the sticker on the bottom of one of my cheap-o Walmart routers. I also used my original box with the Realtek NICs. I made no other changes to that box.

here is the logs from the past 4-5 days:

Jun 5 20:01:59	php-fpm	30559	/index.php: Successful login for user 'admin' from: 192.168.1.105 (Local Database)
Jun 4 15:51:45	root		rc.update_bogons.sh is ending the update cycle.
Jun 4 13:28:32	root		rc.update_bogons.sh is beginning the update cycle.
Jun 4 11:06:43	root		rc.update_bogons.sh is sleeping for 8509
Jun 4 11:06:43	root		rc.update_bogons.sh is starting up.
Jun 4 10:51:02	root		Could not extract fullbogons-ipv4.txt
Jun 4 10:51:02	root		Could not download https://files.pfsense.org/lists/fullbogons-ipv4.txt
Jun 3 13:06:58	php-fpm	31761	/rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'
Jun 3 13:06:57	check_reload_status		Reloading filter
Jun 3 13:06:57	check_reload_status		Restarting OpenVPN tunnels/interfaces
Jun 3 13:06:57	check_reload_status		Restarting ipsec tunnels
Jun 3 13:06:57	check_reload_status		updating dyndns WAN_DHCP
Jun 3 13:06:57	rc.gateway_alarm	86872	>>> Gateway alarm: WAN_DHCP (Addr:173.xxx.xxx.xxx Alarm:0 RTT:61.993ms RTTsd:340.307ms Loss:9%)
Jun 3 13:06:53	php-fpm	25905	/rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'
Jun 3 13:06:52	check_reload_status		Reloading filter
Jun 3 13:06:52	check_reload_status		Restarting OpenVPN tunnels/interfaces
Jun 3 13:06:52	check_reload_status		Restarting ipsec tunnels
Jun 3 13:06:52	check_reload_status		updating dyndns WAN_DHCP
Jun 3 13:06:52	rc.gateway_alarm	57988	>>> Gateway alarm: WAN_DHCP (Addr:173.xxx.xxx.xxxAlarm:1 RTT:39.094ms RTTsd:245.570ms Loss:14%)
Jun 3 13:05:57	php-fpm	30559	/rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'
Jun 3 13:05:55	check_reload_status		Reloading filter
Jun 3 13:05:55	check_reload_status		Restarting OpenVPN tunnels/interfaces
Jun 3 13:05:55	check_reload_status		Restarting ipsec tunnels
Jun 3 13:05:55	check_reload_status		updating dyndns WAN_DHCP
Jun 3 13:05:55	rc.gateway_alarm	8337	>>> Gateway alarm: WAN_DHCP (Addr:173.xxx.xxx.xxxAlarm:1 RTT:3129.560ms RTTsd:8109.103ms Loss:27%)
Jun 3 13:05:44	php-fpm	91976	/rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'
Jun 3 13:05:43	check_reload_status		Reloading filter
Jun 3 13:05:43	check_reload_status		Restarting OpenVPN tunnels/interfaces
Jun 3 13:05:43	check_reload_status		Restarting ipsec tunnels
Jun 3 13:05:43	check_reload_status		updating dyndns WAN_DHCP
Jun 3 13:05:43	rc.gateway_alarm	83446	>>> Gateway alarm: WAN_DHCP (Addr:173.xxx.xxx.xxxAlarm:1 RTT:12.932ms RTTsd:4.719ms Loss:21%)
Jun 2 10:43:07	php-fpm	25905	/index.php: User logged out for user 'admin' from: 192.168.2.101 (Local Database)
Jun 2 10:37:19	php-fpm	21933	/index.php: Successful login for user 'admin' from: 192.168.2.101 (Local Database)
Jun 1 11:00:57	php-fpm	25905	/index.php: Successful login for user 'admin' from: 192.168.2.101 (Local Database)
Jun 1 06:21:17	root		rc.update_bogons.sh is beginning the update cycle.
Jun 1 03:01:00	root		rc.update_bogons.sh is sleeping for 12017
Jun 1 03:01:00	root		rc.update_bogons.sh is starting up.

Yeah, that is a big difference. It was hundreds of entries per day flooding the logs. Looks like there was something happening on June 3rd for about a minute but I did not notice any interruption.

So I figured this out on my own so, thanks for nothin! LOL I'm just teasing. Nobody get sensitive.

The fact is I just got lucky. Searching the interwebs gave some clues but I did not find any threads that showed or properly explained why spoofing the MAC would fix this problem. I still have no idea what I'm doing really. I don't even know what the log is telling me about the gateway alarm that shows on June 3rd.

Regardless, I am posting what I have stumbled upon. Maybe someone here can explain why this worked so that I might learn something useful... at least something more useful then being good at searching and filtering all the trash on the internet. That skill, I clearly have.

DaddyGo

I wrote on a similar thing here on the forum about 7 months ago, it was just a DOCSIS issue (DOCSIS modem + WAN dynamic IP)

MAC spoofing was useful, because the CMTS and EdgeQAM in the ISP network, were manufactured by Cisco.

pcEngines APU MAC vendor address CMTS doesn't seem to like it and at the moment we spoofed the MAC address of an old E900 Cisco router, the APU pfSense box immediately got the DHCP lease on WAN interface.

(perhaps Cisco to Cisco)

and