A little support for a home user.

behemyth

Ok, so your latency spikes and your actually having packet loss. That would explain why your losing your internet connectivity. I would replace that modem, preferably with a non rented one from the list in my previous post, and i bet all your problems would go away.

If you cant currently afford to buy your own, have spectrum replace that one. There is definitely something weird going on with it.

It's also complaining about your WAN DHCP IP, so I think I'm on the right track with that too.

badfrogg

@behemyth Thank you. I will look at a new modem.

behemyth

It is also possible there is a line problem causing the latency, maybe if the wind blows the wrong way its moving around, but that's not something they will check until you have done everything else.

Once you replace the modem, if you still have problems, remove pfsense from the loop and just run your network off the all-in-one and see if your problem goes away. If it does, then we know for sure its between the modem and pfsense.

badfrogg

I was using an old DD-WRT router for years with this modem without any problems (other than it was just a 10/100 speed). I also tested with a cheap D-link router and that is fine as well.

DaddyGo

@behemyth
I would first test that without pfSense, another router (SOHO) or desktop one is experiencing this issue
because only then is your modem bad category...

it's easy to tell - BAD MODEM - and must be replaced immediately, first prove it's bad what you're throwing away

behemyth

If you used a cheap router already and it worked than it’s your hardware pfsense is running on.

Not using intel NICs is asking for problems. eBay a cheap 2 port server nic for like 20 bucks and I bet your problems will resolve themselves. Someone said this above and their probably right.

I ran pfsense virtualized in VMware for a long time using intel network cards and it ran great. Your using barebones hardware, should have the same experience.

badfrogg

You missed the part where I built a whole new box with intel pro/1000 nic card. Still the exact same problem on two different pfsens boxes.

DaddyGo

yes, I saw that too, @badfrogg "You missed the part where I built a whole new box with intel pro/1000 nic card. "

How localized is the time interval when the issue occurs?
since, then one should pay attention in the logs to what is actually happening

the phenomenon is random or predictable, that’s what I wanted to ask

behemyth

The issue is definitely the latency and packet loss your seeing. Pfsense might be more sensitive to this since it has a check that runs all the time making sure that it’s working correctly than another router probably is.

It’s possible the new network card you bought is causing the problems, but the driver that’s used would be in the version of free-bsd that their building pfsense in. You would have to check your model against supposed versions for that version of free-bsd.

It’s also possible the modem is the problem, and has been a problem but the previous routers you used were less sensitive to the issue.

There is no denying the packet loss and latency spikes in the logs you posted above though.

badfrogg

It is random. Cant find any pattern, Sometimes it is back to back. can happen three times in one hour. I have also seen it go three days without happening at all. No change on my end as a user. I work remote while the TV is streaming with Roku, some usual phone activity. Pretty normal use.

This definitely dose not happen with a cheap walmart router. I know because the TV is streaming and My VPN to work will warn me when I lose internet.

DaddyGo

well, so i'm right, you don't need to throw out the modem

can you possibly monitor (for a longer period of time) and present a log of when this happens ?

badfrogg

So... For no particular reason I decided to spoof the MAC address. I simply typed the MAC address in from the sticker on the bottom of one of my cheap-o Walmart routers. I also used my original box with the Realtek NICs. I made no other changes to that box.

here is the logs from the past 4-5 days:

Jun 5 20:01:59	php-fpm	30559	/index.php: Successful login for user 'admin' from: 192.168.1.105 (Local Database)
Jun 4 15:51:45	root		rc.update_bogons.sh is ending the update cycle.
Jun 4 13:28:32	root		rc.update_bogons.sh is beginning the update cycle.
Jun 4 11:06:43	root		rc.update_bogons.sh is sleeping for 8509
Jun 4 11:06:43	root		rc.update_bogons.sh is starting up.
Jun 4 10:51:02	root		Could not extract fullbogons-ipv4.txt
Jun 4 10:51:02	root		Could not download https://files.pfsense.org/lists/fullbogons-ipv4.txt
Jun 3 13:06:58	php-fpm	31761	/rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'
Jun 3 13:06:57	check_reload_status		Reloading filter
Jun 3 13:06:57	check_reload_status		Restarting OpenVPN tunnels/interfaces
Jun 3 13:06:57	check_reload_status		Restarting ipsec tunnels
Jun 3 13:06:57	check_reload_status		updating dyndns WAN_DHCP
Jun 3 13:06:57	rc.gateway_alarm	86872	>>> Gateway alarm: WAN_DHCP (Addr:173.xxx.xxx.xxx Alarm:0 RTT:61.993ms RTTsd:340.307ms Loss:9%)
Jun 3 13:06:53	php-fpm	25905	/rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'
Jun 3 13:06:52	check_reload_status		Reloading filter
Jun 3 13:06:52	check_reload_status		Restarting OpenVPN tunnels/interfaces
Jun 3 13:06:52	check_reload_status		Restarting ipsec tunnels
Jun 3 13:06:52	check_reload_status		updating dyndns WAN_DHCP
Jun 3 13:06:52	rc.gateway_alarm	57988	>>> Gateway alarm: WAN_DHCP (Addr:173.xxx.xxx.xxxAlarm:1 RTT:39.094ms RTTsd:245.570ms Loss:14%)
Jun 3 13:05:57	php-fpm	30559	/rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'
Jun 3 13:05:55	check_reload_status		Reloading filter
Jun 3 13:05:55	check_reload_status		Restarting OpenVPN tunnels/interfaces
Jun 3 13:05:55	check_reload_status		Restarting ipsec tunnels
Jun 3 13:05:55	check_reload_status		updating dyndns WAN_DHCP
Jun 3 13:05:55	rc.gateway_alarm	8337	>>> Gateway alarm: WAN_DHCP (Addr:173.xxx.xxx.xxxAlarm:1 RTT:3129.560ms RTTsd:8109.103ms Loss:27%)
Jun 3 13:05:44	php-fpm	91976	/rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'
Jun 3 13:05:43	check_reload_status		Reloading filter
Jun 3 13:05:43	check_reload_status		Restarting OpenVPN tunnels/interfaces
Jun 3 13:05:43	check_reload_status		Restarting ipsec tunnels
Jun 3 13:05:43	check_reload_status		updating dyndns WAN_DHCP
Jun 3 13:05:43	rc.gateway_alarm	83446	>>> Gateway alarm: WAN_DHCP (Addr:173.xxx.xxx.xxxAlarm:1 RTT:12.932ms RTTsd:4.719ms Loss:21%)
Jun 2 10:43:07	php-fpm	25905	/index.php: User logged out for user 'admin' from: 192.168.2.101 (Local Database)
Jun 2 10:37:19	php-fpm	21933	/index.php: Successful login for user 'admin' from: 192.168.2.101 (Local Database)
Jun 1 11:00:57	php-fpm	25905	/index.php: Successful login for user 'admin' from: 192.168.2.101 (Local Database)
Jun 1 06:21:17	root		rc.update_bogons.sh is beginning the update cycle.
Jun 1 03:01:00	root		rc.update_bogons.sh is sleeping for 12017
Jun 1 03:01:00	root		rc.update_bogons.sh is starting up.

Yeah, that is a big difference. It was hundreds of entries per day flooding the logs. Looks like there was something happening on June 3rd for about a minute but I did not notice any interruption.

So I figured this out on my own so, thanks for nothin! LOL I'm just teasing. Nobody get sensitive.

The fact is I just got lucky. Searching the interwebs gave some clues but I did not find any threads that showed or properly explained why spoofing the MAC would fix this problem. I still have no idea what I'm doing really. I don't even know what the log is telling me about the gateway alarm that shows on June 3rd.

Regardless, I am posting what I have stumbled upon. Maybe someone here can explain why this worked so that I might learn something useful... at least something more useful then being good at searching and filtering all the trash on the internet. That skill, I clearly have.

DaddyGo

I wrote on a similar thing here on the forum about 7 months ago, it was just a DOCSIS issue (DOCSIS modem + WAN dynamic IP)

MAC spoofing was useful, because the CMTS and EdgeQAM in the ISP network, were manufactured by Cisco.

pcEngines APU MAC vendor address CMTS doesn't seem to like it and at the moment we spoofed the MAC address of an old E900 Cisco router, the APU pfSense box immediately got the DHCP lease on WAN interface.

(perhaps Cisco to Cisco)

and