Packet loss at certain time every night

waqark3389

I have an issue where every night at 1:08AM I see packet loss for about 5 minutes for anything going through the pfsense firewall. I have 3 interfaces, LAN, WAN and DMZ and packet loss is not restricted to one of these, it happens for every interface.
Monitoring bandwidth shows no spikes at all at this time and traffic is the same as it is during any other time of the day.

I am on the latest version, pfsense is installed on a bare bones machine and everything is wired only.
Also I've tried to bypass the pfsense firewall to determine if it is my ISP, and during this I saw no packet loss at that time. It seems this is only happening when using pfsense.

Can someone please help, where do I start? If you require any more information please let me know.

GoldFish

Need more info. Did you try to ping a site or internally. Is there a switch in between?

waqark3389

I tried to ping internal and external IP.

172.16.1.0/24 = DMZ network on pfsense
10.1.0.0/16 = LAN network on pfsense

When I run a constant ping from 10.1.0.0 range to 172.16.1.0 range I see packet loss at 1:08AM

When I run a constant ping from either 172 or 10 to an external IP, for example, 8.8.8.8 I see packet loss at 1:08AM.

I even tried to ping from pfsense itself to internal and external site and I still see this problem.

There is a switch in between but I did bypass the switch i.e. ping directly from the pfsense firewall to external 8.8.8.8 IP and still see the problem.

The layout is as below:

LAN machines/DMZ machines -> switch -> pfsense -> ISP router

Ping from LAN to LAN = No packet loss at all
Ping from DMZ to DMZ = No packet loss at all
Ping from DMZ to LAN machine = Packet loss at 1:08AM around 80% for 4-5 minutes
Ping from pfsense to 8.8.8.8 = Packet loss at 1:08AM around 80% for 4-5 minutes
Ping from ISP router to 8.8.8.8 = No packet loss at all

Thanks

Gertjan

Hi,

SSH into pfSense before 1:08 and run "top". See what happens.
Also : inspect your cron jobs, and check what might run at that moment.
Also : you use packages ? (that update regularly ?)

Disconnect WAN and ping from LAN to DMZ at that moment - still seeing the same thing ?

Implement stats like these : https://www.test-domaine.fr/munin/brit-hotel-fumel.net/pfsense.brit-hotel-fumel.net/cpu.html ;)

waqark3389

I will run top tonight and report back.

I have nrpe package installed but this is not updating every day. Plus nothing in cron anywhere doing anything at that time.

Ive set up graphs like the one you show and I can see nothing at 1:08. I am using snmp to monitor and no spikes in cpu/traffic or anything at that time!

I didnt try and disconnect that WAN, I will do that as well tonight.

waqark3389

Nothing to report on this. I tried all above. CPU is normal at 1:08AM and traffic is also normal. By normal I mean its the same as it is during any other part of the day.

Disconnected WAN, still packet loss from DMZ to LAN network at that time.

Thanks

Gertjan

Humm. Interesting.

Stop the ntpd daemon in the GUI, goto shell access, and launch :

date

Note the time. Is it ok ?
Change the time with date. An hours or so.
The question is : the issue happens again, at what time ?
If the source of the issue comes from pfSense, the time will change. If the source is from somewhere else, like your PC that start a packet hail storm at 01h08, then it will still happen at the real 01h08.

Install the Cron package if you didn't do so already.

What does

ps ax

shows ?

And another shell access in parallel :

top -t -ocpu