update_alias_url_data stalls packet flow

pitchfork

What is causing minicron to call this

php-fpm 359 /rc.update_alias_url_data: Configuration Change: (system): URL alias data updated via minicron

at 2:18am every night when the only relevant entry in config.xml is set to the default 12:30:

This is crucial because around the time this is called pfSense seems to stall and connectivity for all the machines behind it is interrupted. I'd like to change the time this happens, or ideally make it happen in a way that does not interrupt packet flow.

/var/db/aliastables/ has a single entry in it. a file with less than 70 single IPs

stephenw10

Install the cron package and change it. Or just edit the config file and reboot.

If you're running 2.6 you may need the recommended workaround patch for this: https://redmine.pfsense.org/issues/12827

Steve

pitchfork

@stephenw10 damn I wish I knew this earlier.

FreeBSD example.com 12.3-STABLE FreeBSD 12.3-STABLE RELENG_2_6_0-n226742-1285d6d205f pfSense amd64

A) could I pls get the patched kernel?

B) just for reference, "Applying the recommended workaround patch in System Patches makes this issue go away." - where is this "System Patches" so I could look next time for a fix there?

C) Just so I understand (even if I can't change it unless I install the cron pkg or edit the config file): where is the trigger to run this at 2:18am coming from?

stephenw10

See: https://github.com/pfsense

The System Patched package can be installed via the gui package manager and recent versions now contain a list of recommended patches:
https://docs.netgate.com/pfsense/en/latest/development/system-patches.html

The reload there is probably being triggered by pfBlocker if I had to guess. Maybe Snort/Suricata if you're running those. Most of those packages now include a random element in their update time settings.

Steve

pitchfork

@stephenw10 I cannot post an issue on Github. How do I requeste a patched kernel? The patch has been applied but it still stalls packet flow most of the time it runs.

On another notes, I don't run Snort/Suricata. Is pfBlocker an additional or default package?

thanks Steve.

stephenw10

pfSense bugs are tracked on redmine not github.

What kernel patch do you want? The pf counters patch can be applied via the System Patches package, it's not a kernel patch.
The root cause is fixed in 2.7 though.

pfBlocker is an additional package.

Steve

pitchfork

@stephenw10 yes, pf counters patch is applied but it doesn't fix the issue entirely. it still happens on occasion.

this is the patched kernel i was looking for https://redmine.pfsense.org/issues/12827#note-22. unless this has been superseded by the pf counters patch, in which case I'd have to upgrade to a non stable version. since I can't do that, i will have to figure out a way to disable this reload until 2.7 is out.

stephenw10

Just how much interruption are you still seeing with the counters patch in place?

I see nothing here with that.

There is no 2.6 kernel with those patches in, they were only committed to 2.7. Running a 2.6 kernel with those patches is completely untested. More so than 2.7. By a lot! I would definitely recommend at least testing 2.7 before attempting that. If only to prove they will actually help your situation.

Steve

pitchfork

@stephenw10 hmm, i don't have an exact count but I will guess 50-60% less? not enough days have passed since I installed the patch, so even a precise count would be statistically meaningless.

I don't have pfBlockerNG installed. no Snort/Suricata either. i would like to pause updating this list entirely until 2.7 is stable. since i don't know what is triggering it, i can only remove the list entirely.

stephenw10

But what sort of interruption are you seeing? Lose one ping in a ping stream? Completely loss of connectivity?

pitchfork

@stephenw10 my applications require sub second liveliness and they get knocked out for a good 30 to 40 seconds. I wouldn't care if it was just a monitor saying the connection is iffy... the applications are failing to perform and those are the alarms i see every night.

the uptime monitors (minute resolution) don't even register the downtime.

stephenw10

Ah, OK. Then that is something different. Even without the patch that delay loading the filter was only ever a few seconds.
What do you actually see logged at that time?

pitchfork

It is possible that it harms performance for 30 to 40 seconds because if the application is offline for a few seconds it has to catch up with the lost time.

BUT I have done pings (one per second between 2:15 and 2:20) and there are several failures over that 30-40 sec time span:

the 02:15:02.413719768 can be ignored. it's a random failure. but the cluster from 02:17:45.909412786 to 02:18:36.380112182 is when the outage occurred.

PS: any rough idea when 2.7 will ship? as in 1 month? 6 months?

stephenw10

Not at this time but 22.05, which includes that same fix, is imminent.

Steve

pitchfork

@stephenw10 said in update_alias_url_data stalls packet flow:

Not at this time but 22.05, which includes that same fix, is imminent.

Steve

hmmm, I'm inclined to upgrade sooner or later, but the same fix won't solve it... happy to go private with you if you'd like to dig further. thank you very much for all your help!

stephenw10

That's not the counters workaround it's the fixes to pf to prevent the delays. Same as 2.7.

If you manually the update command: /etc/rc.update_alias_url_data
Do you see the same thing?

pitchfork

weird, running /etc/rc.update_alias_url_data manually doesn't show up in the log at all (and no downtime either)

good to know, i will look to upgrade. any rough idea when the free enrollment will be over? Happy to be a paying customer, just a solo guy though, budget is tight.

stephenw10

Mmm, yeah it seems likely that's just part of something else that is called at that time. Nothing else is logged at that point?

pitchfork

@stephenw10 nada!