PfSense crash after using Traffic Shaping Wizard
-
Hi all,
I am running pfSense 2.3.2-RELEASE (amd64) on an APU.1D4 board with one WAN interface and three LAN interfaces (one of which is used to access the VDSL modem using the same physical port as the WAN PPPoE interface). pfSense has been very stable for me, but today I encountered several crashes when I tried to turn on traffic shaping using the "Multiple Lan/Wan" wizard (I made a test a weeks ago using HFSC for all interfaces and didn't see any crash then, but I don't remember which release this was on).
Today I tried to enable traffic shaping with CBQ. As soon as I finish the wizard the pfSense box locks up - the web GUI doesn't respond anymore, I can't ping the internal interface and there is no output on the serial console. Sometimes it takes several reboots for it to come up again properly. After I removed the traffic shaper again, everything is back to normal. Several reboots worked without problems.
Unfortunately it seems that there is no crash log.
Did anybody else see this issue?
Thanks
Arno -
I did a little bit more testing. The crash does not occur when I choose HFSC instead of CBQ.
My interfaces are configured like this on APU1D4:
- WAN: PPPoE on re0_vlan7
- LAN: re1, internal network with all regular clients
- VOIP: re2, internal netwrok with only VoIP clients
- MODEMACCESS: re0 (same physical interface as WAN), internal network to configure the DrayTek Vigor130 Modem
Is it possible that this specific configuration leads to the crash I've been seing? Maybe the VLAN on the WAN interface?
-
I had a very similar issue when I moved from 2.2.6 to 2.3.2. See my post below.
https://forum.pfsense.org/index.php?topic=116682.0
I found that the problem stopped after disabling all my traffic shaping rules and a few packages.
After a few days without the issue I reinstalled snort and waited a few days. No problems.
After another few days, I deleted the disabled shaping rules and recreated them (or as close as I could remember). Still no problems.
I was thinking pfBlockerNG may have been causing the issue, since that was the last piece of the original setup that I have still not set back up. But after reading this post I am thinking the traffic shaping rules were the cause.
One thing to note, I do have a few VLANs and the original setup had shaping rules on the VLANs. I would have to check my configuration to see if I recreated those shaping rules or not.
-
I can too observe this crash on latest pfSense.
4 interfaces: re0, igb0, igb1 and igb0_vlan7. Before wizard, two limiters have been configured if this is important BTW. I have configured CBQ on all interfaces (WAN, LAN, DMZ, link4 …) with traffic_shaper_wizard_multi_all.xml. I have omitted VOIP and Games part of the wizard, just wanted to get initial skeleton for tweaking to my needs.
After saving configuration and reloading firewall, web gui becomes unresponsive, ssh connection too. New connections are not possible. ICMP - nothing. During reboot, ping is working for about 3-5 seconds, and then everything freezes. Attached on console, I can see machine freezes when starting firewall.
Booted in single user mode and restored latest pre-shaper config.xml, booted in normal mode ok.
No such crash if I decide to use HFSC during wizard.
-
Same thing here - what the fuck!
Didn't use the wizard just removed the shaper and added it again.
Good thing we have a secondary legacy T1 as a backup or we'd have been completely hosed.
-
Same thing here - what the fuck!
Didn't use the wizard just removed the shaper and added it again.
Good thing we have a secondary legacy T1 as a backup or we'd have been completely hosed.
Great bug report, thanks.
The details you include are especially useful.
-
Same thing here - what the fuck!
Didn't use the wizard just removed the shaper and added it again.
Good thing we have a secondary legacy T1 as a backup or we'd have been completely hosed.
Great bug report, thanks.
The details you include are especially useful.
I was too busy freaking out to contribute.
When I got on site there were no crashes. All it did was apply the traffic shaping rules and some how the wan interfaces were offline. I had access to the gui through this and it was a simple reboot after removing the codel rules.
I'm fairly certain it's the realtek ethernet ports going wonk - I had put a quad intel nic in there and have put off transitioning 2 of the ports because it was working fine.
I noticed a month or so back when I applied traffic shaping rules or removed them the system would speed up tremendously, so every 5 days I'd add and remove traffic shaping rules. Worked flawlessly so it was quite a shock last night.
I will admit, pfsense had lulled me to sleep from working so well. I hadn't had an issue like this in a year and a half.
-
One thing to note from my setup, not sure if this will help or not.
I have a realtek interface built into my MB that I am using for my WAN.
When I experienced the problem I had a CBQ traffic shaping rule enabled on the WAN. Not sure about the details of the rule since I deleted it a few weeks back and don't have a backup config with me to check against.
I no longer have any traffic shaping rules on my WAN and have been problem free for a few weeks now.
-
Yes I will keep this thread posted with my developments as well.
I tred through the nightmare of moving to the rest of my intel ports today - every curveball issue dealt with. 8)
There may be more to this issue than pfsense - comcast business had a hell of a time giving me my settings back after I factory reset their modem to clear the mac out.
What a freaking nightmare!
LOL
I can laugh now that I'm on the other side of that scenario.
-
The plot thickens.
The arp table mac address is one character off from the mac address the business modem is reporting.
We don't have any issues, iperf3 was giving me as fast as my upload to the site allowed - but it is making the gateway monitor throw a shit-fit about packet loss on ipv4 gateway but ipv6 gateway is fine.
I don't know if i've seen any stranger things in my day. ;)
Edit:
Well it was a blown up cable modem from our thunderstorm that worked just enough to be difficult to spot.
It was merely a coincidence I was changing the traffic shaping rules when the site went down.
::)Jot this down as things that won't be missed when we finally get fibre - hopefully in my lifetime.
-
I've just upgraded to 2.3.2 today (first opportunity with no users to upset) and loading my old config caused a hang during firewall initialisation. Incrementally adding my old config to the default (System, DNS, DHCP etc.) was fine until the shaper section.
Certainly appears to be an instability here.
I'll investigate further tomorrow.
Steve
-
Has anyone run into this issue recently or know if there is a ticket to follow up? I just enabled QoS, using the multi-link wizard with CBQ. I have added a couple of custom floating rules for particular services on our network. Shortly after booting, the pfsense seems to lock up. When I reboot, it is only alive again for a few minutes before it locks again. When I removed traffic shaping, the problem disappeared. I'm running 2.3.4-RELEASE-p1. Thanks!
-
It looks like this is a more recent thread on the same issue: https://forum.pfsense.org/index.php?topic=129267.15