After Update to 23.09 Performance and stabillity issues
-
Im having some performance problems after the Upgrade to 23.09 and OpenVPN with Nordvpn. When under heavy load the VPN connection gets unstable and is not responding to websites etc. until it goes back to low load.
The Gateway shows some packetloss and the log says that there is latency problems. When directly connecting to WAN it works and there are no connection losses, i allready tried different Nordvpn Servers but its the same behavior. With 23.05 there wherent any problems like this, it started with 23.09. Overall tunnel performance went down to ~80% of line capacity, before with 23.05 it was at maximum.
I did not change anything in config, its the same as before the update.
Anyone any idea whats going on here ? -
D DaddyGo referenced this topic on
-
@sig1980 said in After Update to 23.09 Performance and stabillity issues:
Im having some performance problems after the Upgrade to 23.09 and OpenVPN
Hi,
@sig1980 in what HW environment did this happen?
- rebooted the firewall properly?
- do you constantly see these "numbers"?
-
Hi @DaddyGo,
its a Intel x86 NUC with a N3150 CPU and 2 Realtek cards, its kind of my "test" System.
I had trouble with the Update process, it hang at 1st reboot and after 20min waiting i had to power cycle it, but i did not gave that much thought because it happend before with older versions.
As soon there is heavier load it happens, i can reproduce it with a fast download. -
@sig1980 said in After Update to 23.09 Performance and stabillity issues:
and 2 Realtek cards,
Hi,
Unfortunately, it seems to me that no parallel can be drawn between the two cases, although I had hoped YES
I don't know exactly what Realtek chip your NIC is using, but here on the forum we never recommended Realtek stuff for firewalls...With that said, it is no longer exact for comparison although there are undoubtedly some identical momentums.
Somehow I felt that the restart (yours) did not go without problems, it can always mean some minor or major problem.Unfortunately, since we are talking about NUC, you can't switch to a correct NIC chipset (Intel!!!) and I think this Realtek thing will always get in your way.
have you tried going back to the previous version and if so what does it show?
-
Hi,
i was aware that Realtek NIC´s are not the best choice for pfsense but i didn´t had any problems with this NUC when using it as my main Router for years. When i find time next few days i will try to downgrade to an earlier version and report back.
-
@sig1980 said in After Update to 23.09 Performance and stabillity issues:
i will try to downgrade to an earlier version and report back.
I would thank you for this
as I cannot downgrade the our NGFW in question as it is a semi-production environment... -
there is something wrong with 23.09, for downgrading i had to do it the "hard" way since i was still on UFS. I installed 2.70CE and restored a config i had saved, no latency and speed issues. Then i upgraded to 23.05.1 and everything was still fine, then i upgraded to 23.09 and this time the reboot did not get stuck, and performance problems are back. Im now back at 23.05.1 thanks to ZFS Boot environment which is a great feature.
-
@sig1980 said in After Update to 23.09 Performance and stabillity issues:
upgraded to 23.09 and this time the reboot did not get stuck, and performance problems are back.
I suspected it would be like this.... :-) (23.05.1 = OK)
today I had time to check the related settings, NIC drivers, system tuns., etc.
- as I don't know what the 14.0-CURRENT ones might bring "to in", - but they are all fine...
and then I remembered this is one of the major updates that just came in and it's OpenSSL itself
(I cannot think of anything else that has affected the speed so drastically)now I'm looking for possible HW correlations with crypto accelerations, which nearly halved the speed of OVPN tunnels
@stephenw10 "Hi Steve - any ideas, have you come across this question elsewhere?
+++edit:
I'm always one of the last to update, I don't know why I did it now....
(that was very amateurish
) -
@sig1980 said in After Update to 23.09 Performance and stabillity issues:
there is something wrong with 23.09,
Hi @sig1980
Clearly 23.09 brings "in" this VPN performance problem, I just downgraded it with the "ZFS Boot Environments" feature, back to 23.05-1...
(this does not mean that the third party HW + this 23.09 release + OpenSSL "up" mix is causing this performance drop)- it's another case that this feature ("ZFS Boot Environments") doesn't work properly either, I have not had to use it so far, but I thought it was a stable feature - is NOT
- the NGFW GUI did not start properly after I reverted back to 23.05.01, so I had to go back to 23.09 with Supermicro IPMI / HTML5...
- the CLI (shell) was fine and firewall worked fine, even with PHP error, so I was able to measure the VPN speed under 23.05.1 and again it was fine
back to here 23.05.01 - VPN performance...
then restored 23.09 - I had to come back because as I wrote there was no GUI
BTW:
PHP error using "ZFS Boot Environments", if anyone has seen this before -
(the GUI is not responding to anything, but the firewall is working, I haven't looked into what might be causing this)
Has anyone tried in this chaotic system (Netgate - new licensing policy) to downgrade from 23.09 to CE2.7.1, since the example shows I can't get back to 23.05.01 and from there to CE2.7?
+++edit: @stephenw10 I'd still be happy if you could add something to this, either for VPN performance or ZFS Boot Environments with PHP error, or even a revert to CE2.7, thanks.
-
Hi,
there was a new Version released 23.09.1 which fixed this for me. As it looks like there was like you already suspected something wrong with AES-NI Acceleration.
-
@sig1980 said in After Update to 23.09 Performance and stabillity issues:
something wrong with AES-NI Acceleration
Hi,
Thanks for the info, just upgraded and it is indeed better...
I still think it's less, ..... my usual speed (600-650), but this may be time interval dependent, I'll measure at other times.
CDN77 gives the transit network to us, and it's darn well loadable, with no typical fluctuations.
What I find funny is that this hasn't been a problem for anyone but us?
There was a dead silence on this
PS:
long ago here on the forum, if there was such a VPN performance problem, - the thread would have spun up...
Thanks again for pointing this out to me (23.09.01), now I'm about to revert to CE everywhere in our deployments, but I'm already testing OPNsense as well.