23.09 QAT question (just upgraded 6100 from 23.05.1 and /boot/kernel/if_ovpn.ko is no longer listed by vmstat)
-
@sandie There might have been changes...something about user-space. Both my boxes are showing QAT enabled...
-
@NollipfSense Yes, I understand, you are QAT "capable" like me. The question is why this module is NOT loaded anymore in 23.09? Should it be or not? Dashboard info does not change anything:
I am NOT having DCO enabled in ovpnc1 in 23.09. I had no DCO in ovpnc1 in 23.05.1 too, but if_ovpn.ko module was loaded. And... Maybe that was why I observed some IRQs in 23.05.1 and I am no longer observing them in 23.09? Please tell me how to load if_ovpn.ko on boot in 23.09 so I can try to revalidate 6100 behaviour after that and see whether QAT IRQs are coming slowly.
Cheers -
if_ovpn is only used for DCO mode. I would expect to see it loaded if you have DCO enabled on any OpenVPN instance.
-
@stephenw10 Hi Stephen,
Ok, understood. Trust me or not - it was loaded in 23.05.1 even though I did NOT use DCO in ovpnc1 (Site-To-Site link).
No changes on my side. Module is no longer loaded. I do not run OpenVPN server on that 6100 box just client connection.
I will try to enable DCO in ovpnc1 in 6100 shortly, but there is some obvious difference and in my case in 23.09 module is NOT loaded, but WAS loaded in 23.05.1 (even though I never setup DCO on that box!).
Maybe nothing to worry, just my 5 cents. -
I don't think it's anything to worry about. I can't replicate that here though. In 23.05.1 it only loads the module if I enable DCO on an instance.
You can see in the logs where openvpn changes from using tun to using if_ovpn:
Nov 11 23:08:08 kernel tun1: changing name to 'ovpns1' Nov 11 23:09:19 kernel ovpn0: changing name to 'ovpns1' -
@stephenw10 Thank you for following this thread. Do not start/setup ovpns1 please. Test using only ovpnc1 non-DCO please. I can boot 23.05.1 for you with that ovpnc1 non-DCO if you would like to access it remotely to check. But please trust my copy&paste - it is not cheated.
Maybe it is something related to OpenSSL 3.0 which no longer loads that module in non-DCO client connections? (???)
That module was present in 23.05.1 (I know it sounds strange it was loaded when running ovpnc1 non-DCO but it was!). No single config change on my side. -
@sandie said in 23.09 QAT question (just upgraded 6100 from 23.05.1 and /boot/kernel/if_ovpn.ko is no longer listed by vmstat):
The question is why this module is NOT loaded anymore in 23.09? Should it be or not? Dashboard info does not change anything:
Okay, I see what you mean now...maybe why it takes almost 3 minutes to load the QAT driver, but it was doing the same in 23.05.1...
the kernel should be saying: pci/qat?[23.09-RELEASE][admin@NollipfSense.nolli.lan]/root: vmstat -i | grep qat
[23.09-RELEASE][admin@NollipfSense.nolli.lan]/root:
And[23.09-RELEASE][admin@NollipfSense.nolli.lan]/root: kldstat -v Id Refs Address Size Name 1 59 0xffffffff80200000 339f7e8 kernel (/boot/kernel/kernel) Contains modules: Id Name 497 alq 587 mlx4 588 mlx4en 515 if_lo 565 cubic 574 pfsync 573 pflog 575 pf 570 ipsec 571 tcpmd5 556 ng_socket 563 carp 568 ip6_mroute 569 mld 562 igmp 564 ip_mroute 496 shell 495 elf32 494 elf64 463 pseudofs 513 if_gre 567 netdump 510 if_enc 512 if_gif 572 netlink 514 if_lagg 519 if_vlan 508 bridgestp 518 if_tap 517 if_tun 516 if_tuntap 509 if_bridge 536 ng_car 545 ng_ksocket 535 ng_bridge 531 ng_UI 544 ng_iface 534 ng_bpf 543 ng_hole 561 ng_vlan 560 ng_vjc 559 ng_tty 558 ng_tee 557 ng_tcpmss 542 ng_framerelay 541 ng_ether 555 ng_rfc1490 554 ng_pred1 532 ng_async 553 ng_pptpgre 552 ng_pppoe 540 ng_eiface 551 ng_ppp 539 ng_echo 550 ng_pipe 538 ng_deflate 549 ng_one2many 548 ng_mppc 537 ng_cisco 547 ng_lmi 546 ng_l2tp 503 sysvmsg 504 sysvsem 505 sysvshm 506 aio 591 krpc 577 nfslockd 576 nfssvc 460 nfsd 596 xdr 457 nfscommon 462 procfs 461 nullfs 458 nfs 455 devfs 592 ufs 493 cd9660 456 msdosfs 464 tmpfs 459 nfscl 500 acl_posix1e 499 acl_nfs4 8 cam 10 pmp 124 efidev 19 sa 383 uether 452 pci/xl 14 cd 9 aprobe 262 nvd 295 rndtest 378 uhub/rue 142 pci/fxp 120 pci/dc 477 g_part_mbr 476 g_part_gpt 475 g_part_ebr 474 g_part_bsd 18 pass 13 probe 17 da 12 nvme_probe 7 xpt 16 ada 11 nda 15 ch 20 enc 62 ata 79 pci/ata_national 78 pci/ata_micron 77 pci/ata_marvell 76 pci/ata_jmicron 172 pci/iwn 75 pci/ata_ite 74 pci/ata_intel 422 pci/vmx 421 acpi/vmgenc 420 virtio_pci/virtio_console 419 virtio_mmio/virtio_console 418 virtio_pci/virtio_scsi 417 virtio_mmio/virtio_scsi 73 pci/ata_highpoint 416 virtio_pci/virtio_balloon 415 virtio_mmio/virtio_balloon 414 virtio_pci/virtio_blk 413 virtio_mmio/virtio_blk 72 pci/ata_cyrix 71 pci/ata_cypress 412 virtio_pci/vtnet 411 virtio_mmio/vtnet 410 pci/virtio_pci_modern 70 pci/ata_cenatek 409 pci/virtio_pci_legacy 69 pci/ata_ati 679 nexus/apic 678 pci/ioapic 68 pci/ata_amd 407 vge/miibus 406 pci/vge 164 pci/iwm 67 pci/ata_ali 405 uhub/ukbd 676 vgapci/vgapm 675 isa/vga 674 isa/sc 673 pci/vmd 672 scrndr-vga 671 speaker 404 uhub/uhid 66 pci/ata_acard 670 pci/smartpqi 403 uhub/uvscom 669 pci/qla80xx 402 uhub/uvisor 401 uhub/uslcom 666 nfe/miibus 665 pci/nfe 400 uhub/uplcom 399 uhub/umodem 398 uhub/umct 397 uhub/uftdi 396 uhub/ufoma 395 uhub/uark 394 uhub/u3g 393 uhub/zyd 392 uhub/urtw 391 uhub/ural 390 uhub/if_upgt 664 pci/ixl 389 uhub/uath 65 atapci/ata 387 uhub/run 663 pci/ice 386 uhub/rum 662 pci/iavf 64 pci/atapci 661 ax/miibus 660 pci/axp 384 uhub/rsu 659 hostb/agp_via 658 vgapci/agp_i810 657 hostb/agp_amd64 160 pci/iwi 382 miibus/ruephy 381 udav/miibus 380 uhub/udav 379 rue/miibus 377 mos/miibus 376 uhub/mos 375 uhub/kue 374 uhub/cue 373 uhub/cdce 372 axge/miibus 371 uhub/axge 370 axe/miibus 369 uhub/axe 368 aue/miibus 367 uhub/aue 366 uhub/uhub 365 usbus/uhub 159 pci/isp 42 acpi/acpi_smbat 63 isa/ata 364 uhub/umass 363 saf1761otg/usbus 362 dwcotg/usbus 361 octusb/usbus 360 uss820dci/usbus 359 musbotg/usbus 358 xhci/usbus 357 ehci/usbus 656 acpi/fpupnp 356 uhci/usbus 655 root/nexus_acpi 355 ohci/usbus 32 acpi/apei 155 pci/ipw 354 pci/xhci 353 pci/uhci 352 pci/ohci 351 pci/ehci 154 pci/ips 350 puc/uart 349 pci/uart 348 isa/uart 347 acpi/uart 153 ips/ipsd 654 isa/sysresource 653 nexus/ram 652 root/nexus 41 acpi/acpi_syscontainer 152 pci/igc 151 iichb/iicbus 346 pci/tws 150 iicbb/iicbus 345 pci/ti 344 tcp_log_dev 61 ale/miibus 651 legacy/cpu 650 nexus/legacy 149 pci/ida 343 scterm-scteken 342 scterm-sc 341 scterm-dumb 340 pci/sym 339 stge/miibus 338 pci/stge 337 ste/miibus 336 pci/ste 335 midi 148 ida/idad 334 sound 147 pci/hptiop 649 qpi/pcib 648 nexus/qpi 647 isa/pcibus_pnp 646 legacy/pcib 645 isa/orm 644 acpi/atdma 643 isa/atdma 60 pci/ale 642 xenpv/isa 641 legacy/isa 640 acpi/attimer 639 isa/attimer 146 pci/gem 638 acpi/atrtc 637 isa/atrtc 145 gem/miibus 144 miibus/inphy 143 fxp/miibus 333 hdac/snd_hda 141 firewire/sbp 332 pci/snd_hda 40 acpi/acpi_sysresource 636 acpi/dmar 331 hdaa/snd_hda_pcm 330 hdacc/snd_hda 635 cpu/powernow 634 cpu/p4tcc 633 cpu/hwpstate_intel 632 cpu/hwpstate 631 cpu/est 140 pci/fwohci 329 pci/snd_via8233 328 pci/snd_ich 327 pci/snd_es137x 326 emu10kx/snd_emu10kx_midi 630 nexus/kvm_clock 629 pci/isci 325 emu10kx/snd_emu10kx_pcm 324 pci/snd_emu10kx 323 csa/snd_csapcm 628 io 627 acpi/vmbus_res 626 vmbus/hv_et 625 acpi_syscontainer/vmbus 624 pcib/vmbus 322 pci/snd_csa 321 pci/snd_cmi 623 vmbus/hv_timesync 622 vmbus/hv_shutdown 621 vmbus/hv_heartbeat 620 vmbus/hv_vss 619 vmbus/hv_kvp 320 nexus/smbios 618 vmbus/storvsc 319 sk/miibus 617 vmbus/hn 616 vmbus/vmbus_pcib 615 vmbus/hv_kbd 614 vmbus/hvsock 318 skc/sk 613 isa/fdc 612 acpi/fdc 611 fdc/fd 317 pci/skc 23 aac/aacp 610 pci/bxe 316 sis/miibus 609 acpi/psmcpnp 608 isa/psmcpnp 607 atkbdc/psm 315 pci/sis 606 acpi/atkbdc 605 isa/atkbdc 604 atkbdc/atkbd 603 pci/arcmsr 602 acpi/acpi_timer 314 siis/siisch 601 pci/acpi_pcib 600 acpi/acpi_pcib 313 pci/siis 599 acpi/acpi_pci_link 139 fwohci/firewire 312 sge/miibus 598 pcib/acpi_pci 311 pci/sge 597 acpi/hpet 310 sdhci_pci/mmc 309 pci/sdhci_pci 595 xenstore/xenbusb_back 594 xenstore/xenbusb_front 308 sdhci_acpi/mmc 307 acpi/sdhci_acpi 59 alc/miibus 58 pci/alc 138 udl/fbd 137 drmn/fbd 136 fb/fbd 135 exca 134 uinput 31 acpi/acpi_acad 133 e6000sw/miibus 297 uhub/rtwn_usb 296 pci/rtwn_pci 132 e6000sw/etherswitch 294 rl/miibus 293 cardbus/rl 292 pci/rl 291 re/miibus 290 pci/re 498 cpu/cpufreq 131 mdio/e6000sw 26 pci/aacch 57 ahc 56 ahd 39 cpu/acpi_perf 38 acpi/acpi_lid 130 et/miibus 129 pci/et 30 nexus/acpi 128 pci/igb 127 pci/em 126 nexus/efirtc 37 acpi/acpi_isab 123 miibus/pnphy 284 pci/ral 122 miibus/dcphy 283 pci/puc 121 dc/miibus 282 puc/ppc 281 pci/ppc 280 isa/ppc 279 acpi/ppc 278 ppbus/ppi 277 ppc/ppbus 36 acpi/acpi_ec 276 ppbus/lpt 275 pci/vgapci 25 pci/aac 274 pci/pcib 55 pci/ahd 492 eisab/isa 491 isab/isa 116 cc/vcc 115 cxl/vcxl 114 cxgbe/vcxgbe 273 pcib/pci 272 pci/isab 271 pci/ignore_pci 270 pci/hostb 269 pci/fixup_pci 268 pci/cbb 113 t6nex/cc 267 pci/ocs_fc 112 t5nex/cxl 266 pci/oce 111 t4nex/cxgbe 110 pci/t6nex 265 pci/nvme 264 ahci/nvme 109 pci/t5nex 108 pci/t4nex 54 pci/ahc_pci 53 isa/ahc_isa 261 null 260 nge/miibus 259 pci/nge 52 atapci/ahci 258 netmap 257 nexus/netgate 51 pci/ahci 35 acpi/cpu 50 ahci/ahciem 49 ahci/ahcich 252 pci/mxge 29 acpi/acpi_4200 250 pci/mwl 34 acpi/acpi_cmbat 249 pci/mvs 248 sata/mvsch 247 mvs/mvsch 107 pci/t6iov 246 msk/miibus 245 mskc/msk 244 pci/mskc 106 pci/t5iov 105 pci/t4iov 104 cxgbc/cxgb 243 pci/mrsas 103 pci/cxgbc 102 cpu/ichss 101 pci/ciss 48 age/miibus 240 pci/mpt 47 pci/age 28 pci/aacraid 24 aac/aacd 33 acpi/acpi_button 100 cas/miibus 237 pci/mps 99 pci/cas 98 cbb/cardbus 97 pci/bwi 236 pci/mpr 96 pci/bnxt 235 mmc/mmcsd 234 pci/mlx 233 mlx/mlxd 232 miibus/xmphy 590 nexus/cryptosoft 231 miibus/vscphy 230 miibus/ukphy 229 miibus/truephy 228 miibus/tdkphy 227 miibus/smscphy 226 miibus/smcphy 225 miibus/rlphy 224 miibus/rgephy 223 miibus/rdcphy 222 miibus/qsphy 221 miibus/pnaphy 220 miibus/nsphyter 219 miibus/nsphy 218 miibus/nsgphy 217 miibus/mv88e151x 216 miibus/mcommphy 215 miibus/lxtphy 214 miibus/jmphy 213 miibus/ip1000phy 589 pci/lkpi_mlx5_core_pci_table 212 miibus/icsphy 211 miibus/gentbi 210 miibus/e1000phy 209 miibus/dp83867phy 208 miibus/dp83822phy 207 miibus/ciphy 206 miibus/brgphy 586 pci/lkpi_mlx4_pci_table 205 miibus/bmtphy 204 miibus/axphy 203 miibus/atphy 202 miibus/amphy 201 miibus/acphy 584 uhub/usb_linux 200 mfi/mfisyspd 199 mfi/mfid 198 pci/mfi 95 bnxt_mgmt 197 mem 46 ae/miibus 94 bge/miibus 93 pci/bge 195 pci/malo 454 miibus/xlphy 453 xl/miibus 583 lkpi_iicbb/iicbb 582 drm/lkpi_iicbb 581 drmn/lkpi_iicbb 580 lkpi_iic/iicbus 579 drm/lkpi_iic 578 drmn/lkpi_iic 451 xenpv/xendebug 450 xenpv/gntdev 92 bfe/miibus 194 lge/miibus 193 pci/lge 449 xenpv/privcmd 448 xenpv/evtchn 91 pci/bfe 447 xenpv/xsd_dev 446 xenstore/xs_dev 445 xenpv/xenstore 192 pci/le 444 pci/xenpci 443 xenpv/xentimer 442 xenbusb_front/xe 191 kbdmux 90 bce/miibus 441 xenbusb_back/xnb 190 jme/miibus 189 pci/jme 89 pci/bce 438 acpi/xen_cpu 45 pci/ae 88 ath_hal 437 xenstore/xctrl 436 xenpv/xc 435 nexus/xenpv 44 cpu/acpi_throttle 188 pci/ixv 434 xenbusb_back/xbbd 27 aacraid/aacraidp 433 xenbusb_front/xbd 187 ix/mdio 432 xenstore/xenballoon 186 pci/ix 87 pci/if_ath_pci 430 pci/wpi 429 watchdog 428 vte/miibus 427 pci/vte 43 acpi/acpi_tz 86 pci/ata_via 85 pci/ata_sis 84 pci/ata_sii 83 pci/ata_serverworks 82 pci/ata_promise 81 pci/ata_nvidia 80 pci/ata_netcell 426 nexus/vtvga 425 vr/miibus 424 pci/vr 423 pci/pvscsi 478 g_raid 241 mpt_raid 483 g_raid_md_promise 465 g_bde 482 g_raid_md_nvidia 125 efirt 481 g_raid_md_jmicron 468 g_disk 490 g_uzip 480 g_raid_md_intel 467 g_dev 479 g_raid_md_ddf 466 g_eli 239 mpt_cam 473 g_part 472 g_mirror 439 pvefirt 471 g_label 470 g_vfs 196 g_md 242 mpt_user 484 g_raid_md_sii 593 g_class 469 g_flashmap 169 iwm7265Dfw_fw 168 iwm7265fw_fw 167 iwm7260fw_fw 166 iwm3168fw_fw 289 random_device 165 iwm3160fw_fw 117 t4fw_cfg_fw 288 rt2860_fw 287 rt2661_fw 286 rt2561s_fw 285 rt2561_fw 502 firmware 408 virtio 489 g_raid_tr_raid5 488 g_raid_tr_raid1e 530 wlan_xauth 487 g_raid_tr_raid1 529 wlan_sta 528 wlan_rssadapt 527 wlan_ratectl_none 486 g_raid_tr_raid0 388 runfw_fw 485 g_raid_tr_concat 501 rootbus 385 rsu_rtl8712fw_fw 22 rc4 119 t6fw_cfg_fw 526 wlan 118 t5fw_cfg_fw 185 iwn6050fw_fw 184 iwn6000g2bfw_fw 183 iwn6000g2afw_fw 263 nvme 182 iwn6000fw_fw 181 iwn5150fw_fw 525 wlan_wep 524 wlan_tkip 523 wlan_ccmp 180 iwn5000fw_fw 522 wlan_amrr 521 wlan_wlan_acl 256 mxge_rss_ethp_z8e_fw 255 mxge_rss_eth_z8e_fw 254 mxge_ethp_z8e_fw 507 zlib 253 mxge_eth_z8e_fw 251 mwl_fw 179 iwn4965fw_fw 178 iwn2030fw_fw 177 iwn2000fw_fw 176 iwn135fw_fw 175 iwn105fw_fw 174 iwn100fw_fw 440 xenpv/granttable 173 iwn1000fw_fw 171 iwm8265fw_fw 306 rtwn_rtl8821aufw_fw 305 rtwn_rtl8812aufw_fw 304 rtwn_rtl8192eufw_fw 303 rtwn_rtl8192cfwU_fw 302 rtwn_rtl8192cfwT_fw 301 rtwn_rtl8192cfwE_B_fw 431 wpi_fw 300 rtwn_rtl8192cfwE_fw 299 rtwn_rtl8188eufw_fw 298 rtwn_rtl8188eefw_fw 238 mpt_core 170 iwm8000Cfw_fw 533 netgraph 585 if_infiniband 21 xz 511 ether 520 iflib 667 rdrand 668 nehemiah 677 x86bios 566 ertt 2 1 0xffffffff835a1000 1e2b0 opensolaris.ko (/boot/kernel/opensolaris.ko) Contains modules: Id Name 1 opensolaris 3 1 0xffffffff835c0000 76f8 cryptodev.ko (/boot/kernel/cryptodev.ko) Contains modules: Id Name 2 cryptodev 4 1 0xffffffff835c8000 5d6ea0 zfs.ko (/boot/kernel/zfs.ko) Contains modules: Id Name 3 zfsctrl 5 zfs 6 zfs_zvol 4 zfs_vdev 5 1 0xffffffff84420000 2220 cpuctl.ko (/boot/kernel/cpuctl.ko) Contains modules: Id Name 680 cpuctl 6 1 0xffffffff84423000 3240 ichsmb.ko (/boot/kernel/ichsmb.ko) Contains modules: Id Name 682 pci/ichsmb 681 ichsmb/smbus 7 1 0xffffffff84427000 2178 smbus.ko (/boot/kernel/smbus.ko) Contains modules: Id Name 8 1 0xffffffff8442a000 4378 qat.ko (/boot/kernel/qat.ko) Contains modules: Id Name 693 nexus/qat 9 6 0xffffffff8442f000 14d60 qat_hw.ko (/boot/kernel/qat_hw.ko) Contains modules: Id Name 691 pci/qat_c4xxx 686 pci/qat_200xx 690 pci/qat_dh895xcc 687 pci/qat_4xxx 689 pci/qat_c3xxx 685 pci/qat_c62x 688 pci/qat_4xxxvf 10 9 0xffffffff84444000 2ff70 qat_common.ko (/boot/kernel/qat_common.ko) Contains modules: Id Name 683 qat_common 11 8 0xffffffff84474000 68cd8 qat_api.ko (/boot/kernel/qat_api.ko) Contains modules: Id Name 684 qat_api 12 1 0xffffffff844dd000 161f38 qat_dh895xcc_fw.ko (/boot/kernel/qat_dh895xcc_fw.ko) Contains modules: Id Name 692 qat_dh895xcc_fw_fw 13 1 0xffffffff8463f000 11240 qat_c2xxx.ko (/boot/kernel/qat_c2xxx.ko) Contains modules: Id Name 694 pci/qat_c2xxx 14 1 0xffffffff84651000 20e8 coretemp.ko (/boot/kernel/coretemp.ko) Contains modules: Id Name 695 cpu/coretemp [23.09-RELEASE][admin@NollipfSense.nolli.lan]/root: -
@NollipfSense I referred to if_ovpn.ko which somehow was loaded in my setup in 23.05.1 (non-DCO ovpnc1 running), when it is not loaded anymore for same OpenVPN 23.09 setup.
QAT IRQs are kept =0 on 23.09 - I wanted to understand if that was intended difference and that module is obsolete for non-DCO client connections. If OpenVPN or anything else benefited from it being loaded I would prefer to keep it loading in 23.09 too :)BTW. 23.09 delivers OpenVPN 2.6.5 which is affected by crappy and fresh CVE-2023-46849 (DoS) and CVE-2023-46850. Are you guys going to patch these problems soon? You always fix problems in reasonable time so I am generally sleeping well.
23.05.1 is affected too as it runs OpenVPN 2.6.2. -
@sandie I am trying understand how QAT is implemented in pfSense +...according to here: https://ieeexplore.ieee.org/document/8842961
"The hardware accelerators work by offloading the computing from CPU. In this work, we first analyze the hardware encryption models with OpenSSL and Intel
QuickAssist Technology (QAT) accelerators implemented in user space and kernel space."
So, is it for both user and kernel like this below pic as taken from here: https://networkbuilders.intel.com/docs/networkbuilders/intel-quickassist-technology-nginx-performance-white-paper-1675662546.pdf
-
@sandie said in 23.09 QAT question (just upgraded 6100 from 23.05.1 and /boot/kernel/if_ovpn.ko is no longer listed by vmstat):
BTW. 23.09 delivers OpenVPN 2.6.5 which is affected by crappy and fresh CVE-2023-46849 (DoS) and CVE-2023-46850. Are you guys going to patch these problems soon?
Urgh. Doesn't look like 2.6.7 is in FreeBSD pkgs yet so it will be at least that long.
-
-
Yes, but it looks like it may have a significant bug.
