Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    /usr/lib/sys/rcu_bj /usr/lib/sys/rcu_udev

    Scheduled Pinned Locked Moved General pfSense Questions
    7 Posts 2 Posters 1.2k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • G
      gessel
      last edited by

      At least one other person (https://www.reddit.com/r/PFSENSE/comments/10osmp6/anyone_know_anything_about_rcu_udev_rcu_libk_or/) and myself appear to have been compromised. PfSense, SSH only via public key, limited users, VPN by key, but something is killing the system - it runs a few minutes and then locks up, console streams no swap errors.

      seems to be calling stuff like:
      /bin/sh /usr/lib/sys/rcu_libk
      /bin/sh /usr/lib/sys/systemd
      /bin/sh /usr/lib/sys/rcu_udev
      /usr/lib/sys/rcu_bj

      the last sorts to max memory one instance using 28.8%.

      Any hints. Is /usr/lib/sys a real directory?

      G 1 Reply Last reply Reply Quote 0
      • G
        gessel @gessel
        last edited by

        ah, there's a crontab installed
        */5 * * * * /bin/sh /usr/lib/sys/systemd

        G 1 Reply Last reply Reply Quote 0
        • G
          gessel @gessel
          last edited by

          Hmm, updated all the packages, locked out all user ssh (perhaps a compromised user key?) and removed the unexpected directory. Anyone familiar with expected processes see anything suspect in this dump of active processes?

          /root: ps -auxw
USER      PID  %CPU %MEM    VSZ    RSS TT  STAT STARTED     TIME COMMAND
root       11 333.8  0.0      0     64  -  RNL  14:04   66:02.95 [idle]
root    33315  45.2  5.3 488208 442820  -  Ss   14:23    0:01.05 /usr/local/bin/snort -R _18213 -D -q --suppress-config-log --daq pcap --daq-mode passive --treat-drop-as-alert -l /var/log/snort/snort_bge018213 --pid-path /var/run --nolock-pidfile --no-interface-pidfile -G 18213 -c /usr/loca
root    25319  45.2  1.2 146500  96016  -  S    14:23    0:04.31 /usr/local/bin/zeek -i bge0 -U .status -p zeekctl -p zeekctl-live -p standalone -p local -p zeek local.zeek zeekctl zeekctl/standalone zeekctl/auto
unbound 77453  19.1  0.6  76648  47584  -  Ss   14:06    1:24.06 /usr/local/sbin/unbound -c /var/unbound/unbound.conf
root     9187  12.0  0.1  35428   8180  -  S    14:24    0:00.72 freebsd
root       12   1.8  0.0      0    336  -  WL   14:04    0:19.91 [intr]
root    26300   1.3  0.0  11504   2628  -  S    14:24    0:00.20 sh -s 16761
root    61156   1.3  0.0  12224   2840  -  Ss   14:06    0:04.42 /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid
root       14   0.6  0.0      0     48  -  DL   14:04    0:10.95 [geom]
root     9272   0.6  0.0  11352   2460  -  Ss   14:06    0:03.82 /usr/sbin/syslogd -O rfc3164 -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/syslog.conf
root    25167   0.6  0.0  12100   2520  -  SC   14:24    0:00.01 /usr/local/libexec/sshg-blocker -w /usr/local/etc/sshguard.whitelist
root    24516   0.3  0.0  11540   2624  -  Ss   14:24    0:00.01 /bin/sh /usr/local/sbin/sshguard -i /var/run/sshguard.pid
root    24831   0.3  0.0  10632   1940  -  S    14:24    0:00.00 /bin/cat
root    24906   0.3  0.1  17432   4180  -  SC   14:24    0:00.00 /usr/local/libexec/sshg-parser
root    25422   0.3  0.0  11540   2624  -  S    14:24    0:00.00 /bin/sh /usr/local/sbin/sshguard -i /var/run/sshguard.pid
root    25622   0.3  0.0  11508   2616  -  S    14:24    0:00.00 /bin/sh /usr/local/libexec/sshg-fw-pf
root        0   0.0  0.0      0    592  -  DLs  14:04    0:00.35 [kernel]
root        1   0.0  0.0   9528    716  -  SLs  14:04    0:00.03 /sbin/init --
root        2   0.0  0.0      0     16  -  DL   14:04    0:00.00 [crypto]
root        3   0.0  0.0      0     16  -  DL   14:04    0:00.00 [crypto returns 0]
root        4   0.0  0.0      0     16  -  DL   14:04    0:00.00 [crypto returns 1]
root        5   0.0  0.0      0     16  -  DL   14:04    0:00.00 [crypto returns 2]
root        6   0.0  0.0      0     16  -  DL   14:04    0:00.00 [crypto returns 3]
root        7   0.0  0.0      0     32  -  DL   14:04    0:00.02 [cam]
root        8   0.0  0.0      0     16  -  DL   14:04    0:00.00 [soaiod1]
root        9   0.0  0.0      0     16  -  DL   14:04    0:00.00 [soaiod2]
root       10   0.0  0.0      0     16  -  DL   14:04    0:00.00 [audit]
root       13   0.0  0.0      0     64  -  DL   14:04    0:00.00 [ng_queue]
root       15   0.0  0.0      0     16  -  DL   14:04    0:00.00 [sequencer 00]
root       16   0.0  0.0      0    240  -  DL   14:04    0:00.05 [usb]
root       17   0.0  0.0      0     16  -  DL   14:04    0:00.00 [soaiod3]
root       18   0.0  0.0      0     16  -  DL   14:04    0:00.00 [soaiod4]
root       19   0.0  0.0      0     16  -  DL   14:04    0:00.00 [sctp_iterator]
root       20   0.0  0.0      0     16  -  DL   14:04    0:00.78 [pf purge]
root       21   0.0  0.0      0     16  -  DL   14:04    0:00.09 [rand_harvestq]
root       22   0.0  0.0      0     48  -  DL   14:04    0:00.25 [pagedaemon]
root       23   0.0  0.0      0     16  -  DL   14:04    0:00.00 [vmdaemon]
root       24   0.0  0.0      0     80  -  DL   14:04    0:00.10 [bufdaemon]
root       25   0.0  0.0      0     16  -  DL   14:04    0:00.01 [vnlru]
root       26   0.0  0.0      0     16  -  DL   14:04    0:00.09 [syncer]
root       27   0.0  0.0      0     16  -  DL   14:04    0:00.00 [ALQ Daemon]
root      364   0.0  0.3 102844  22528  -  Ss   14:05    0:00.05 php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm)
root      365   0.0  0.5 134844  40988  -  S    14:05    0:06.24 php-fpm: pool nginx (php-fpm)
root      366   0.0  0.5 105224  39116  -  I    14:05    0:06.31 php-fpm: pool nginx (php-fpm)
root      387   0.0  0.0  11376   2508  -  INs  14:05    0:00.04 /usr/local/sbin/check_reload_status
root      389   0.0  0.0  11352   2356  -  IN   14:05    0:00.00 check_reload_status: Monitoring daemon of check_reload_status (check_reload_status)
root      556   0.0  0.0  10008   1188  -  Ss   14:05    0:00.06 /sbin/devd -q -f /etc/pfSense-devd.conf
root     4733   0.0  0.5 134572  40484  -  I    14:06    0:05.07 php-fpm: pool nginx (php-fpm)
root     8835   0.0  0.1  17796   7104  -  S    14:20    0:00.01 /usr/local/sbin/lighttpd_pfb -f /var/unbound/pfb_dnsbl_lighty.conf
root     9933   0.0  0.0  10700   1920  -  Is   14:06    0:00.00 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid /usr/local/bin/ping_hosts.sh
root    10045   0.0  0.0  10724   1932  -  I    14:06    0:00.00 minicron: helper /usr/local/bin/ping_hosts.sh  (minicron)
root    10404   0.0  0.0  10700   1924  -  Is   14:06    0:00.00 /usr/local/bin/minicron 300 /var/run/ipsec_keepalive.pid /usr/local/bin/ipsec_keepalive.php
root    10526   0.0  0.0  10720   1932  -  I    14:06    0:00.00 minicron: helper /usr/local/bin/ipsec_keepalive.php  (minicron)
root    10777   0.0  0.0  10700   1924  -  Is   14:06    0:00.00 /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /usr/local/sbin/fcgicli -f /etc/rc.expireaccounts
root    11237   0.0  0.0  10724   1936  -  I    14:06    0:00.00 minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.expireaccounts  (minicron)
root    11423   0.0  0.0  10700   1924  -  Is   14:06    0:00.00 /usr/local/bin/minicron 86400 /var/run/update_alias_url_data.pid /usr/local/sbin/fcgicli -f /etc/rc.update_alias_url_data
root    11481   0.0  0.0  10724   1936  -  I    14:06    0:00.00 minicron: helper /usr/local/sbin/fcgicli -f /etc/rc.update_alias_url_data  (minicron)
root    14587   0.0  0.0  13964   2508  -  Is   14:06    0:00.82 /usr/local/bin/dpinger -S -r 0 -i ATT_Fiber -B 23.114.97.242 -p /var/run/dpinger_ATT_Fiber~23.114.97.242~23.114.97.254.pid -u /var/run/dpinger_ATT_Fiber~23.114.97.242~23.114.97.254.sock -C /etc/rc.gateway_alarm -d 1 -s 500 -l 
root    15185   0.0  0.0  16024   2548  -  Is   14:06    0:01.11 /usr/local/bin/dpinger -S -r 0 -i SonicG -B 23.114.97.242 -p /var/run/dpinger_SonicG~23.114.97.242~173.228.36.129.pid -u /var/run/dpinger_SonicG~23.114.97.242~173.228.36.129.sock -C /etc/rc.gateway_alarm -d 1 -s 500 -l 2000 -t
root    15745   0.0  0.0  16024   2548  -  Is   14:06    0:01.03 /usr/local/bin/dpinger -S -r 0 -i WANGW -B 23.114.97.242 -p /var/run/dpinger_WANGW~23.114.97.242~66.93.181.129.pid -u /var/run/dpinger_WANGW~23.114.97.242~66.93.181.129.sock -C /etc/rc.gateway_alarm -d 1 -s 500 -l 2000 -t 6000
root    22704   0.0  0.0  11292   2372  -  S    14:24    0:00.00 cron: running job (cron)
root    23321   0.0  0.0  11000   2248  -  Ss   14:24    0:00.02 /usr/sbin/newsyslog
root    24241   0.0  0.0  13184   3804  -  I    14:23    0:00.02 /usr/local/bin/bash /usr/local/share/zeekctl/scripts/run-zeek -1 -i bge0 -U .status -p zeekctl -p zeekctl-live -p standalone -p local -p zeek local.zeek zeekctl zeekctl/standalone zeekctl/auto
root    26034   0.0  0.0      0      0  -  Z    14:24    0:00.00 <defunct>
root    30413   0.0  0.1  19104   5888  -  Ss   14:06    0:00.57 /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid
root    31011   0.0  0.1  28352   6984  -  Is   14:06    0:00.01 nginx: master process /usr/local/sbin/nginx -c /var/etc/nginx-webConfigurator.conf (nginx)
root    31244   0.0  0.1  28916   7456  -  S    14:06    0:00.00 nginx: worker process (nginx)
root    31510   0.0  0.1  28916   7456  -  S    14:06    0:00.00 nginx: worker process (nginx)
root    31696   0.0  0.1  28916   7456  -  S    14:06    0:00.00 nginx: worker process (nginx)
root    31828   0.0  0.1  19756   6928  -  Ss   14:06    0:02.24 /usr/local/sbin/openvpn --config /var/etc/openvpn/server1/config.ovpn
root    31992   0.0  0.1  28916   7456  -  S    14:06    0:00.00 nginx: worker process (nginx)
root    32304   0.0  0.1  28916   7456  -  S    14:06    0:00.00 nginx: worker process (nginx)
root    32362   0.0  0.1  30328   8956  -  S    14:06    0:00.70 nginx: worker process (nginx)
root    32627   0.0  0.1  30684   9104  -  S    14:06    0:00.25 nginx: worker process (nginx)
root    32902   0.0  0.1  28916   7472  -  S    14:06    0:00.00 nginx: worker process (nginx)
root    34446   0.0  0.0  10588   1920  -  INC  14:23    0:00.00 sleep 60
dhcpd   39245   0.0  0.1  23540  10904  -  Ss   14:06    0:00.29 /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid bge1
root    42810   0.0  0.1  19512   7104  -  Ss   14:06    0:00.03 /usr/sbin/sshd
root    51937   0.0  0.1  17204   6388  -  Ss   14:06    0:00.03 /usr/local/sbin/openvpn --config /var/etc/openvpn/server2/config.ovpn
root    54459   0.0  0.4  59208  36772  -  S    14:23    0:00.40 /usr/local/bin/php -f /usr/local/pkg/pfblockerng/pfblockerng.inc dnsbl
root    73647   0.0  0.5 104836  38204  -  S    14:22    0:00.49 /usr/local/bin/php-cgi -q /usr/local/bin/notify_monitor.php
root    81549   0.0  0.0  10832   2076  -  Ss   14:06    0:00.65 /usr/local/sbin/dhcpleases -l /var/dhcpd/var/db/dhcpd.leases -d blackrosetech.com -p /var/run/unbound.pid -u /var/unbound/dhcpleases_entries.conf -h /etc/hosts
root    83228   0.0  0.0  11292   2364  -  Ss   14:06    0:00.05 /usr/sbin/cron -s
root    95031   0.0  0.1  19736   8748  -  Ss   14:18    0:00.05 sshd: admin@pts/0 (sshd)
root    96195   0.0  0.5  47248  37472  -  Ss   14:23    0:00.18 /usr/local/sbin/arpwatch -N -z -Z -f /usr/local/arpwatch/arp_bge1.dat -i bge1 -w gessel@blackrosetech.com
root    53251   0.0  0.0  10796   2060 v0  Is+  14:06    0:00.00 /usr/libexec/getty Pc ttyv0
root    74003   0.0  0.0  11592   2392 v0- IN   14:06    0:00.48 /bin/sh /var/db/rrd/updaterrd.sh
root    53258   0.0  0.0  10796   2060 v1  Is+  14:06    0:00.01 /usr/libexec/getty Pc ttyv1
root    53508   0.0  0.0  10796   2060 v2  Is+  14:06    0:00.00 /usr/libexec/getty Pc ttyv2
root    53623   0.0  0.0  10796   2060 v3  Is+  14:06    0:00.00 /usr/libexec/getty Pc ttyv3
root    53931   0.0  0.0  10796   2060 v4  Is+  14:06    0:00.00 /usr/libexec/getty Pc ttyv4
root    54033   0.0  0.0  10796   2060 v5  Is+  14:06    0:00.01 /usr/libexec/getty Pc ttyv5
root    54238   0.0  0.0  10796   2060 v6  Is+  14:06    0:00.00 /usr/libexec/getty Pc ttyv6
root    54569   0.0  0.0  10796   2060 v7  Is+  14:06    0:00.00 /usr/libexec/getty Pc ttyv7
root     9264   0.0  0.0  11616   2684  0  R+   14:24    0:00.00 ps -auxw
root    33201   0.0  0.0  12960   3720  0  S    14:18    0:00.03 /bin/tcsh
root    74721   0.0  0.0  11432   2612  0  Is   14:18    0:00.01 /bin/sh /etc/rc.initial
          1 Reply Last reply Reply Quote 0
          • G
            gessel
            last edited by

            I did not have the presence of mind to capture the scripts, but I have a core dump if anyone is interested.

            1 Reply Last reply Reply Quote 0
            • jimpJ
              jimp Rebel Alliance Developer Netgate
              last edited by

              The freebsd process in that list is also suspect.

              I wouldn't trust that system as-is. You should take a config backup, comb through it looking for anything that doesn't belong (compare to old backups), wipe and reload.

              Something on there had to have been exposed to the attacker. It might have been remote, or it could be a device on your local network somewhere. ssh seems unlikely, but if someone's credentials were compromised on their device, it's possible.

              Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

              Need help fast? Netgate Global Support!

              Do not Chat/PM for help!

              G 1 Reply Last reply Reply Quote 0
              • G
                gessel @jimp
                last edited by

                @jimp I'll do a wipe n reload (actually hardware replace, mine runs on an old IBM x336 dual socket with 10k drives, which was in the olden days of cheap electricity, pretty sweet). diffing the config is good advice, not sure why that didn't occur to me.

                I did do a full search for compromised files on the network and run RK hunter, no hits. The problem hasn't returned. I do apologize for not capturing the contents of the injected files, that was an embarrassing oversight.

                I also added, perhaps redundantly, rules to lock out all admin ports from the LAN side. That's also a bit of a duh step. It is possible that an SSH key was compromised, but as logs are local and not infinite, they'd already been rotated past the infection event.

                So far, peace and operating stability have returned. It's a bit of a blob of text dump, but I have the the ps -auxw and pkg info dumps and am comfortable posting them for posterity if useful or not if a waste of bits.

                1 Reply Last reply Reply Quote 0
                • jimpJ
                  jimp Rebel Alliance Developer Netgate
                  last edited by

                  In addition to a fresh install, I would consider any secret that touched that firewall to be compromised, including the admin password.

                  Make sure you change everything on there. Passwords, VPN keys, anything considered private/secret.

                  See this recipe for ideas:

                  https://docs.netgate.com/pfsense/en/latest/recipes/changing-credentials.html

                  Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                  Need help fast? Netgate Global Support!

                  Do not Chat/PM for help!

                  1 Reply Last reply Reply Quote 0
                  • S SteveITS referenced this topic on
                  • S SteveITS referenced this topic on
                  • First post
                    Last post
                  Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.