Java log4j vulnerability - Is pfSense affected ?

johnpoz

As a PSA sort of thing - if your taking the time to look to see if you have any susceptible to this issue. It also a good time to just review your overall security stance.

Do you have stuff open to the internet, could it be just turned off? Could it be more restrictive in any way. Limit source IPs? To either known good clients IP or IP ranges, or limit to geoIP based. Are all your clients only in specific geoip area, if so limit to only that region (pfblocker is great for this sort of thing).

Could you use a VPN to access these resources while remote vs just port forwards from the internet.

On your local network segments, does your guest wifi network for example need access to the pfsense web gui? Do any of your local networks need access to your other networks? If they only need access to specific thing in another vlan, make sure your rules only allow that specific thing vs any, etc.

If your just all on 1 flat lan network, is it time you thought about segmenting your different sort of devices to their own vlans, say iot devices, stuff that is available via the public, etc.

Never a bad idea to do an overall review.. Maybe you were allowing something from vlan x to vlan y.. Is that something still in use? If not remove the rule that allowed that, etc.

Are all your things, fully patched, on the most current version.. If your not running current of pfsense for example.. Good time as any to make sure your on current 2.5.2, 21.05.2 etc..

nimrod

@stephenw10 said in Java log4j vulnerability - Is pfSense affected ?:

@honest_matt said in Java log4j vulnerability - Is pfSense affected ?:

CVE-2021-44228

The only thing it's listed against in FreeBSD is Graylog:
http://vuxml.freebsd.org/freebsd/3fadd7e4-f8fb-45a0-a218-8fd6423c338f.html

pfSense does not ship with graylog. It's not available as a package.

log4j is not even in our repo as Johnpoz showed so you'd need to try very hard to install it.

Steve

Yup. Just found that same info. Thanks for the confirmation Steve.

@johnpoz

I only use pfBlocker, Snort, and Squid. I have two PC`s connected to pfSense box. One machine is running FreeBSD, the other one is running Linux. No such packages are installed on either of those machines. As far as im concerned, this thread can be marked as solved.

Thank you everyone.

qctech

This is maybe an opportunity to educate users a little as well.

I have 20+ years experience in this game but am happy to admit that I don't really know how the log4j library is included in projects.

I am quite capable of logging onto a pfsense box or linux server and checking what packages have been installed by the package manager but I don't know, for example, if log4j could be included as part of another package in such a way as it does not show separately (compiled right in for example).

Confirmation from @stephenw10 and @johnpoz (thanks guys) that the underlying OS is not showing any known problems (short of a package that isn't actually available for pfSense) is great.

Maybe this isn't the right place to discuss how log4j might be included in other packages.
Maybe we can never be 100% sure anyway.
Maybe someone knows how it's normally used and could comment?

johnpoz

@qctech said in Java log4j vulnerability - Is pfSense affected ?:

log4j could be included as part of another package in such a way as it does not show separately (compiled right in for example).

While not saying something like that is not possible.. I have never noticed such a thing. If some package wants or needs a library to function, they normally just have the library package as a dependency that is installed along with their package.

It would be bad practice to do such a thing, where you hard code some library into your package. And could cause issues if the the package that includes such library functions was installed along with package that has the library included with it. Say some other package that uses the library package normally as a dependent package.

That for sure would make it harder to find and patch issues with library or other dependencies.

bmeeks

@qctech said in Java log4j vulnerability - Is pfSense affected ?:

This is maybe an opportunity to educate users a little as well.

I have 20+ years experience in this game but am happy to admit that I don't really know how the log4j library is included in projects.

I am quite capable of logging onto a pfsense box or linux server and checking what packages have been installed by the package manager but I don't know, for example, if log4j could be included as part of another package in such a way as it does not show separately (compiled right in for example).

Confirmation from @stephenw10 and @johnpoz (thanks guys) that the underlying OS is not showing any known problems (short of a package that isn't actually available for pfSense) is great.

Maybe this isn't the right place to discuss how log4j might be included in other packages.
Maybe we can never be 100% sure anyway.
Maybe someone knows how it's normally used and could comment?

I will try to explain (and rant a little bit as well) --

When you install a package on pfSense or any other system (FreeBSD, Linux, Windows, etc.), it is highly likely that dependent libraries will get pulled in and automatically installed. The Snort and Suricata packages are two great examples that I am familiar with since I maintain both of them and created one of them as well.

Snort formerly had the capability of exporting log data to a sub-system called Barnyard2. Barnyard2 pulled in Snort binary logging data and stored it into a database for easy searching and reporting. The default database used by Barnyard2 was MySQL. So when Snort was compiled, and the Barnyard2 feature was enabled (as it was on pfSense), a MySQL database connectivity client was automatically compiled and installed alongside of Snort and Barnyard2. So you got the Snort binary, the Barnyard2 binary, and a shared system library that allowed Barnyard2 to connect to and send data to a MySQL database engine running on a separate box. But over the years Barnyard2 ceased to be maintained, and that shared MySQL database connector library did not get updated. Turns out there were some security vulnerabilities uncovered in that library later on, and since Barnyard2 was no longer maintained, there was nobody to take charge and update the database client with a patched version. So I made the move some time back to drop Barnyard2 support and thus remove that vulnerable library.

The upstream Suricata team elected to start coding all new protocol decoders/detectors in Rust. So now, when you compile Suricata, it will drag in the full Rust language library. And because Rust itself is just a very basic programming language, it offers pretty much no built-in features for mundane things like math, string handling, file I/O and so forth. As a developer you have to get that by linking to and compiling in utilities created by others (meaning not the Rust core developers). These necessary plumbing parts you must have to do useful things are called Crates in Rust land. All of this (Rust and the Crates) is getting updated very, very frequently. And it makes me quite nervous. But I can't do anything about it because upstream decided Rust was the new magic pixie dust ... .

At least on FreeBSD, these dependencies are managed by the pkg utility. It determines what version of a given shared library an application being installed needs, then it goes and downloads and installs the needed library. Note that it is highly likely that the dependent libraries have, in turn, their own required runtime dependencies. That's why you will sometimes notice, if you pay close attention, that 50 or 60 things might get installed or updated when you install a single package you want. This is why us old guys caution all the newbies that want to install all kinds of third-party packages on their firewall. All those dragged-in dependencies can break something. And they also create a much larger potential attack surface.

Because pkg manages what gets installed and updated on pfSense, you can always run this command to see what is actually installed:

pkg info

Now for the rant part ---

You may have guessed by now that I am an older retired computer coding guy. And in fact, I could possibly be properly called a "curmudgeon". I've been around long enough to remember when Java was created and hailed as the greatest thing since sliced bread. We know now that was a bit of an overstatement. Java was akin to Rust in that you had the core language, but then you used Beans to do useful work. After Java, the next saviour to mankind in the programming world was Ruby and its Gems. But alas, that really didn't work out all that great so along came GO, and now there's Rust, too. Then there are all the scripting thingies such as Lua, Rexx, and the list goes on. Each of these programming tools has its collection of fan boys who fight endlessly over which is best.

I'm not a big fan of this approach of jumping on the next new hula-hoop thing. But new developers love to take the easy road, so they will use the latest "fad" language and then just import tools willy-nilly to do tasks like logging, or connecting to a database, drawing fancy charts and graphs, etc. The problem is you really have no good idea what kinds of security vulnerabilities you are opening up when doing that. That's exactly how something like the log4j2 tool got used all over the place - dragging a critical security flaw along with it.

There was actually a logging package I looked at for Suricata a few years ago that used Java! Soon as I found that out, I dropped it like a hot potato. Who in their right mind would put anything Java on a firewall? What thought process led that utility creator to write a Java-based logging package for firewall security software? If you are going to do that, why don't you just go ahead and include Adobe Flash as well ... .

tquade

@nimrod Only log4j2 is vulnerable. The older log4j (no "2") is not vulnerable. But people often say "log4j" to include either version. This is according to Johannes Ullrich, SANS Institute.

Ted Quade

bmeeks

@tquade said in Java log4j vulnerability - Is pfSense affected ?:

@nimrod Only log4j2 is vulnerable. The older log4j (no "2") is not vulnerable. But people often say "log4j" to include either version. This is according to Johannes Ullrich, SANS Institute.

Ted Quade

My bad -- fixed it in my post. Thanks for the hint.

nimrod

@bmeeks said in Java log4j vulnerability - Is pfSense affected ?:

@qctech said in Java log4j vulnerability - Is pfSense affected ?:

This is maybe an opportunity to educate users a little as well.

I have 20+ years experience in this game but am happy to admit that I don't really know how the log4j library is included in projects.

I am quite capable of logging onto a pfsense box or linux server and checking what packages have been installed by the package manager but I don't know, for example, if log4j could be included as part of another package in such a way as it does not show separately (compiled right in for example).

Confirmation from @stephenw10 and @johnpoz (thanks guys) that the underlying OS is not showing any known problems (short of a package that isn't actually available for pfSense) is great.

Maybe this isn't the right place to discuss how log4j might be included in other packages.
Maybe we can never be 100% sure anyway.
Maybe someone knows how it's normally used and could comment?

I will try to explain (and rant a little bit as well) --

When you install a package on pfSense or any other system (FreeBSD, Linux, Windows, etc.), it is highly likely that dependent libraries will get pulled in and automatically installed. The Snort and Suricata packages are two great examples that I am familiar with since I maintain both of them and created one of them as well.

Snort formerly had the capability of exporting log data to a sub-system called Barnyard2. Barnyard2 pulled in Snort binary logging data and stored it into a database for easy searching and reporting. The default database used by Barnyard2 was MySQL. So when Snort was compiled, and the Barnyard2 feature was enabled (as it was on pfSense), a MySQL database connectivity client was automatically compiled and installed alongside of Snort and Barnyard2. So you got the Snort binary, the Barnyard2 binary, and a shared system library that allowed Barnyard2 to connect to and send data to a MySQL database engine running on a separate box. But over the years Barnyard2 ceased to be maintained, and that shared MySQL database connector library did not get updated. Turns out there were some security vulnerabilities uncovered in that library later on, and since Barnyard2 was no longer maintained, there was nobody to take charge and update the database client with a patched version. So I made the move some time back to drop Barnyard2 support and thus remove that vulnerable library.

The upstream Suricata team elected to start coding all new protocol decoders/detectors in Rust. So now, when you install Suricata, it will drag in the full Rust language library. And because Rust itself is just a very basic programming language, it offers pretty much no built-in features for mundane things like math, string handling, file I/O and so forth. As a developer you have to get that by linking to and compiling in utilities created by others (meaning not the Rust core developers). These necessary plumbing parts you must have to do useful things are called Crates in Rust land. All of this (Rust and the Crates) is getting updated very, very frequently. And it makes me quite nervous. But I can't do anything about it because upstream decided Rust was the new magic pixie dust ... .

At least on FreeBSD, these dependencies are managed by the pkg utility. It determines what version of a given shared library an application being installed needs, then it goes and downloads and installs the needed library. Note that it is highly likely that the dependent libraries have, in turn, their own required runtime dependencies. That's why you will sometimes notice, if you pay close attention, that 50 or 60 things might get installed or updated when you install a single package you want. This is why us old guys caution all the newbies that want to install all kinds of third-party packages on their firewall. All those dragged-in dependencies can break something. And they also create a much larger potential attack surface.

Because pkg manages what gets installed and updated on pfSense, you can always run this command to see what is actually installed:
pkg info
Now for the rant part ---

You may have guessed by now that I am an older retired computer coding guy. And in fact, I could possibly be properly called a "curmudgeon". I've been around long enough to remember when Java was created and hailed as the greatest thing since sliced bread. We know now that was a bit of an overstatement. Java was akin to Rust in that you had the core language, but then you used Beans to do useful work. After Java, the next saviour to mankind in the programming world was Ruby and its Gems. But alas, that really didn't work out all that great so along came GO, and now there's Rust, too. Then there are all the scripting thingies such as Lua, Rexx, and the list goes on. Each of these programming tools has its collection of fan boys who fight endlessly over which is best.

I'm not a big fan of this approach of jumping on the next new hula-hoop thing. But new developers love to take the easy road, so they will use the latest "fad" language and then just import tools willy-nilly to do tasks like logging, or connecting to a database, drawing fancy charts and graphs, etc. The problem is you really have no good idea what kinds of security vulnerabilities you are opening up when doing that. That's exactly how something like the log4j2 tool got used all over the place - dragging a critical security flaw along with it.

There was actually a logging package I looked at for Suricata a few years ago that used Java! Soon as I found that out, I dropped it like a hot potato. Who in their right mind would put anything Java on a firewall? What thought process led that utility creator to write a Java-based logging package for firewall security software? If you are going to do that, why don't you just go ahead and include Adobe Flash as well ... .

I am experienced Arch Linux user, and i know very well what it means to keep things simple and clean. This is why i always install and use only what is needed. The thing with Linux distros and their package managers is that sometimes they mark certain packages as system essential packages and they will not show up when you try to list them with package manager. Probably due to security reasons and to avoid accidental removal.
However, these "hidden" packages do show up once you start full system update, and package managers clearly shows that they need to be updated. Im not use how it works on FreeBSD, but on Arch Linux every single package can be listed with:

sudo pacman -Q >list.txt

And every single package with all its dependencies can be forcefully removed by using:

sudo pacman -Rnsdd <package_name>

Of course, i dont need to point out that this command can be extremely destructiive if you do not know what you are doing.

Its good that there is nothing Java related in pfSense and its packages. All the info you shared here is useful, thank you sir.

NollipfSense

@bmeeks said in Java log4j vulnerability - Is pfSense affected ?:

The Snort and Suricata packages are two great examples that I am familiar with since I maintain both of them and created one of them as well.

I knew you were hardcore to the max...had to go off topic to give you a shout-out.

ramtech

@bmeeks said in Java log4j vulnerability - Is pfSense affected ?:

The Snort and Suricata packages are two great examples that I am familiar with since I maintain both of them and created one of them as well.

Sorry for the off-topic, but RESPECT @bmeeks ! Great rant, and couldn't agree more. (maybe something to do with us probably being similar vintage and background )

bmeeks

@ramtech said in Java log4j vulnerability - Is pfSense affected ?:

@bmeeks said in Java log4j vulnerability - Is pfSense affected ?:

The Snort and Suricata packages are two great examples that I am familiar with since I maintain both of them and created one of them as well.

Sorry for the off-topic, but RESPECT @bmeeks ! Great rant, and couldn't agree more. (maybe something to do with us probably being similar vintage and background )

Ha-ha! !

And all you damn kids get off our lawns! ...

ramtech

@bmeeks said in Java log4j vulnerability - Is pfSense affected ?:

Ha-ha! !

And all you damn kids get off our lawns! ...

darcey

I saw alot! of this CVE in my suricata (IDS mode) log from earlier today:

ET HUNTING Possible Apache log4j RCE Attempt - Any Protocol lower Bypass (CVE-2021-44228)
ET HUNTING Possible Apache log4j RCE Attempt - Any Protocol upper Bypass (CVE-2021-44228)
ET EXPLOIT Apache log4j RCE Attempt - 2021/12/12 Obfuscation Observed M1 (CVE-2021-44228)
ET EXPLOIT Apache log4j RCE Attempt - lower/upper TCP Bypass (CVE-2021-44228)
ET EXPLOIT Apache log4j RCE Attempt (http dns) (CVE-2021-44228)
ET EXPLOIT Apache log4j RCE Attempt (tcp dns) (CVE-2021-44228)
ET HUNTING Possible Apache log4j RCE Attempt - Any Protocol (CVE-2021-44228)

From the reported src/dst addresses/ports, these alerts seem to be associated with the following traffic:

syslog-ng on pfsense sending eve log to logstash on a local elastic server.
filebeat on a webserver sending nginx logs to elasticsearch on that same elastic server.

AIUI this particular pfsense installation and syslog-ng aren't vulnerable to the exploit. However the various elastic components involved may be.

This shows the distribution of these alerts:

Screenshot at 2021-12-13 13-25-48.png

From this, I'm guessing the rules appeared in my ruleset with a midnight update. Suricata then began furiously detecting and logging (no blocking on the firewall interface concerned). What interests me is how the alerts peak and decay. Then stop altogether despite the fact the same sort of traffic is almost certainly still crossing that interface.

Re log4j, it seems a very popular library. Several netwrok enabled applications I use daily make use of it. Elastic stack being one of them.

darcey

It's just occurred to me, the logging of these alerts triggers more of the very same alerts! Is it the case suricata rolls off alerting/logging, hence the pattern seen in the logging volume?
I'm guessing I need to disable my remote syslogging of suricata eve to logstash or disable some of those CVE-2021-44228 rules.

bmeeks

@darcey said in Java log4j vulnerability - Is pfSense affected ?:

It's just occurred to me, the logging of these alerts triggers more of the very same alerts! Is it the case suricata rolls off alerting/logging, hence the pattern seen in the logging volume?
I'm guessing I need to disable my remote syslogging of suricata eve to logstash or disable some of those CVE-2021-44228 rules.

It would depend on exactly what the triggering rule is looking for. Could be the rule is just looking for anything log4j2 related. That would mean the potential for false positives exists.

If your logging server is well isolated and protected on your LAN or other more secure subnet, I would not immediately suspect any malicious activity in that scenario. I would investigate with maybe a few packet captures and use Google research to validate if the alerts are something that can be suppressed for the IP of your remote logging server. And obviously you would want to get any log4j2 utility on there patched up.

bingo600

While NOT pfSense related

There's a list of possible vulnerable products here.
https://github.com/YfryTchsGD/Log4jAttackSurface

I have two of these installed , on "Other servers"

And have "patched/updated both"

None of these are exposed to the "Outside" , but fixed anyway.

I know a few others here are using Unifi ...

/Bingo

johnpoz

@bingo600 said in Java log4j vulnerability - Is pfSense affected ?:

I know a few others here are using Unifi

Yeah I updated to 6.5.54 from .53 as soon as it came out. I normally do anyway.

bmeeks

@bingo600 said in Java log4j vulnerability - Is pfSense affected ?:

While NOT pfSense related
I know a few others here are using Unifi ...

/Bingo

Yes, the latest 6.5.54 version of the Unifi Network Application (a.ka. "Controller") is patched. Just installed it on my system this morning. It was released on December 11th, I believe.

mer

@bingo600
Another list, seems comprehensive from teh Dutch Cyber Security folks.

https://github.com/NCSC-NL/log4shell/tree/main/software

bingo600

@mer said in Java log4j vulnerability - Is pfSense affected ?:

@bingo600
Another list, seems comprehensive from teh Dutch Cyber Security folks.

https://github.com/NCSC-NL/log4shell/tree/main/software

Thanx .. looks good