pfSense as an ICS / SCADA environment internal segmentation firewall

JKnott

@ka3ax said in pfSense as an ICS / SCADA environment internal segmentation firewall:

johnpoz, bmeeks, thanks for your inputs.

I still do not have full list of important ICS firewall differences (from regular firewall). I remember few points discussed:

no access to the Internet or to email should be allowed from ICS networks

ICS networks should be defended from other plant networks, especially those with Internet access

limited or on-demand (normally blocked) remote access
Those requirements are conflicting with our typical access and maintenance scenarios...

The no internet access is easy enough. You just don't connect it to anything. Critical networks are often completely separate from those used for other purposes. What triggers the "on demand" access?

bmeeks

@ka3ax said in pfSense as an ICS / SCADA environment internal segmentation firewall:

johnpoz, bmeeks, thanks for your inputs.

I still do not have full list of important ICS firewall differences (from regular firewall). I remember few points discussed:

no access to the Internet or to email should be allowed from ICS networks

This is to protect those vulnerable components from malware infestation. The primary exploit vectors these days are through browser or browser plugin flaws (Adobe Flash being one example) and email attachments or hyperlinks. If vulnerable components and networks can't access the Internet or email, those attack vectors are mitigated.

ICS networks should be defended from other plant networks, especially those with Internet access

This is another layer of protection from the threats I described above. Trusted networks can act as a "carrier" of a virus that then infects your sensitive networks. You want to severely limit "trust" in ICS networks.

limited or on-demand (normally blocked) remote access

Remote access equals bad cyber practice! In the industry I retired from recently, remote access to control systems was absolutely prohibited by government regulations. You got into serious trouble for violating that one! We used those data diodes to enforce that rule. If I needed to make any change to a control network, my butt went into the physical plant, through all the physical security and stood in front of the actual device (or least a terminal within a physical security boundary that was attached to the control network). And speaking of physical security, you need to have absolute physical access control to all parts of your Control Layer network. That means locked rooms where Control Layer components are located. It also means some type of alarm system for locked cabinets that might have to be located outside your typical physical security boundary (think a device that might have to be outside the fence).

Those requirements are conflicting with our typical access and maintenance scenarios...

I understand it can be a pain in the butt, but remote access to critical Control Layer components is just never a good idea. It is very hard to secure at a very high confidence level. You just can never eliminate human error.

One thing I did not see you mention, but that is a HUGE concern, is what are called portable media and mobile devices. This area includes USB memory sticks, flash memory cards, CD/DVD discs and even cell phones being charged via USB cables. You need very tight controls on these devices around Control and even Monitoring Layer networks. It would be so easy to compromise a system via a USB stick or even some employee plugging in their cell phone's USB charger cable into a vulnerable PC. In fact, USB media is exactly how the infamous Stuxnet virus infiltrated the Iranian nuclear processing facility and then went on to damage multiple centrifuge machines there. A worker was used to unknowingly carry the virus into the secure facility on a USB stick. When he plugged the USB stick into a computer on the secure and isolated network, that allowed it to spread across the sensitive Control Layer network and do its work.

bmeeks

I'll say one last thing to maybe more directly answer one of your questions about the differences between a typical firewall and an ICS firewall.

Functionally, there is no real difference. One firewall is fundamentally the same as another when speaking of today's stateful inspection technology. It's mostly a difference of features, support and of course cost. Cost runs the gamut from free to thousands and thousands of dollars for the hardware and software support contracts.

I love pfSense, and it is a very secure firewall out of the box. But it has one shortcoming when used with ICS networks. It wants to be able to talk to the Internet. The "home" screen, immediately upon opening, attempts to query the pfSense/Netgate servers to check for new firmware updates. The package and update system of pfSense is also heavily dependent on Internet connectivity. You can, with much trouble, create a sort of offline package server, but it takes quite a bit of skill to pull this off.

For ICS networks, especially those most sensitive Control Layer networks, you want a firewall that is happy being 100% isolated from the Internet. One that allows you to update it with a single file by uploading the file to the firewall from a local management network connection. So when looking for your ICS/SCADA firewall, ask the vendor about their support for 100% isolated setups where the firewall has absolutely no Internet access. Can the firewall work in that environment without issue, and does the vendor provide an offline update process that is workable for you? There are firewalls out there that fit this bill, but unfortunately none of the ones I'm familiar with are free.

ka3ax

bmeeks, your post inspired me to read more about data diodes and unidirectional networks. Also found few articles explaining "why firewalls are not recommended for securing SCADA systems"...

Luckily our site is not a power plant network and remote access to control systems is not prohibited by government regulations in our region... :)

It is just a production plant with few assembly machines. If disaster strikes we can restore everything quite quickly, our estimation is 8-48 hrs. And management is happy with this estimation. So I hope that reasonable / descent security level will help us to improve our security score without going to very expensive solutions and paranoid security mode. :)

bmeeks

@ka3ax said in pfSense as an ICS / SCADA environment internal segmentation firewall:

bmeeks, your post inspired me to read more about data diodes and unidirectional networks. Also found few articles explaining "why firewalls are not recommended for securing SCADA systems"...

Luckily our site is not a power plant network and remote access to control systems is not prohibited by government regulations in our region... :)

It is just a production plant with few assembly machines. If disaster strikes we can restore everything quite quickly, our estimation is 8-48 hrs. And management is happy with this estimation. So I hope that reasonable / descent security level will help us to improve our security score without going to very expensive solutions and paranoid security mode. :)

That's why my first post mentioned first determining the "value" of the network you are protecting. It's one thing if the loss is just monetary, but quite another if loss or sabotage of the network could result in injury or death to people. For a typical factory scenario, I would also say a data diode and all of the other physical security expenses I mentioned are overkill. However, if you are protecting a nuclear power station and its reactor, or a chemical processing facility or say the national electric grid, then more robust controls are definitely called for.

However, even with just a factory network, you will benefit from tight portable media and mobile device control. It would be a bad thing for the company to lose thousands of dollars of production because Joe brought in some pictures of his grandkids on a compromised USB stick and plugged it into a PC in the plant control room to show his co-workers.

johnpoz

@ka3ax said in pfSense as an ICS / SCADA environment internal segmentation firewall:

We have several outdated WinXP computers in our production area

If this was the case in any of the scenarios you mentioned, National Elec grid, Nuc planet - I would seriously be like WTF People!!!

bmeeks

@johnpoz said in pfSense as an ICS / SCADA environment internal segmentation firewall:

@ka3ax said in pfSense as an ICS / SCADA environment internal segmentation firewall:

We have several outdated WinXP computers in our production area

If this was the case in any of the scenarios you mentioned, National Elec grid, Nuc planet - I would seriously be like WTF People!!!

WTF is an appropriate response, but that old nemesis money comes into play. You have vendors that provide highly specialized components (sometime custom designed for just one site) and they want many millions of dollars to "upgrade" their component for you and to certify their old software on new operating systems. It becomes, unfortunately, a money decision made on a risk vs cost to mitigate scale. Sometimes totally isolating the network and associated devices is the most cost effective approach for a short-term solution. Longer term you plan on replacing that component or system. However, in the case of power plants, consider the impact of spending millions and millions of dollars to upgrade systems on consumers' power bills. They will yell and scream and the utility rate regulators are stingy with granting rate increases.

NogBadTheBad

@johnpoz said in pfSense as an ICS / SCADA environment internal segmentation firewall:

If this was the case in any of the scenarios you mentioned, National Elec grid, Nuc planet - I would seriously be like WTF People!!!

https://www.theregister.co.uk/2018/11/09/royal_navy_old_os_at_sea/

johnpoz

Yeah that really is a WTF... but if that is true
"It’s all standalone, not connected to the outside world,”

That is something

JKnott

@johnpoz said in pfSense as an ICS / SCADA environment internal segmentation firewall:

I would seriously be like WTF People!!!

Unfortunately, there is a lot of gear around where the manufacturer can't or won't update it. In some cases the hardware doesn't have the needed resources to support newer software. I have used some EXFO test equipment that ran XP and even with it, it was painfully slow. In that case, the only recourse is to buy newer equipment.

JKnott

@NogBadTheBad said in pfSense as an ICS / SCADA environment internal segmentation firewall:

https://www.theregister.co.uk/2018/11/09/royal_navy_old_os_at_sea/

Ah yes, Windows Moron Edition.

bmeeks

I had to deal with even a Windows NT 4.0 system, but at least it was a completely standalone single machine located in a high security area.

You can make these machines secure by locating them behind high security physical barriers, locking down USB access with key-lock physical blocking devices inserted into the ports, and by putting them on data-diode isolated networks so two-way communcations with the "outside" is impossible. You would only use the data-diode option if it was necessary for the older machine to send its data to some other network. If everything was local, then most times no network is even connected and the RJ45 port is also physically blocked closed with a key-lock plug.

JKnott

@bmeeks said in pfSense as an ICS / SCADA environment internal segmentation firewall:

I had to deal with even a Windows NT 4.0 system, but at least it was a completely standalone single machine located in a high security area.

IIRC, Windows NT only met Orange Book C2 security by having no connections to anything and floppies/USB sticks blocked.

bmeeks

@JKnott said in pfSense as an ICS / SCADA environment internal segmentation firewall:

@bmeeks said in pfSense as an ICS / SCADA environment internal segmentation firewall:

I had to deal with even a Windows NT 4.0 system, but at least it was a completely standalone single machine located in a high security area.

IIRC, Windows NT only met Orange Book C2 security by having no connections to anything and floppies/USB sticks blocked.

Luckily for me that NT machine was in a display-only role that simply received data from a remote sensor over a proprietary analog interface and displayed the output for trending. It was for local use only and had no external connection.

JeGr

@johnpoz said in pfSense as an ICS / SCADA environment internal segmentation firewall:

"It’s all standalone, not connected to the outside world,”
That is something

Somehow with the whole twist this thread has been taking (and the horror scenarios...) while reading "isolated from the network" I had to remember something I read years before about "tempest shielding" like some sort of "bunker" isolation. Reminds me of that.

ka3ax

Thank you all guys for all the info and motivating scary stories. :)

Back to my original question: Are there manuals or guidelines for turning pfSense into an ICS / SCADA firewall?

johnpoz

@ka3ax said in pfSense as an ICS / SCADA environment internal segmentation firewall:

ICS / SCADA firewall?

You have yet to call out the specifics of this.. I don't even think its a thing to be honest.. There are firewalls, and then there are firewalls.. There isn't X firewall, and then B firewall for what devices you put behind them.

You can put you ISC network behind a firewall.. You can then allow or block whatever traffic you want or don't want based on all kinds criteria. You can run IPS on the traffic you allow if you so desire. What else do you need?

If I put my plex behind a firewall, I could call it my plex firewall if I wanted ;)

bmeeks

@johnpoz is correct. There is no difference between an ICS/SCADA firewall and any other typical firewall in terms of security or performance. Of course there are differences in appearance, perhaps differences in features and certainly differences in cost. An ICS/SCADA firewall is not more secure than any other firewall assuming both are configured with the proper firewall rules.

Where there can be differences are in these two areas:

Hardware touted as an ICS/SCADA firewall is more likely to be rated for use in harsher environmental conditions (higher temps, maybe higher humidity, etc.).
Going back to one of my earlier posts, a true ICS/SCADA firewall is more likely to be able to operate isolated from the Internet. That means it does not need to constantly check a home web site for firmware updates and it is more likely to support an offline update process (meaning you can update the firewall's software without needing an Internet connection directly from the firewall).

In terms of feature set differences, a vendor selling you so-called dedicated ICS/SCADA firewalls may well offer some pre-packaged firewall rules as features. Say something like having some canned Modbus rules ready for you, or perhaps rules for other standard industrial control protocols. Of course when it comes down to it, ports are ports and UDP and TCP are the same everywhere, so you could replicate any of these pre-packaged rules on your own. But you might have to do more research to find the proper ports and protocols, for example.

A vendor selling you what they term an ICS/SCADA firewall is likely to be more attuned to industrial control terminology and technology, so their after-the-sale tech support may be superior to what you might get from say a more generic perimeter firewall vendor should you have questions related to industrial control technologies.

So to sum up what I'm saying, in terms of security there is no inherent advantage of one firewall over another. It's the features one brand provides over another that is most likely to matter to you. And of course don't discount the operating environment. Typical industrial locations are much more harsh than a nice climate-controlled server room in a data center.

In terms of making pfSense an ICS/SCADA firewall -- sure you can do that. But there may be some headaches. How big the headaches are depends on whether or not pfSense will have Internet access. If the firewall has unrestricted Internet access, then it can be a fine ICS/SCADA firewall. Just pay attention to any physical environment issues and buy hardware appropriate for the environment. If you need to put pfSense into a situation where it will not have direct Internet access, then you start to have the headaches. When the home page opens after login, it will attempt to contact the Netgate servers to check for updates. The GUI will be very slow during this time if there is no Internet access. Also, lack of direct Internet access will be a real burden if you want to load optional packages such as Snort or Suricata. Those all expect Internet connectivity in order to function.

NogBadTheBad

@bmeeks said in pfSense as an ICS / SCADA environment internal segmentation firewall:

Hardware touted as an ICS/SCADA firewall is more likely to be rated for use in harsher environmental conditions (higher temps, maybe higher humidity, etc.).

Going back to one of my earlier posts, a true ICS/SCADA firewall is more likely to be able to operate isolated from the Internet. That means it does not need to constantly check a home web site for firmware updates and it is more likely to support an offline update process (meaning you can update the firewall's software without needing an Internet connection directly from the firewall).

In terms of feature set differences, a vendor selling you so-called dedicated ICS/SCADA firewalls may well offer some pre-packaged firewall rules as features. Say something like having some canned Modbus rules ready for you, or perhaps rules for other standard industrial control protocols. Of course when it comes down to it, ports are ports and UDP and TCP are the same everywhere, so you could replicate any of these pre-packaged rules on your own. But you might have to do more research to find the proper ports and protocols, for example.
A vendor selling you what they term an ICS/SCADA firewall is likely to be more attuned to industrial control terminology and technology, so their after-the-sale tech support may be superior to what you might get from say a more generic perimeter firewall vendor should you have questions related to industrial control technologies.
So to sum up what I'm saying, in terms of security there is no inherent advantage of one firewall over another. It's the features one brand provides over another that is most likely to matter to you. And of course don't discount the operating environment. Typical industrial locations are much more harsh than a nice climate-controlled server room in a data center.

If your running Snort it may be an idea to enable the SCADA Preprocessors, they're disabled by default.

Tinus087

@ka3ax

@ka3ax said in pfSense as an ICS / SCADA environment internal segmentation firewall:

Thank you all guys for all the info and motivating scary stories. :)

Back to my original question: Are there manuals or guidelines for turning pfSense into an ICS / SCADA firewall?

Since I did not see a reply on your question or that this topic is closed, allow me to pitch in from my perspective.

Let me elaborate first: I have some novice experience with PfSense but not in an ICS environment. However I do know a thing or two about ICS/SCADA environment. Sometimes referred to as Operational Technology (OT).

The most important question we need to answer is if PfSense is able to handle industrial protocols like Modbus/TCP (port 502), Profinet (ports 34962 - 34964), etc.

For those who are not aware of these protocols: these are used to communicate between i.e. a Human Machine Interface (HMI) and a Programmable Logic Controller (PLC) and are widely used, from water treatment plants, chemical factories to opening and closing a bridge. PLCs are also used for changing the traffic lights from red to orange/green. Tunnels (for car traffic) are most of the time packed with OT devices to control things like air flow, alarms for fire, vehicles that are to high, etc. So we come across these systems more then we realize in our daily life. Every manufactured good in your house, from plastic storage boxes, electrical wires to processed foods like coffee beans and bags of chips, have been made and gone through quality checks using OT environments.

As was stated earlier, these systems can sometimes be very old (think 1970's, 1980's) and originally communicated using serial connections (COM ports like RS-323 and RS-485). When TCP/IP based networking was introduced into manufacturing (OT) environments these protocols have been adjusted (encapsulated with TCP/IP markers around the original traffic) to allow them over rout-able networks.

However, the issue is that these protocols, or at least most of then, where not designed to be handled by IT network based routing. So I expect that when using PfSense you might run into interesting issues. Especially when you add deep packet inspecting in the mix (although I do not think PfSense is capable of doing that)

Now let's become pragmatic. If you have the ability to create a test environment to see if PfSense is a viable option, I would do that. You can set up a (virtual) test environment, run some tests and check the logs for specific issues. As you can see the rout-able protocols have port numbers that you can allow, block, drop, etc. So, in general, I think that yes, the firewalling function would support industrial protocols as it is also port based. You should be able to specify which PLC is allowed to connect to which PLC and/or HMI based on IP address or MAC address as you would in an IT environment. You can also set access to HMI and PLC for i.e. vendor support but keep in mind that using a stepping stone server in a DMZ would always be preferred over a regular connection using a VPN into the business network.

Also invest time to look into latency that the PfSense will bring. Especially when real time information in the SCADA systems is important. So discuss with the business how much latency between the levels would be acceptable for the manufacturing process if you haven't done so already of course.

On more pitfall would be a dual homed system. A system that circumvents the firewall by having two network connections, one going to Level 2 or 3 networks and one going down to the Level 1 network. (for more on ICS networking look up the purdue model)

I know it has almost been three years but I hope I have been able to help you out.