The problem with EACMS

The NERC CIP Glossary is foundational to the (let us not forget "mandatory and enforceable") CIP Standards

One of the terms defined there is Electronic Access Control or Monitoring System (EACMS):

"Cyber Assets that perform electronic access control or electronic access monitoring of the Electronic Security Perimeter(s) or BES Cyber Systems. This includes Intermediate Systems."

Disregard the last sentence for a moment. There are a few examples of throw-away statements like this added in NERC CIP for convenience, or exceptions where no one could think of wording that would be universally applicable.

Focus on the meat of the definition: it's an access control and access monitoring system. Outside of NERC CIP Standards, this is generally known as AAA: Authentication, Authorization, and Accounting, which actually captures the steps involved in granting and monitoring access much better than the EACMS definitions. There is also tons of guidance and information on implementing AAA in the broader IT Security realm.

But ignore that for a moment too.

The real problem with NERC CIP Standards and the applicable systems that they list, is that some systems that fall into the EACMS category (plus a number which don't) actually pose a much more significant risk than simple access management. They actually perform configuration management via "service accounts" with elevated privileges.

So for example, an Active Directory system is an EACMS, even though it not only controls access, it also controls configurations. But there is no requirement in NERC CIP to monitor the configuration changes made during a session, only the access to the system. Specifically, the failed and successful login attempts.

SCCM is not specifically an EACMS, even though it has an agent installed on Windows devices, and has an elevated privilege service account with Domain Administrator equivalent permissions. But it doesn't control or monitor access attempts.

There is no requirement to protect these systems any differently than any other user-accessible system, like perhaps a data portal on a web server. There is no requirement to separate user or system access within an ESP based upon roles, or impact levels (another problem with NERC IP is impact level is based upon the facility's physical impact on the Bulk Electric System, not the Cyber System's impact on operations). Once you're in, you're in and there is no requirement and not even any explicit security objective to do more than guard the perimeter. As I've mentioned before, this is the "hard crunchy shell, soft gooey center" model from 20 years ago.

On top of that, there's an exemption to remote access requirements for machine-to-machine communications. A management system located outside the Electronic Security Perimeter isn't even required to have encryption, and has no special requirements above and beyond the simple baseline, change management, and logging requirements applied to any system used to support the BES.

Weird.

Electronic Security Perimeter

Over at his blog, Tom Alrich commented to me:

"objectives-based requirements are the only way not to have the situation you discussed (drawing from Lew Folkerth's recent article in RF's Newsletter)- where an entity can be doing great things for cyber security that the auditors can't even consider because they're not in their "zone of authority" (in this case, the ESP). 

Objectives-based requirements would have to be built on top of a framework of concepts that goes beyond just BCS and ESP's, to include all of the entity's computing infrastructure (including "IT" networks)."

I definitely agree that the ESP construct is inherently limiting. For one thing, it is established by arbitrarily drawing a line, identifying where the access point (with controls applied) is, and it only exists at OSI Model Layer 3: routable protocols. Lest we forget, not all security takes place at Layer 3, it's simply the easiest place to measure.

Right now, guidance from NERC and the regions tells us that we cannot use a Layer 2 switch with some ports inside the ESP and some outside. Some would argue you can't use any switch, even a Layer 3 or 4 switch this way. However, there is no requirements language that says this is the rule. It's not a clearly established and absolute security best practice. The reason there can't be any NERC CIP requirements around this is because the ESP is defined strictly at Layer 3, and it was done that way on purpose to exclude Layer 2 connections from needing ACLs (access control lists) and firewalls. Here's the relevant reasoning from CIP-005  Guidelines & Technical Basis: 

"This requirement applies only to communications for which access lists and ‘deny by default’ type requirements can be universally applied, which today are those that employ routable protocols. Direct serial, non-routable connections are not included as there is no perimeter or firewall type security that should be universally mandated across all entities and all serial communication situations. There is no firewall or perimeter capability for an RS232 cable run between two Cyber Assets. Without a clear ‘perimeter type’ security control that can be applied in practically every circumstance, such a requirement would mostly generate technical feasibility exceptions (“TFEs”) rather than increased security."

It's not easy to measure security being applied to layer 2, so we define this type of connectivity out of relevance. Good security would be to apply controls upstream to those devices which can participate, but the device-centric focus of early CIP standards language leads people away from thinking in these terms. Security doesn't have to be "perimeter" based, and generally in a modern layered approach security functions are distributed across multiple devices/systems within your network. They're not all "on the perimeter" so the current NERC CIP standards apply poorly if at all to this strategy. We need some changes to the dominant paradigm. At the same time, we don't need auditors crawling into every ancillary system in our IT department.

Some concepts that add to the discussion:

Accessing a device means the ability to view/modify it's configuration settings. This can be because the device interacts only in the data plane and doesn't have a separate management plane, i.e. a Windows Server. It could be access to a dedicated management port.

Traffic transiting a device means that the packets are forwarded via that device. It doesn't necessarily mean that the user originating the traffic has the ability to view or modify the device in question, in most cases they may not even be aware that the device exists in the traffic stream. Packets crossing a router, switch, or firewall in the data plane would be examples.

Management plane is the logical function which allows a user to interact with the configuration settings of a system. It includes any dedicated management ports, the user interface (terminal emulation of graphical users interface), and the network connectivity to these if accessing remotely (really remote access, which means from anywhere not directly connected, not NERC CIP remote access which only means from outside the ESP).

Control Plane is the logic embedded in the code base of the system. Admins can modify the control plane logic, but usually only by replacing the code base (upgrades, patching, etc.)

Data plane is just packet switching. traffic is generated by users or devices, and sent to a destination IP address. It is switched, routed, forwarded to devices inline along the way. This traffic has no access to the Control Plane or Management Plane on a properly configured network appliance (including hypervisors).

Determining Metrics & Requirements

On Thursday, I wrote a bit about defining BES Cyber Assets, in response to some older discussion on the WICF Forum that is still an undecided topic of discussion today.

It's funny how often the definition of BES Cyber Asset has to be discussed. Tom touches on it again here in talking about the difficulty of an auditor determining whether something is a BCA based upon legal semantics. The BES CA definition is:

"A Cyber Asset that if rendered unavailable, degraded, or misused would, within 15 minutes of its required operation, misoperation, or non-operation, adversely impact one or more Facilities, systems, or equipment, which, if destroyed, degraded, or otherwise rendered unavailable when needed, would affect the reliable operation of the Bulk Electric System. Redundancy of affected Facilities, systems, and equipment shall not be  considered when determining adverse impact. Each BES Cyber Asset is included in one or more BES Cyber Systems."

In NERC CIP Standards, definitions are often contrary to plain language or commonly-understood terminology, and all-too-often include definition by exclusion, even when it tortures reading comprehension. In the definition above, they are intending to close a loophole that one might exploit to get out of compliance scope. And to the point that an identically-configured system might be just as vulnerable to a flaw or cyber attack, it's a valid point. But there are many scenarios where redundancy alone makes the idea of a functional impact into a very remote possibility.

The same article from Tom crosses over to touch upon the Streetlight Effect I wrote about here and here, and is particularly apt with the tagline I ended the second article with: "When your auditor insists upon a simple and clean way to measure compliance at the device level, they may be doing us all a disservice."

It boils down to a common management problem. I can't manage what I can't measure, because to determine the effectiveness of processes I must determine whether they change an outcome and whether that change is positive or negative. So determining what to measure and how it is related to a specific process is necessary, but the problem is that this is not always easy. Some measurements are meaningless. They either don't relate to the process, or the relate to the wrong (or misunderstood) process and therefore indicate results unrelated to the change I'm trying to make. If I don't choose the right test points and interpret the test results correctly, I am not gaining anything other than a thin cover story for why it's not my fault when things go drastically wrong. We have lots of people measuring lots of things and producing tons of documents to prove that they're testing them.

But are we looking at the right things or just the easy ones?

More on Streetlight Effect, and: "The Way We've Always Done It."

So the other day I wrote a bit about looking for things in the wrong place simply because it is easier. It's a pretty common problem in a variety of human endeavours. But it's not just laziness that causes people to look where the light shines. Often there simply is no perfect means of examining a problem. In some cases we have to rely on proxy metrics because we can't directly measure the goal. It might be a quality we're trying to determine rather than quantity, so we choose something that seems to track along with our desired outcome, and measure that.

Another reason we might do something like this is because we don't have a good understanding of the underlying mechanisms that produce a given result. We don't understand, but we desire a particular outcome, so we try to screw the inscrutable by going through the forms that we've seen other people doing, but without understanding why they did. It's as though, controlling something that looks like something else will influence it (sympathetic magic). It's like a cargo cult ritual where the form is the only thing happening; there's no technical substance to the action.

A long time ago in a galaxy not so far away, a paradigm was born; of a hardened perimeter that would keep attackers out of our networks. As security techniques go, it was primitive but had some significant positive effect. Indeed perimeter or boundary defense is still part of a solid security strategy. But it's only a part. Although we've long since passed the point where a simple OSI model layer 3-4 access control list based upon source IP, destination IP, and TCP port is sufficient for security, people are still doing it "The Way We've Always Done It™". Attacks are far more sophisticated now than when the "hard crunchy shell" first surrounded the "soft gooey center" of a network. (Now I want a candy bar!)

We need a layered defense that includes controls internal to our network. We also need security mechanisms that are not based on integrity of the function they are trying to protect. This means that not all security mechanisms can or should be all-in-one. While NERC CIP uses "systems" language at the Requirements level, many of the Measures, the RSAWs and the VRF/VSLs implicitly require a given device to provide the control. This is yet another example of wanting the auditing approach to be simple and easy (looking only where the light is shining).

Unfortunately that sort of simple-to-audit approach precludes, or at least discourages, layered defense mechanisms where components of the security mechanism are provided by network-based functions that integrate multiple types of devices. For example, AAA or Authentication, Authorization, and Accounting functions (which in the NERC CIP world for some bizarre reason is called "Electronic Access Control & Monitoring System or EACMS) generally should not reside entirely upon the single device being accessed, if for no other reason than users requiring access to this device means that attacks on the security mechanisms embedded in it are possible. Far better to have this device act as a AAA client and receive Authentication and Authorization from that network-based function, while sending Accounting information to it. This isolates the code base of the security appliance(s) from the code base of the device that users require access to, and limits opportunity for privilege escalation.

When your auditor insists upon a simple and clean way to measure compliance at the device level, they may be doing us all a disservice.

Streetlight Effect Apparent in NERC CIP Requirements

This comment is in response to an article in the ReliabilityFirst Newsletter - "Virtual Systems and Zones of Authority".

I'm going to quote a bit liberally, since the original on page 7 is difficult to link to directly. But first, read this:

https://en.wikipedia.org/wiki/Streetlight_effect

"Responsible Entity personnel see their entire network as a whole, with the parts of that network subject to CIP compliance as part of a larger security picture. They can see the protections afforded to all systems, and can see how the protections applied to non-CIP assets increase the security of CIP assets as well.

CEA personnel, on the other hand, only have CIP assets within their purview. They cannot consider non-CIP assets as adding to the entity's security posture, as those assets are not under the CEA's regulatory authority. Non-CIP assets are not subject to audit by the CEA and may change at any time with no notification to the CEA.

This difference in viewpoint can lead to conflicting views of virtual systems such as virtual networks. Responsible Entity personnel see the protections applied to the non-CIP networks that might share, for example, a physical switch with CIP networks. They see the multiple layers of protection and the controls surrounding the security of these non-CIP networks.

CEA personnel, on the other hand, do not have the authority to review the security level of non-CIP assets or networks. The CEA personnel must therefore assume that any non-CIP assets or networks could be compromised and used in attacks on the in-scope CIP assets and networks. The resulting differences in the perception of risk can be a source of misunderstanding between Responsible Entity personnel and CEA personnel.

I call this difference in perspective “mixed zones of authority.” The Responsible Entity's zone of authority is all of its owned assets, both CIP and non-CIP. The CEA's zone of authority is limited to assets that are in scope for CIP.

For this reason, and others that I don't have the space to go into here, I strongly recommend that

Responsible Entities refrain from implementing Cyber Assets or networks that mix CIP in-scope and out-of-scope assets, network traffic, or data. The reason for not mixing in-scope and out-of-scope is not, as is commonly discussed, that “untrusted” configurations are implemented.

The biggest issue, in my view, is that without being able to view all aspects of the systems used for BES reliability, there is no way for the CEA to ensure that weak or high-risk configurations are not implemented."

{emphasis added in the last two paragraphs.}

That’s an interesting perspective. Once again, the crux of the argument is not whether or not virtualization adds to or subtracts from security. It’s the difficulty of auditing that drives the recommendation.

This, my friends, is a terrible basis for driving standards. It is what is known as a perverse incentive. It drives one to make decisions that, on the basis of achieving the objective, one would not otherwise make. It's a common problem in security, that one does what is visible, and easy to measure rather than closing the worst (invisible) gap. In fact, this is a problem far beyond security, it happens in all kinds of production environments with easily gathered statistics, in management, etc.

Standards should not be about making the network easy to audit, it should be about making it difficult for a bad actor to compromise reliability and security. Industry knows how (or individual entities can quickly learn how) to secure a virtual environment in shared infrastructure mode. Industry properly securing the virtual environment isn’t the sticking point. Even the Federal Government's more security-conscious entities have processes for contracting compliance in Cloud Computing. Surely an entity is capable of adequate control in its own networks.

Take a look in the PCI standard and its glossary. The concept of trusted and untrusted network is defined there as:

Trusted Network	Network of an organization that is within the organization’s ability to control or manage.
Untrusted Network	Network that is external to the networks belonging to an organization and which is out of the organization’s ability to control or manage.

There is no logical basis to assume that an entity capable of providing a compliant solution inside their ESP is simultaneously unable to or unwilling to provide security to Cyber Assets under their own control outside the ESP, simply because the auditing agency doesn’t have explicit control over this latter subset of Cyber assets. Requirements-based language may be the problem, where an objective-based standard wouldn’t have quite the same problem. For example, proof that you meet the objective of “isolation” would tend to consist of controls and measurements applied inside, outside, and on the perimeter. It doesn’t require the bright line of layer 3 “ESP” as the sole measure of compliance and it works well with security zones as a concept.

As long as Critical Infrastructure Protection is driven by compliance rather than security, the Grid will be at unnecessarily elevated risk.