Virtualization in the CIP Environment Drives Discussion of Applicable Systems Classifications

Here's a draft of a Whitepaper I intend to submit to NERC.

Alt Title:

More Problems With EACMS

Summary:

One of the topics entwined in the NEC CIP virtualization discussion is the risk posed by the virtual management consoles. This is related to consolidated interfaces and automation where management console access can change or delete entire infrastructures including virtual servers, networks, and storage. A short-hand phrase has been coined calling this the “Fewer, Bigger Buttons” problem.

Because this is a valid concern that is not really addressed at all in NERC CIP, the scope of discussion quickly grew to address similar Centralized Management Systems (CMS) in the physical arena as well, and from there to all kinds of systems which pose, or appear to pose similar risks.

Statement of the Issues:

The 2016-02 SDT has been discussing several options publicly. One option is to classify CMS (a heretofore undefined term in NERC CIP, and not an industry standard definition in Cyber Security either) as applicable systems and apply specific security requirements to them.

Another option the SDT has discussed is including CMS into the existing EACMS definition. This is more or less by default the approach taken to date with all of those legacy enterprise automation tools. Obviously, this capability and risk has existed, largely unacknowledged in CIP standards, for a long time without being confined to virtualized environments. HP Openview, IBM Tivoli, Solarwinds Orion, and CiscoWorks (to name just a few enterprise automation tools) have had the ability to affect the entire enterprise all at once for literally decades now.

Anti-malware and software patching systems likewise. If a system in my network operates via a system account with administrative privileges that allow it to modify the configuration of a BCS, isn't that tool both a target and a potential attack vector? And if such a system inside my network that performs this function is an EACMS, then isn’t Microsoft’s Software Update Services site, and Ubuntu’s Linux Repository, or Symantec or McAfee’s antivirus signature update sites on the Internet as much an EACMS as any SCCM or Anti-virus server inside my network?

However, this perpetuates and exacerbates an issue where EACMS has become a "catch-all" category of CIP-related Cyber Assets with one-size-fits-all requirements regardless of the degree of risk or technical constraints posed by the particular system.

An example is the Intermediate System. In order to make the IS subject to CIP requirements it has to be categorized somehow as a type of applicable system. To function as an intermediate in practical terms (and by definition per the NERC Glossary) it has to be outside the ESP. Apparently the IS has therefore been categorized as an EACMS simply because that is the only category currently available that allows for applicable systems to be subject to CIP requirements outside the ESP.

Thus, the otherwise-unconnected phrase “This includes Intermediate Systems” was tacked onto the end of the EACMS definition. It is notable that no other examples had previously been given.

The Definition of Intermediate Systems:

“A Cyber Asset or collection of Cyber Assets performing access control to restrict Interactive Remote Access to only authorized users. The Intermediate System must not be located inside the Electronic Security Perimeter”

Here we see a presumably audit-able CIP requirement set in the definition of Intermediate System rather than in a table of requirements or in a security objective. We see a cyber security function (Authentication and Authorization) defined as a specific type of applicable device, and we see the security benefit of such a function truncated to apply only to users who interactively access Cyber Assets inside the ESP from outside the ESP rather than applying the benefit of robust authentication, authorization and accounting to all remote access.

Another example of the problem with the one-size-fits-all approach to compliance requirements for EACMS is what is known as the “hall of mirrors” effect. Specifically, there may be some types of Ecyber Security Systems that should be required to be protected behind a firewall. However, that requirement can’t exist for all EACMS without defining a new category because a firewall is itself an EACMS. Without defining a new category, the result would be every EACMS needing to be inside an ESP and protected by another EACMS which creates a recursive "hall of mirrors" effect without end.

In addition to the catch-all and recursive problems I've just noted, there is also a missing component: Risk-based assessment and mitigation. For example, a system that only monitors and logs access (such as monitoring systems like Splunk, Tripwire, etc) does not pose the same level of risk as a management console for a large virtualized Control Center infrastructure. In addition, the technical controls to mitigate the risk may differ. A SIEM system presents a risk of leaking BES Cyber System Information; an electronic access control (AAA) system presents a risk of unauthorized access to or modification of a BES Cyber System’s operational parameters; and a Centralized Management Console (physical or virtual) presents an infrastructure reliability risk.

Risk-based assessments and mitigation would help with this. It would allow for acknowledging that there are a number of systems that both monitor and provide part of the solution of controlling access, but which do not actually control traffic at the point of entry. These devices or systems may or may not benefit from being inside a protected boundary, or they may form part of the strategy that protects BES Cyber Assets. The technical means of implementing some multi-part systems may require that components be outside or that they span the ESP.

All of this appears to be an unfortunate artifact of the single-level security mindset inherent to the ESP approach (hardened perimeter defense) and the “All-In” nature of CIP Applicable Systems. Part of this problem (creeping scope of applicability to increasingly peripheral systems) stems from using the ESP to define scope of applicability, rather than using risk to BES Cyber Systems and security objectives to define the scope of necessary controls. FERC does not allow a Registered Entity to assess and accept their own risks due to the interconnected nature of risk to the BES. At the same time, NERC and the Regions have zero incentive to accept risks on behalf of Entities. NERC and the Regions bear none of the cost of mitigation, and would receive the lion's share of criticism in the case of a failure of reliability or security breach. It is very difficult to create standards that are effective, comprehensible, and inclusive of different technical capabilities, based upon the existing definition of EACMS or even new definitions structured with the same ESP-as-hardened-perimeter mindset. ESP is easy to visualize. Drawing a “red dotted line” around assets needing protection is convenient. It’s simply not sufficient. In contrast, a modern defense-in-depth, systems- rather than device-based approach doesn’t require torturing definitions.

With these examples in mind, the issues caused by having EACMS as a “catch all” category are obvious. Some applicable systems were defined into existence for compliance purposes and these standards are incompatible with broader standardized cyber security best practices, and with one-size-fits-all requirements applied to the whole artificial group.

Recommendations:

So what do we do about it?

1. The audit process needs to envision an approach more concerned with meeting an objective than a performance requirement. Then it doesn’t matter where something resides, as long as the applied combination of protections achieves the objective. It doesn’t matter much what is providing the control, as long as the assets needing protection receive the control. For example, the implied security objective of CIP 5 is not to “have inbound and outbound rules”. That is a limited method of achieving the real objective, which is to protect the BCS from unauthorized and potentially malicious traffic. If you have a better method, you should be allowed to use it.

2. It might be necessary to break up the EACMS category of applicable systems into discrete functions (bullet points below) so that the appropriate security objective and requirements for each can be derived and applied whether the systems in question are physical or virtual.

• SIEM - Security Incident & Event Monitoring systems.

This would subtract the “M” for “Monitoring” from EACMS. These are systems that strictly monitor and collect information about the ESP and BES Cyber System electronic communications or status but do not control access. A great deal of literature, discussion, guidance, and best practices are published across a broad range of industries as to how to securely implement SIEM. The risk presented by compromise of these systems revolves around the information (such as configurations and event logs) they contain. The crux of the concept here is that the protections already defined for BES Cyber Systems Information (BCSI) are adequate and effective at providing protection for this information, and it is the information that needs protecting, not necessarily the SIEM system.

A rather large issue with EACMS is CIP-004 and its applicability of most personnel-oriented controls (training, background checks, etc.) to anyone with potential access to an EACMS. This kills sharing any service whatsoever (AAA, SIEM, etc.) because anyone with any form of “access” to an EACMS gets sucked into CIP-004. Have an account on a EACMS AAA server but NO access to any BCS? Too bad, you still must have a CIP background check and be trained on the CIP program (beyond your ‘need to know’). It’s a symptom of the “all-in” issue.

Splitting these monitoring systems out and adjusting the requirements would may allow entities to more easily use outsourced managed security service providers or global/enterprise-wide SIEM systems and correlate event information in their CIP operational environments with those in their non-CIP environments to provide increased security and reliability benefits. The concern is that under current standards the CIP program and device-level CIP audits might be deemed to encompass Cyber Assets which do not actually affect the reliable operation of the BES in the wider enterprise network or at the service provider and could therefore dilute attention from BES Cyber Security functions in favor of paperwork exercises.

• EACS - Electronic Access Control System. The fundamental defining characteristic of an Electronic Access Control system is that it performs authorization of traffic or users. This is the gatekeeper function- the classic Authentication and Authorization functions of standard AAA.

In many cases these systems do not perform any active filtering of the traffic passing through any particular interface. The primary duty of EACS is to authenticate and authorize. Additional components of electronic access security strategies are accounting (logging) systems and gateways which actually pass or drop traffic. In comparison to the SIEM discussion above, EACS & EAG move beyond the risk of unauthorized access to meta-information about an environment to unauthorized access to and modification of operational parameters of the actual BES Cyber Systems.

In contrast, theoretically an application level IPS could send a control signal from anywhere in the enterprise to anywhere in the enterprise telling a specific host not to respond to a given type of packet, with a certain payload, from a particular address- and it could do this dynamically based upon heuristics rather than signatures, but this outstanding security solution would not be a compliant solution under our current regime.

Or a network level intrusion protection system combined with dynamic firewall rules may send a control message to a firewall instructing it to dynamically change an Access Control List (ACL) in response to traffic patterns indicating a threat. The IPS does not itself filter traffic in this scenario, but it is involved in controlling access. An Active Directory server may enforce a lock-out on a user account after hours or subsequent to a number of incorrect password attempts. It is involved in controlling access (authentication and authorization), but it is not blocking traffic at layer 3 and from the current NERC CIP ESP perspective is therefore irrelevant in boundary protection. All boundary protection requires access controls, but not all access control is boundary protection.

A metaphor for Access Control Systems that do not reside in or on an ESP/ESZ is: a pair of military units with interlocking fields of fire supporting each other against frontal assaults and flanking movements. Defense relies upon being positioned to assist one another, not on “being inside a fence”. Electronic Access Control Systems (AAA) can work to protect Cyber Assets inside the ESZ from anywhere.

• EAG - Electronic Access Gateway. The fundamental defining characteristics of an EAG are that it hosts the EAP and performs the active function of filtering or forwarding traffic at the demarcation point (boundary protection). Primarily it is firewalls and routers that perform gateway functions at the layer 3 ESP boundary demarcation point. Virtual firewalls and virtual routers inside a hypervisor perform the same function in the same manner. However, hypervisors themselves may not be EAGs if they are not configured with a virtual firewall or virtual router function to provide a gateway function.

Modern security methods typically employ a defense in depth strategy using distributed AAA (Authentication, Authorization, and Accounting) systems to authenticate and authorize access to the Electronic Security Zone based upon characteristics of the user and traffic, while the EAG subscribes to the AAA service for user permissions and filters (permits or denies) traffic based on the source, destination, and port or protocol. Further, Electronic Access Control strategies often employ multiple devices, each containing a part of the AAA solution, such that compromise of one element of AAA does not result in the entire system failing. Vendor and platform diversity within a defense-in-depth systems-based approach to Access Control are generally an element of securing the entire system from vulnerabilities common to specific classes of devices (e.g. all-Windows or all-Linux environments may have common configuration or malware vulnerabilities). Often the Accounting (logging) function is used to determine (and formulate a strategy to correct) any failures in Authentication and Authorization. 

Although some EAG devices are also capable of performing various levels of functions to authenticate and authorize traffic, many are not capable of complete AAA solutions in themselves and therefore differ enough from EACS to warrant different technical control measures. Requirements that acknowledge the difference and allow for handling them differently will prevent any "hall of mirrors" effects as described above only. 

For conceptual discussion purposes EAG acts somewhat like the legacy ESP as a logical demarcation point for conceptual discussions to delineate PCA and BCS from non-CIP-Applicable Cyber Assets. It may even be useful to replace ESP completely using ESZ with EAG for demarcation points.

• CMS - Centralized Management System. System using an elevated privilege account either on behalf of an interactive user or in an automated fashion, allowing mass modification of BES Cyber Systems.

As discussed, these systems are the ones driving the SDT's apparent thought process. The risk posed by these systems is not just unauthorized access to information or BES Cyber Systems, but the ability to modify or destroy the infrastructure the BCS rely upon, or the BCS themselves. These will have unique requirements over and above the others. It would obviously not be beneficial to simply create a reclassification and documentation exercise for entities who would not see sufficient benefit.

Proposed definitions:

AAA: Authentication, Authorization and Accounting systems are Cyber Systems that control ‘Gatekeeper’ functions (electronic access methods and permissions, e.g. authentication of users or control messages that alter dynamic ACLs) for Cyber Systems.

CMS: Centralized Management Systems are Cyber Systems that perform automated management tasks and mass configuration of BCS whether scheduled or on demand using a dedicated service account credential.

EACMS: Deprecated.

EAG: An Electronic Access Gateway is a Cyber Asset that performs active electronic traffic control (filtering and/or forwarding) for ESZ boundary protection based upon the criteria given to it.

EAP: An Electronic Access Point is the logical interface on an EAG where traffic filtering operations take place. 

ESZ: An Electronic Security Zone is the logical container providing separation or isolation from threats or attack vectors to grouped Cyber Assets, said Cyber Assets being characterized by similar operational criticality, or sensitivity to compromised data confidentiality and integrity, as well as needing similar access controls, audit logging and/or monitoring requirements.

SIEM: Security Incident & Event Monitoring is a Cyber System that performs electronic monitoring of Electronic Security Zone(s) or Cyber Systems. 

Rapidly evolving threats & slow-moving regulatory standards

Over at the Anfield Group Blog, Chris Humphreys posts: 

The DOE’ s Quadrennial Energy Review Report states that:

“The current cybersecurity landscape is characterized by rapidly evolving threats and vulnerabilities juxtaposed against the slower-moving prioritization and deployment of defense measures.” I lump regulatory standards and requirements into the “slower-moving prioritization and deployment of defense measures” as one of the key components to preventing a truly proactive stance on cybersecurity. Additional focus on recovery and resiliency needs to be a foundational element of any cybersecurity program because the idea that an organization can combat against 100% of cyber intrusions is false. What becomes critical is the recovery of the system if/when a successful cyberattack occurs."

I couldn't agree more. We will never eliminate all risk.So it behooves us to have a backup plan- resilient recovery strategies. NERC CIP's specific language around redundancy doesn't dismiss the importance of redundancy, but a lot of NERC CIP compliance folks do. The language says one cannot exclude a Cyber Asset from scope of CIP simply because the system is redundant. Fair enough. Redundancy doesn't protect from software vulnerabilities, malware, or mis-configuration. But too many people seem to think that this means redundancy doesn't matter, and in fact, there doesn't appear to be any requirement to have redundancy for Cyber Assets. 

Something that NERC CIP doesn't do well: make clear that assessing technical controls for high availability at the systems level rather than at the device level can provide a more accurate perspective on real cyber security, and this high availability is achieved through redundancy of underlying infrastructure (perhaps switching and virtual network systems, or hypervisor infrastructure) that has little or nothing to do with BES functions, BES Information etc. Building resiliency in and eliminating reliance upon single devices (or as I like to call them, "single points of failure") is a key part of virtualization's benefit.

The entire mindset behind and promoted by the NERC Glossary and the definition of BES Cyber Asset is to blame for this lack. Add that to the prescriptive requirements, device-centric example measures, and the device-oriented Severity Level tables, and you get a self-reinforcing  echo chamber about how to achieve reliability that makes it difficult to look outside the way it has always been done.

From the EnergySec website:

"EnergySec published comments on the Transmission Owners Control Center and Virtualizations white papers the Standards Drafting Team is seeking comments on. Since EnergySec does not formally vote on SDT ballot issues, we are not formally submitting these comments to the SDT. However, we are making them available for our members to use or revise as they see fit. The documents are available in the Members section of the EnergySec Community wiki."

Not only are they not formally submitted, it would appear that almost no one on the SDT or the NERC staff associated with it actually has access to read it. You would think that something like this would be more use to their membership if it were not behind a pay wall.

CIP 9 and Hypervisors

Carlo asked a question about CIP 9 in comments to a previous post.

I hope this answers it well enough:

Backup solutions are going to be specific to your Hypervisor choice. If you're working with a Type I Hypervisor (Bare Metal) your backup solution (for the Hypervisor itself) may very well be “install fresh from vendor media” because there is very little specific data to be restored and which Host the Guest resides upon is transparent to the Guest and/or users of the system. If you have a Type II Hypervisor that includes an operating system, and possibly performs functions other than Hypervisor in addition (which I would strongly recommend AGAINST), then your backup solution may be more complex.

In most cases, backups for Hypervisors are not urgent because you plan and implement swap space capacity into your infrastructure. You should have more Hypervisor capacity than is necessary for the Guests in each. This allows for maintenance of Hypervisors (patching, upgrades) without taking any functionality offline. You simply move Guests from the Hypervisor being taken offline to other Hypervisors for the duration of the procedure. In an unplanned outage, the same process is used. Guests don’t rely on any specific Hypervisor, they simply need a Hypervisor.

Backup solutions for Guests can work exactly the same as your traditional network-based backup management software. I wouldn't do it that way, but it's possible.

Most Virtual Cluster Management consoles allow for "snapshots" of the running state of your Guest. This is much more complete than a backup of files and directories, and requires merely rebooting to a previous state rather than a process of restoring files and settings to a base image.

The advantage here is that if your target to be restored is participating in a security domain such as Active Directory, you're merely restoring a previous state, not deleting and recreating AD objects with unique and possibly conflicting GUIDs. It's more complicated if it is an AD Domain Controller; this may require authoritative or non-authoritative restore procedures (depending on the state of the AD domain and how corrupted it may be).

Depending on your Hypervisor choice, the Guest’s configuration data (number of processor cores, RAM, storage targets etc.) may be contained in the image of the Guest or in your private cloud cluster manager. This configuration data is rather static and rarely changes. In most cases, the cluster manager is a virtual machine itself and can be rebooted from a snapshot. It could also be backed up across the network using a traditional backup application.

If you’re using standalone, unmanaged Hypervisors (perhaps for cost reasons) then you have more manual planning to do. You have to make sure that you have a process to identify target destination Hosts with adequate resources and you should also maintain functional redundancy or security zone separation for manual moves of Guests during planned or unplanned outages. For automating Guest movements, this is managed by setting affinity of certain Guests to targeted Hosts.

For CIP-009-6 specifically, nothing changes until CIP-009-6 R1.3. Here we need to note that the Hypervisor doesn’t have any BES Cyber System Information. The Hypervisor is just a container, and doesn’t interact with the code base of the BES Cyber Systems themselves. So “processes for the backup  and storage of information required to recover BES Cyber System functionality”  {emphasis added} apply more strictly to the Guests individually (Cyber Asset) and to the Hypervisors as a general function (Cyber System).

R1.4 and 1.5 are also going to have to be applied at a Cyber Systems level rather than Cyber Asset.

 

 

 

Virtualization Webinar Today (NERC CIP Standards Drafting)

 

Industry Webinar Project 2016-02 Modifications to CIP Standards Virtualization in the CIP Environment April 18, 2017 | 3:00 – 4:30 p.m. Eastern   Click here for Webinar Registration Dial-in: 1-415-655-0002 | Access code: 731 913 110
Background In addition to the Order No. 822 directives, the Project 2016-02 Modifications to CIP Standards Drafting Team (SDT) is addressing four issues identified by the CIP Version 5 Transition Advisory Group (V5 TAG). These issues are: ·         Cyber Asset and BES Cyber Asset Definitions; ·         Network and Externally Accessible Devices; ·         Transmission Owner (TO) Control Centers Performing Transmission Operator (TOP) Obligations; and ·         Virtualization   Virtualization of Cyber Assets provides advantages for the availability, resiliency, and reliability of applications and functions hosted in such an environment when implemented in a secure manner. The SDT is offering a series of webinars in an effort to provide a resource for technical information related to several concepts relevant to virtualization. During the first webinar, held on March 21, 2017, the SDT discussed logical isolation and Centralized Management Systems (CMSs), in addition to introducing storage virtualization.   Webinar Objectives This second webinar will discuss the Hypervisor with a special focus on template considerations, as well as multi-tenancy and the concepts of underlay hardware and Electronic Security Zones. This webinar will also expound on the introduction to storage virtualization provided in the first webinar, to include discussion of storage area networks and the manner in which underlying virtualization concepts are similar to server virtualization. Finally, this webinar will discuss storage virtualization scaling and data leakage.
For more information or assistance, contact Katherine Street (via email) or at (404) 446-9702 or Mat Bunch (via email) or at (404) 446-9785
3353 Peachtree Road NE Suite 600, North Tower Atlanta, GA 30326 404-446-2560 | www.nerc.com

 

Cyber Assets vs Cyber Systems Confusion in the Virtual Environment

I'm concerned to still see discussion in the Nerc-isphere about how to categorize a VM in the most simple of clear-cut examples: the virtual server/hypervisor combination. Some folks still disagree that it is necessary to treat each virtual machine and hypervisor as separate Cyber Assets. They think:

"The hypervisor (parent) is the device or software which runs the virtual machine (child). The virtual machine (VM) cannot operate without the hypervisor. This shared relationship means that neither can be separate Cyber Assets. For example, if a VM has been identified as a BES Cyber Asset (BCA); the hypervisor that runs the VM is also a BCA; which also applies to PACS, EACMS, and PCA’s

Treating the VM and hypervisor as separate Cyber Assets can cause mixed-trust virtual environments; the hypervisor runs CIP and corporate VM’s. CIP controls are only being applied to the CIP VM and not the hypervisor; even though the hypervisor “if rendered unavailable, degraded, or misused” can impact the CIP and corporate VM’s."

 Allow me to counter:

The hypervisor (parent) is the device or software which runs the virtual machine (child).

The Hypervisor or Host merely provides a container or environment for the virtual machine. In this aspect the control plane of the Hypervisor operates certain control plane functions on behalf of the guest. However, the Hypervisor is not involved in the control plane¹ decisions made by the guest.  The Hypervisor does not need to (and should not be configured to) interact in the data plane of the Guests. They make computing decisions entirely independent of each other, including their reactions to inputs and malware. If an RE decided to treat Host and Guest as one Cyber Asset for compliance reasons, there is no logically consistent framework to require the long-standing and well-known security best practice of separating the management and data plane of the Hypervisor and Guests. The guest should be completely unaware of and unable to interact with the Host in a secure virtual environment. The Host and Guest more often than not run different operating systems, and it would be difficult to categorize that as one Cyber Asset for any practical purpose. 

The virtual machine (VM) cannot operate without the hypervisor.

This is not strictly correct. More precisely, the VM cannot operate without a hypervisor. It is not dependent upon any specific hypervisor (rather, it can exist on any Hypervisor in the cluster and this is one of its biggest advantages). Therefore treating them as one Cyber Asset is inappropriate because this approach would not require distinct vulnerability assessments, patching, baseline configuration etc.

This shared relationship means that neither can be separate Cyber Assets.

The assumption in this statement doesn’t work in both directions. Presuming that a Guest could not be a separate (meaning independent) Cyber Asset, does not preclude a Hypervisor from being an independent Cyber Asset. After all the Hypervisor is a complete hardware, operating system, (and potentially software) stack in itself. Depending on whether it is a Type I or Type II, it might even be the same operating system as the Guests with Hypervisor function software merely installed on top of a generic operating system. It exists with a hostname, an address, a particular set of open ports/APIs, network connection, and responds to network traffic. The Hypervisor can exist and operate without a single Guest inside.

For example, if a VM has been identified as a BES Cyber Asset (BCA); the hypervisor that runs the VM is also a BCA; which also applies to PACS, EACMS, and PCA’s

This may be true; it doesn’t negate the necessity of treating the Host and Guest as distinct Cyber Assets (for multiple reasons). At most it makes the entire assemblage a BCS.

Treating the VM and hypervisor as separate Cyber Assets can cause mixed-trust virtual environments; the hypervisor runs CIP and corporate VM’s.

This is known as argumentum ad consequentiam (appeal to consequences) and is a known logical fallacy. Whether or not shared infrastructure is allowed and whatever the consequences of doing so, it does not change the distinct character of the Guest vs the Host. It also presumes that sharing infrastructure between CIP-applicable systems and “Corporate” VMs is impossible to secure and must be prohibited. This remains to be proven.

CIP controls are only being applied to the CIP VM and not the hypervisor; even though the hypervisor “if rendered unavailable, degraded, or misused” can impact the CIP and corporate VM’s.

Again, this is an unproven assumption. Treating the Hypervisor and the Guest as separate Cyber Assets does not require a difference in controls. If both are BES Cyber Assets then the same requirements apply to both. Additionally, the technical configuration controls applied to a Hypervisor are generally different than those applied to a Guest (Mal-ware, for example). Again, The Host and Guest may run different operating systems with different open ports, APIs, software vulnerabilities, and baseline capabilities. 
The question is whether a BCA Hypervisor can host a PCA or non-CIP Applicable System safely. The advisability of this approach depends upon whether or not sufficient security controls can be put in place to render negligible the risk of unavailability, degradation, or misuse of the BCA Hypervisor and associated BCA Guests. Risk assessment and acceptance are highly subjective and specific questions that depend more upon the overall architecture and defense in depth posture, than upon a simplistic question of "to virtualize, or not to virtualize."

(1)Control plane functions are not directly accessible by users of the system, they are embedded in the logic of the code base, and are generally require modification to the code base to change them. Virtualizing a server involves abstracting the hardware interactions much like the Hardware Abstraction Layer (HAL) does in Windows. The difference being that the Guest operating system simply sends its hardware access requests to the Hypervisor rather than to firmware. This has a three-fold security benefit. 

1. The users and software accessing the Guest in the Data plane cannot substitute malware in place of authorized device drivers. Since device drivers are one of the worst vectors for malware after Phishng attacks, this narrows the attack surface of the Guest OS.
2. The Hypervisor can be a different operating system than the guest, which means attacks in the data plane against the guest firmware will be ineffectual against the Guest due to no device drivers present and no direct access to hardware, and will be ineffective against the Hypervisor because there is no Data plane access between the two, and the device drivers that actually do interact with hardware are for a different operating system than what is visible to the attacker.
3. Drivers that actually interact with the hardware are generally a smaller subset of better-vetted drivers approved by the Hypervisor vendor, as long as you make the smart choice and go with a Type I Bare Metal Hypervisor.

Trial Balloon

Open up the current CIP-005, and go to page 15 (towards the bottom) to see the original, then compare it side by side to this (unofficial, non-authoritative, completely speculative, proposed, draft) language... What do you think?

*********************************************************************************

Requirement R1 requires isolation of BES Cyber Systems from other systems of differing trust levels by requiring Boundary Protection and controlled Network Ports, Protocols, and Services via identified Electronic Access Points between the applicable BCS and Non-CIP Cyber Systems. Electronic Security Perimeters are also used to identify a defense boundary for some BES Cyber Systems that may not inherently have sufficient cyber security functionality, such as devices that lack authentication capability.


All applicable BES Cyber Systems that are connected to a network must reside in a defined Electronic Security Zone (ESZ). Even standalone networks that have no external connectivity to other networks must have a defined ESZ. The ESP is a demarcation of the security zone containing the BES Cyber System, and it also provides clarity for entities to determine what systems or Cyber Assets are in scope and what requirements they must meet. The ESP is used in:

• Defining the isolation boundary between CIP-applicable Cyber Assets and Non-CIP Cyber assets, including the location where certain controls are applied, i.e. Electronic Access Points (EAP).
• Defining the scope of ‘Associated Protected Cyber Assets’ that must also meet certain CIP requirements.
• Defining the boundary inside of which:
• All of the Cyber Assets meet the requirements of the highest impact BES Cyber System that is in the zone (the ‘high water mark’) –or-
• Cyber Assets reside in security zones characterized by specific sets of controls applied to a class or classes of BCS according to risk or impact criteria

The CIP Cyber Security Standards do not require network segmentation of BES Cyber Systems by impact classification. Many different impact classifications may be mixed within an ESP. However, all of the Cyber Assets and BES Cyber Systems within the ESP must either be protected at the level of the highest impact BES Cyber System present in the ESP (i.e., the “high water mark”) where the term “Protected Cyber Assets” is used, or grouped into security zones with discrete security controls applied to each zone.

-The CIP Cyber Security Standards accomplish the high water mark by associating all other Cyber Assets within the ESP, even other BES Cyber Systems of lesser impact, as “Protected Cyber Assets” of the highest impact system in the ESP. For example, if an ESP contains both a high impact BES Cyber System and a low impact BES Cyber System, each Cyber Asset of the low impact BES Cyber System is an “Associated Protected Cyber Asset” of the high impact BES Cyber System and must meet all requirements with that designation in the applicability columns of the requirement tables.
-The alternative to the high water mark approach is to categorize BCS and Associated PCA into security zones according to their risk or impact and apply those controls which are required for the impact rating to the zone in which they reside. Security zones are isolated from other zones of differing risk or impact rating by controlling traffic, allowing only that which is explicitly identified as necessary.

If there is external routable connectivity to any CIP-applicable Cyber Asset, then an Electronic Access Point (EAP) must be identified where inbound and outbound access controls are applied to traffic traversing the ESP. Responsible Entities should know what traffic needs to cross an EAP and document those reasons to ensure the EAPs limit the traffic to only those known communication needs. These include, but are not limited to, communications needed for normal operations, emergency operations, support, maintenance, and troubleshooting.

The control strategy implemented at the EAP should apply to both inbound and outbound traffic. The standard added outbound traffic control, as it is a prime indicator of compromise and a first level of defense against zero day vulnerability-based attacks. If Cyber Assets within the ESZ become compromised and attempt to communicate to unknown hosts outside the ESP (usually ‘command and control’ hosts on the Internet, or compromised ‘jump hosts’ within the Responsible Entity’s other networks acting as intermediaries), the EAPs should function as a first level of defense in stopping the exploit. This does not limit the Responsible Entity from controlling outbound traffic at the level of granularity that it deems appropriate and large ranges of internal addresses may be allowed.

The SDT’s intent is that the Responsible Entity knows what other Cyber Assets or ranges of addresses a BES Cyber System needs to communicate with and limits the communications to that known range. For example, most BES Cyber Systems within a Responsible Entity should not have the ability to communicate through an EAP to any network address in the world, but should probably be at least limited to the address space of the Responsible Entity, and preferably to individual subnet ranges or individual hosts within the Responsible Entity’s address space.

The SDT’s intent is not for Responsible Entities to document the inner workings of stateful firewalls, where connections initiated in one direction are allowed a return path. The intent is to know and document what systems can talk to what other systems or ranges of systems on the other side of the EAP, such that rogue connections can be detected and blocked.
This requirement applies only to communications for which access lists and ‘deny by default’ type requirements can be universally applied, which today are those that employ routable protocols. Direct serial, non-routable connections are not included as there is no perimeter or firewall type security that should be universally mandated across all entities and all serial communication situations. There is no firewall or perimeter capability for a serial cable run between two Cyber Assetsand such a requirement would mostly generate technical feasibility exceptions (“TFEs”) rather than increased security. However, the technical security control does not need to be applied at the device level. The security control can be applied at the system level.

For example, a relay may be controlled via a serial connection (RS-232, etc.) from a device with an Ethernet port. This device, generally a terminal server or computer, may have multiple serial connections, a console port, and one or more Ethernet connections capable of interacting with a remote access client via a routable protocol. The terminal server generally has the capability to act as an AAA client, requesting authentication, authorization, and providing or participating in logging for Accounting purposes with a network based service or combination of services such as RADIUS, TACACS+ or LDAP directory services. This provides the opportunity for enhanced security such as multi-factor authentication which is typically not natively available in relays or terminal servers.

The security objective of providing controlled access and isolation is achieved by controlling the external addressability of the serial device rather than placing a security mechanism between the serial ports of the device and its immediate upstream serial controller. An IP-serial converter that has an Ethernet port outside and serial connection(s) inside is externally addressable, and on a practical level passes through that external addressability to the device receiving the serial connection. The control should be applied to the security zone where the Ethernet connection resides, upstream of the relay.

As for dial-up connectivity, the Standard Drafting Team’s intent of this requirement is to prevent situations where phone number alone can establish direct connectivity to the BES Cyber Asset.  If a dial-up modem is implemented in such a way that it simply answers the phone and connects the line to the BES Cyber Asset with no authentication of the calling party, it is a vulnerability to the BES Cyber System. The requirement calls for some form of authentication
of the calling party before completing the connection to the BES Cyber System. Some examples of acceptable methods include dial-back modems, modems that must be remotely enabled or powered up, and modems that are only powered on by onsite personnel when needed along with policy that states they are disabled after use. If the dial-up connectivity is used for Interactive Remote Access, then Requirement R2 also applies.

The standard adds a requirement to detect malicious communications for Control Centers. This is in response to FERC Order No. 706, Paragraphs 496-503, where ESPs are required to have two distinct security measures such that the BES Cyber Systems do not lose all perimeter protection if one measure fails or is misconfigured. The Order makes clear that this is not simply redundancy of firewalls, thus the SDT has decided to add the security measure of malicious traffic inspection as a requirement for these ESPs. Technologies meeting this requirement include Intrusion Detection or Intrusion Prevention Systems (IDS/IPS) or other forms of deep packet inspection. These technologies go beyond source/destination/port rule sets and thus provide another distinct security measure at the ESP.

VoIP Phones, BES Cyber Asset or not?

Over at the WICF forum, there's an older discussion on inclusion of VoIP phones as a NERC CIP-applicable Cyber Assets.

Folks keep trying to differentiate between VoIP and POTS (Plain Old Telephone System) analog lines. Here's a newsflash:

A PBX (Private Branch Exchange) system is certainly a programmable electronic device, whether it is VoIP or Analog. Even the majority of Analog PBXs have the ability to be administered remotely via Telnet or similar protocols. Unless the Responsible Entity has analog phone lines that are directly-supplied local loops from the Telco, they do have a programmable device in their voice system. Even then, the Telco switch is programmable, but would fall under a communications system exemption due to the RE not having control over it.

Here's the problem, a VoIP phone does have an OS/firmware that can be updated, including with a hacked copy. It's a simple TFTP operation. It has a configuration that can be modified by the administrator and to a certain extent by the end user. So do smart cell phones (I've used a jail-break to hack my own phone in the past) and even some relatively dumb cell phones.

What it probably doesn't have is the ability to directly affect the BES as long as it is not included inside an ESP with zero internal technical controls (legacy philosophy alert: hard crunchy shell around soft gooey center) that is the minimum acceptable solution at the moment. This could really be a fine example of a situation where short-sighted compliance actually reduces security by forcing you to include VoIP systems inside your trusted perimeter rather than keeping them properly segregated as they should be.

The current comment form on Virtualization is trying to address similar issues. The definition of Cyber Asset is up for discussion. It's also asking specifically whether CIP-005-5 ESP Requirements are adequate to address isolation in a virtualized world, so the parallel may not be obvious, but many of the risks identified for virtualization are unaddressed for nearly-identical risks that apply to physical Cyber Assets.

Bringing in the concept of security zones and granular internal technical controls applied to Applicable Cyber Systems by grouped risk or impact rating has a lot more implications than just for virtual Cyber Assets.

Virtualization Comment Form- Q&A Session

NERC has comments open right now on Virtualization and CIP Standards. Registered Entities, hie thee hence and make known your will. After all, you get the governance you deserve, not the one you want. Especially if you don't participate.

Here's more or less what we came up with in my neck of the woods:

Q1. Version 5 introduced the BES Cyber System concept, and requirements reference applicability at the BES Cyber System level. However, language in the measures shows that, implicitly, many controls are expected to be implemented at the BES Cyber Asset or device level. The SDT assumes that most auditors expect entities to demonstrate compliance at the device level. Do you agree with the SDT’s assumption?

A1. It has been our experience that guidance provided to auditors leads them to expect and look for controls to be applied to the Cyber Asset. Also, they seem sceptical of implementations where a given device performs a portion of the control function and additional components of the security strategy are implemented across multiple devices on the network. Auditors might consider only the device portion of an overall control and evaluate it outside of the network-based defense-in-depth strategy.

One way to address this inconsistency would be to normalize the use of the term “system” across the example measures rather than “device” wherever applicable. The SDT should add Guidance in the Technical Basis sections to clarify that defense in depth strategies are desirable. The EACMS paradigm should be revised in line with standard IT Security practice and terminology as performing Authentication, Authorization, and Accounting. It is also important to explicitly allow for distributed systems to perform this AAA function for a security zone rather than the legacy concept of hardened perimeter.

There may also be a need to revisit the RSAWs in light of system vs device to provide better guidance to auditors attempting to apply the questions in the RSAW to an entity’s particular security posture.

Q2. The SDT proposes that each virtual machine and hypervisor are separate Cyber Assets. 2. Do you agree with this position?

A2. The proposed definition of Cyber Asset must include virtual machines. Both “virtual machine” and “hypervisor” are  well-understood terms with formal definitions and broad IT Industry acceptance, thus do not need further definition in the NERC Glossary. We agree that each Hypervisor and Virtual Machine is a distinct Cyber Asset. Controls and strategies for securing virtual machines across a variety of industries have been published by agencies such as NIST and SANS.

The key issue the SDT appears to address in this revised definition is clarifying the scope or boundaries of a given virtual cyber asset in order to apply requirements and controls to each. Clarifying the definition is only necessary to address gaps in current requirements language that allow for mis-applying the requirement, not because Industry doesn't understand how to do it.

Q3. Do you agree that the proposed Cyber Asset definition clarifies the term programmable? Please provide a rationale to support your position.

A3: The SDT’s proposed definition clarifies programmable means that such a device is subject to configuration and software changes by the end user. This clarification and scope limitation is useful.

However, the device does not "consist of" all the data stored in the device. Data inside the device is peripheral and irrelevant to the operation of the device. It may be necessary to operation of the BES, but unless you want the BES to be one, singular BCS, it is ridiculous to scope it that way.

Much data inside the device is peripheral and irrelevant to the operation of the device. Even data which is required to perform the function of the device is generally not part of the device. This “data in the device” portion of the definition apparently is a mis-interpretation of wording from Section 215 of the Energy Policy Act of 2005 which actually reads: “programmable electronic devices and communication networks including hardware, software and data that are essential to the reliable operation of the bulk power system.” Even this is problematic in that standard IT Security best practice separates the concept of protecting a system (better known as Information Assurance, Source: NIST SP 800-50, CNSSI-4009) from the mechanisms of protecting data transiting or resident on that system (the latter being Information Security, Source: NIST SP 800-59; SP 800-53; SP800-53A; SP 800-60; CNSSI-4009; FIPS 199; 44 U.S.C., Sec. 3542). For example; a SCADA server can be deemed perfectly functional with no real SCADA data present. A newly-deployed SCADA server functions at the instant of being commissioned before real SCADA data is input.

The introduction of the concepts of management plane and data plane which are referenced in question 8 is a useful addition to the NERC CIP discussion because it enables appropriate controls to specifically protect systems or data.

Q4. Such (shared infrastructure) configurations are not addressed explicitly in CIP-005-5. Are modifications required to address the issue? 

A4. CIP-005-5 would benefit from being modified to a security objective-oriented standard rather than a requirements-based standard. The security objective in this case would be isolation of CIP-applicable Cyber Assets from Cyber Assets that are out of scope of CIP controls. The mechanisms of that protection are primarily Boundary Protection and Control of Network Ports, Protocols, and Services (SANS 20 Critical Security Controls).

CIP v5 narrowly focuses on routable protocols and Layer 3 controls and does not address the other layers of the OSI model. For example, under CIP-005-5 Layer 2 protocols are not addressed and can convey malware as well as allow information exfiltration and cyber-attacks even if no routable IP communications are present. Controls for these protocols should not be limited to or defined by Layer 3/4 ACLs on a firewall or router as the only or even the best means of achieving in-bound and out-bound access control. Entities need the opportunity to provide technical controls at whatever conceptual layer is appropriate to meet the security objective.

I recommend retiring the ESP construct in favor of security zones. When framed in terms of Boundary Protection, a security zone is more inclusive and granular because it is not limited to routable protocols at the OSI Model’s Layer 3. A security zone construct does not force any particular interpretation or control onto serial or other non-routable means of transporting data or accessing the management plane of any systems. Security zones apply to physical and logical separation equally. An example of logical isolation provided by other than ACLs would be when a hypervisor provides isolation between Guest VMs or between Virtualized Network Functions. This isolation is implemented in the control plane by means of logic embedded in code base (NIST SP 800-125A – Draft, Section 1.2: Hypervisor Baseline Functions) and not at a conceptual network layer.

Current CIP standards take a broad stroke approach by requiring the protection of all cyber assets within an ESP in a singular manner at the highest impact level of controls ("High Water Mark"). Further, High Water Mark is applied to the impact rating of the BES Asset or Facility, and not to the impact rating or risk level of the Cyber System itself. This is not cost effective or flexible enough for individual entities’ needs. Security zones provide a scalable means of appropriately protecting Cyber Systems of differing security risks by further isolation within the zone.

Q5. The SDT asserts that VLANs providing logical isolation are not addressed explicitly in CIP-005-5, and controls may be necessary to isolate BES Cyber Systems. Are the current requirements of CIP-005-5 sufficient to address logical isolation using VLANs? Please provide your rationale.

A5. The same modifications necessary to make CIP-005-5 adequate to question 4 would apply to addressing the specific question of 802.1 Q VLANs providing adequate logical isolation in question 5. The security objective should be to provide isolation by means of Boundary Protection as well as Control of Network Ports, Protocols, and Services. Legacy software vulnerabilities that have long since been patched or known exploits that are mitigated through proper configuration should not require the blanket rejection of VLANs as a component of a particular entity’s specific security scheme. Newly discovered exploits and vulnerabilities are addressed through CIP-007 testing and patching, CIP-010 baseline configuration, change management, vulnerability scanning and assessment.

Q6. Do you agree with the proposed definition of CMS (Centralized Management Systems)? If not, please provide alternative language for the definition and your rationale.


A6. The SDT’s proposed definition of CMS captures most types of systems that support automation with a large span of control and privileged access. A similar span of control risk exists in that EACMS is a type of CMS for electronic access but the term EACMS is specific to NERC CIP and used nowhere else in IT Security or Information Assurance in any other industry. This inherently limits the amount of expertise, guidance, and documentation available for solving the root problem of controlling access to CIP-applicable systems.

The SDT should retire the NERC CIP defined term Electronic Access Control & Monitoring System from the NERC Glossary and adopt the industry solution Authentication, Authorization, and Accounting System (AAA System). Non-standard jargon should be avoided when adequate terms and concepts exist already.

Further, the SDT should clarify in Guidelines and Technical Basis, that:

• AAA clients that subscribe to AAA services (e.g. via a protocol such as LDAP, RADIUS or TACACS+) but do not maintain any account information are not AAA Systems in themselves
• Remote access clients or terminal emulators that are used to connect to a CMS, are not a CMS in themselves

Q7. Do you agree with the SDT’s approach to reference the CMS specifically as a type of applicable system in the CIP standards? Please provide your rationale.

A7. The inherent risk in such (centralized management) systems’ span of control and privileged access to CIP-applicable Cyber Systems should be addressed in support of the security objective of protecting BES Cyber Systems from threats in the data plane and isolation of the management plane (out of band management).

Q8. Do you agree with the SDT’s approach to require the isolation between the data plane and the management plane?

A8. Cautiously agree with the SDT’s proposal to require isolation between Data Plane and Management Plane for centralized management systems when system capability allows and risk justifies it (i.e. Hgih and Medium Risk Control Centers). We caution the SDT against overly rigid prescriptions for providing isolation. Combinations of other controls may afford the same or better protection in a particular circumstance. When the use of automated tools can improve security and manageability, it is important to avoid discouraging automation with overly burdensome compliance requirements.

Q9. Do you agree with limiting the applicability to high and medium impact Control Centers?

A9. Limiting applicability only to those facilities such as High and Medium Control Centers with the highest level of risk is reasonable, and there may be exceptions to those as well. Combinations of other controls may afford the same or better protection in a particular circumstance.

Streetlight Effect Apparent in NERC CIP Requirements

This comment is in response to an article in the ReliabilityFirst Newsletter - "Virtual Systems and Zones of Authority".

I'm going to quote a bit liberally, since the original on page 7 is difficult to link to directly. But first, read this:

https://en.wikipedia.org/wiki/Streetlight_effect

"Responsible Entity personnel see their entire network as a whole, with the parts of that network subject to CIP compliance as part of a larger security picture. They can see the protections afforded to all systems, and can see how the protections applied to non-CIP assets increase the security of CIP assets as well.

CEA personnel, on the other hand, only have CIP assets within their purview. They cannot consider non-CIP assets as adding to the entity's security posture, as those assets are not under the CEA's regulatory authority. Non-CIP assets are not subject to audit by the CEA and may change at any time with no notification to the CEA.

This difference in viewpoint can lead to conflicting views of virtual systems such as virtual networks. Responsible Entity personnel see the protections applied to the non-CIP networks that might share, for example, a physical switch with CIP networks. They see the multiple layers of protection and the controls surrounding the security of these non-CIP networks.

CEA personnel, on the other hand, do not have the authority to review the security level of non-CIP assets or networks. The CEA personnel must therefore assume that any non-CIP assets or networks could be compromised and used in attacks on the in-scope CIP assets and networks. The resulting differences in the perception of risk can be a source of misunderstanding between Responsible Entity personnel and CEA personnel.

I call this difference in perspective “mixed zones of authority.” The Responsible Entity's zone of authority is all of its owned assets, both CIP and non-CIP. The CEA's zone of authority is limited to assets that are in scope for CIP.

For this reason, and others that I don't have the space to go into here, I strongly recommend that

Responsible Entities refrain from implementing Cyber Assets or networks that mix CIP in-scope and out-of-scope assets, network traffic, or data. The reason for not mixing in-scope and out-of-scope is not, as is commonly discussed, that “untrusted” configurations are implemented.

The biggest issue, in my view, is that without being able to view all aspects of the systems used for BES reliability, there is no way for the CEA to ensure that weak or high-risk configurations are not implemented."

{emphasis added in the last two paragraphs.}

That’s an interesting perspective. Once again, the crux of the argument is not whether or not virtualization adds to or subtracts from security. It’s the difficulty of auditing that drives the recommendation.

This, my friends, is a terrible basis for driving standards. It is what is known as a perverse incentive. It drives one to make decisions that, on the basis of achieving the objective, one would not otherwise make. It's a common problem in security, that one does what is visible, and easy to measure rather than closing the worst (invisible) gap. In fact, this is a problem far beyond security, it happens in all kinds of production environments with easily gathered statistics, in management, etc.

Standards should not be about making the network easy to audit, it should be about making it difficult for a bad actor to compromise reliability and security. Industry knows how (or individual entities can quickly learn how) to secure a virtual environment in shared infrastructure mode. Industry properly securing the virtual environment isn’t the sticking point. Even the Federal Government's more security-conscious entities have processes for contracting compliance in Cloud Computing. Surely an entity is capable of adequate control in its own networks.

Take a look in the PCI standard and its glossary. The concept of trusted and untrusted network is defined there as:

Trusted Network	Network of an organization that is within the organization’s ability to control or manage.
Untrusted Network	Network that is external to the networks belonging to an organization and which is out of the organization’s ability to control or manage.

There is no logical basis to assume that an entity capable of providing a compliant solution inside their ESP is simultaneously unable to or unwilling to provide security to Cyber Assets under their own control outside the ESP, simply because the auditing agency doesn’t have explicit control over this latter subset of Cyber assets. Requirements-based language may be the problem, where an objective-based standard wouldn’t have quite the same problem. For example, proof that you meet the objective of “isolation” would tend to consist of controls and measurements applied inside, outside, and on the perimeter. It doesn’t require the bright line of layer 3 “ESP” as the sole measure of compliance and it works well with security zones as a concept.

As long as Critical Infrastructure Protection is driven by compliance rather than security, the Grid will be at unnecessarily elevated risk.

Your humble author participated in this webinar, presented here for your viewing pleasure:

Slides and Recording Posted Industry Webinar Project 2016-02 Modifications to CIP Standards Virtualization in the CIP Environment Webinar Date: March 21, 2017
Click here for Slide Presentation Click here for Recording
For more information or assistance, contact Standards Developer, Mat Bunch (via email) or at (404) 446-9785.
3353 Peachtree Road NE Suite 600, North Tower Atlanta, GA 30326 404-446-2560 | www.nerc.com