Master Planning: The First Step to a Robust SCADA System

Supervisory Control and Data Acquisition systems are an asset that can be used to promote safe, efficient operations and continuity of service to utility customers. As an asset, they require maintenance and renewal programs to stay current, help address utility challenges and mitigate vulnerabilities introduced as systems grow over time. 

This article explores the use of SCADA master planning as a tool to establish programmatic improvements to address challenges faced by most utilities in maximizing the use of and maintaining their SCADA system assets.

When SCADA System assets are not maintained, compounding issues impacting the security, resiliency, reliability and optimization occur over time. 

This relates to a higher cost of operation and maintenance as resources are diverted to react to issues rather than permitting utility owners to focus on normal operations, process improvements and preventative maintenance.

Without a SCADA system master plan, the problems of today become the problems of tomorrow.

The Problem

SCADA systems are not typically treated as an asset requiring maintenance, updates and renewal. As a result, what is installed exposes utility owners to vulnerabilities that may threaten the security and reliability of their systems, as well as their ability to overcome failures that can shut down their operations and present risk to personnel and public safety.

Some of the global challenges impacting utilities include:  

• System hardware is obsolete with limited available supply and support increasing the risk of failure and cost of maintenance.
• A leaner workforce requiring mobility to stay connected with the control system to perform their duties.
• Protection of the control system network from attacks that may result in costly losses to operations or threaten the security of staff and the public.
• Software is not routinely updated or obsolete resulting in security risks.
• Inability to efficiently recover after a catastrophic system failure.
• Inundated with alarms requiring operations to manage by exception impacting efficiency and safety of equipment and personnel.
• Inconsistent delivery of design or implementation of projects through documented and enforceable standards resulting in a patchwork system that is difficult to operate and maintain.
• Not using the installed or available technology to optimize operations and improve efficiency.
• Limited use of available data to inform operational and business decisions through situational awareness.
• Secure integration of operational data into business systems to optimize enterprise efficiencies.
• Lack of documentation to support knowledge transfer and training as an aging workforce prepares for retirement.

Often owners find that they have limited planning in place to address these global challenges and that as their systems have grown and changed over time, the proposition of taking on all of these challenges can be intimidating.

To introduce more complexity, these challenges are relational in that addressing any one of them may have unintended consequences on the others, requiring a holistic approach and consideration for sequence of events.

A master plan that identifies the current state of the installed system, defines the requirements for the desired future state, and identifies the steps and cost to achieve those requirements is crucial to addressing and overcoming the challenges that threaten the optimized operation, reliability, resiliency and security of a utility’s SCADA system.

A tried-and-true approach for master planning includes the following five major tasks:

1. Document the current state of the system.
2. Define the requirements for the desired future state of the system.
3. Identify the gaps between the current state and the requirements, determine how to close those gaps and define the scope of improvements to reach the desired future state.
4. Consider alternative approaches where multiple viable methods exist to overcome the gaps and achieve the desired future state.
5. Develop a master plan that includes prioritized improvements with dependencies including schedule and anticipated cost.

Current State

The foundational step of a successful SCADA master planning effort is documenting the current state of the installed system. SCADA systems can be geographically dispersed and include many assets, so documenting the current state can require significant effort. An extensive current-state assessment may take several months to complete depending upon the size of the system.

A typical current-state assessment has two milestones: data collection and data evaluation. During data collection, information is collected that best represents the installed or current state of the system. This is a hands-on effort that requires an investigation of SCADA system components including:

• Available documents such as piping and instrumentation diagrams, panel drawings, Input/Output lists, loop drawings, network diagrams and control descriptions.
• Controllers and control panels.
• Network equipment such as switches, routers and radios.
• Workstations and servers.
• Installed software including Human Machine Interface, historians, asset inventory, antivirus and operating systems, and how they use and share data.

All subsequent master planning tasks will build upon the information collected during this step so spending time to collect and organize information for reference is critical.

Do not rush data collection. Establish an organized collection process and where possible use technology and data collection tools to develop relational databases and reports to share information for ease of future reference.

If possible, the date of installation and condition of equipment should be determined during the investigation. This information is used to determine the position of the installed hardware and software components in the product lifecycle and to determine obsolescence. The condition is used to determine the extent of future improvements. For example, if a controller requires replacement due to obsolescence or planned obsolescence and the panel is in poor condition, future improvements to replace the controller should also consider the replacement of the associated panel.

It is also important during this step to discuss how members of the utility’s team interact with and use SCADA to perform their work. This is typically done in an interview setting with members of operations, maintenance, engineering, information technology and management.  

During the interviews, information is gathered on how data is used by staff, what features users feel are broken or what can be improved to help staff perform their duties most efficiently. 

Evaluating the data collected is primarily a desktop exercise. Some of the objectives of the data evaluation include:

• Determine hardware and software position in the product lifecycle so that need for replacement can be planned.
• Determine how data flows in the SCADA system and if best practices for securing the control system are in place.
• Document the baseline system architecture.
• Identify vulnerabilities in the overall system architecture that present security or reliability risks.
• Determine how the system operates and identify opportunities for optimization.

Defining the requirements for the future state of the SCADA system is a collaborative effort and is a pivotal step in the planning process. 

The data evaluation is used to benchmark the utility system against best practices that are employed at similar utilities and other industries. To name a few, the current-state assessment should include sufficient data collection and evaluation to create a benchmark for the utility in the following categories:

• Human-machine interface system architecture
• Control philosophy
• Data management
• Network architecture
• Mobility
• Cybersecurity
• Automation and predictive analytics
• Alarm management
• Reporting
• Integration with business systems
• Disaster recovery
• Maintenance and renewal

Once the current-state assessment is complete, and the benchmark is established, the next step is to determine the requirements for the desired future state of the SCADA system.

Define the Requirements

This step of the master planning process establishes the vision and objectives for the future-state SCADA system. 

In most cases, utilities will consider how they compare to other utilities in the benchmark, however many will innovate to solve problems that are unique to their operations and service to their customers.

The vision and objectives are typically established in concert with members of the utility’s organization that will approve the expenditures for improvements identified during the planning process. 

If any improvement does not meet the objectives defined to achieve the utility’s vision, then it does not make it into the plan. This visioning process is extremely important to align improvements that will be approved and receive funding to achieve the organization’s vision.

Defining the requirements for the future state of the SCADA system is a collaborative effort and is a pivotal step in the planning process. The success or failure of the SCADA master planning process and successive plan execution is dependent upon this initiative. 

Typically, success or failure hinges on stakeholder involvement. The larger the sample size of stakeholders, the better. 

In other words, obtaining input on how the SCADA system can be changed to improve the interaction and use of the system should be gathered in 
a collaborative setting with as many utility stakeholders as possible including representatives from operations, maintenance, information technology, operational technology, engineering and management.

Requirements for how the system will look and operate in the future should be defined for each of the categories in the current-state assessment benchmark. 

A first pass priority for each improvement is assigned at this stage of the planning process and typically includes the following classifications:

• Critical: The requirement is critical to operational mission of the utility and must be addressed immediately.
• High: Essential to the operational mission of the utility and must be addressed in the near future.
• Medium: Important requirement that does not directly impact the operational mission but may have operational, economic or management benefits.
• Low: Nice-to-have features but not operationally critical.

Once a list of prioritized requirements is defined, the next step is to complete a gap analysis.

Gap Analysis

Gaps are obstacles that must be overcome to transition from the current state to the desired future state as defined by the requirements to achieve the utility’s vision. This step identifies the gap and defines the work required to leverage or improve the current state to meet the requirements of the future state.

Classification of the gaps identified may look different for each master plan, but in general, gaps typically fall within one of five categories as defined below:

• Present: The requirements to achieve the desired future state can be met by the current assets without significant modification or with minor adjustments. No significant investment is required. An example is introducing alarm deadbands through configuration of the existing controller programs to dampen signals and reduce nuisance alarming.
• Expansion: The requirements can be achieved without changes to the system’s underlying infrastructure, but expansion is required. Moderate investment may be required. This could represent the addition of remote terminal units or remote I/O cabinets and wiring to add monitoring and control of existing processes without reconfiguration of the process or communication infrastructure.
• Modification: Changes to the infrastructure are required to meet the requirements. Significant investment may be required. For example, modifying the design of the SCADA system network architecture to add a process information network or buffer between the business and SCADA system networks with tier-2 historians for securely sharing data with data management and maintenance management systems.
• Infeasible: Cannot meet the requirement without unjustifiable cost, effort or resources. This typically applies to potential alternatives and could represent adding miles of fiber to replace radio infrastructure for a large geographically dispersed collection of remote sites that have already been constructed and are currently in service.
• Other: Requires effort, either not classified by expansion or design, or requires deferment to additional study to make a recommendation. For example, establishing criteria for organizational changes to add an operational technology group to support the utility’s SCADA system.

Once the gaps are classified, preliminary scopes are defined for short-term and long-term improvements, and potential costs are defined with a breakdown of management, engineering and construction/implementation costs. Priority assignments from the requirements step should be retained.

As the gap analysis is concluded, alternatives may be considered as a value engineering exercise to maximize return on investment.

Alternatives Analysis

Not all SCADA master plans will require an alternatives analysis. This analysis represents an additional study step for the master planning process typically to inform the utility on available options or paths to achieve a requirement, or to evaluate alternatives for gaps classified as Expansion or Infeasible. Examples may include telemetry alternatives to compare infrastructure technology to achieve communication with remote sites, or comparison of hardware or software platform alternatives that will provide the best option for supporting future features to protect and maximize the investment.

Once the alternatives analysis is completed, modifications to the gap analysis may be required to adjust scope and cost for impacted improvements. At this stage of the planning process, formal definition of an actionable plan can begin.

Develop Master Plan

Each step of the master planning process builds on the previous step. Benchmarking drives the definition of requirements and the requirements drive the definition of improvements which work together to form the basis for the master plan.

In this step, coordination of funding requirements, possible execution strategies and delivery models is paramount to developing a kinetic plan. In some cases, the anticipated cost of improvements is used for funding requests, or maximum year-over-year funding may already be defined in which the plan will be established to best use that funding and determine the duration of the plan.

Consideration is also given to planned capital improvements that can be utilized to execute elements of the SCADA master plan or to determine if additional capital improvements will be needed to achieve defined improvements. 

As funding is considered, priority assignments for identified improvements are revisited and finalized with the utility management team and improvements are grouped for execution based upon dependencies and available funding. A schedule is developed and year-over-year cost accrual with recommended escalation is further defined. 

Master plan execution durations of five to 10 years are typical to allow time for phasing and investment for both near-term and long-term improvements, and to minimize the impacts of changes in technology during execution of the master plan.

Once this step is complete and the plan is accepted by the utility, then execution begins. 

Critical for Success

The cost to develop a typical master plan can be approximately 1% – 5% of the overall SCADA improvements program value, depending upon the size and complexity of the utility’s system, and the planning process requires a significant investment of time by the utility to complete.

For this reason, the ultimate goal from the onset of the master planning process should be to develop an executable plan rather than a report that sits on a shelf collecting dust. If the focus is only on singular improvements rather than a program, a master planning effort will not be beneficial and may be more of a hindrance than a help.

Once the decision is made to proceed with a master planning effort, key factors to success include:

• Experience: Whether self-performed by the utility or led by a third party, make sure the plan is led by a team experienced with program planning and implementation.
• Core Team: The utility should define a core team that will be involved in all aspects of the master plan as representatives of SCADA users and stakeholder groups for the duration of the planning effort to provide governance throughout the process.
• Stakeholder Engagement: An executable master plan can be achieved only if the utility engages as many team members as possible that use SCADA to perform their duties. This promotes ownership and consensus that is important for success of the master plan.

Conclusion

To avoid having the problems of today become the problems of tomorrow, it is important for utilities to recognize that their SCADA system is an asset that is critical to the success of their operations and continuity of service to their customers. This asset not only requires attention and investment to avoid costly outages and obsolescence cliffs, but also to capitalize on untapped potential to improve overall operations. 

The SCADA master plan is a tool that starts as a road map to address challenges faced by utilities, but it should not end there. Master planning should become an integral part of the organizational structure as a continuous process requiring renewal and maintenance that does not end after development of the initial master plan.