Substation automation: fundamentals of substation automation

Traditional substations (mechanical relays, limited visibility, first RTUs with IO)

High availability and constant operation of an electrical substation has always been the focus of an electrical company. More faults mean more interruption of service to clients and it translates to less revenue that is not desirable to any company. From the early age of electrical systems, engineers and operators have always been interested in collecting useful information on different devices in a substation so they can better evaluate the health of their system, predict potential problems and – in case of a fault – to analyze and troubleshoot the problem as soon as possible to protect their high values assets and to improve their continuous service to their clients

Early substations consisted of mechanical relays and meters that barely supported recording and had no means of communication. Fault recorders were capturing information mainly in the form of paper charts so reading and analyzing the information was not a straightforward process.

Lack of communication caused any maintenance or troubleshooting to be long and costly because personnel had to be sent to substations that were often far away and hard to reach.  

Modern substations

With the introduction of microprocessor technology, digital protection and control devices became more intelligent. New intelligent electronic devices (IEDs) can collect and record information on many different parameters of a system, process them based on complex logics in a fraction of a second and make decisions on abnormal situations to send control commands to switches and breakers to clear the fault.

In addition to their superior processing capability, modern substation devices can also hold information in their internal storage for a certain period and transfer this information to third-party applications for further study and analysis. IEDs can now send information to a local or remote user via different types of communication. This gives operators more flexibility on how and when to process the information to provide a fast recovery time from an interruption in the substation.    

With more information remotely available, new supervisory systems were developed to facilitate the task of a system administrator in the control center. A Supervisory Control and Data Acquisition (SCADA) system can collect information from various IEDs in an electrical system via different methods of communication, control and monitor them using different visualizing technologies and even automate the supervision task based on predefined parameters and algorithms.

A Human Machine Interface (HMI) is deployed in each substation to provide operators with local control and monitoring capabilities which are often necessary during the configuration, commissioning or maintenance of the substation. 

Digital substation, autoconfiguration and standards

Digital control and protection technology has been evolving since the first introduction of digital devices. The more intelligent and capable the devices became, the more responsibilities that tended to be transferred from human to the device. Unlike early digital technologies where an operator had to work with bits and bytes on a primitive user interface to define every parameter of the system and make sure all elements of the system are correctly configured to make the processing and communication work, new technologies let users focus more on high-level aspects of the system architecture by taking care of the tedious task of defining every single detail in the system configuration.

In the beginning of the digital era each manufacturer had its own way of interpretation and implementation of different elements in an intelligent system which led to the lack of interoperability and caused vendor dependency.  New standards have been developed to make sure devices from different vendors will function in the same predefined way. This gives users more flexibility and freedom to choose functions that suit them better without having to focus too much on manufacturer.    

Although remote access to information provides operators with much more visibility to the system, it also introduces new concerns and challenges. Having information exchanges with remote entities and often via shared media, makes cyber security one of the most important considerations in any system deployment.  

Big data, non-operational data processing

In the earlier years of digital technology, limited data points were available on each device and the high cost of communication as well as slow data exchange rate would render it impractical to collect high amount of data from each substation. Only necessary operational data was sent to control centers and communication lines were cautiously programmed to minimize the bandwidth and communication cost.

Rapid evolution of communication and process technologies now offers system administrators the luxury of polling more and more operational and non-operational data points from their substations. This information can now be processed in a variety of ways using different software to more efficiently monitor an electrical system - providing for improved sight on overall health and useful information for other applications such as condition-based maintenance and asset monitoring.

Traditional protocols (DNP3, MODBUS, proprietary)

Most of the early protocols in the electrical automation industry were proprietary protocols developed by device manufacturers. Although proprietary protocols work especially well with the devices from the same manufacturer, lack of interoperability along with vendor dependency pushed electrical companies toward standard and open source protocols. Today, device manufacturers have adopted popular standard protocols and proprietary protocols have been almost completely phased out from the industry. Like other protocols, substation automation communication protocols have evolved along with communication infrastructure improvement. Unlike slow and error-prone older protocols, newer protocols can deal with different communication mediums, recover from communication sever failures and deliver information in a more robust way.

Although older protocols like MODBUS are still used in substation automation, most of the systems has already adopted protocols like DNP3 (North America) and IEC 60870 (Europe) as their de facto default protocol.

In addition to the set of rules, control headers and error recovery mechanism, traditional protocols also define the structure of a “point list.” A point list is the list of all data points the communication parties want to exchange as well as additional information such as point address and point type. The point list is defined during the configuration of a communication instance and will be deployed in the devices that will use that communication instance.

The most deployed protocol architecture in substation automation industry is the Master-Slave (server-client) architecture where one or several devices called slave (or server) are polled by a master (client) device or software in some predefined intervals. in some protocols, slaves can also initiate the communication to send information to the master using a mechanism called “unsolicited response,”

Although traditional protocols require more time and effort during configuration and commissioning, they are popular in the automation industry because they are easy to understand, configure and troubleshoot. 

IEC 61850

Faster and more reliable network infrastructure opened the possibility of implementing higher-level protocols that make the task of configuration, commissioning and testing easier - even though the protocol itself is more complex. These newer protocols tend to move from an IT-oriented paradigm to an OT-oriented paradigm where users focus mostly on “what” a device should do rather than “how” it should do it.

Parallel efforts started in the early 1990s to develop an object-oriented protocol that focuses more on the actual functions and information of a devices rather than low-level implementation detail such as register addresses and data type.

Interoperability was another main force behind new protocol developments as utilities tried to move toward vendor-agnostic solutions. New protocols should make sure devices from different vendors would be able to exchange information with the least amount of configuration.

The IEC 61850 standard was accepted by a majority of the utilities as a modern protocol that can address the shortcoming of the traditional protocols. Unlike older protocols, IEC 61850 is more a suite of standards that address different aspects of a modern substation rather than just a communication protocol. It defines in detail a standard model for each function in a substation plus the communication standards to support such a model as well the methods on how the map this model into the lower level communication. IEC 61850 also addresses necessary hardware requirements for a substation-grade device and defines a communication language that can be used to exchange a substation or a device model.

Although traditional protection systems tend to be completely separated from the automation and control system and are still relying on dedicated hardwire signals between CTs and PTs and relays, IEC 61850 presents a system model where protection data points could be exchanged on a shared Ethernet link. It implements necessary measures to make sure this information will be delivered in a deterministic way within a redefined time period.

GOOSE and Sample Values concepts in IEC 61850, define the object models and communication criteria that can be used to exchange protection information (e.g. voltage, current, breaker status) over a dedicated ethernet link called Process Bus (in less than 4 ms to comply with protection system time constraints). This reduces the amount of wiring in a protection system because all the wires between CTs, PTs and protection relays can now be merged into one Ethernet cable.

IEC 61850 also includes testing methods a user can refer to during the commissioning or maintenance phase of a project to make sure all devices are functioning according to the requirement of the project and to isolate problems during a troubleshooting session.

Being connected to a substation and retrieving vital information from remote devices has always been a challenge for system designers. Not all the substations are of the same size or importance. Many are in remote areas where communication could be the biggest challenge when it comes to monitoring a substation.

Early remote monitoring started by using modems on telephone or leased lines. At the time when most of the devices in the field had very limited communication capabilities, these methods of communication were sufficient for most situations. Initial efforts were made using a gateway device in the field that would concentrate information it was receiving from serial devices and would send information to a master station using a modem based on a predefined time schedule. Concentrating information would improve the communication (and the cost) as sending data points in one burst would reduce the communication time as compared to sending small data packets from different devices over a longer time period.

In modern substations, most devices communicate via Ethernet links. Data from different devices is sent to the control centers via various communication mediums. Utilities generally prefer to install their own inter-sub communication infrastructure using fiber-optic links or radio mesh systems but, in some cases, especially in remote areas or smaller substations, using cellular modems becomes more practical.  

Although using a public infrastructure such as cellular networks can reduce maintenance costs, it also raises new concerns about security and availability. 

Redundancy

Although redundancy is not a new concept, new technology is easier to implement and manage redundant devices. In a hot standby configuration, two devices (e.g. a gateway) can be configured in a group where one acts as ‘’active’’ while the other one stays in ‘’standby." The standby device constantly monitors the status of the active device while the active device receives information from other parties, updates its internal database as well as the database of the standby unit and sends information to one or multiple clients. If the standby device detects that the active unit is not communicating anymore, after a predefined period of time it assumes that it is not functional anymore and takes over the control and continues sending/receiving.

Sophisticated redundant systems also support virtual addresses. A virtual address will hide physical device addresses so this transition stays transparent as long as the virtual address is used for the communication.

Time synchronization

Time synchronization devices and methods like GPS clocks and IRIG-B signals have been used in substations for quite a while. The goal is always to keep the internal clock of the devices in a system synchronized so timestamps from different sources can be precisely compared in a system analysis. Time synchronization is also critical in a protection system.

New substation automation technologies offer new methods of time synchronization. Unlike older systems where the time signal was distributed using hard-wired links (IRIG-B wires, serial cables), new time protocols leverage the communication infrastructure to distribute time signals. Some communication protocols (e.g. DNP3 or IEC-104) as well as NTP (Network Time Protocol) and SNTP (Simple Network Time Protocol) can already provide sufficient precision for many applications. However, the high-time precision that is required in mission-critical applications cannot be achieved using this methods.

In recent years, Precision Time Protocol (IEEE standard 1588) was introduced to leverage existing network infrastructure to provide sub-microsecond time accuracy for devices in control and protection systems.

System logs, event files

With more processing power and internal storage, IEDs can now produce more information on their internal activities. Logging this information and sending them to a control center will give operators more visibility on what happens in the device and, in case of a problems, gives hints on where to start the troubleshooting. A Syslog server can also be installed in the system to collect these log files from different IEDs and store them in a central repository for further analysis.

Event files can also be generated based on some changes on internal status of a device or data points. For example, an oscillography file captures the value changes on some system parameters (e.g. voltage, current, phase angle) during a fault. Analyzing such a file provides system engineers with valuable information on the status of the system right before and after a fault occurs.

Like log files, event files can be collected centrally and made available for a wider range of users. It also makes sure information won’t be lost from the devices as they still have limited storage space as compared to a local server. 

• Gateway (automation platform)
  • Data concentration
  • Protocol translation
  • Data distribution
  • Logic processing
• Distributed IO
• IEDs (protection relay, smart meters, etc)
• SCADA
  • Single-line diagram
  • Real-time trending
  • Historian
  • Event and alarm management
  • User notification

• Substation HMI
  • Single-line diagram
  • Alarm management
  • System health information
  • Commissioning tools

Data concentration

As a data concentrator, a gateway can collect information via several serial or ethernet ports from the devices in a substation and make them available to remote users. Although the data concentration function is not as important as previously, it still adds a lot of flexibility to the system. This is especially so in cases where a cellular modem is used as the link to a remote substation. Concentrating information and sending in a chunk instead of individually collecting information from the IEDS reduces modem usage and lowers communication costs. A data concentrator can also offer more storage to maintain log and event files as compared to internal storage of the IEDs.

Using a data concentrator also simplifies system configuration on the SCADA side. Instead of individually setting up the devices in the substation in the SCADA, only one gateway with one communication link and one set of points is required to be integrated in the SCADA. When adding, removing or changing a device in the field, the SCADA system only needs to update the  point list without changing the communication link parameters.

Protocol translation

As a protocol translator, a gateway can receive information from different devices via different protocols, translate the inputs in another protocol and send it to local or remote users. Although growing use of standard protocols reduces the need for a protocol translator there are situations where utilities still have installations with legacy devices, but they need to upgrade the outbound protocol for performance or security reasons. A protocol translator can facilitate such an upgrade by keeping the legacy devices intact.

Data distribution

Once data points are concentrated in the gateway, they can be available to various remote and local users via different protocols. This feature of the gateway is especially useful in cases where a device has limited outbound communication. Different users with varying interests may want to access the same device at the same time. 

Logic processing

Since a gateway collects data points from different devices in a substation, it is the ideal place to implement some logic for control and operation reasons. Using a well-known programming language like IEC 61131, input points can be created and output commands can be issued based on some predefined logics. These points can also be sent to a control and monitoring system in the master station. 

Distributed IO

Although the terms RTU and gateway are used interchangeably these days, first-generation RTUs were fact devices with limited communication capabilitiesused to convert hardwired signals into digital binary or analog data points. They generally had high IO capacities as most of the system parameters were not yet available in digital format and they communicated over serial links.

With the evolution of digital relays, most system parameters became digitally available directly from the relays and meter-over-ethernet links and via new protocols and the demand for high capacity RTUs was reducted. However, there are still some hardwire signals (e.g. breaker monitoring and control, cabinet door safety switch, transformer oil gauge) that need to be monitored or controlled by remote users – sometimes even separately from the protection system.

A distributed IO device can convert a limited number of Input/Outputs into digital values and communicate them via a standard protocol through serial or ethernet links.

IEDs (protection relay, smart meters)

An IED (Intelligent Electronic Device) is a microprocessor-based device with some processing and communication capabilities. The biggest category of IEDs in a substation is protective relays. This device can receive information from CTs, PTs or other type of sensors, make control or protection decisions based on some algorithms and issue commands to other devices such as breaker and switches. Although sensor signals are still mainly in hardwired form, modern IEC 61850-based substations can communicate digital information between sensors and relays using Sample Values or GOOSE protocols. A digital relay can also generate and save log, event and oscillography files.

Digital meters are another type of IEDs that can measure and record main system parameters and communicate them to a control center. 

Single-line diagram

A single-line diagram is an interactive graphical representation of the grid system via which an operator can monitor different parameters of the system and issue commands in case it is necessary. A SCADA single line diagram generally consists of an overview of the system plus multiple detailed pages for different components of the system to which an operator can navigate. 

Real-time trending

Unlike single-line diagrams that show the components and connection of the system, the real-time trending function provides the operator with a real-time chart that monitors the values it receives from devices in the substation. An operator can add one or several points to the chart and follow the real-time value changes for better analysis of the system.

Historian

Recording information is another important function in a SCADA system. Except for some buffering capabilities, most IEDs and gateways have insufficient internal storage to maintain a record of real-time value changes for an extended period of time. One of the main tasks of a SCADA system is to record the real-time values it collects from the devices in the field. This information is saved in a relational database and can be surveyed based on different filters using the historian function. The recorded information can also be accessed directly from the database using a third-party application for further analysis.  

Event and alarm management

Event and alarm management is also part of the standard functions offered by a SCADA system. An alarm can be raised by the SCADA system in an alarm window based on predefined criteria. The operator can then acknowledge the alarm and clear it when the value of the point the alarm was created on goes back to its normal status.

Like alarms, events can also be generate based on the status of the data points collected from the field. Contrary to the alarm management system, an event management system doesn’t require an operator’s intervention – as generally events are not considered critical.

User notification

One of the main tasks of a SCADA system is to provide necessary information to the right people in a timely manner. In a new SCADA system, the software administrator can assign notifications for different alarms and events to specific users or group of users and send them email or text message notification based on that list. 

Single-line diagram

The concept of the Single Line Diagram in a HMI system is the same as in a SCADA system. A graphical representation of the system helps the operator to visually investigate the current state of the substation and to send commands to the control. A HMI has fewer single-line diagram pages as it only needs to represent its own substation.

Alarm management

Except for the fact that HMI alarm management only takes care of local alarm and events, the rest of the functionality is similar to a SCADA system alarm management system.

System health information

Another function of a HMI system is to show some form of a summary on the general health of a substation. Information such as number of successful and failed communications, gateway CPU and memory usage, software version can be shown in a graphical form so an operator can evaluate the overall condition of the system at a glance.

Commissioning tools

Commissioning tools is a set of tools on a HMI system that offer an operator different functions that can improve or speed up the process of testing and commission during an installation or troubleshooting session in a substation.

Showing real-time value of the points received by a gateway, the ability to simulate some values and other functions such as reading logs or events can always provide a better insight into the current status of the system and potential faults that may occur during the operation.

In the early years of digital communication, substations were either connected through low-bandwidth links or not connected at all. Only a small set of information was sent to the control center due to the communication limits and this information was only used in real-time control systems. Devices in the substation had limited processing and communication capabilities and only provided the necessary information required by the control system.

The rapid growth in communication and processing technologies, changed digital devices into intelligent units capable of sending various information on a fairly high speed. Communication links between control centers and substations can now carry a large amount of information with a lower cost, so control systems have access to a rich set of operational and nonoperational data they can use in many different paradigms.

Unlike early digital control systems, control centers are no longer only interested in operational data. The nonoperational data collected from the substations can now be fed into many different applications to predict and prevent future errors, provide improved insight into the fleet of devices, manage devices in a more secure way and limit operator direct access to the devices or decrease maintenance on-road time.

Firmware update

Unlike old mechanical devices, digital device health is highly dependent on its firmware health. IED firmware changes over time as new features or bug fixes become available, so utilities may need to update the firmware version of their IEDs. Updating firmware on a big fleet of devices from various manufacturers is quite a challenging task. An application that can automatically monitor firmware and do batch updates in case of a new release saves a lot of time and effort and prevents errors.

Secure remote access

These days, most digital devices can be remotely accessed by an operator during a maintenance or programming session. Utilities also need to track and log these accesses, especially for security auditing or troubleshooting. Native Vendor Tools (NTV) provided by IED manufacturers often have limited log capability (if any) and that doesn’t meet the requirements for an enterprise-level system.

A central remote access system provides secure remote access to a device while it logs every exchange between the user and the device and saves this log in its internal database for further investigation if necessary. This type of system can also leverage a central authentication system such as MS Active Directory to authenticate and authorize users before they can access a device. 

Configuration tracking

Keeping track of the configuration of the devices in a small substation is fairly easy. On the other hand, keeping track of all devices in a utility with many (often remote) substations is quite a big challenge. With standards such as NERC CIP that need every configuration change to be reported, companies need an enterprise-level application that can automatically monitor the configuration of each device in the system and notify the right people in case of a change. A user should also be able to compare two configuration files to better investigate the changes and to spot unauthorized modifications, if any.  

Event and oscillography files

Right before and during a fault, data fault recorders generate event and oscillography files. These may help a protection engineer to better investigate the cause of a fault and take necessary measure to readjust the system to prevent  future similar faults.

Such files are internally stored in a protection device and normally need to be manually downloaded from the device. Even if the device is remotely accessible, a user should take time to go through the files to find the new ones and then download and share them in case more than one person requires access to these files.

A better approach would be to have an automated system monitor the protection devices and download the new files when they become available. Such a system can also send notifications to the stakeholders with the files attached to the message or a link to the folder where they can find a copy of the files. This system can cut the operational cost by automating the download process and decreasing the time it takes for the information to become available to the operators.

Condition-based maintenance

By analyzing information coming from devices in the field, an application can predict when and why a particular device or equipment may need maintenance. Condition-based maintenance can eliminate sudden and unpredictable substation shutdowns that could happen when a piece of equipment fails abruptly. It can also help utilities to lower their maintenance costs by planning their maintenance in advance and combining multiple maintenance sessions into one based on the information they get from a condition-based maintenance system.

Cyber security was not a concern during early years of digital control systems. Most of the communication was done using dedicated such as telephone lines and modems that where not visible outsiders Security was not even an issue in the initial design of the internet infrastructure but the more it became available to the public, the more vulnerabilities were detected. Newer technologies have tried to address the problems caused by a shared link accessible to anonymous users.

Security threats can be categorized into two main types:

• Voluntary threats can be caused by people inside or outside of a company that deliberately try to initiate a failure or to access and steal information
• Involuntary problems most likely happen because of a poor system design

A good security infrastructure should make sure the right information will be accessible only to the right people at any time using proper authentication, authorization and access control.

• NERC-CIP
• User-access management
• Secure remote access and VPN
• Password management
• Activity log and trace
• Firewall, malware protection

NERC-CIP

North American Electric Reliability Corporation (NERC) developed a suite of standards for Critical Infrastructure Protection (CIP). This suite includes nine different standards that covers different areas from physical security to electronic and cyber security. Its main purpose is to limit access and grant it only to authorized people, to secure connections and to log every access to an element in the system while documenting any changes.

User access management

A major requirement in a secure system is to make sure users can only access or modify the assets they are given permission to by a system administrator. Such a system can leverage a corporate level access management system like MS Active Directory and adopt it in a substation automation environment to grant granular access to the operators, system engineers and other users that need to access the devices in the system.

A user management system should also log and timestamp every access to the system and make sure access privileges will be revoked once a user leaves the company. 

Secure remote access and VPN

To eliminate travel time to a substation, utilities prefer to use a remote access system that works in conjunction with user access management tools. The main drawback of a remote access system is that security may be compromised especially when shared links such as cellular modems or internet connection is used to establish such a communication. In addition to various encryption methods, a Virtual Private Network (VPN) can also be used to secure a communication link and to hide it from unauthorized users that may use the same shared connection.   

Password management

Digital devices are normally password protected to prevent unauthorized people from accessing and modifying a device. Security standards require a device password to be changed routinely. Managing password on hundreds of devices cannot be done manually, so an automated system is required to can keep track of the password changes.

More sophisticated password management systems not only monitor the passwords and update them when necessary, but also hide them from the users by tying them to a user’s account. In such a paradigm all that users would need to access the system is to log in remotely using their corporate credentials. The password manager system then provides the access management system with the device password without exposing it to the user. The access management system will internally use this password to grant access to the device with predefined user privileges. 

Activity log and trace

Logging user interaction with a device can provide system administrators or other users with valuable information - especially in case of a failure. Logs and traces can be investigated to spot any potential mistake during a commissioning or maintenance session. An automated system can collect this log information and save in a central database for future use. 

Firewall, malware protection

Although firewall and antivirus software are generally part of the IT infrastructure in a substation control system, more sophisticated devices also support some of these security features. As a gateway can sometimes be used as the single point of access to a substation, it needs to support security measures that block an unauthorized access.

A gateway firewall will block all communication ports except the ones which are necessary for its normal operation. The malware protection system on a gateway uses a whitelisting approach to constantly monitor the codes run on the gateway and to block codes that are not digitally signed by a trusted source and it halts the gateway operation if it detects any suspicious code.