As industrial control systems have moved from proprietary hardware and software to commercial off-the-shelf (COTS) equipment, they have become exponentially more vulnerable to cyber threats. New threats appear in rapid succession. For this reason, security patches developed by Microsoft, antivirus updates by security companies such as Symantec, and hotfixes by industrial control equipment companies are distributed on a regular basis. This can present a considerable problem for those in charge of keeping everything up to date.
Which patches should be implemented? Not all are relevant to all users. When should updates be installed? While some can be installed while the control system operates, others require a reboot of the various portions of the system, which can lead to downtime if not properly coordinated. Also, which patches go with which systems? Putting a patch in the wrong place can have serious consequences.
Eli Lilly and Company in Indianapolis faced exactly this problem. For the facility’s 15 DeltaV distributed control systems (DCS) — both offline and online running multiple batch operations — keeping track of everything in regard to administration and system maintenance took up a great deal of time and cost a substantial amount of money.
Eli Lilly began to experiment with ways to automate the patching process when Emerson Process Management introduced its Guardian Software Update Delivery Service (GSUDS) and Automated Patch Management Service. Lilly became early adopters and worked with Emerson on refinement of this Lifecycle Services offering.
The systems at Lilly’s Indianapolis facility (Figure 1) consist of one upstream server and a number of downstream servers (in what Emerson calls a one-to-many arrangement). The upstream server has Internet access and hosts Microsoft Windows Server Update Services (WSUS) and GSUDS.
The multiple downstream dedicated non-DeltaV server machines (one for each DeltaV system) host Microsoft WSUS, Guardian WSUS Interface (GWI) and Symantec Live Update Server – Distribution Center (LUS-DC). Each downstream server can also talk to other DeltaV servers via plant LAN connections where available.
The GSUDS client solicits system hot fixes and approval information for Microsoft security updates from Emerson via the Internet. GWI is a software application that periodically checks with the GSUDS Client for new DeltaV hot fixes and the latest approval information for Microsoft security updates. It then programmatically injects them into WSUS.
Depending on the particular system, the downstream servers also support other needs, including Emerson’s Backup and Recovery Services and Mimic (simulation software from MYNAH Technologies). The potential also exists for the downstream servers to be virtualized. Configuration for WSUS is handled through a separate Group Policy Object (GPO).
ENTERPRISE PATCHING APPROVALS
All Microsoft security patches are automatically approved at the upstream server and synchronized to each downstream server. DeltaV hotfixes are approved at the downstream server level, with each system administrator handling approvals for his/her own system. The advantage here is that it saves time as the patches and updates are automatically placed on the computers where they are required. Patches and hotfixes are installed through Windows Update on each computer. DeltaV hotfixes do involve some manual interaction during the installation process.
It’s worth noting that it is still necessary to follow all release notes prior to installation, and all hot fix prerequisites must still be performed. Oversight of the patching process is strongly encouraged to verify distribution.
There are often significant differences among the multiple systems in a plant, which means that the set of patches must be customized for each. The automated system delivers the right patches to the right machines via the network. For those patches that require a reboot, it gives the option to automatically reboot, automatically install or just automatically download. In Lilly’s case, automated patching saves several days of effort for each patching event. It reduces the time needed to gather information, delivers updates to systems all over the plant site, makes sure the right updates are in the right places, and installs the updates.
In Eli Lilly’s experience, automatic patching saves anywhere from a few hours to a few days for each Emerson Process Management DeltaV system. With 15 systems in place, that can add up to a significant amount of time and money saved, while minimizing human error associated with patching.
ABOUT THE AUTHOR
Kurt Russell is a Consultant Engineer at Eli Lilly and Company in Indianapolis, IN, working in the Biotechnology & Life Sciences manufacturing area. Kurt has worked in automation and control system engineering with multiple platforms over the past 20 years. His current work involves managing multiple DeltaV DCS implementations from an operational, system administrative and project delivery perspective.