Maintenance engineers often spend countless hours justifying the need to spend money. Consulting services, hardware and software upgrades, maintenance agreements, headcount and spare parts all come with a price. And all are necessary expenses when the cost of downtime is arguably greater than the cost of change.
But how does one calculate and justify the price of peace of mind? In the number of hours of sleep lost thinking about what could go wrong at a given facility? In the number of hours spent on reactive maintenance to fix persistent problems? In reality, peace of mind is a priceless commodity. It’s the qualitative gravy one gets when X dollars are spent on control system upgrades that do result in quantitative, measurable benefits.
Mallinckrodt Pharmaceuticals recently completed a control system upgrade at its bulk active pharmaceutical ingredients (API) plant in St. Louis, Mo. This plant is among several manufacturing lines on the 40-acre campus that makes ingredients used for narcotics and other final, salable products.
Of the 500 employees in the St. Louis facility, only two people understood the 1986-era PLC technology running the plant. Additionally, spare parts were increasingly hard to find and very costly to acquire. The plant’s maintenance staff set out to upgrade the aged technology and bring the plant to a common control system platform. The added bonus? Peace of mind.
IN WITH THE NEW
The multi-year, multi-phased upgrade was an exhaustive endeavor. Mallinckrodt Pharmaceuticals’ process runs 24 hours a day, seven days a week, so the staff had to make sure the shutdown time was minimal. Additionally, the system needed to run efficiently — and at full capacity — as soon as it was brought back online. To make sure we stayed on track, we evaluated the risks and compared them to the overall success of the project at every phase: design, configuration, testing and qualification.
Fortunately, it only required five weeks of system downtime to install, thanks in part to smart engineering, migration tools and comprehensive functional testing. We replaced the legacy PLC system with the process automation system from Rockwell Automation, which established plant-wide control and information. New HMI graphics were redrawn from the legacy HMI at the facility, encompassing 140 screens of process graphics needed to visualize and control the API line. The goal was to keep the operations interface very similar to existing plant equipment so operators could easily transition to the new system when it was brought online.
Uptime has dramatically improved on the line since the system was installed. Another bonus is that the architecture is virtualized via thin clients in the control room. There is no longer a need for a large capacity, expensive server, and the virtual environment provides a robust and reliable architecture using fewer servers to run batch and HMI.
Because other facilities on campus had already transitioned to the new platform, it’s been easier for engineering staff across campus to learn the system and apply the technology’s benefits. It’s also eliminated the risk associated with single-source responsibility for maintenance. In the building discussed here, Mallinckrodt Pharmaceuticals no longer lives on borrowed time with its obsolete control technology. Selecting control modules and HMI that had been validated in other parts of the company’s campus became a major driver helping the organization institute a common platform.
MANY HANDS, MANY HOURS, VIRTUALIZED
Replacing the brains of a manufacturing process in a round-the-clock facility requires careful planning. The system design was completed in two phases. The first phase — handled by Mallinckrodt Pharmaceuticals’ maintenance team and Rockwell Automation engineers — involved developing a programming standard. Once finished, Rockwell’s team completed the documentation and the functional and design specification. It then built, tested and validated the programming code.
During development, Rockwell configured a duplicate system at its Ohio facility to help troubleshoot and finalize programming before startup. In fact, Mallinckrodt Pharmaceuticals retained a service contract so it could call on Rockwell’s technical support engineers to assist when issues arise or to propose possible system improvements.
In phase two, engineers updated the company’s functional specifications using a new programming standard. Mallinckrodt Pharmaceuticals was also able to leverage existing line documentation, which helped to reduce project risk. It also helped ease the transition to the new system because the plant engineering, quality and validation team members were already familiar with the documentation. This helped engineers efficiently test and validate the system. After several reviews, specifications were finalized and the facility was programmed based on these specifications.
FOUR MONTHS TO FAT
Factory acceptance testing (FAT) was completed at the facility with the help of local system integrators and Rockwell Automation partners. It took four months to complete the FAT, using water batching to assure all system components operated properly before adding the chemicals used to produce the ingredients. To get the most out of its plant assets, Mallinckrodt Pharmaceuticals built a new control system based on the existing virtualization infrastructure. The virtualization system enhances system security, which mitigates significant operational risk.