Maintenance engineers often spend countless hours justifying the need to spend money. Consulting services, hardware and software upgrades, maintenance agreements, headcount and spare parts all come with a price. And all are necessary expenses when the cost of downtime is arguably greater than the cost of change.
Mallinckrodt Pharmaceuticals upgraded to a process automation system at its bulk active pharmaceutical ingredients plant in St. Louis, MO, enabling plant-wide control and information.
But how does one calculate and justify the price of peace of mind? In the number of hours of sleep lost thinking about what could go wrong at a given facility? In the number of hours spent on reactive maintenance to fix persistent problems? In reality, peace of mind is a priceless commodity. It’s the qualitative gravy one gets when X dollars are spent on control system upgrades that do result in quantitative, measurable benefits.
Mallinckrodt Pharmaceuticals recently completed a control system upgrade at its bulk active pharmaceutical ingredients (API) plant in St. Louis, Mo. This plant is among several manufacturing lines on the 40-acre campus that makes ingredients used for narcotics and other final, salable products.
Of the 500 employees in the St. Louis facility, only two people understood the 1986-era PLC technology running the plant. Additionally, spare parts were increasingly hard to find and very costly to acquire. The plant’s maintenance staff set out to upgrade the aged technology and bring the plant to a common control system platform. The added bonus? Peace of mind.
IN WITH THE NEW
The multi-year, multi-phased upgrade was an exhaustive endeavor. Mallinckrodt Pharmaceuticals’ process runs 24 hours a day, seven days a week, so the staff had to make sure the shutdown time was minimal. Additionally, the system needed to run efficiently — and at full capacity — as soon as it was brought back online. To make sure we stayed on track, we evaluated the risks and compared them to the overall success of the project at every phase: design, configuration, testing and qualification.
Fortunately, it only required five weeks of system downtime to install, thanks in part to smart engineering, migration tools and comprehensive functional testing. We replaced the legacy PLC system with the process automation system from Rockwell Automation, which established plant-wide control and information. New HMI graphics were redrawn from the legacy HMI at the facility, encompassing 140 screens of process graphics needed to visualize and control the API line. The goal was to keep the operations interface very similar to existing plant equipment so operators could easily transition to the new system when it was brought online.
Uptime has dramatically improved on the line since the system was installed. Another bonus is that the architecture is virtualized via thin clients in the control room. There is no longer a need for a large capacity, expensive server, and the virtual environment provides a robust and reliable architecture using fewer servers to run batch and HMI.
Because other facilities on campus had already transitioned to the new platform, it’s been easier for engineering staff across campus to learn the system and apply the technology’s benefits. It’s also eliminated the risk associated with single-source responsibility for maintenance. In the building discussed here, Mallinckrodt Pharmaceuticals no longer lives on borrowed time with its obsolete control technology. Selecting control modules and HMI that had been validated in other parts of the company’s campus became a major driver helping the organization institute a common platform.
MANY HANDS, MANY HOURS, VIRTUALIZED
Replacing the brains of a manufacturing process in a round-the-clock facility requires careful planning. The system design was completed in two phases. The first phase — handled by Mallinckrodt Pharmaceuticals’ maintenance team and Rockwell Automation engineers — involved developing a programming standard. Once finished, Rockwell’s team completed the documentation and the functional and design specification. It then built, tested and validated the programming code.
During development, Rockwell configured a duplicate system at its Ohio facility to help troubleshoot and finalize programming before startup. In fact, Mallinckrodt Pharmaceuticals retained a service contract so it could call on Rockwell’s technical support engineers to assist when issues arise or to propose possible system improvements.
In phase two, engineers updated the company’s functional specifications using a new programming standard. Mallinckrodt Pharmaceuticals was also able to leverage existing line documentation, which helped to reduce project risk. It also helped ease the transition to the new system because the plant engineering, quality and validation team members were already familiar with the documentation. This helped engineers efficiently test and validate the system. After several reviews, specifications were finalized and the facility was programmed based on these specifications.
FOUR MONTHS TO FAT
Factory acceptance testing (FAT) was completed at the facility with the help of local system integrators and Rockwell Automation partners. It took four months to complete the FAT, using water batching to assure all system components operated properly before adding the chemicals used to produce the ingredients. To get the most out of its plant assets, Mallinckrodt Pharmaceuticals built a new control system based on the existing virtualization infrastructure. The virtualization system enhances system security, which mitigates significant operational risk.
Today the company has two identical physical server racks housed in two separate buildings — something that’s significantly increased system uptime. Having the servers in separate buildings provides redundant protection in case the other server fails catastrophically. With virtualization, islands of information are consolidated. In the event of a data center system failure, Mallinckrodt engineers use the backup to run the system without any lost production or downtime. This also allows the company to run the backup during scheduled maintenance outages, enabling them to keep the production line running.
Staff engineers estimate virtualization will save at least 8 and as many as 24 hours to rebuild the system and get the line back into service if a failure occurs. With three buildings involved and one acting as the main power supplier for all other buildings, virtualization provides a quick disaster recovery option and saves them from catastrophic downtime.
The environmental health and safety, production, plant-floor and leadership teams now have access to continual process data via the PlantPAx system, allowing Mallinckrodt to meet regulatory reporting requirements and run a better operation.
Batch records allow Mallinckrodt to track process trends and indentify production issues to head off problems quickly, minimizing waste and quality concerns.
COMMON PLATFORM, BETTER REPORTING
It took only a few days for Mallinckrodt Pharmaceuticals maintenance and controls engineers to understand the benefits emanating from the control system upgrade. Having a common platform for control and information has decreased downtime and allowed the company to turn away from expensive, reactive maintenance and turn to more proactive cost effective maintenance regimes —something not possible with the company’s old technology.
From Mallinckrodt Pharmaceuticals maintenance team’s perspective the project was a complete success. As an organization, the benefits being realized go beyond the maintenance department. The company’s environmental health and safety (EH&S), production, plant-floor and leadership teams now have access to continual process data via the system’s historian and that allows Mallinckrodt Pharmaceuticals to meet regulatory reporting requirements and run a better operation.
Data from the historian application is accessible to anyone in the facility who has Excel, and it can be analyzed from any building on campus. This allows the company’s EH&S staff to analyze data essential to demonstrate that the company is not sending hazardous chemicals into the environment. Daily production reports provide engineers and building managers with process data correlated from batches completed over several days, which helps streamline overall production. Operators enter production data on the HMI and generate an electronic batch record required for compliance reporting. Having this capability reduces the risk of human error commonly caused by handwritten reporting.
Finally, Mallinckrodt Pharmaceuticals operators can call up trend screens on the shop floor at any time to aid troubleshooting. Batch records allow engineers to track process trends, including tank levels, temperatures and other data needed to streamline and pinpoint potential bottlenecks. When engineers can identify the times production is experiencing higher impurities, staff can head off problems quickly, thus minimizing waste and quality concerns. Previously, the company’s daily production reports would pinpoint problems, but only after they had occurred. Now operators can identify potential problems and respond faster, keeping quality at an elevated state.
CONTINUOUS IMPROVEMENT
Sometimes, it takes a village to bring improvements to manufacturing facilities, especially those that have evolved like Mallinckrodt Pharmaceuticals over its 140-year history. Having a dedicated team that understood the plant’s pharmaceutical syntax and vocabulary reassured the decision-making team and allowed the company to meet an aggressive timeline with minimal disruption and problems.
Within days of the upgrade, the control system helped curtail downtime and made proactive maintenance a feasible, practical reality. While it may have been a maintenance issue that prompted the control system and reporting upgrade, the entire operation has reaped the benefits associated with better data and a better operating system. Mallinckrodt Pharmaceuticals now has the ability to build tools for all engineering departments and have a control system infrastructure that empowers all the company’s operations.
