Last year, MedImmune received ISPE’s Facility of the Year “Overall Winner” award for its new large-scale mammalian cell culture-based production facility adjacent to its Frederick Manufacturing Center (FMC) in Frederick, Maryland. It wasn’t so much the facility itself that was impressive but the process by which it was designed, built, and started up.
A key to the success was the approach to automating the site—namely, the development of a simulator that enabled the team to execute commissioning and qualification of the Process Control System offline; and the reliance upon the principles of ANSI/ISA-88 and standardized control modules to enable 44 different skids to be tested prior to shipping, ensuring ease of integration.
We spoke with Brent Hill, director of automation within MedImmune’s Global Engineering organization, and Victor Ronchetti, Sr. VP and technical director for Automated Control Concepts, which worked with MedImmune on the system integration. The two presented together last fall at the Rockwell Automation Fair in Chicago, and answer questions collectively below.
“Automation is always to blame for slow startup,” said Hill recently, speaking at the Rockwell Automation Fair in Chicago. This project was an attempt to change that. Said Ronchetti, his co-presenter, “The S-88 approach and tools allowed a true ‘black box’ testing approach . . . We took the spotlight of the project schedule off of us.”
PhM: Your Automation Fair slides were titled, “Using Ordinary Tools in Extraordinary Ways.” What was the impetus to do something extraordinary and, in your mind, what is truly groundbreaking about the project?
Hill, Ronchetti: We had recently just finished a large “Pilot Plant” project similar in nature to the one in Frederick, Maryland. We had learned that we would need to do something that was extraordinary to meet the aggressive schedule. Therefore, we set in motion several systems, such as the Factory Acceptance Test Process Automation Core (FATPAC) and code control, to position ourselves for success.
As most projects go, there are unforeseen events that involve patience and adaptability. Near the completion of the project we were given a completely new set of built P&ID’s. The control system had already been designed around the design documents. We were now faced with the task of re-coding and re-execution of the Process Control System (PCS) SAT.
Due to competing deadlines, MedImmune was now faced with a nearly vertical schedule where the constraint was the physical equipment in the facility. To stay on schedule, the automation team used Rockwell Softlogic’s software to completely replicate the process control system in its entirety. As shakedown activities took precedence in the schedule, it was necessary for the project team to perform activities related to the commissioning and qualification of the PCS in parallel, without interrupting shakedown runs. Under the strain of limited time on equipment as a result of competing project phases (start-up and debugging of applications, train production operators, and running test batches), the project team developed a separate PCS simulator (in addition to the PCS simulator used for operator training) that allowed them to commission and qualify major aspects of the PCS without having to perform the work on the plant floor. This system, which simulates every Programmable Logic Controller (PLC) in the facility, provided a safe, equivalent environment to perform testing. Today, this simulator is used for qualification and tech transfers and has become the Medimmune standard for how we execute automation projects.
PhM: What about S-88 (aka ANSI/ISA-88) “worked” for this project? Do you think the drug industry is finally coming around to its value and using S-88 principles as common practice?
Hill, Ronchetti: Giving the specific control modules, equipment modules and phases to each of the appropriate vendors was key to the project’s success. With the regulatory nature of the pharmaceutical industry, companies are becoming more informed about the value of only validating these standard modules and then using them time and time again. We are seeing the majority of the companies in the pharmaceutical industry going this way on most, if not, all hardware/software platforms.
PhM: You had 44 skids from different vendors all over the world and expected them to have common data storage, a single domain controller, form, fit, function, etc. Did this prospect seem ludicrous when the project began?
Hill, Ronchetti: At the beginning of the project, we sent out in the bid specifications, our requirement that all skid vendors were to use our standard modules and our intersystem communications specification. Once the vendor understood what we were trying to accomplish, all but one not only accepted it, but were enthusiastic about it. They realized how much easier it would make start-up and integration of the system as a whole.
PhM: You provided software modules to Type 1 vendors (e.g., bioreactor manufacturers) three years before startup. Is this type of advance requirement practical and applicable for most projects?
Hill, Ronchetti: Yes, being able to integrate skids into a system or systems with minimal downtime and communication are critical to the success of any project. MedImmune has integrated this practice into all automation projects, regardless if it is green field, retrofit, or additional scope.
PhM: How did vendors respond? Were there a couple that didn’t, shall we say, “see the light”? Any that backed out of the project?
Hill, Ronchetti: All but one vendor was able to implement this. For this group, with the help of ACC, MedImmune worked closely with them to help write the code. This was a seamless integration of all systems during start-up.
PhM: One of the key aspects of the project was controlling and using FAT for validation. What lessons did you learn in this regard?
Hill, Ronchetti: MedImmune and ACC had just finished a very similar but smaller project and had learned that if we did not control the code throughout the entire project that the project cost and schedule may be impacted. During automation system start-up in a cGMP environment, it is typical practice to leverage work completed in FATs, installation, and commissioning. More often than not, these activities are limited to hardware IQ checks. Functional OQ test validity is always in question due to lack of code control. Installation and testing of a system this size, with seven million lines of automation software code and 44 different skids, involves careful oversight. In addition, the leveraging strategy dictated by MedImmune added a complexity to start-up activities, which posed a major challenge to the team: how to track, maintain, and control PCS code revisions while simultaneously performing leveraged start-up activities.
To proactively address this problem, the automation group used an off-the-shelf Rockwell configuration management application to manage coding. The tool, which has automatic revision control, requires programmers to “check out” code from a repository. This allows an auditor to go back to any point in time and see what changes were made to code throughout the development and testing process.
MedImmune was able to control the code from the first control modules issued to the skid vendor all the way through ICQ. By controlling the code there were no questions as to what changed or the impact those changes would have on the FATed or commissioned equipment
PhM: Explain a bit more about FAT PAC (Factory Acceptance Test Process Automation Core). What is it, and what did it enable?
Hill, Ronchetti: A common issue in automation projects, especially of this size, is interfacing skidded systems with the PCS and with each other. Using a proactive approach to solve this potential issue, the team developed a FATPAC, a portable interface package to support FAT. This package of servers replicated MedImmune’s high-level process network and allowed MedImmune to test the equipment in the appropriate environment, at each site.
The FATPAC included a domain controller used to preset user access, a PLC, and an HMI client from the PCS system. Communications were set up with the PCS PLC/HMI and used in the FAT for each of the skids. Any problems identified during FAT were resolved and retested prior to shipment. When the skids arrived onsite, minimal setup was required to integrate them into the PCS system. We shaved at least six to seven months of typical delays in start-up, including commissioning and validation groups addressing interface issues.
PhM: The process simulator allowed you to do commissioning and qualification of the PCS offline. How extensive was the simulation project, and is this a template that you can now apply to most any process going forward?
Hill, Ronchetti: As discussed earlier, the PCS simulator was a complete replication of the live PCS system. If a change was needed due to errors found in testing or errors found on the live system the code was changed on the live system first. Each night the code from the live system was then downloaded into the simulator. Asset Centre was used to maintain code equivalence between systems and allowed for traceability. Test scripts were then generated and the errors were then retested to insure the quality and integrity of the codes. This process is now the standard for all of MedImmune Control system related projects.
PhM: Explain, briefly, the four-tier military training scheme and how this enabled operators to get up to speed quickly.
Hill, Ronchetti: MedImmune needed to transition manufacturing operators accustomed to a small-scale, single product, semi-automated facility to a large-scale, multi-product, and highly automated facility. This had to be completed in a short period of time without risking equipment damage to any critical process and support systems or risking loss of product materials due to incorrect use of the PCS.
The solution: a four-tiered, blended learning approach commonly used in the military, but rarely implemented in the biopharmaceutical industry. This approach was developed as a collaborative effort between a number of site and corporate functions, and included:
o Concept Training – concept training consisted of interactive Computer Based Training (CBT), which allowed employees to gain a general understanding of how the facility and the process control system would operate. The CBTs, developed according to successful adult learning theories, clearly state the training objectives, and then lead the student from overview and concept understanding to higher-level knowledge of the PCS. The CBTs included detailed checks for understanding after completion of each module. Also included were Printable Quick Reference Guides to be used as job aids while using the PCS HMI. These guides included information on the PCS structure, User Access
Rights, a summary of alarm conditions, locations of PCS stations throughout the
building, and definitions of terms.
o Review of Operational SOPs – this component involved the requirement of students to review the SOPs as preparation prior to attending hands-on training.
o Hands-On, Instructor-Led Training – a dedicated training lab was built by creating a pared-down version of the PCS. The lab included a subset of the process control functionality found on the manufacturing floor. The lab allowed operators to train on a “live” system that looked, felt, and behaved like the real PCS. A series of instructor-led sessions which mirrored actual production scenarios were also were created. These sessions allowed operators to use the PCS HMIs to perform tasks, such as media preparation, transfer operations, and cell culture, harvest, and purification operations.
o On the Job Training with a PCS Simulator – the project team understood that proficiency on the live system would require additional practice using the PCS. To minimize knowledge loss between operator training and live PCS usage, the project team developed a PCS Simulator for all operators who had completed the instructor-led training in ICQ activities. Use of this simulator helped ensure proper use of equipment through the PCS.
PhM: In the presentation, Brent said, “Automation is always to blame for slow startup.” Is your Frederick Building 633 project proof that this statement really can be a thing of the past—where automation is an early-stage, fundamental consideration for all projects?
Hill, Ronchetti: The very nature of automation is considered a risk. The errors or problems in coding have an unknown duration, which is what has historically has given automation a bad name. By careful planning, the industry as a whole can help make the paradigm shift. Once it is obvious that although critical to the success of any project, the automation need not be one of the greatest unknowns of a project.