Optimizing Automation Project Management: Best Practices from Baxter

March 26, 2007
Success requires defining goals and priorities clearly, adopting a project methodology and developing test plans for the entire project life cycle.

In pharmaceutical industry automation project management, every minute counts. Any new drug has only a limited time to capture a market, gain a following among physicians and extract maximum value from sales before its patent runs out and generic versions hit the market. Millions of dollars worth of investment are riding on the project team, which must prevent time-consuming haggles, turf battles, disagreements, false starts, mistakes, unnecessary work and missed schedules.

The control system team must be ready when the plant construction is complete. The plant must manufacture its compliance lots on schedule and achieve FDA validation within the shortest possible time frame. This article will suggest some steps that will help achieve project goals, and give examples of how these best practices are being applied at Baxter Healthcare facilities.

Set Goals and Priorities

Once the project is approved by senior management, the project manager must define its goals and priorities, and communicate them to the entire team. For example, the goals may be to have Plant A in operation by the second quarter of 2008 (please note that, generally, it is not a good idea to specify a date) and be able to produce any of three target drugs currently in the pipeline with a staff of X number of people.

Unless care is taken on the front end of the project to prioritize the team’s efforts, the project can very easily get off-track. For any project, three general priorities compete: functionality, budget and project schedule. They’re all important, but they can’t all be equally important. Select the most important one, initially, to set the pace for the rest. (In Figure 1 below, adherence to schedule was chosen as the critical factor.)

Figure 1: All project goals can’t be equally weighted. Begin by deciding which one is most important or the team may make the wrong assumptions. Click photo to view larger image.

A kickoff meeting should then be held, during which the project’s priorities and goals should be set and agreed to by all project team members, including: Project Engineering and Management, Process Engineering, Quality, Validation, Procurement, Construction Management and Plant Operations. Priorities and goals should also be set with each vendor at kick-off meetings.

Occasionally, priorities will change due to factors outside the project team’s control. For example, if a competitor launches a product that competes with the new drug, the need to reduce time to market may outweigh previously established budget constraints. This situation is manageable as long as everyone on the team realizes that disruptions will have serious consequences for overall project efficiencies. New expectations should be aligned quickly so that the team can correct its course with minimal loss of efficiency.

For example, one Baxter Healthcare grassroots facility was designed to make two distinct groups of products using different processes. Initially, the plan was to complete validation and start up the facility with one process, then start work on the second process. However, due to delays in validating the first process and in response to changing market demands, Baxter decided to start the commissioning and validation of the second process while the first was still being validated.

This decision could have wrought havoc, but any impact was avoided by assembling a cross-functional team. Including controls engineers, representatives from quality, manufacturing and engineering and from the project’s system integrator, Emerson Process Management, the team developed a new project plan for the second process that could work around the validation efforts of the first process.

The team made extensive use of the FDA’s risk-based approach to validate as much of the automation software as possible on a simulated system. This allowed validation of the first process to proceed on the production equipment while the software for the second process was validated on the simulation system. Reusing much of the software for the first process greatly reduced the amount of validation work required for the second process. Finally, by combining both projects on the same schedule, we were able to quickly identify and resolve conflicts between the projects.

Communication is Vital

On another recent project, the project engineer made it clear to everyone at the beginning of the project that the project’s budget was limited. He said it was everyone’s job to meet the automation objectives and the schedule while staying within that budget. He then encouraged the entire team to look at every item with a critical eye and ask the question: Is this required to meet the project’s automation goals or is this a luxury (and a sign of “creeping elegance”)?

From the very beginning, the project manager did an excellent job of communicating that he was serious about meeting the budget. Any work that would increase project cost had to be offset by eliminating other work, or he would not approve the change. The project manager also made sure that his team, as well as the vendor’s teams, knew that they were all responsible for meeting the budget. He consistently monitored the expected cost at completion and made the necessary trade-offs between parts of the project to accommodate unexpected costs that could not be avoided. As a result, the project was completed on budget and on time, with excellent quality.

Avoid Data Hoarding

Sometimes, with the best of intentions, project managers may keep priorities a secret or assume that they are all equally important, leading to a situation where everyone else on the team must scramble to figure out which goal is most important. Very often, they wind up visualizing the wrong goal, leading to misunderstandings and possibly finger pointing later on.

Why would any project manager fail to communicate this information? In some cases, it is because they are concerned that team members, particularly vendors, may use it to minimize the work they do for the project. Such fears are usually groundless, and even if a few individuals try to use this information to their advantage, the amount lost on the overall project will be dwarfed by the gains achieved by having everyone working toward the same goal.

Adopt a Project Methodology

Figure 2. The basic functions that must be addressed when setting up a project methodology. Planning work that is done up front will pay big dividends as the project proceeds.

Paradoxically, the shortest time to a project startup is not necessarily the route that minimizes engineering work on the project. It’s necessary to develop a project methodology (Figure 2, above) that considers the project from the development of process descriptions and Piping & Instrumentation Drawings (P&IDs) to ongoing operation of the plant. This methodology must take into consideration the entire project life cycle to eliminate any unnecessary gaps or overlaps. This is especially critical on projects in which the final result must be validated by the FDA, since the cost of making changes to documentation during the project are high, as are the costs of maintaining those documents during the life of the plant.

Since the overall goal in life sciences design a methodology that takes into account all steps, from the beginning of the process to the making of saleable product. One of best ways to understand the impact of methodology decisions on the schedule is to create a detailed plan schedule (Figure 3, below) that shows the dependencies of activities and considers the impact that changing the content of one deliverable will have on another deliverable.

Figure 3. Click on photo to view larger image.

Another important task is to document clearly the expected content and purpose of each step of the methodology and make sure that all parties involved review and approve the plan. Ideally, this documentation should include examples of each deliverable; this will ensure that there is no misunderstanding about the purpose and content of each deliverable. This approach also ensures that all the different groups working on the project know what others’ deliverables are for and what the content will be. Taking this approach should eliminate overlaps and gaps, reduce costs and protect the schedule.

Care must be taken to identify not only which documents and steps are needed, but why they’re needed — in other words, the purpose and extent to which each step must be taken. Failing to map this out can result in inadequate guidance, and extra work. For instance, if the project’s methodology includes the development of process descriptions, functional specifications and detailed designs but does not define the content of all three documents in the beginning, there is a good chance that the process descriptions will either have too much or too little information from which to create functional specifications.

Document content must take into account the skills and backgrounds of those writing the documents. For example, one would not want the process descriptions to define the S88 architecture and divide the process into classes if the individuals writing the documents are not highly skilled batch automation engineers with S88 experience. Too much information in process descriptions will create extra work for the automation engineers or place restrictions on them when developing the functional specifications. On the other hand, too little information will force the automation engineers to question the process people repeatedly, or even cause them to miss an important requirement is decreasing time to market, it is vital to until after implementation is complete, when the cost of making the change is much higher in both money and time.

In one recent project, Baxter assembled a staff of both process engineers and control engineers to complete the process description and the functional specifications in house. Process description and “user requirements” were defined in a high-level document written by a team of process and controls engineers with guidance from Quality. Since the facility was highly automated, the controls engineers took the lead in writing functional specifications in sufficient detail to send out to system integrators for bids. Because of the detailed functional specifications, the selected system integrator (Emerson Process Management) was able to move quickly to the design phase, resulting in significant savings in the project schedule.

As each portion of the design was completed, the in-house controls engineers carefully and efficiently reviewed them. Baxter stayed in close contact with Emerson Process Management to ensure that knowledge from one part of the design review was incorporated into the remaining designs. Because of this careful design review, implementation of the software was completed with few changes and no schedule delays.

Develop Test Plans for the Entire Life Cycle of the Project

In developing the testing methodology for the project, it is again vital to consider the project from beginning to end. All too often, questions about what test plans are required or what testing needs to be done are answered based on a very limited view of the overall project. Engaging Quality and Validation early in a project life cycle and taking into account the overall project make it possible to develop a methodology that minimizes cost and maintains the schedule while delivering a high-quality, well-documented solution. Automation vendors are frequently told by their pharma clients to use “whatever your standard methodology is,” without considering the overall project plan. This type of direction will likely increase the cost and time required, compared with a project that was reviewed.

The best way to avoid these problems is to create a series of project quality plans that document the testing and test documentation that will be created at each step of the project.

Considering the entire project life cycle and the type of project — a new facility, retrofit or an expansion — helps to develop a methodology that best fits the goals of a specific project and eliminates unnecessary duplication of effort. For example, if the project is a retrofit to an existing, operating facility and avoiding downtime is critical, then the best strategy may be to do as much testing and documentation as possible before installation and shutdown of the plant. While this may result in some duplication of effort, the goal would be to minimize loss of production.

On the other hand, if the project involves building a new plant, it may be best to move most of the formal testing and documentation to the IQ/OQ phase to minimize the cost and keep to the schedule. In a new plant, significant changes may be required during commissioning that would reduce the value of early testing. Keep in mind that documents detailing requirements should be kept updated throughout the process, as these will be needed to develop the IQ/OQs.

In a highly automated, grassroots facility, the automation software must be developed and tested before any equipment can be operated or tested. In a recent project, Emerson implemented the software and then tested it internally before turning it over to Baxter for a factory or site acceptance test (SAT). Since detailed software validation OQs would be done later, the SAT consisted of functionally testing all recipes on a simulation system. During the SAT, Emerson people remained on site to correct code issues and to make changes as they were identified.

Following the SAT, the software was released to the equipment commissioning teams. Commissioning protocols were executed to ensure that both the code and the equipment would meet process requirements. The advantage of executing the SAT and the commissioning protocols prior to validation is that changes to the code did not need to be made under change control. Changes were simply made to the design and functional specifications to reflect code changes.

Since the automation project was a large one, testing, commissioning and validation were broken up into several parts. As commissioning was completed on each part of the software, software validation was started on that portion of the code.

Breaking the code into portions allowed commissioning and validation to occur concurrently, which greatly sped up the project. However, software validation and commissioning work often used much of the same equipment, which created scheduling challenges. In order to minimize scheduling conflicts, we did most of the software testing on a simulation system that mimicked the actual process equipment. A risk-based approach was adopted to determine which part of the code could be validated in simulation only. Code that did not require a field element for testing was deemed “low risk” and could be validated in simulation only. Examples include calculations and algorithms, parameter mapping and logical transitions. After completion of the simulation OQs, validation of items requiring field elements, such as valves and instruments, were tested in software field OQs.

Decisions about the project methodology can affect the longterm operating cost of the plant. For example, which documents will be “living documents,” maintained throughout the plant’s lifecycle, and what information will they contain? Which documents will be created for a specific point in time, and not updated as information changes? Typically, design documents such as functional specifications are living documents and documents such as executed test protocols are not. A decision to have overlapping or unnecessary information such as standard product features in living documents can create extra work during startup and operation without providing any real value.

All of these decisions are consistent with the risk-based approach to validation recently being encouraged by the FDA in the cGMPs for the 21st century, and should be part of a formal risk analysis prepared at the beginning of a project. This analysis should involve all parties that will be working on the project.

Take a Broad View

The project team should be made up of everyone who affects the automation or is affected by it. At a minimum, it should consist of people from Operations, Quality, Validation, Process Design and Automation. It is important to assemble this team very early in the project life cycle, because decisions made by one group probably will affect other groups.

Often, we see projects where each individual group tries to minimize the cost and schedule of its part of the project without consideration for the overall project, an approach that often leads to failure. Without developing a team in which everyone is doing his or her best to make the entire project successful, it will be difficult or impossible to complete the project on time and on budget.

Keys to success include choosing trusted team members; communicating goals, priorities and direction; and developing a plan and methodology with input from the team. It works best if the goals of all the members of the team can be tied to the overall success of the project instead of just their part. If each individual group’s goals are not tied to the overall success of the project, then their goals will encourage them to optimize their portion of the project without concern for the effect on the overall project.

For example, a decision made by the process engineers to minimize the cost of physical equipment by not standardizing the equipment could drive up the cost and schedule of the automation, commissioning and validation.

Large grass roots facilities typically have formal project teams assembled for several years from within the company and sometimes through additional hiring. Teams for large projects must represent all relevant disciplines and should include Process and Controls Engineering, Quality and Regulatory, Manufacturing and Maintenance.

However, even smaller projects require a cross-functional team to ensure success. In a recent example, new hardware and software was being added to an existing unit. Since the plant was close to startup, the hardware had to be installed and the software developed, tested and validated in a tight time frame, with no delays. We formed a team that included a process engineer to oversee hardware installation, a senior controls engineer to oversee software development and a quality engineer to ensure that all protocols met corporate requirements. Regulatory personnel also were consulted early to understand the impact of the project on the plant’s FDA license.

There will be trade-offs during any project in which increasing the cost or schedule in one place can decrease the overall cost and schedule. It is also imperative to spend the necessary time in the early stages of the project to clearly define what needs to be done, and set standards before getting a large group of people implementing tasks. Often the need for perceived progress early in the project causes people to start doing things before it is clear what needs to be done.

The most important action is to get every discipline involved in a true team effort where everyone benefits from the overall success of the project. Document and communicate the goals and priorities of the project as well as the project methodology and schedule.

In the smaller project described above, the Baxter project team communicated daily with Emerson due to the tight software development schedule. The main communication tool was project scheduling software that was updated and shared by the Baxter group and Emerson each time an event occurred that affected the schedule. This scheduling tool was invaluable in spotting schedule delays early, so the team could take corrective actions required to keep the project on schedule.

In one case, due to unavoidable circumstances, Emerson’s project lead had to leave the project during the software design phase. This caused a two-week schedule delay to bring a new project lead up to speed on the software design. Because of the plant startup, the project end date could not be moved. Since a good working relationship had been built, we were able to work together to find ways to recover the lost time in the schedule. By adding system integrator resources to the implementation and testing, and by Baxter accepting some low level of risk in implementing parts of the code with only preliminary design approval, we were able to recoup the two weeks needed to finish the project on time.

About the Author

Steven Gorsler | Baxter Healthcare