The same drive toward automation and digital control that has transformed modern manufacturing is reshaping the pharmaceutical production world. At one level, FDA addresses this by requiring validation of control and information systems that interact with production, the same as it does for reactors or tableting machines. On another level, the FDA, reacting to the industry's desire to take advantage of the efficiencies of increased automation, has sought to regulate the means by which software-based systems obtain, transmit or store digital information. The envisioned goal is true paperless documentation of production and related data.The stakes are high--and getting higher--for automation vendors to the pharmaceutical industry and the companies that supply them. Market-research firm IDC, Framingham, Mass., forecasts a $6-billion annual market for FDA-regulated industries in automation systems that can meet the evolving recordkeeping requirements. The same pressure that the FDA exerts on frontline pharmaceutical manufacturing concerns gets passed on to contract manufacturers, business partners, subcontractors and system suppliers themselves. "Many of the largest pharmaceutical companies will not do business with you unless you can validate your systems to their satisfaction," notes Winnie Cappucci, computer systems validation coordinator for Berlex Laboratories, Inc. of Montville, NJ. "You do the work initially to meet FDA requirements, but you are also aiming for a cross-pollination that validates your systems to clients and customers."The need for validated procedures is even more clear-cut for a vendor providing devices or systems to pharmaceutical manufacturers. "We go through a customer qualification at our site just about every other month," says Don Smith, technology strategist at OSIsoft, Inc., San Leandro, Calif. "The auditor will review our software development procedures, then pull several programmers aside and use our code and test procedures in a live test. It can be a very demanding process, but we've had internal development methods and procedures in place for over a decade to cover this." OSIsoft is the vendor of the PI system, a real-time data historian that can be used to record and store batch production data, then transmit it to manufacturing execution systems or other higher-level enterprise systems.Industry CollaboratesSeeing that numerous software developers are audited by pharmaceutical manufacturers, and recognizing the burdens this places on both the pharmaceutical company and the vendor, the Parenteral Drug Assn. (PDA) of Bethesda, Md., with some involvement from the FDA, has developed the Audit Repository Center (ARC) in Pottstown, Pa., as a source of standardized audits that can be used by multiple pharmaceutical companies. Ideally, the audits, which are performed by ARC-trained auditors, can take the place of some or all individual customer audits, thus saving the resources that both vendors and customers have to devote to this process. ARC will re-audit a system whenever there is a major upgrade or, in any case, at least every other year. Pharmaceutical companies, who pay a subscription fee to ARC for the service, can download the audits as needed.Although the process is proceeding somewhat slower than initially forecast, the effort is beginning to pay off. "We currently have 95 auditors trained and 52 subscribers from pharmaceutical companies," says Harvey Greenawalt, president of ARC. He says that new subscribers are coming in at a rate of a couple per month; meanwhile, in some cases, individual audits have been downloaded or referenced dozens of times, "and that represents a huge savings" for both the vendor and subscriber. "As word of this center gets better known, we'll grow bigger," he says.Along the way, several interesting benefits have appeared. Some vendors will apply for an audit on their own as a means of pre-qualifying their products to customers. In addition, having the trained auditors as a resource helps everyone in the business get to a common standard of what should be in an audit. Berlex's Cappucci, who is also on the board of directors of ARC, says that "I am a big fan of this, even though I often use the audits as just a part of an overall evaluation" because it saves her and her company travel costs and time. "You could look on it as buying two free days of consulting work." In practice, Cappucci says that her group still performs numerous in-person audits to meet Berlex's individual standards, but that the ARC material can speed up the auditing process and be used almost as-is for any follow-up or requalification of the vendor. System Life Cycles"Validation is a process, not an event." That is how Dan Matlis, vice president of business development at Stelex TVG, Bensalem, Pa., sums up the ideal approach to validation. "When you consider that 90 percent or more of the lifecycle of an automation system is spent in a maintenance phase, monitoring ongoing production--you have to think about controlling those ongoing costs to get the ROI you expect." Indeed, there are a variety industry statistics floating around that indicate that only 25 percent of the lifetime cost of an automation system is spent in buying the hardware and software. The rest is consumed in installing the software, linking it with various networks, validating it and handling the ongoing upgrades and revalidations.Stelex-TVG is unusual in that its scope ranges from systems integration and implementation through validation and business-practices consulting, and includes packaged software. Its software offering is ComplianceBuilder, an enterprise-level, file-and-message security system that uses private/public key infrastructure (PKI) code to authenticate electronic signatures. The company has worked with Square D/Schneider Electric of Palatine, Ill., to audit Schneiders' software-development procedures and to provide PKI-type security for electronic signatures. "One of the traditional strengths of programmable logic controllers like Schneiders' is that they are easy to program--or re-program," notes Stelex-TVG's Matlis. "That strength becomes a weakness when a reprogramming requires the appropriate level of authority, and when it could necessitate a revalidation of the control system." Adds Mark Liston, sales director for Schneider, "Unlike traditional approaches that handle security and data management at the server level, [this solution] manages these functions at the controller level, which is the only way to guarantee access control."Stelex-TVG has also worked with Foxborough, Mass.-based Intellution (now part of GE Fanuc Automation) to evaluate its automation system packages--FIX, iBatch, iHistorian, among others--for Part 11 compliance. These human-machine interface (HMI) and control products have been used to run PLC-based control systems in both batch and continuous manufacturing. Changed controller settings is a situation that Phil Clark, sales director at CimQuest, Inc., Exton, Pa., a consulting company focused on regulated industries, has often seen. "We do a lot of retrospective validation of legacy systems," he says. "You often find a control or SCADA [supervisory control and data acquisition] system with updated PIDs, but the people who originally installed the system have moved on, new computers might have been installed and not revalidated, and FDA finds an out-of-compliance system when it runs an inspection." Clark says that pharmaceutical manufacturers are too willing to stint on funding quality-assurance (QA) operations at their plants. "You'll see QA being budgeted as part of the capital expense of a plant, and you'll know that the QA function is being minimized. QA should be revered, with its own budget and management."Monitoring and updating the status of already-installed automation systems is one of the key objectives of the Asset Management System (AMS) offering from Emerson Process Management, Austin, Tex.. AMS's original scope has been to monitor the status, calibration and maintenance information of field-control devices like transmitters, valves or flowmeters, explains Krisi Bailey, customer relations manager for the company's Asset Optimization Service. Most importantly, it provides an audit trail--a secured time-and-date stamp and record of the as-is and as-left status of a field instrument," she says. Emerson is now combining this with similar systems for rotating machinery and electrical and utility systems and will offer up the whole service with access through a Web-based portal. The solution is scheduled to come out this June. "Initially, this portal will provide read-only access, but eventually we will make it interactive," Bailey says.Add-On AuditingSystem vendors, striving to comply with Part 11 requirements, have been extending the scope of their systems to include the documentation and audit-trail requirements that pharmaceutical manufacturers need. To some extent, there is a tussle between the plant-floor systems, which perform actual control, and the enterprise systems, which record data and provide for production scheduling and overall plant management.Emerson, for example, announced this spring an alliance with Decision Management International, Inc. (DMI) of Bradenton, Fla., to combine their product offerings and jointly sell and develop products. In Emerson's case, the lead product is its DeltaV control system; for DMI, it is Regulus, a document management system that is tailored for batch records, materials management and other production functions. "This alliance will go to market quickly because of the decisions we made early on in how we structured our control software," says Robert Lenich, business development manager, life sciences, at Emerson. "When we brought the two products together, we were able to get them communicating in less than six hours." Lenich says that the joint offering is impressing potential clients who have seen it, and will have commercial installations very soon.Invensys, Foxboro, Mass., is tackling the problem by recombining several of its diverse business units, including its Foxboro I/A automation system and the Protean and Prism business units to form the Invensys Production Solutions unit. Within this unit, another division, Invensys Validation Technologies, will be maintained to offer lifecycle validation and auditing services. On a software level, the company is depending on its Windows XP operating system foundation, along with interoperable Unix workstations running the Solaris 8 operating system from Sun Microsystems.On the software side, a sense of the scale of meticulousness necessary to meet FDA requirements can be found in a project described by Douglas Tracy, director, safety & regulatory within Pfizer Pharmaceuticals Group, New York. In a presentation at this spring's Interphex meeting, Tracy showed how an unspecified automation project was handled by an internal staff of programmers and systems developers. To begin the project, a detailed requirements plan was developed using CaliberRM from Starbase (now part of Borland, Scotts Valley, Calif.). To handle testing of the evolving program, tools from Mercury Interactive, Sunnyvale, Calif. were used. Test results, process workflows and change management were all recorded in the PVCS Dimensions product from Merant Inc., Hillsboro, Ore. "Our concern with this project was that up to 30 percent of the project cost and time is spent in validating what you have done," Tracy says. "We think that by running the entire development in a paperless, electronic fashion, we were able to lower those costs." Why would a manufacturing company like Pfizer invest so in software development tools? "It puts us ahead of the curve," says Tracy. "We're taking a slow approach to automation systems validation, and this gets us out of the mode of validation as a reactive process. It's a good business reason to go in this direction." 21 CFR Part 11: An Update on the Evolving Saga The issue is a seemingly simple one, but its resolution has been tangled and is still somewhat up in the air: how to move from paper-based to electronic data storage. For years, the pharmaceutical industry has sought to take advantage of the same speed, accuracy and efficiency benefits that automation supplies to any manufacturer. At the same time, the need to generate and store these data in a reliable fashion must be met in a manner that FDA can rely on. This effort culminated with a final rule, first published in 1997, as Part 11 of Title 21 of the Code of Federal Regulations, a.k.a. "21 CFR, Part 11," or simply "Part 11." In the intervening years, industry, FDA, control and IT vendors gradually came to grips with the capabilities of new IT technology and how it would be regulated. In January 2002, a guidance document entitled "General Principles of Software Validation" was issued, pertaining to the specific concerns about validating software systems (as opposed to the data they generate, transmit or store). Then, last August, FDA unveiled a broad initiative, "Pharmaceutical Current Good Manufacturing Practices for the 21st Century" (cGMP), that signaled a re-evaluation of Part 11 and all FDA's regulation of manufacturing practices. In February, FDA issued a guideline (still being discussed) announcing that a new direction would be taken with Part 11: the scope of Part 11 compliance would be limited; a risk-based prioritization of what systems or subsystems should be reviewed or monitored would be allowed (especially for "legacy" systems that predate Part 11 guidance); and the task of enforcement of Part 11 would shift from the Dept.of Regulatory Affairs to the Center of Drug Evaluation and Research. The narrowing of the scope of Part 11 might sound like industry is getting a free ride on what are thought to be onerous regulations, and indeed, there is some grumbling among the ranks of automation system vendors, who have invested heavily in developing technology and services to facilitate compliance. However, FDA's changed direction also allows it to sharpen its focus on the most pressing regulatory concerns. "For many years, FDA has been unable to consistently achieve biennial inspections of all drug manufacturers because of the combination of the increasing number of manufacturing-related establishments and declining resources available for such inspections," it stated in an explanatory document about the change.Even so, FDA has been very clear that it will still enforce validation and cGMP regulations as they pertain to actual production (Parts 210 and 211 of 21 CFR) and laboratory quality controls (Part 860). From the perspective of Part 11, which technically only has to do with the documentation of those parts, the rules in those parts are termed "predicate rules." A predicate rule for releasing a product from production to distribution might require, for example, that an analysis of the concentration of active pharmaceutical ingredients (APIs) must be performed. That will continue to be fully enforced. The relevant Part 11 requirement might or might not require that the documentation for that test be recorded and stored in a certain way.Part 11's previous guideline generated tremendous debate over its requirements for ensuring authentic electronic signatures for human interaction with automation systems, and secure recordkeeping. The former, broader interpretation would, in certain circumstances, require storage and validation of individual keystrokes--say, opening a view of a file that happened to be on a system. The current, narrower interpretation (whose details are still being worked out) will categorize such interactions as "merely incidental" to the operation of the automation system.