Cause for Alarm?

Unfortunately, alarm systems in the batch-oriented pharmaceutical industry, as with other industries, are often a source of concern. It’s not unusual for a manufacturing facility to average over 1,000 alarms per month, and many operators in the industry are frustrated by both the number of nuisance alarms they receive and the total number of alarms.

In optimizing alarm systems and their management, it is important to consider several key questions:

• Is the process being considered truly in control?
• Do alarms for batch manufacturing operations reflect the batch nature of such processes?
• What automation systems validation issues does the alarm system pose?
• Has sufficient attention been paid to rationalizing and configuring alarms?
• With off-the-shelf systems, have the vendors provided sufficient functionality for the given application?

This paper will address these questions, offering broad perspectives on how a drug manufacturing facility’s alarm management can be improved.

Is the Process in Control?

It is well known that quality cannot be tested into a product but must be built into the systems that make the product. In other words, the determination that a manufactured lot of product is acceptable to market assumes that it has been made using a formally validated process, as well as having passed certain in-process and/or final product tests.

A validated process is one that is “in control” and reproducible, while meeting expectations that it will produce a safe and effective product. If you were an FDA inspector and saw that a plant was generating over 1,000 alarms per month, would that raise a concern as to whether the plant was “in control?” Would seeing only 500 alarms per month be a concern?

Any system in which operator display consoles constantly show active alarms, or in which the historian shows a high total number of alarms per batch can be red flags to FDA inspectors that the process may not be in control. They’ll usually prompt further investigation. Thus, it is important to audit systems to ensure that they reflect a state of control.

Matching Alarm Systems to Batch Processes

Most automation technology, including process control computer systems and data historians, originated in continuous steady-state processes, such as petrochemicals processing. Only in the last few years have systems started to accommodate the additional requirements of batch processes (e.g., with batch historians), which are common in the pharmaceutical industry.

However, functionality gaps still remain with many commercial systems. First, consider the basic differences between batch and continuous processes. Batch processes are typically comprised of a sequence of many steps/phases with few, if any, steady-state characteristics. For example, control of a fermentation process will typically require the following sequence:

1. Fermentor CIP
2. Fermentor media filling
3. Fermentor sterilization
4. In-line probe calibration (e.g. pH and dissolved oxygen, DO₂)
5. Inoculation with live cells
6. Cell growth phase
7. Induction (shift from cell growth to product production)
8. Product production phase
9. Harvest

Most batch process alarms are functions of the process/step/phase. Therefore, alarms, their limits and other attributes, activation and deactivation, should be configured as a function of the batch step or phase. In addition, whenever possible, the process step or phase should be included in the alarm record tag when logging alarm events.

Some batch-step control parameters may be time varying, so automation systems need to allow for time-varying calculated control loop setpoints and alarm limits.

Think Batch Time, Rather than Calendar Time

Operators, scientists and engineers who work with batch processes tend to think of their processes in terms of relative batch time, so that Time = 0 indicates the start of the batch or batch step.

However, almost all automation systems collect, display and store process data, including alarm records, in calendar time. This causes significant inefficiency when analyzing such data, as users must constantly translate calendar times to batch-relative times to provide the appropriate context.

Pharmaceutical manufacturing professionals also often desire to analyze process data based on batch lot number. This allows them to compare one particular lot to another, facilitates the generation of batch reports and allows them to access information relating to specific abnormal situations that may have occurred during a batch. Unfortunately, many commercial automation systems do not include lot number in their process data or alarm record tags.

The challenges and nuances of batch processes are so numerous that ISA developed S-88, to define standard terminology as well as the many “states” within a typical batch process (e.g., running, holding, pausing, aborting and stopped). The majority of automation logic is not associated with the normal running of a batch process, but rather with the other possible states of a process and the transitions between them. This suggests that alarms and their attributes also can be configured as a function of the process “state.”

Computer System Validation

Computer systems used in the manufacture of pharmaceutical and biopharmaceutical products must generally be validated. Documented evidence must be provided that the system “does what it purports to do, and can be expected to continue to do so.” This definition will be examined from a number of different perspectives.

The Human-Machine Interface (HMI) – One aspect of what the HMI portion of an automation system typically “purports to do,” is to provide accurate, reliable and timely information to operators regarding process upsets such that the operator understands the situation and can appropriately respond to it.” In reality, many plant operators are frustrated by a large number of nuisance alarms, which do not represent an abnormal situation, and/or do not require a response. In addition, the frequency of alarms often exceeds what an operator can reasonably be expected to handle, as codified by EEMUIA [1]. The combination of “information overload” and insufficient prioritization of the information that is supplied during process upsets is a common operator complaint. In cases where operators receive frequent undifferentiated alarm signals, they can sometimes miss “real” abnormal situations while responding to (or ignoring) nuisances.

If the abnormal situation involves a Critical Process Parameter (CPP) in which the Proven Acceptable Range (PAR) has been exceeded, then the lot must be formally investigated as to its suitability for further processing, acceptability for use in clinical trials, and/or marketing for sale to the public. Anecdotal evidence suggests that excessive alarms correlate with an increase in process deviations that must be formally investigated.

Computer validation requires that operators be trained to do their job, and part of their job is to respond appropriately to alarms. However, in a system where 4,000 alarms are configured per operator (Figure 1) [2], how can any operator be expected to know how to respond to so many different conditions?

Figure 1. Number of Configured Alarms per Operator (reproduced with permission of PAS – ref. 2)

SOPs and operator manuals may exist but may not be accessible when a major abnormal situation occurs on the plant floor. Sometimes, basic documentation is missing. For instance, when rationalizing individual alarms, some project design teams fail to document the rationale in selecting attribute values and fail to document the expected operator response to abnormal situations. Recall that an effective alarm should alert, inform and provide guidance to the expected response.

FDA Expectations

A computer-based alarm system is really a subset of the overall automation system and so is generally subject to the same cGMP requirements as are other aspects of electronic process control and data historian parts of the system. However, a few unique additional expectations can exist with alarm systems, one of which is “measurement uncertainty.”

The FDA expects that “measurement uncertainty” will be considered in determining alarm limits, at least for product quality in which control acceptability is guided by critical process parameters. These parameters, typically identified by scientists and engineers during product development, are associated with a proven acceptable range (PAR), usually based on experiments run during process development. The natural inclination is to set alarm limits at a CPP’s PAR.

However, instrumentation, as well as other components in the signal path a measurement takes from sensor to computer software, is associated with some uncertainty. For example, some process sensors have a possible error of up to 1% of the instrument’s span. Therefore, if an alarm limit is set at the PAR value, then it is statistically likely that a real process deviation could be missed, due to measurement uncertainty. Formulas exist that allow for the determination of measurement uncertainty, the use of which then often results in the setting of alarm limits inside the PAR values (Figure 2).

Figure 2. Setting Alarm Limits for Critical Process Parameters

While many of the principles of computer validation lead to “building quality into a system,” it is a paradox that another primary tenet of computer validation, “management of change,” frustrates the effort to improve application software, including alarm systems. The pharmaceutical industry often perceives change control as burdensome, time consuming and bureaucratic, stifling changes and improvements to alarm configuration and other aspects of automation systems.

In reality, such changes are often very easy to accomplish, technically. It’s the “people issues” that get in the way. For example, a change to an alarm limit that might take five minutes to accomplish via a control system console may actually take weeks to implement, given the large number of people in different organizational components (e.g., tech service, quality control, automation, operations) who must justify, review, document and/or approve such changes.

User Discipline: Less is often more

Project engineers and automation groups often deal with automation systems that can have thousands of input/output (I/O) points. Since the cost of configuring alarms to each I/O is so low, it can be tempting to configure one or more alarms to most of the I/O points, as well as to calculated variables and other information available within the system. But this practice has led to an overwhelming increase in the number of configured alarms per operator [2]. The recent proliferation of “smart” sensors and valves increases the temptation to configure alarms to some of the additional information now available per I/O. When tempted, just ask the question, does this really represent an abnormal process upset requiring a response from the operator? If the answer is no, its not an alarm and should not be configured as one. Use the same logic for alarm remediation projects for existing systems.

Not only are there too many undifferentiated alarms, but many facilities also have programmed routine operator messages to the alarm system. Thus, many operators learn via an alarm that a process operation has successfully completed.

Notifications are not alarms and, vendor software permitting, should not be configured to the alarm system. Instead, this information should be stored and displayed in a separate part of the HMI. If this can’t be done with a particular automation system, then every effort must be made to display the information so that operators can distinguish the notifications from alarms (for instance, using different colors or audio signals).

Adhering to the basic definition of an alarm, as indicating an abnormal situation requiring a response, as well as reducing any chattering alarms and situations where multiple alarms are generated per individual event, can reduce total alarms by more than 75% at a typical facility. Most operators truly appreciate such reductions.

If All Else Fails, Consider Your Car

If engineers become defensive in justifying a huge number of alarms in a system, ask them to consider the total number of alarms that exist on their own automobiles. Although cars are complicated systems involving many different components, operations, load changes and even critical safety operations, only about five alarms are usually installed on most automobiles: low gas level, seat belt not fastened, unauthorized entry, engine maintenance needed. Each of these alarms represents an abnormal situation requiring a response, and presents information in an easily understandable format, and in a timely and accurate way. One single alarm is used for each type of abnormal situation.

Project engineers and automation personnel should have the end clearly in mind when rationalizing and configuring alarms, and right from the start of a project’s life. The need to think through how information is to be organized and prioritized for operators so that operators know what is important during possible “alarm floods.” They need to consider who is likely to access and use the records and for what purposes. For instance, the data may be of value in generating batch reports at the end of a manufacturing run to include any and all product quality (i.e., CCP) related alarms. Such a report could help plants quickly determine, in part, what abnormal events during a batch need to be formally investigated as possible “product quality” deviations. This example “end in mind” would suggest that alarm records be tagged with the alarm category, such as product quality, safety or environmental concern. Or, if not part of the tag, software utilities should exist within the data historian (e.g. dictionaries, databases, etc.) to align alarm records with their appropriate category. The same may be true regarding alarm priority and process lot number.

Off-the-Shelf Functionality

Most automation vendors’ roots are in the continuous process industries and have only recently been addressing the unique needs of batch processes as well as incorporating the latest regulatory system validation expectations (e.g., electronic records). Pharmaceutical companies should be aware of this when beginning a new automation project. The documented System Functional Requirements typically pursued near the beginning of a project, may influence vendor selection and/or need for customization of the system. Many projects, assuming that the computer system would provide all the alarm (and other process control) functionality that would be needed, have ended up with frustrated users when it was discovered too late in the project that the system didn’t provide all the desired functionality. Several pharmaceutical plant project teams have noted the alarm management limitations in process control and historian vendor COTS(?) and have interfaced other 3rd party products (e.g., alarm loggers, paging systems, expert systems) to their core DCS/PLC/historian automation systems.

Part of the problem with overall alarm management is that alarm functionality is actually distributed among several parts of an overall automation system, so no one computer system “does it all.” ISA Standard S-95 offers some perspective on this, defining several different levels of automation. Figure 3 notes these levels as well as some of the different components that an overall automation system could have that are associated with the different levels. For example, alarms can be generated in many different parts of a system, from field devices to at-line analyzers and PAT systems, to alarms determined within PLC/DCS environments.

Figure 3. Distribution of Automation Components Involved with Alarms

The challenge is to consolidate the alarm information in a single environment in a single format with a single user interface. Having alarm displays in a single place will help the operators do their jobs and having alarm records in a single database will greatly facilitate mining data for their information and knowledge content.

In summary, the key considerations for alarm systems and management is focus, defining the “end in mind,” adherence to the definition of an alarm and implementation of alarm management best practices.

The following checklist can help pharmaceutical process and automation engineers avoid some of the pitfalls in designing and implementing alarm management systems and in working more effectively with vendors.

• Consider desired alarm functionality when defining Functional Requirements (near the beginning of a project) when time is available for any customization needed (for example, to output alarms to pagers). Do not assume that vendor COTS will provide all desired alarm functionality.

• Define the “end in mind” (near the beginning of a project) regarding what use will be made of alarm records. This will influence the selection of alarm categories, priorities, etc. when designing/rationalizing individual alarms.

• With minimal exceptions, configure alarms only for abnormal situations requiring a response. Use other areas of the HMI/MMI (system permitting) for informational messages, or use color and other graphical means to distinguish between alarms and informational messages.

• Determine and document the expected operator response when designing/rationalizing alarms. Consider coding the expected response on-line. Remember that an effective alarm alerts, informs and guides.

• Include tools in historians (or third-party products such as alarm loggers) to allow for the querying, sorting, charting and batch reporting of alarm records. For example, a pareto chart could be used to depict alarm frequency.

• Implement a monitoring program in which continuous improvement is pursued (including reduction of nuisance alarms).

• Maintain the mindset that nuisance alarms can be a major frustration for operators, may contribute to them missing real abnormal situations, and, if excessive, may call into question the validation of the system.

• Consider developing a more “risk-based” change control procedure to expedite minor changes to automation logic. (For example, it should not take weeks to administer the adjustment of an alarm deadband value to reduce the incidence of a chattering alarm.)

• Recognize that alarm management is really a shared responsibility between process engineers (who are manufacturing process experts who should specify what alarms are required), automation engineers (who are computer experts and should implement the requested alarms) and operators (the customer). Assigning alarm management responsibility to only one or two of the above groups is often a formula for failure.

• Be familiar with cGMPs, Alarm Management Best Practices [e.g., 1,2], and ISA’s S-88 regarding Batch Processes. Become familiar with ISA’s S-18 Standard on Alarm Management when it becomes available (expected in 2008).

Vendors should be able to:

• Provide the ability to configure all alarm attributes (vs. hard coding).

• Ensure the ability to change any alarm attribute as a function of batch process state or batch step/phase.

• Provide several user definable fields in alarm record tags (e.g., users often want to include lot number, batch step/phase).

• Provide the ability to configure alarms using all appropriate information in the automation system (e.g., via “if-then-else” rules).

• Comply with 21 CFR Part 11 regarding electronic records/signatures (i.e., ensure that alarm records cannot be edited without appropriate authorization and audit trail).

• Provide the ability to view alarm information in relative time (i.e., from the beginning of the batch) in addition to calendar time.

• Provide software utilities (e.g., pareto charts) to help users analyze alarm information.

References

1. Alarms Systems: A Guide to Design, Management and Procurement, EEMUA, Pub. 191, 1999.

1. Hollifield, B., and Habibi, E., The Alarm Management Handbook; A Comprehensive Guide, PAS, 2006.

1. Alford, J, Stankovich, R., and Kindervater J., Alarm Management: Regulatory Expectations and Selected Best Practices, CEP, Spring, 2005.

About the Author

Joseph Alford recently retired as an engineering consultant for a major pharmaceutical / biotech company after a 35-year career. During his career, he was recognized within industry as has having advanced the state of the art of bioprocess analytical instrumentation, automation, modeling, process control and data mining technologies and applications. His employer recognized his contributions with several corporate engineering, IT, and product development technology awards. He was also honored with the Douglas H. Annin Award from the Instrumentation, Systems, and Automation Society (ISA) for his achievements in using advanced technologies in automated control applications, and in 2003, was elected Fellow of ISA. He has also received the AIChE Computing Practice Award.