Genentech's DAWN Patrols Critical Assets Across Campus

Dec. 3, 2014
Biotech giant’s Data Acquisition Wireless Network provides unprecedented infrastructure transparency for transient lab equipment
Figure 1

As part of the DAWN infrastructure deployment, wireless repeaters like this one were installed at strategic points around the campus.

As wireless technologies expand farther into industrial automation and process manufacturing, organizations across the world are adopting and implementing wireless solutions to provide unique identifiers, along with the ability to transfer data over a network without requiring human-to-human or human-to-machine interaction. 

As the heart of its U.S. R&D center of excellence, San Francisco is a special place for biotech giant Genentech. Employing about 10,000 people, Genentech’s campus occupies a large area, encompassing 60 buildings within a two-mile radius. To support overall operational excellence, the site contains a dedicated facility-monitoring system that allows equipment owners to view live process values and monitor equipment for performance and alarm conditions via a common, accessible and unified system. Data collected by the system is also sent to a historian to preserve data generated by the many assets across campus.

Using this capability, Genentech’s operators monitor the condition of critical assets around the clock, ready to contact individual equipment owners if and when an alarm condition occurs.

Figure 2

Each individual ultra-low freezer has a group of devices installed, including a wireless module, antenna and power supply that works together to collect data from an array of sensors and sends it back to the central monitoring and control system through the network.

MOBILE ASSETS, WIRED INFRASTRUCTURE
As business requirements evolve at Genentech, critical portable lab equipment migrates from one building to another, causing discontinuity and generating the additional paperwork required for the myriad assets that require tracking. Trying to accommodate asset mobility while serving the business and regulatory imperatives associated with centrally monitoring and managing data with a wired infrastructure would be difficult at best. Traditionally, Genentech would decide to shift equipment into a building or lab without an existing facility-monitoring infrastructure. This caused delays because the labs needed to be retrofitted with a new data logger and associated wiring. Cost and delays during equipment migration could add up quickly depending on the condition of the building or lab.  

According to Vikas Bakshi, Genentech’s monitoring system owner for equipment similar to this, the advantages offered by the wireless technology were multifold. “Significant cost savings realized during the initial implementation due to additional channel capacity were extended by reduced resource requirements for data administration during normal equipment moves,” he said.

While there are thousands of critical assets across the site, ultra-low freezers are one of the most common equipment types. Many ultra-low freezers contain aging compressors that can potentially fail at any time. Monitoring the health of these compressors proved to be challenging due to the limited amount of I/O allowed on existing wired infrastructure. As discussed earlier, freezers often migrate from one location to another, so adding additional wiring to monitor the health of compressors helped drive costs higher during equipment migration and implementation. Monitoring specific equipment for compressor status and similar functional conditions like door open/close status was also needed to provide additional data to support predictive maintenance and reliability regimes and identify at-risk equipment prior to failure.

Figure 3

The basic DAWN module connects directly to the sensors through the I/O terminals. It interfaces with the larger network regardless of its location within the campus.

A NEW DAWN
Yokogawa and Genentech partnered to develop DAWN (Data Acquisition Wireless Network). It operates on the 900 MHz frequency band and employs a frequency hopping, self-healing and self-organizing mesh network. This mesh networking strategy allows the radio signals from the equipment to find the closest and most efficient route back to the gateway during migration without any reprogramming or reconfiguration (see sidebar below). In addition, mesh networking allows full network redundancy. If an infrastructure repeater goes down, the individual radios automatically connect to the next closest repeater within range.

Genentech’s DAWN delivers temperature, compressor condition and alarm status readings from each ultra-low freezer. The radio provides secure, reliable 128-bit encrypted wireless communication. Without additional hardware, each radio has the ability to act as a gateway, access point/repeater, and I/O device based on its configuration. The radio includes eight discrete I/O, four analog I/O and one Type T thermocouple module input. The system supports the Modbus TCP communication protocol.  

The majority of non-GMP equipment at Genentech is monitored by DAWN across 15 buildings. Each building is equipped with repeater radios strategically positioned across the facility to ensure wireless coverage for every lab, room and cubicle. The system can handle up to 2,400 radios across the entire site going back to five individual endpoint gateways. Each DAWN radio is configured to monitor temperature, compressor amps and door status to ensure the reliability and health of the equipment.

Endpoint gateways communicate via Modbus TCP and OPC back to the facility monitoring system so operators can monitor the health of each piece of equipment and take action when alarm conditions occur.

Genentech began to immediately realize benefits from its DAWN system despite the fact that its expansion across the campus had not been completed. Seamless equipment migration between buildings without reconfiguration or onerous paperwork generated significant data administration cost savings.

Similarly, managers commissioning new labs no longer have to purchase dedicated data loggers, wiring services and faceplates to network critical assets assigned to that particular lab. The ability to scale and expand the wireless system to monitor virtually any piece of equipment without being limited to the channel capacity of a traditional data logger was another cost-containing aspect of the system’s implementation. Live and historical compressor data and trends are monitored by a SCADA system.

Through DAWN and the data it delivers, technicians conducting ultra-low freezer preventive maintenance can predict the potential for compressor failure and prevent the disruptions and losses that might occur if its contents were to thaw.

Industrial Wireless Networking: Matching the Tool to the Job

Use of wireless networks in industrial environments has grown a great deal in the last five to 10 years along with developments of networking technology to make such applications more effective and economical. While sending data via radio is not new, earlier deployments were typically single point-to-point installations and weren’t very efficient or reliable.

New wireless networking technologies have been developed that are designed to move information bi-directionally from a large number of field devices to a central control system in a way that provides a similar kind of data transfer as wired networks.

These days it’s hard to find a place where there are no wireless Ethernet (Wi-Fi) networks, and many of the same techniques have been applied for ZigBee, ISA100.11a, WirelessHART, and many more. These networking strategies developed to fulfill some specific user case that called for a group of operational requirements. Usually those are defined in reference to: determinism (minimizing latency), bandwidth (amount of data), distance, power consumption, and reliability.Mesh networking turns every node in a network into a combination of receiver and transmitter. At some points a node is receiving, listening to the traffic on the network. Other times it is transmitting, sending along data collected from its sensors. In an ideal situation, when a node has a packet of data to send to the gateway, it will go there directly. However, a given node might not be able to contact a gateway for various reasons. In that case, it sends its data to a nearby node that has a better connection, and that node relays the information. The concept of self-organization gives the network the capability to establish these connections automatically. The network can also adjust the way it communicates when circumstances change and existing connections no longer work.The downside of mesh networking is that it can introduce latency when a given bit of information has to travel from node to node to reach the gateway. That time may be a matter of a few seconds or less, but in some applications this can present a problem. Effective network management can minimize latency across all data transmissions, and some signals can be assigned higher priority than others.
With the Yokogawa DAWN wireless system, each radio can include up to 13 I/O configured in various combinations. Each radio’s internal controller interacts with each I/O point and transmits information back to the gateway. While as many up to 2,400 radios can be used, none generate particularly large amounts of information.

Most industrial wireless networks cover distances of a kilometer or less, although radio signals may have to pass through walls and around large steel structures. Using more powerful radios and efficient antennas can overcome those obstacles, but increase power consumption.

For some networking approaches, power consumption is a major issue. Field devices that have to operate for many years with no power source other than an internal battery no bigger than one used in a flashlight, have to be carefully optimized to conserve power.

In the application at Genentech, this has not been a factor given that each node is attached to a freezer or other unit that has to be supplied with mains power to run its compressor. Nonetheless, each individual node can be outfitted with a battery that allows it to continue to sending an alarm message if power to the unit is lost.

For the DAWN system, Genentech and Yokogawa created a configuration suited to the needs typical of pharmaceutical and biotechnology research and manufacturing. The same sort of operational needs apply in many other types of applications in process and discrete manufacturing.

About the Author

Aaron Lee | DAQ Product Manager