The use of Ethernet for industrial automation has grown dramatically. One of the main benefits of moving from legacy fieldbus to Ethernet is the ability to connect the front office to the manufacturing system. This is possible because Ethernet is not a proprietary communication protocol. The non-proprietary nature of Ethernet allows engineers to mix and match equipment from different vendors and get competitive bids. This combination of better office-factory communication and open standards helped industrial Ethernet gain recent widespread acceptance.
But with these benefits come potential problems. As networks and the services they provide evolve and servers or user machines are replaced and upgraded, the likelihood of passing unwanted, often obsolete, protocols within the network increases.
Potentially more challenging is the existence of unknown protocols that may degrade the performance of the network. Unknown protocols are often caused by well-intended but uninformed employees who attach unauthorized devices, such as wireless access points, to the network. They can also be caused by traffic such as streaming audio from employees listening to Internet radio stations while working.
The impact can be devastating. Networks that control manufacturing processes need to deliver the right information at exactly the right time, without fail. If an instruction is delayed because unwanted or unauthorized traffic took priority, the entire manufacturing process can come to a stop. The resulting costs and production failures are the nightmare of every production engineer.
Each situation is unique, but knowing where to look and what to look for is critical. Identifying top talkers — the bandwidth hogs — can show which users are saturating the network. It can also help identify non-business related activities and prove valuable in isolating traffic patterns that are associated with viruses or worms.
Bandwidth Hogs
Bandwidth hogs tend to fall into three main categories
- Excessive broadcast traffic that affects end stations. Managers need to determine if the traffic is relevant.
- Unwanted protocols may indicate an obsolete or other incorrect device configuration, or unauthorized use of the network.
- Factory-default switch port settings that can cause large amounts of unnecessary traffic and contribute to intermittent network sluggishness.
Finding the sources of unwanted network traffic and taking steps to correct or eliminate the root causes can enhance network performance and help avoid future problems, but it can also be a time-consuming task without the proper tools and troubleshooting techniques. Using a tool specifically designed to analyze Ethernet traffic will help. In the following examples, Fluke Networks’ EtherScope Network Assistant is used.
Excessive Broadcasts
Broadcast traffic is a necessary part of virtually every network, but since each end station that receives a broadcast packet may need to do some processing, it is desirable to reduce the overall volume of broadcast traffic. Excessive broadcasts could also indicate a hardware or configuration problem or even potentially malicious activity.
In a typical network, the amount of broadcast traffic may be very small, or it could potentially overload the network. The first step is to measure the amount of broadcast traffic, then determine if it is excessive for the given situation. Since every network is different, this requires making several measurements over a period of time to establish a normal baseline. Then if there is suspicion of excessive network traffic (such as users complaining of a slow network), the current network traffic can be measured and compared to the baseline.
A portable network analysis tool such as EtherScope can track traffic both by type and by MAC address. MAC, which stands for Media Access Controller, is a specific device’s unique hardware number. By tracking broadcasts to the MAC address, you can quickly see which devices are generating the most broadcast traffic. EtherScope includes an automated device discovery capability and will associate received network traffic with the source device to create a "Top Talkers" view. By using this view and selecting "Broadcasts," you can instantly view the top sources of broadcast traffic.
(Click to enlarge image)A view of the Top Talkers can show which device is generating the largest amount of traffic.
EtherScope also discovers the Layer 2 topology (the switching layer) of your network. During the discovery process, EtherScope determines the switch and switch port that connects end devices to the network, allowing you to take a appropriate action which may include temporarily disabling the switch port while the problem is being investigated.
Unwanted Protocols
A portable network analysis tool allows a user to monitor all network traffic and review the types of traffic on the network. EtherScope automatically provides protocol statistics for an extensive list of protocol types and TCP and UDP ports. Combining protocol statistics with device discovery provides a simple way to determine what protocols are running on the network and who is using them.
(Click to enlarge image) A function such as Trace Switch Route will show the location of a device generating unwanted protocols.
When connected to a span port, the EtherScope analyzer monitors network traffic and automatically provides protocol statistics for an extensive list of protocol types. Unwanted protocols can be quickly identified. Clicking on a suspected rogue protocol will locate the top source devices generating that protocol. Clicking again on a specific device will locate the offender.
Factory Default Switch Configurations
Unwanted network traffic and even temporary network problems can occur as a side effect of factory default settings in a normally healthy network. Consider SpanningTree Protocol (STP) used in almost every switched network. Most vendors enable spanning tree on each switch port by default. This is a reasonable choice as it makes it easy to quickly connect a new device and also protects the network from forwarding loops as the network grows. When the state of an interface changes, for example, and connectivity to another switch is lost, STP utilizes a special Bridge Protocol Data Unit (BPDU) called a Topology Change Notification (TCN). This mechanism works very efficiently in a stable network and the presence of TCNs is normally not an issue.
(Click to enlarge image) An analysis tool can also be used to access and change the connectivity configurations within a switch.
A problem that can cause unexpected consequences is when the spanning tree is enabled on ports that do change state frequently. Since a TCN is generated when a port that was in the forwarding state goes down or when a port transitions to the forwarding state, including each time an end user connects to the network, the TCN process starts and affects each bridge in the spanning tree. In the worst case of a large network with many users connecting and disconnecting, the network can be in topology change status almost constantly. The impact on the network is that the bridge forwarding aging time (nominally five minutes) is reduced to an effective 15 seconds which can lead to a very high level of flooding as switches re-learn each link.
Use of a portable network analyzer can both determine if TCNs are generating unnecessary traffic and provide the user with a means to change a factory default switch port setting. EtherScope's Telnet or Terminal Emulator functionality can access the switch and set port configurations. Consult the switch documentation for the applicable switch configuration commands.
Conclusion
Unwanted traffic on an industrial Ethernet network is not only a nuisance to users, it can also halt production causing delays and major expenses. Understanding the possible causes and sources for unwanted traffic can be an important part of keeping a network clean and running efficiently. Combining the knowledge of where and what to look for with automated tools helps ensure uptime.
Visit www.flukenetworks.com/industrial for more details in industrial Ethernet troubleshooting.
About the Author
Mara White is Fluke Networks’ Marketing Manager for industrial Ethernet. She is an active ISA and ODVA member.
