Table of contents:
- How end-to-end network measurement is performed
- How the network path is discovered
- Reasons for loss result variation
- Tips for further isolating network path issues
- Further reading
If end-to-end measurement detects packet loss, the following figure demonstrates how said loss is presented to the user:
Figure.1: Packet loss detected in the end-to-end measurement
Figure.2: Path Visualization view
As outlined above, paths from each agent (nodes on the left-hand side) towards the target (the single node on the right-hand side) are rendered and merged where appropriate. Forwarding loss, if it would be detected by the path discovery process, would be indicated with a red ring around the relevant node in the Path Visualization graph.
For additional information about Path Visualization and why certain information might be missing (white nodes), consult the Reasons for missing information on the Path Visualization View article.
Timing differenceAlthough both, network end-to-end measurement and path discovery processes occur within a specified interval, they may not start at exactly the same moment. If loss events are sporadic or clear rapidly, they may resolve before path discovery is performed.
Additionally, tests are not guaranteed to be run at exactly the same moment from all agents assigned to the test.
Packet count differencePath discovery only sends three packets per TTL value while end-to-end measurement sends a stream of 50 packets directly to the target. Therefore, an individual lost packet has a greater statistical percentage in Path Visualization (33% of packets sent) compared to the end-to-end loss, where a single packet represents only 2% of packets sent. This variation in statistical probability causes differences in the reported loss between the two views.
Packet loss on the targetThere might be actual packet loss on the target while the path is clean. The traffic drop on the target might be caused by one of the following conditions:
- The SYN backlog on the target is full. The target can't accept new TCP connections and is ignoring all new SYN packets. This only applies to SYN-based TCP measurements.
- The target has reached the ICMP rate limit threshold and is therefore not generating new ICMP responses. This only applies to ICMP-based measurements.
- The target has implemented a rate limiting algorithm and is currently dropping incoming traffic. This condition usually manifests itself as an initial test round showing zero packet loss, followed by one or more test rounds with high packet loss. This alteration between no loss and high loss is often periodic, producing a clearly visible pattern.
- The recipient has an old TCP/IP stack with a "buggy" selective acknowledgment (SACK) implementation. This condition usually manifests itself as a constant low amount of packet loss in SACK-based TCP measurements.
- The target's computing resources have been exhausted. The server can't cope with the network traffic. Although not very likely, this option should not be discarded.
The forwarding loss might not be visible to all agents performing the test, i.e. when the loss is occurring in a network segment that is only traversed by traffic from a single agent. If packets from a single agent are traversing the lossy network segment and one of the Path Visualization probe packets does not get responded with the corresponding ICMP TTL exceeded message, then Path Visualization will not render the affected node with a forwarding loss (red circle). Instead, it will show the missing response as a parallel path with a blank node representing the missing response.
Once the network segment is traversed by traffic from multiple agents, then the missing Path Visualization probe responses are marked as nodes that exhibit forwarding loss, regardless if such loss is detected by one or multiple agents.
Inspect multiple test rounds of Path Visualization dataIf a small percentage of loss is consistently observed by the end-to-end measurement for a significant length of time, eventually even the path discovery process will hit the statistical "sweet spot" and show the forwarding loss. To see it, one may need to check multiple rounds of Path Visualization results to spot it.
Adding more agents to the testAdding more agents to an individual test decreases the statistical influence of false positives or transient events caused by problems local to any individual agent. More agents in different locations will give a more real-world perspective on traffic conditions by using various providers and paths to reach the target.
Adding more agents is especially helpful when the packet loss is occurring closer to the target. Paths from source to target are often wildly different at the start, but then they converge more and more the closer they get to the test target. This makes more and more packets' TTL counters to expire on nodes closer to the target, resulting in a more sensitive forwarding loss detection.
Running more parallel Path Traces
Isolating return path issues
Changing the test target
- In your original test's Path Visualization view, identify the furthest node that is functioning normally on the Path Visualization and will respond to pings.
- Configure a new Agent-to-Server test targeting the node identified above. Use the ICMP protocol.
- Once the initial test results are in, manually verify that the path displayed in the Path Visualization is identical to the original test.
If the results of this new test are free of packet loss and if the discovered path from the new test matches the one observed in the original test, this confirms that the forward path is working as expected, at least up to the node tested in the second test. The loss observed in the original test is, therefore, occurring either on the path between this furthest node tested in the second test and the target or on the reverse path from the target back to the agent.
The following resources provide further information:
- Network tests explained article delves into the packet-level details of how ThousandEyes network measurements are performed.
- Reasons for missing information on the Path Visualization view article explains conditions under which various bits of information might be absent from the Path Visualization.