The ThousandEyes Network test provides a Path Visualization view which is similar to the traceroute command line utility that is available for many operating systems (Windows systems have the tracert command). This article describes the reasons why devices may not appear on the Path Visualization or not display information, as well as steps that customers can take if they need to render devices or information which are not currently displayed.
A ThousandEyes Cloud Agent or Enterprise Agent performing a Path Visualization sends either TCP packets with SYN bit set or ICMP packets (Echo request; ICMP type 8) depending on the Network test's Protocol setting. Packets are sent sequentially to each "node" (routing device) between the ThousandEyes Agent and the target of the test. Each successive set of packets has the Time to Live (TTL) field in the IP header set to a value which should expire (TTL = 0) at the node being probed. Normally, a routing device discards a packet whose TTL = 0 and returns an ICMP Time-to-Live Exceeded packet (ICMP Type 11) to the Agent. For more information on the mechanics of traceroute, see the traceroute Wikipedia page.
Once a TCP or ICMP packet reaches the test target, the target normally returns to the Agent a packet which is either a TCP SYN-ACK packet if TCP is the test protocol (or TCP RST packet if the port is not open), or an ICMP Echo Reply (ICMP type 0) packet if ICMP is the test protocol.
A Path Visualization will display intermediate nodes between the ThousandEyes Agent and the target. The Path Visualization will also provide information about each node, such as the node's IP address and DNS hostname (if a DNS PTR record is available) as well as packet round-trip response time from the Agent. The graphic below shows a Path Visualization from the Cloud Agent in Denver to the test target 22.214.171.124, with the cursor over the fourth node from the Agent, displaying the device's IP and other information.
Node information not displayed
One or more nodes in the path may lack information (white node) or may be missing altogether from the Path Visualization. This often occurs with an Enterprise Agent installed behind a firewall. Below is a Path Visualization and results table from an Enterprise Agent behind a firewall (which is located before the first node in the path, but does not itself appear as a node; see below for explanation).
In the above Path Visualization, the firewall does not have a node displayed or a line in the table because it is configured not to decrement the IP header's Time-to-Live field, so generates no ICMP Time-to-Live Exceeded packets.
The subsequent intermediate nodes send ICMP Time-to-Live Exceeded packets in response to the Agent's TCP or ICMP packets (whichever the test is configured to use). However, the firewall filtering ruleset does not permit ICMP Time-to-Live Exceeded packets to reach the Agent, resulting in a "No reply (100%)" entry for Hops 1 through 6.
Once the Agent's TCP or ICMP packets reach the target (126.96.36.199, lb-c1.thousandeyes.com), the returned packets are no longer ICMP Time-to-Live Exceeded packets. The target responds with TCP SYN-ACK or ICMP Echo Reply, depending on the packets initially sent by the Agent. The firewall in this example can statefully associate these returned packets with the initial packets sent by the Agent, and permits them to reach the Agent. This is seen in Hop 7 of the table below the Path Visualization. From this information, the Path Visualization can tell that there are six prior hops, although the responses from those hops never reached the Enterprise Agent, hence the nodes being rendered in white (no node information).
One or more nodes in the path may be missing from the Path Visualization for several reasons. In the previous example, we saw a node (firewall) that was not configured to return ICMP Time-to-Live Exceeded packets, so was not rendered as a node.
If an Agent receives excessive numbers of consecutive non-responses at any point during the test, the Agent terminates the test and does not display results for the partial test, beyond a single "No reply" line. Only the Agent will appear on the Path Visualization. An example is below.
If an Enterprise Agent is placed behind a proxy server, and the Path Visualization is using TCP rather than ICMP, nodes beyond the proxy will not be displayed. See the ThousandEyes Knowledge Base article Path Visualization--edge firewall incorrectly shows single hop to destination for more information.
Increasing the amount of information displayed
To increase the amount of information displayed in the Path Visualization, try both TCP and ICMP as the Network test's Protocol setting. One may yield the desired information, or at least display more information. If the results of using a different protocol are not sufficient, the following steps can be used to increase the amount of information displayed. The steps are split into two groups: one for the intermediate nodes, and one for the target of the test.
If the Path Visualization does not display intermediate nodes, or provides incomplete or no information about intermediate nodes, use the following steps:
- Determine which devices in the Path Visualization do not have a node displayed or do not show complete information for the node
To determine if a node is missing, you will need knowledge of the devices that are actually in the path. Usually this information is only known about the networks under the control of your organization or an organization which can respond to inquiries from your organization.
- Determine which devices in the Path Visualization are controlled by your organization or an organization which can respond to inquiries from your organization
If the issue resides with a device you cannot affect, you will not be able to change the Path Visualization. For devices belonging to unrelated network providers, it will typically not be possible to have changes made on the provider's device. However, a node belonging to an unrelated organization which is not correctly displayed may be due to a problem on a device under your control. See the next step for more information.
- For each device your organization controls or can inquire about, determine the answers to the following questions:
- Does the device perform packet filtering?
Firewalls, routers, load balancers and other network devices can perform packet filtering. Also, the host operating system for a Linux package Enterprise Agent may have a packet filter such as iptables, or the target host may employ a packet filter which blocks the outbound ICMP or TCP packets from the Agent, or blocks the inbound ICMP Time-to-Live Exceeded packets returning to the Agent.
It is important to keep in mind that a rule permitting the outbound test packets from the Agent may not be sufficient, even in a stateful firewall or stateful packet filter. A second rule or setting permitting the inbound ICMP Time-to-Live packets may be required if the software does not associate and permit the inbound ICMP Time-to-Live packets.
The effect of packet filtering on the Path Visualization graph depends on which packets are filtered:
- Filtering the outbound TCP or ICMP packets will prevent nodes between the filter (possibly including the filtering device) and the target from appearing on the Path Visualization graph. This will typically include the target node.
- Filtering the returned/inbound ICMP Time-to-Live Exceeded packets returning to the Agent will render nodes between the filter (possibly including the filtering device) and the target as blank nodes on the Path Visualization graph. This will typically not include the target node.
In addition to packet filtering, network devices such as firewalls, routers, and load balancers can perform network address translation. Translation rules must be in place for NAT'ing the outbound ICMP or TCP packets from the Agent and for NAT'ing the inbound ICMP Time-to-Live Exceeded packets returning to the Agent.
Address translation can be done in several ways. For example, most implementations of port-based address translation (PAT, also called one-to-many NAT) typically will not translate the inbound ICMP Time-to-Live Exceeded packets returning to the Agent because ICMP packets lack port numbers in their headers. In contrast, a static one-to-one NAT should allow the needed ICMP Time-to-Live Exceeded packets returning to the Agent to be correctly translated.
The effect of missing address translation rules on the Path Visualization graph are similar to the effect of packet filtering.
To address address translation issues, check your device's translate rules, consult your security administrator or the documentation for the device, or contact technical support for the device's manufacturer.
Some devices are configured not to issue ICMP packets or a subset of ICMP packets. For example, a firewall may be configured not to send ICMP packets from itself in order to avoid network scanning or device fingerprinting techniques.
If a packet is received by a device which does not issue ICMP Time-to-Live Exceeded packets, the TTL may still be decremented. But after decrementing, if the TTL = 0 then the packet will be discarded without generating an ICMP Time-to-Live Exceeded packet. The effect is to render the non-TTL generating device as a white node (unknown) on the Path Visualization. See the Path Visualization Legend for an example.
To permit a device to issue Time-to-Live packets, consult your security administrator or the documentation for the device, or contact technical support for the device's manufacturer.
If your organization's security concerns require that a device not provide information (not issue Time-to-Live Exceeded packets) to unapproved destinations, a packet filter rule can typically be used to drop the packets before they leave the device if the destination is not the ThousandEyes Agent.
If the device does not decrement the value in the TTL field by 1 before routing the packet, then the device cannot issue an ICMP Time-to-Live Exceeded packet (ICMP type 11) because the condition for issuing ICMP type 11 packets is that TTL = 0. A firewall may be configured not to issue ICMP packets in order to avoid network scanning or device fingerprinting techniques, or a router in a multi-protocol label-switched (MPLS) network may not decrement the IP header's TTL for packets on a label-switched path.
If not decrementing the packet's TTL, the device will always forward the packet to the next node. The effect is to omit the non-TTL decrementing device from the Path Visualization graph.
To permit a device to decrement the Time-to-Live field of routed packets, check your device's configuration settings, consult your security administrator or the documentation for the device, or contact technical support for the device's manufacturer.
If the Path Visualization shows 100% packet loss at the target, use the following steps:
- If the Protocol field of the Network test is set to TCP, verify that the target has a service listening on the port configured in the Network test's Port field.
- Verify that there is no host-based packet filter on the target or a firewall just before the target that is blocking the TCP or ICMP packets sent from or to the Agent (Linux package-based Agents only).
NOTE: Determining the answers to these questions will often require packet captures on the appropriate interface of the device in question. Keep in mind that if capturing the Time-to-Live Exceeded packets, the source IP address of these packets is not the IP address of the test target, but rather IP addresses belonging to interfaces on the intermediate hop devices. Any packet capture filters or display filters in your packet capture application need to take this into account.
- Using the Path Visualization View describes the components in the Path Visualization view, including a legend for the graph's node rendering scheme.
- Firewall configuration for Enterprise Agents describes the firewall rules required for all the ThousandEyes tests which may be run on Enterprise Agents.