1. Home
2. /
3. Server & Cloud
4. /
5. Network diagnosis with MTR

Network diagnosis with MTR

13 min read

MTR is an ICMP (ping) tool that allows diagnosing and narrowing down network problems. Through continuous traceroute/tracert, the routers (hops) that the IP packet passes through to the queried computer are displayed.

Ping is often used to test the network connection between two hosts. During a ping test, the source host sends an ICMP echo request to the target host. The target host responds with an ICMP echo reply. MTR reveals the hops between the source and target hosts by incrementing the TTL (Time to Live) of the IP packet by one step at a time. The TTL ensures that IP packets are not routed endlessly on the Internet. Each router through which an IP packet passes decreases the TTL by one. When a router sets the TTL to 0, it sends a message to the source host.

Installation

Linux

Debian / Ubuntu

apt update && apt upgrade
apt install mtr-tiny

CentOS / RHEL / Fedora

yum update
yum install mtr

Windows

For Windows, there is a port of MTR called “WinMTR”. This application can be downloaded from the Upstream-WinMTR website

MacOS

On MacOS, MTR can be used either with Homebrew or MacPorts .

brew install mtr

To install MTR with MacPorts, execute the following command:

port install mtr

MTR on Unix based systems

Generate MTR reports using the following syntax:

mtr -rw [destination_host]

For example, to test the route to the destination host example.com

mtr -rw example.com

Your server: bidirectional MTR reports

An MTR report for your server can be generated from your local computer. Replace the IP with the IP address of your server.

mtr -rw ip

Establish an SSH connection to your server. Replace the IP with the IP address of your computer (potentially your company or home network). If you don’t know your public IP address, you can use WhatIsMyIP.com .

- The -r option flag generates the report (short for —report).
- The -w option flag uses the long version of the hostname, allowing our technicians and you to see the full hostname of each hop (short for —report-wide).
- The -c option flag specifies how many packets are sent and recorded in the report. If not used, the default setting is 10.

MTR reports

The following MTR report examples were generated for the destination host www.internetx.com using the —report-wide option.

$ mtr —report-wide www.internetx.org
HOST: example                                                 Loss%   Snt   Last   Avg  Best  Wrst  StDev
 1. kabel                                                     0.0%    10    1.6   2.0   1.6   2.7   0.3
 2. ip53a9b757.static.kabel-deutschland.de [83.169.183.87]    0.0%    10   13.5  13.5  10.8  17.5   1.9
 3. ip5886cb91.static.kabel-deutschland.de [88.134.203.144]   0.0%    10   13.5  15.5  12.7  19.3   2.7
 4. 145.254.3.100                                             0.0%    10   12.0  15.8  12.0  19.5   2.8
 5. 145.254.2.185                                             0.0%    10   13.6  14.4  11.8  18.5   1.8
 6. bb01.muc02.net.internetx.com                              0.0%    10   13.5  16.3  11.8  21.9   3.7
 7. b01-vl2.muc01.net.internetx.com                           0.0%    10   20.0  18.6  13.1  27.6   5.6
 8. c02-vl902.muc01.net.internetx.com                         0.0%    10   16.7  15.4  12.8  19.2   2.1
 9. 247-56-236-85.customer-net.de                             0.0%    10   11.6  14.9  11.6  22.0   3.4

The —report option sends 10 packets to the target host. Without this option, MTR runs continuously. In most cases, the —report mode provides sufficient data in a suitable format.

Each numbered line in the MTR report represents a hop. Hops are the routing internet nodes that IP packets traverse to reach their destination. 

Since MTR works with ICMP ping, this tool determines the round trip time between the source host and the hops. The packet round-trip time is specified in milliseconds in MTR. The names of the hops are determined through reverse DNS lookups. If you wish to skip the rDNS lookup, you can use the —no-dns option. The full hostname is visible with -rw, which is short for —report-wide.

MTR provides statistics on the stability of this connection in the following seven columns.

• The “Loss%” column shows the percentage of packet loss at each hop.
• The “Snt2 column counts the number of packets sent.

The next four columns Last, Avg, Best, and Wrst are all measurements of the packet round-trip time in milliseconds.

• “Last” is the packet round-trip time of the last sent packet.
• “Avg” is the average packet round-trip time of all packets.
• “Best” and “Wrst2 show the best (shortest) and worst (longest) packet round-trip time to that host.
• “StDev” is the standard deviation of the packet round-trip time for each host. The higher the standard deviation, the greater the difference between the measurements.

In most cases, the average (“Avg”) column should be the focus of your attention.

Troubleshooting

In most cases, you can divide the MTR output into three network segments.

• Source Network: The first hop is the gateway of the source network. If you generate the MTR report from a corporate network, additional internal routers may be visible.
• ISP Networks: The second or third hop is your regional ISP. This is often followed by a Tier 1 ISP.
• Destination Network: The last two to three hops belong to the data center where the destination host is located. You can identify the redundant network backbone of InternetX by the net.internetx.com subdomain.

If you notice any irregularities in the initial hops near the source host while running MTR locally, contact your network administrator or local ISP. If you see irregularities near the target, please contact InterNetX Technical Support and send the MTR report to support@internetx.com.

Most issues in ISP networks indicated by MTR reports are temporary and resolved within 24 hours. In many cases, the Internet service provider’s monitoring has already reported the problem. If the service is impacted for an extended period, you can alert a provider to the issues you are experiencing. When contacting a service provider, send MTR reports and any other relevant data you may have. 

ICMP rate limiting

When analyzing MTR output, you often look for two main things: packet loss and latency.

However, with some service providers, it is common practice not to prioritize responses to the ICMP packets generated by MTR or even to discard ICMP packets. As a result, packet loss and latency measured for ICMP packets may not accurately reflect the conditions for IP packets encapsulating HTTP or SSH data. ICMP rate limiting often leads to latencies that are measurable only for specific hops:

$ mtr —rw www.icann.org
HOST: example                            Loss%   Snt   Last   Avg  Best  Wrst  StDev
 1. fw1.muc01.net.internetx.com          0.0%    10    0.3   0.4   0.2   1.2   0.3
 2. c02-vl13.muc01.net.internetx.com     0.0%    10   25.2  40.9  25.2 136.1  34.9
 3. 2001:4178:1::11d                     0.0%    10   10.0  13.8   0.7  20.5   5.9
 4. ipv6.decix-munich.core1.muc1.he.net  0.0%    10    1.0   1.1   1.0   1.5   0.2
 5. 100ge14-2.core1.fra2.he.net          0.0%    10    6.9   9.4   6.7  28.5   6.8
 6. e0-53.core1.ams2.he.net              0.0%    10   13.8  14.4  13.8  15.9   0.9
 7. 100ge8-1.core1.lon3.he.net           0.0%    10   19.7  19.1  16.9  24.2   2.8
 8. 100ge14-1.core1.lon2.he.net          0.0%    10   17.0  18.0  17.0  25.3   2.6
 9. 100ge13-2.core1.nyc4.he.net          0.0%    10   85.3  85.5  85.2  86.0   9.6
10. 100ge16-1.core1.ash1.he.net          0.0%    10   90.4  97.9  90.3 112.0   9.6
11. 100ge2-1.core1.lax1.he.net           0.0%    10  144.8 144.9 144.7 145.1   0.1
12. equinix-lax.icann.org                0.0%    10  144.3 144.3 144.1 144.5   0.1
13. www.icann.org                        0.0%    10  144.2 144.4 144.1 144.7   0.2

Latency and ICMP packet loss are often displayed simultaneously.

$ mtr —rw www.icann.org
HOST: example                            Loss%   Snt   Last   Avg  Best  Wrst  StDev
 1. fw1.muc01.net.internetx.com          0.0%    10    0.3   0.4   0.2   0.9   0.2
 2. c02-vl13.muc01.net.internetx.com     0.0%    10   25.2  25.5  25.1  27.2   0.6
 3. 2001:4178:1::11d                     0.0%    10    0.5   0.6   0.4   0.9   0.2
 4. ipv6.decix-munich.core1.muc1.he.net  0.0%    10    0.9   1.2   0.9   1.7   0.2
 5. 100ge14-2.core1.fra2.he.net          0.0%    10    6.6   6.7   6.5   6.9   0.1
 6. e0-53.core1.ams2.he.net             70.0%    10   16.4  15.2  14.2  16.4   1.1
 7. 100ge8-1.core1.lon3.he.net          20.0%    10   17.1  17.1  16.8  17.5   0.2
 8. 100ge14-1.core1.lon2.he.net         30.0%    10   16.9  17.2  16.9  17.4   0.1
 9. 100ge13-2.core1.nyc4.he.net          0.0%    10   85.2  85.4  85.2  85.7   0.2
10. 100ge16-1.core1.ash1.he.net          0.0%    10   91.1  97.1  90.4 104.9   5.2
11. 100ge2-1.core1.lax1.he.net           0.0%    10  144.6 144.7 144.5 145.1   0.2
12. equinix-lax.icann.org                0.0%    10  144.1 146.4 144.0 166.7   7.1
13. www.icann.org                        0.0%    10  144.2 144.4 144.2 144.7   0.2

ICMP packet losses can be attributed to routers implementing ICMP rate limiting and not responding to all ICMP packets. Asymmetric routing can also account for measured packet losses. In this scenario, ICMP packets reach their destination without errors but may struggle to follow the return route. Therefore, it’s best to collect MTR reports in both directions when an issue occurs.

MTR sends one ICMP packet per second but can also run slower or faster:

sudo mtr -rw -i 0.2 [destination_host]

If there is packet loss during a faster execution of ICMP ping, it does not necessarily indicate network overload. Routing network components implement ICMP rate limiting by default to counter actions like Denial of Service (DoS) attacks.

It is not necessary or practical to investigate or report all instances of ICMP packet loss in your connections. Internet protocols are designed to be resilient to network degradation, and the routes data travels over the internet can fluctuate in response to load, brief maintenance events, and other routing issues. If your MTR report shows small amounts of loss around 10%, there is no cause for genuine concern, as non-ICMP data streams are compensated for temporary loss by the transport and application layers.

MTR reports for common network problems

Some network issues are novel and require escalation to the operators of the upstream networks. However, there are common MTR reports available that describe frequent network problems. If you encounter a network problem and want to diagnose your issue, consider the following examples.

Network Latency

MTR can help pinpoint the cause of latencies between the source host and the destination host. Due to physical constraints, latency increases with the number of hops in a route. The increases should be consistent and linear. Unfortunately, latency is often relative and heavily dependent on the quality of the connections of both hosts and their physical distance.

When evaluating MTR reports for potentially problematic connections, consider previous correct reports in addition to known connection speeds between other hosts in a specific network segment.

The following MTR report shows that Hop 2 is causing latency:

$ mtr —rw www.internetx.com
HOST: example                                                       Loss%   Snt   Last   Avg  Best  Wrst  StDev
 1. 192.168.176.1                                                   0.0%    10    0.5   0.8   0.3   1.2   0.3
 2. dslb-084-057-017-003.084.057.pools.vodafone-ip.de (84.57.17.3)  0.0%    10   17.7  17.7  17.5  17.8   0.1
 3. 188.111.216.234                                                 0.0%    10   17.3  17.4  17.2  17.6   0.2
 4. 92.79.215.18                                                    0.0%    10   17.8  17.8  17.7  18.1   0.5
 5. 145.254.2.185                                                   0.0%    10   18.2  18.2  18.1  18.3   2.3
 6. bb01.muc02.net.internetx.com                                    0.0%    10   18.1  18.0  17.6  18.3   0.4
 7. b02-vl2.muc01.net.internetx.com                                 0.0%    10   18.2  18.1  18.1  18.2   2.1
 8. c02-vl904.muc01.net.internetx.com                               0.0%    10   18.4  20.9  18.5  23.6   2.2
 9. 246-56-236-85.rev.customer-net.de    
 

The latency jumps significantly between Hop 1 and 2 and remains high. In the subsequent hops, the latency increases practically linearly. Therefore, the DSL internet connection can be pinpointed as the cause of the overall high latency between the source host and the destination host.

Latencies in the measured packet round-trip time could also be caused by an issue with the return path in ICMP ping. The return path is not displayed in the MTR report, and packets can take completely different routes to and from a specific destination. Therefore, if possible, bidirectional MTR reports should be created to narrow down latency issues.

Firewall blocking ICMP responses

In the next example, it appears that the ICMP packets are not reaching the destination host, although this is not the case.

$ mtr —rw load-balance.customer-net.de
HOST: example                       Loss%   Snt   Last   Avg  Best  Wrst  StDev
 1. 192.168.2.1                     0.0%    10    0.5   0.6   0.5   0.7   0.3
 2. load-balance.customer-net.de  100.0%    10    0.0   0.0   0.0   0.0   0.0

The MTR report shows a 100% loss because Hop 2 is not sending an ICMP response. This could be due to network or firewall rules (e.g., iptables) blocking ICMP responses.

Residential Routers

Residential gateways sometimes cause misleading reports:

$ mtr —rw www.internetx.com
HOST: example                                                         Loss%   Snt   Last   Avg  Best  Wrst  StDev
 1. 192.168.176.1                                                     0.0%    10    0.5   0.8   0.3   1.2   0.3
 2. dslb-084-057-016-001.084.057.pools.vodafone-ip.de (84.57.17.3)  100.0%    10   17.7  17.7  17.5  17.8   0.1
 3. 188.111.216.234                                                   0.0%    10   17.3  17.4  17.2  17.6   0.2
 4. 92.79.215.18                                                      0.0%    10   17.8  17.8  17.7  18.1   0.5
 5. 145.254.2.185                                                     0.0%    10   18.2  18.2  18.1  18.3   2.3
 6. bb01.muc02.net.internetx.com                                      0.0%    10   18.1  18.0  17.6  18.3   0.4
 7. b02-vl2.muc01.net.internetx.com                                   0.0%    10   18.2  18.1  18.1  18.2   2.1
 8. c02-vl904.muc01.net.internetx.com                                 0.0%    10   18.4  20.9  18.5  23.6   2.2
 9. 246-56-236-85.rev.customer-net.de                                 0.0%    10   21.2  21.4  19.0  24.2   2.0

For Hop 2 100 % packet loss is reported, even though the round-trip time for the ICMP packets is also documented.

Routing Loops in the Network

Sometimes routers are misconfigured, and IP packets may never reach their destination, for example, when a router forwards packets in a loop.

$ mtr —rw www.meinserver.com
HOST: example                                                 Loss%   Snt   Last   Avg  Best  Wrst  StDev
 1. kabel                                                     0.0%    10    2.5   2.1   1.7   2.5   0.3
 2. ip53a9b757.static.kabel-deutschland.de [83.169.183.87]    0.0%    10   11.9  13.2  10.7  18.5   2.2
 3. ip5886cb91.static.kabel-deutschland.de [88.134.203.144]   0.0%    10   13.8  14.3  11.8  21.5   3.2
 4. 145.254.3.100                                             0.0%    10   17.8  14.3  11.8  18.7   2.3
 5. 145.254.2.185                                             0.0%    10   12.2  13.2  11.8  15.0   1.1
 6. bb01.muc02.net.internetx.com                              0.0%    10   13.8  19.6  13.2  32.5   7.0
 7. b02-te1-4.muc01.net.internetx.com                         0.0%    10   14.6  15.7  13.0  22.4   3.0
 8. c02-te9-1.muc01.net.internetx.com                         0.0%    10   19.2  14.9  12.8  19.2   2.2
 9. gw-fw.customer-net.de                                     0.0%    10   11.6  14.9  11.6  22.0   3.4
10. c01-te9-1.muc01.net.internetx.com                         0.0%    10    0.0   0.0   0.0   0.0   0.0
11. gw-fw.customer-net.de                                     0.0%    10    0.0   0.0   0.0   0.0   0.0
12. c01-te9-1.muc01.net.internetx.com                         0.0%    10    0.0   0.0   0.0   0.0   0.0
13. ???                                                       0.0%    10    0.0   0.0   0.0   0.0   0.0
14. ???                                                       0.0%    10    0.0   0.0   0.0   0.0   0.0
15. ???                                                       0.0%    10    0.0   0.0   0.0   0.0   0.0

This report indicates routing problems when connecting to the redundant network backbone. InternetX has a redundant and fully meshed backbone. We use the dynamic routing protocol OSPF here. Customer networks are typically redundantly connected to the InternetX backbone using the Virtual Router Redundancy Protocol (VRRP). Housing customers can connect their networks to the backbone using static routes. Issues may arise after modifications or hardware failures, as static routes do not adjust automatically.

Timeout

If MTR does not receive an ICMP response from a router within 5 seconds, it will display ??? for that hop. Timeouts can occur for various reasons. Some routers discard ICMP packets for Quality of Service (QoS) purposes or do not respond at all. Alternatively, there may be an issue with the return path.

$ mtr —report-wide www.internetx.org
HOST: example                                                 Loss%   Snt   Last   Avg  Best  Wrst  StDev
 1. kabel                                                     0.0%    10    2.3   2.0   1.6   2.4   0.3
 2. ip53a9b757.static.kabel-deutschland.de [83.169.183.87]    0.0%    10   18.0  14.3  10.8  25.7   4.5
 3. ip5886cb91.static.kabel-deutschland.de [88.134.203.144]   0.0%    10   11.6  16.7  11.6  21.4   3.3
 4. ???                                                     100.0%    10    0.0   0.0   0.0   0.0   0.0
 5. ???                                                     100.0%    10    0.0   0.0   0.0   0.0   0.0
 6. bb01.muc02.net.internetx.com                              0.0%    10   13.1  14.4  13.1  17.5   1.7
 7. b01-vl2.muc01.net.internetx.com                           0.0%    10   13.8  16.6  12.3  24.7   4.3
 8. c02-vl902.muc01.net.internetx.com                         0.0%    10   16.5  17.2  12.0  21.9   3.5
 9. 247-56-236-85.customer-net.de                             0.0%    10   13.3  15.8  12.1  23.0   4.1

Timeouts are not a cause for concern if ICMP packets reach the destination host without significant latency.

Advanced MTR techniques

Newer versions of MTR can send a TCP SYN packet to the destination host in TCP mode instead of an ICMP packet.

The TCP SYN packet is encapsulated in an IP packet, where the TTL is incremented as usual. This way, routers between the source and destination hosts respond as usual with ICMP when the TTL is set to 0. The destination host responds with SYN ACK if the HTTP port 80 is ready, and then terminates the connection setup with FYN ACK.

mtr —tcp -rw [destination_host]

The —tcp option flag can be used when the destination host is not reachable with ICMP ping packets due to firewall rules.

InterNetX technical support

If you encounter connection problems or have questions about the MTR report, please contact technical support at support@internetx.com. Attach the MTR report to the message for problem analysis.