This is the first in what I hope to be monthly series summarizing CDN performance using data from our cloud speedtest. Because it's the first post, I've also summarized data from all tests run to date - 2011 to present.
The cloud speedtest is a tool we created to measure network latency and throughput between real users and cloud services. When a user starts the test, we use Maxmind's GeoIP database to determine their location (city, state, country) and ISP. We then store the results of every test in a database. Since 2011, we've captured results from over 20 million CDN tests from 5.1 million unique users on 27,000 different ISPs.
The speedtest lets users run 3 types of CDN performance tests. Each test includes a brief warmup followed by multiple test iterations. We capture multiple data points from every test including mean, median and standard deviation. The metric the analysis in this post is based on is median.
- Network Latency: This test uses a small (8 byte) file to approximate network latency between the user and CDN. Latency is essentially how long it takes a CDN to receive and respond to a minimal request. Think of this as how long it takes the CDN to "pick up the phone". The unit for this test is measured in milliseconds (thousandths of a second). Latency is affected most by the distance and network path between user and CDN.
- Small File Throughput: This test downloads random small files ranging in size from 1KB to 80KB, and measures how fast the CDN is able to get them to the user. The unit for this test is megabits per second (Mb/s). A megabit is 1/8th of a megabyte (MB) - so if throughput is 8 Mb/s - this equates to 1 MB transfer per second. Small file throughput is affected more by latency because the requests are brief. Most web pages consist of files in this size range - so this test is more relevant to hosting of a website. Throughput tests use 2 simultaneous download threads.
- Large File Throughput: The methods used and metrics captured for this test are the same as the small files test, but the files are larger - 198KB - 5.1MB. During large file testing, the speedtest picks the largest file it can download 4 times in 10 seconds (bandwidth estimated during warmup and adjusted during testing). If you have a faster connection, it picks larger files. This test is affected more by the network capacity between user and CDN, and how fast CDN POPs are able to get files off disk and start sending bits to the user. A CDN may have great latency, but if their servers are overloaded and/or use slow hard drives, throughput could be slower than what the network is capable of. Many CDNs use SSD drives and caching to speed up delivery time.
CDN is an abbreviation for Content Delivery Network. A CDN is essentially a bunch of web servers you can host web content on. CDN servers are called points of presence (POPs), and are usually located in multiple data centers globally. A CDN speeds up web requests by responding to them using servers that are closer to the user.
For example, consider a situation where you have users in US, Europe and Australia, and a web server in the US. Your website responsiveness would likely be acceptable in the US, slower in Europe, and nearly unresponsive in Australia (where bits are traveling ~8,500 miles each way). Using a CDN in this scenario could dramatically improve performance if the CDN uses POPs on the same continent. Other CDN benefits are fault tolerance and offloading of traffic from your web servers.
Picking the Right POP
The most important task for a CDN is to speed up user requests by responding them using the fastest POP in its network for that user. Often, this POP is the one that is physically nearest the user - but this is not always true. CDNs use different methods to choose a POP when a request is received. The most common methods are:
- DNS: Domain Name System (DNS) resolves a textual hostname like www.google.com to a numeric IP address used by computers to communicate. A CDN can use DNS to send users to faster POPs by responding using different IP addresses depending where that user is located. This is the most common method because it is relatively easy and inexpensive to setup and administer. However, it is not full proof - there are some ways it can result in less than optimal POP selection. For example, if the GeoIP database is out of date or inaccurate (a common problem), or if the location of the client making the DNS query is not the same as that of the user (DNS queries originate from ISP resolvers - not from users). Reliability issues can also occur due to DNS caching if a CDN POP fails or is taken offline. Performance can also be impacted due to more frequent DNS queries.
- IP Anycast1: Uses Internet network routing instructions to automatically send users to the best POP. Setting up and maintaining IP Anycast is complex, costly, and in some areas of the world unfeasible. For these reasons only a few CDNs use this method. When used, Anycast has some advantages over DNS for POP selection:
- Performance can be better because POP selection is based on network distance as opposed to physical distance. Performance is further improved because DNS can be cached for longer periods
- POP selection accuracy is often better because it is based on the user's location, not that of an ISP DNS resolver. It is also not dependent on the accuracy of a third party IP database
- Anycast CDNs are often better at handling failover because routing instructions can be updated very quickly when a POP goes down or is taken offline (no DNS caching issues to deal with)
- Proprietary: Some CDNs use a combination of DNS and proprietary methods like monitoring network conditions between POPs and ISP networks and incorporating this data into POP selections
- DNS + IP Anycast: Some services combine DNS and IP Anycast where DNS is used to initially drop users into larger geographic areas where a CDN IP Anycast network or individual POP is present
- EDNS: A new DNS extension that forwards the user's IP address in DNS queries so the CDN can choose the POP based on that IP address and not that of the user's ISP DNS resolver (which can often be in a different location)
1 IP Anycast for CDN POP routing should not be confused with IP Anycast for DNS. Most, if not all CDNs use IP Anycast for DNS
The performance improvement a CDN can provide is constrained by the number and geographical distribution of its POPs. In North America and Europe it is relatively inexpensive and easy for a CDN to provision new POPs and bandwidth. Where this becomes more challenging and costly is in less connected regions like Asia, Oceania and South America.
This table lists each of the CDNs included in this post including the POP selection methods used and continent based POP counts. The CDN name links to a more detailed profile we've created for each service including pricing.
|CDN||Routing Methods||POPs per continent|
|North America||South America||Europe||Asia||Oceania||Africa|
|Akamai||DNS/Proprietary||Akamai claims to have over a hundred thousand servers in 75 countries. Many of these servers are located within ISP facilities|
|Level 3 CDN||DNS/Proprietary||19||3||13||8||1||0|
1 Some CDNetwork Asia POPs are located in China and Russia and require special permission to use
2 Some EdgeCast Asia POPs are located in China and require special permission to use
3 Limelight POP numbers and locations are estimated because specifics are not disclosed
4 MaxCDN POPs outside of North America and Europe require special provisioning and an additional monthly fee
CDN Performance - 2014 Summary
In this chart, each continent is represented by a different segment the length of which represents median latency for that region. A shorter segment represents lower (better) latency. The segments for North America and Europe are relatively similar for each CDN. Where we see larger differentiation is in the segments representing Asia, Oceania, South America and Africa with some services having much longer segments (higher latency) than others.
|Service||North America||Europe||South America||Africa||Asia||Oceania|
|Level 3 CDN||67.30||69.31||149.99||100.79||179.24||215.00|
Small File Throughput
In this chart, each continent is represented by a different segment the length of which represents median small file throughput (1-80KB files) in megabits per second for that region. A longer segment represents higher (better) throughput. Small file testing is more sensitive to latency because the requests are brief.
|Service||North America||Europe||South America||Africa||Asia||Oceania|
|Azure CDN (Microsoft)||2.82||2.00||0.67||0.99||1.37||1.03|
|Level 3 CDN||2.05||1.91||0.98||0.83||1.46||0.47|
Large File Throughput
In this chart, each continent is represented by a different segment the length of which represents median large file throughput (198KB-5.1MB files) in megabits per second for that region. A longer segment represents higher (better) throughput.
|Service||North America||Europe||South America||Africa||Asia||Oceania|
|Azure CDN (Microsoft)||15.54||13.33||3.40||2.35||5.62||4.88|
|Level 3 CDN||12.32||9.86||3.46||1.45||4.22||1.95|
CDN Performance: North America - 2011 to 2014
The remaining sections provide historical performance data based on test results collected between 2011 (when we first released the speedtest) and present.
North America - Latency
CDN network latency has consistently improved since 2011 based on our test results. Today North America latency test results for all CDNs in this post are closely grouped - around 40-70 milliseconds.
North America - Small File Throughput
Since 2012, CacheFly, EdgeCast and MaxCDN have been top small file performers in North America. Both CacheFly and MaxCDN use Anycast CDN networks in this region which likely helps to achieve better performance.
North America - Large File Throughput
Since 2013, CacheFly, EdgeCast and MaxCDN have also been top large file performers in North America.
CDN Performance: Europe - 2011 to 2014
Europe - Latency
CDN network latency has consistently improved in Europe based on our test results. Today Europe latency is tightly grouped at around 40-80 milliseconds for all CDNs in this post.
Europe - Small File Throughput
Since 2013, CacheFly, CDNetworks, EdgeCast, and MaxCDN have been top small file performers in Europe. Both CacheFly and MaxCDN use Anycast CDN networks in this region which likely helps to achieve better performance.
Europe - Large File Throughput
Since 2013, CacheFly, CDNetworks, EdgeCast, and MaxCDN have also been top large file performers in Europe.
CDN Performance: South America - 2011 to 2014
Testing activity in South America is much lower than North America and Europe. Because of this, the data provided here is based on smaller test populations and thus less robust and reliable compared to analysis in other regions.
South America - Latency
Latency improvements in South America have been less pronounced since 2011. Only 2/3rds of the CDNs in this post currently have POPs in South America. Akamai, Amazon CloudFront and CacheFly have all shown consistent latency improvement in this region with all 3 having 1 or more POPs there. Lack of a POP in South America is less detrimental than Asia or Oceania because distances to US based POPs are shorter.
South America - Small File Throughput
As the only CDN with both a regional POP and IP Anycast in South America, CacheFly is the top performer for small file throughput in South America since 2013.
South America - Large File Throughput
Since 2011, Akamai has consistently been a top large file performer in South America. During this time, Amazon CloudFront and CacheFly have expanded their networks to include POPs in this region and are now positioned with Akamai in the top 3 positions.
CDN Performance: Africa - 2011 to 2014
Testing activity in Africa is the lowest of any continent. Because of this, the analysis provided here is based on a smaller test population and is thus less robust than other regions.
Africa - Latency
Since 2011 Akamai and CDNetworks have consistently provided the lowest latency in Africa.
Africa - Small File Throughput
Akamai and CDNetworks have also been consistent top performers for small file throughput in Africa.
Africa - Large File Throughput
For large file throughput, Akamai and CDNetworks have been top performers since 2011. However, we have also observed good performance from CacheFly and Azure CDN.
CDN Performance: Asia - 2011 to 2014
Asia - Latency
There is a distinct separation between CDNs with Asia POPs and those without. For services with, median latency results generally fall around 100-140 milliseconds. Akamai has been a top performer in this region, but in the past few years it has become more competitive. CacheFly appears to be using mixed DNS and Anycast POP selection in this region (as opposed to straight Anycast in North America, Europe and South America). Our MaxCDN account did not have Asia POPs enabled, and this is likely the cause for its higher latency in this region.
Asia - Small File Throughput
Since 2013, CacheFly, CDNetworks and EdgeCast have been top performers for small file throughput in Asia.
Asia - Large File Throughput
Since 2013, Akamai, CacheFly, CDNetworks and EdgeCast have been top performers for large file throughput in Asia.
CDN Performance: Oceania - 2011 to 2014
Oceania is the continent where lack of a CDN POP presence is most pronounced in our test data. Due to its distance from other major Internet regions (~8000 miles from the US), a regional POP is necessary for a CDN to provide acceptable performance.
Oceania - Latency
There is a notable separation between those services with Oceania POPs and those without. For services with, median latency results generally fall around 80-120 milliseconds. Our MaxCDN account did not have Asia POPs enabled, and this is likely the cause for its lower throughput in this region.
Oceania - Small File Throughput
Since 2013, CacheFly and EdgeCast have been top performers for small file throughput in Oceania.
Oceania - Large File Throughput
Since 2013, CacheFly and EdgeCast have also been top performers for large file throughput in Oceania.