Racklabs - Thoughts on the Hosted IT Revolution

MapReduce at Rackspace

Posted by Stu Hood, Software Engineer, Mailtrust

The Hadoop project has been getting a lot of press recently, and for good reason: the project aims to create an open-source framework for commodity parallel processing. The two primary components of Hadoop are its distributed file system and MapReduce, which allow anyone with a cluster of computers to write simple code to perform tasks on massive datasets quickly and reliably. Although Hadoop’s version number (0.15.2 at the time of writing) indicates that it is beta software, it is quite stable, and already capable of scaling to thousands of machines.

Here at Mailtrust, Rackspace’s mail division, we are taking advantage of Hadoop to help us wrangle several hundred gigabytes of email log data that our mail servers generate each day. We’ve built a great tool for our support team that lets them search mail logs in order to troubleshoot problems for customers. Until recently, this log search and storage system was centered around a traditional relational database, which worked fine until the exponential growth in the volume of our dataset overcame what a single machine could cope with. The new logging backend we’ve developed based on Hadoop gives us virtually unlimited scalability.

The way it works is that raw logs get streamed from hundreds of mail servers to the Hadoop Distributed File System (”HDFS”) in real time, and scheduled MapReduce jobs run to index the new data using Apache Lucene and Solr. Once the indexes have been built, they are compressed and stored away in HDFS. Each Hadoop datanode also runs a Tomcat servlet container, which hosts a number of Solr instances that pull and merge the new indexes, and provide really fast search results to our support team.

Additionally, using MapReduce we are now able to look at our log data in all sorts of interesting ways. For example, we run nightly MapReduce jobs to collect statistics about our mail system, such as spam counts by domain, bytes transferred and number of logins. Now whenever we think of complex question about our customers’ usage patterns, we can pull the answer from our logs within hours via MapReduce. This is powerful stuff.

A few weeks ago we were curious about which regions in the world our customers are logging in from most frequently. So I wrote a MapReduce job that associates user logins to a geographical location (using MaxMind’s GeoIP data), and within a few hours we had the answer. This data was so useful that we’ve scheduled the MapReduce job to run monthly and we will be using this data to help us decide which Rackspace data centers to place new mail servers in as we grow. Below are the results from the MapReduce job if you are curious…

> Percentage of POP/IMAP logins by country for January 2008
>
> 83.2245% United States
> 6.3361 % Canada
> 2.1974 % India
> 1.7406 % Germany
> 1.1941 % Mexico
> 1.0838 % United Kingdom
> 0.3149 % Italy
> 0.2590 % Australia
> 0.2504 % Malaysia
> 0.2006 % China
> 0.1699 % South Africa
> 0.1373 % Brazil
> 0.1321 % France
> 0.1302 % Europe
> 0.1289 % Israel
> 0.1259 % Spain
> 0.1161 % Dominican Republic
> 0.1056 % United Arab Emirates
> 0.0968 % Argentina
> 0.0928 % Philippines
> 0.0899 % Singapore
> 0.0877 % Greece
> 0.0834 % Guatemala
> 0.0711 % Jordan
> 0.0698 % Panama
> 0.0554 % Costa Rica
> 0.0520 % Belgium
> 0.0519 % Vietnam
> 0.0499 % Switzerland
> 0.0477 % Kazakstan
> 0.0468 % New Zealand
> 0.0456 % Japan
> 0.0453 % Ghana
> 0.0417 % Thailand
> 0.0409 % Qatar
> 0.0381 % Netherlands
> 0.0376 % Egypt
> 0.0373 % Afghanistan
> 0.0353 % Romania
> 0.0350 % Hong Kong
> 0.0344 % Indonesia
> 0.0323 % Peru
> 0.0299 % Taiwan
> 0.0284 % Sri Lanka
> 0.0276 % Russian Federation
> 0.0276 % Norway
> 0.0264 % Venezuela
> 0.0264 % Ireland
> 0.0252 % Satellite Provider
> 0.0248 % Colombia
> 0.0238 % Pakistan
> 0.0230 % Cyprus
> 0.0229 % Puerto Rico
> 0.0229 % Kuwait
> 0.0225 % Austria
> 0.0221 % Portugal
> 0.0221 % Trinidad and Tobago
> 0.0220 % El Salvador
> 0.0203 % Poland
> 0.0200 % Czech Republic
> 0.0166 % Turkey
> 0.0165 % Nigeria
> 0.0155 % Iran, Islamic Republic of
> 0.0147 % Chile
> 0.0142 % Ecuador
> 0.0131 % Virgin Islands, U.S.
> 0.0130 % Finland
> 0.0127 % Jamaica
> 0.0126 % Denmark
> 0.0120 % Lebanon
> 0.0117 % Kenya
> 0.0117 % Netherlands Antilles
> 0.0116 % Honduras
> 0.0113 % Korea, Republic of
> 0.0112 % Bangladesh
> 0.0108 % Saudi Arabia
> 0.0096 % Turks and Caicos Islands
> 0.0083 % Ukraine
> 0.0078 % Sweden
> 0.0075 % Uruguay
> 0.0071 % Bahamas
> 0.0063 % Bulgaria
> 0.0057 % Morocco
> 0.0050 % Uzbekistan
> 0.0047 % Mauritius
> 0.0046 % Kyrgyzstan
> 0.0045 % Hungary
> 0.0041 % Cuba
> 0.0041 % Tanzania, United Republic of
> 0.0040 % Cayman Islands
> 0.0038 % Anonymous Proxy
> 0.0038 % Bahrain
> 0.0038 % Bosnia and Herzegovina
> 0.0036 % Belarus
> 0.0035 % Iceland
> 0.0032 % Mauritania
> 0.0032 % Tajikistan
> 0.0032 % Saint Lucia
> 0.0031 % Uganda
> 0.0030 % Oman
> 0.0030 % Bermuda
> 0.0027 % Paraguay
> 0.0025 % Latvia
> 0.0025 % Nepal
> 0.0025 % Tunisia
> 0.0024 % Cameroon
> 0.0024 % Syrian Arab Republic
> 0.0021 % Brunei Darussalam
> 0.0020 % Slovenia
> 0.0020 % Burkina Faso
> 0.0020 % Slovakia
> 0.0019 % Maldives
> 0.0019 % Nicaragua
> 0.0019 % Estonia
> 0.0019 % Iraq
> 0.0018 % Barbados
> 0.0018 % Sierra Leone
> 0.0017 % Asia/Pacific Region
> 0.0016 % Virgin Islands, British
> 0.0016 % Bolivia
> 0.0016 % Monaco
> 0.0015 % Saint Kitts and Nevis
> 0.0015 % Lithuania
> 0.0014 % Antigua and Barbuda
> 0.0014 % Suriname
> 0.0011 % Aruba
> 0.0010 % Libyan Arab Jamahiriya