Racklabs - Thoughts on the Hosted IT Revolution

How to Build a Storage System

Challenge to Rack Labs from the business based on real customer need:

Build us a storage system that is the following:

-Quick to Market:  We needed to develop the system with tools/ technologies that were well known at Rackspace and Mosso.

-Scalable: It needs to provide the same level of performance to 1,000 users as it does 100,000 users.  The system must scale horizontally.  As our customer base and their needs grow, so should our system.

-Dynamic: The space should grow/shrink with demand.  Users shouldn’t need to do capacity planning and over-buy/under-utilize space, or worry about running out of space.  Customers should only pay for what they use.

-Low cost: It needs to be inexpensive.  We were targeting $0.15/ Gig.

-Developer friendly: Make it easily accessible to developers.  It should support generic web-friendly interfaces as well as more traditional, language-specific technologies.

-Reliable performance: It should perform the same whether a customer is storing 5Gigs of data or 5TB of data.

-Redundant: Make sure that data will always be available. The system should keep multiple copies of the data and be built in a manner that is redundant at both the hardware and network layers.

-Secure: Data should be protected.  Ensure that data is secure and only available to the specific customer.  All traffic should be encrypted over SSL and data needs to be stored in an internal private network largely isolated from other Rackspace networks.

Rack Labs interpretation of this request:

How to build a Storage System - by Rack Labs

Business Design, the one liner:

Build us a Storage System that is easy to use, highly available, very secure, fast, extremely scalable,  simple to maintain, cost effective for the customer and oh, it has to be done as inexpensively and as quickly as possible.  A developer’s dream.

Lots of problems.  Let’s knock ‘em out.

1. Problem:  Finding good coders is hard.

Solution

Well first you find the smartest developer in the company and put him in a room and tell him you want a storage system.  Give him the room he needs to experiment, create and exercise his full capabilities.  When he starts showing some results, you add the second smartest guy in the company to the team and tell him to help.  Dim the lights, turn on the AC and supply ample amounts of Ruta Maya coffee and more smart folks will join.

2.  Problem: Gotta be cheap.  No fancy gear, heck, all you get is what we got.

Solution

Rackspace is a hosting company.  We have a ton of perfectly good older servers around that are not in use anymore. First of all you take some of those “seasoned” servers and you rebuild them from the ground up with exactly what you need.  Heck we’ve got the parts. We don’t need lots of processing power. We don’t need much RAM.  We need disk space and a lot of it.

Can we buy some big hard drives? Sure.

3. Problem: Build a super redundant system using a bunch of old servers.

Solution

Use a lot of them and make multiple copies of everyone’s data and stick them in different areas of the DC for good measure.  Build the system so that if any one server dies, just plug another one in.  The system is smart enough to start making copies when it sees a failure anywhere.

4. Problem: This system is going to run on a bunch of servers.  That means provisioning and maintaining them.  People are expensive.

Solution

We built a provisioning system that is completely plug and play.  The servers are easy to build and we do that hundreds of times a day for our other customers.  CloudFS servers can be slotted anywhere in the DC and they boot off the network.  If one fails, we just stick another one in its place.  Fast and easy.  We also built a monitoring and control console that is heavily automated and the entire system can be by just a small crew of folks.

5. Problem: Build a publically accessible system that is privately secure.

Solution

This was a challenge but we figured it out.  First of all we isolated the network so no one else in our DCs has access to it.  Then we layered on all the fun encryption you find in SSL.  We are confident that no one will know what anyone else is storing unless, of course, they want them to.

6. Problem:  This system has to be easy to use.  What is easy to us is not necessarily easy to everyone else.

Solution

Mosso and Rackspace.  The hard-core developers can code directly to language specific APIs but Mosso and Rackspace will be adding interfaces that will make it simple enough for our parents to possibly use (still trying to keep email a mystery to them though!)  Interested in being a partner in this?  The door is open.

7.Problem:  Quick to market.  How many times have we heard this before?

Solution

This one was not easy to solve.  We think “quick” is 2 years.  They think it is 1.  We hired a few more folks to help out and we stay up late at night.  The nice thing is that we get to release something solid.  Does it have all the bells and whistles we want?  Not yet, but we are working on it.  What the Beta testers will see is only the beginning.

The Result - Today Mosso is announcing a private beta for a new storage system called CloudFS. CloudFS is ready for testing. We are looking for a select group of beta testers to help us put it through its paces. We know we’ve got some more work to do to officially release this as a product, but there’s nothing like real users to help “push” us along. If you’re interested head to our CloudFS page on Mosso.

Want to help out?  Know Python or Java?  We need you. Email us: 0XC0FFEE@racklabs.com

(Posted on behalf of the Rack Labs team)