Netflix is the big Kahuna of a Web media businesses, with 33 million subscribers in more than 40 countries. As Netflix's "watch now" streaming service has grown, the company has had to rethink its data and storage strategies to cope with ballooning workloads managed in the cloud. Today, the company is nearly complete in its migration from Oracle to the NoSQL database Cassandra, improving availability and essentially eliminating downtime incurred by database schema changes.
Netflix launched its streaming service in 2007, using the Oracle database as the back end. "We had a single data center, which meant we had a single point of failure," explains Adrian Cockcroft, cloud architect at Netflix. "We were approaching limits on traffic and capacity. Now that people can watch Netflix streaming programming from their phones, from Wii devices, Roku boxes, and many others, the demand for availability increases all the time. We have more customers every quarter, more customers are using streaming, and they're using streaming at a greater rate."
Data has grown as fast as the customer base, Cockcroft says: The number of API requests in January 2011 was 37 times higher than requests in January 2010. The company knew that outages or poor-quality streaming could drive away customers. "We knew we had to get out of the data center, so we could keep running and keep growing," Cockcroft says.
In 2010, Netflix began moving its data to Amazon Web Services. The next step was to replace its Oracle database with Apache Cassandra, an open source NoSQL database known for its scalability and enterprise-grade reliability. "For us, the problem with a central SQL database was that everything was in one place ii which is only convenient until it fails," Cockcroft explains. "And because these databases are expensive, you tend to put everything in there. Then everything fails at once."
Another problem was that schema changes required system downtime. "Every two weeks, we'd have at least 10 minutes of downtime to put in the new schema," he explains. "The limitations of a SQL database impacted our availability and scalability."
The company chose Cassandra from DataStax for its flexibility to create and manage data clusters quickly, particularly in the cloud. Christos Kalantzis, Netflix's manager of cloud and platform engineering, explains that "solutions like Oracle don't run very well on virtualized hardware ... the architecture of Cassandra and the availability and consistency tuning and scalability made it a clear choice." To address these problems, Netflix initially migrated data from Oracle to Amazon's SimpleDB distributed database -- and eventually moved that data to Cassandra.
"With Cassandra, we get better business agility," Cockcroft says. "We don't have to plan capacity in advance, we don't need to ask permission of other people to build things for us, and we don't worry about running out of space or power. I can create a Cassandra cluster in any region of the world in 10 minutes," says Cockcroft. "When the marketing guys decide we want to move into a certain part of the world, we're ready."
In addition, downtime due to schema changes is no longer required. "There are no schemas to change in Cassandra -- therefore, there's no downtime," Cockcroft says.
Of course, with multiple Cassandra instances, there are more systems to manage than with the monolithic Oracle database. But Cockcroft points out, "They tend to break at different times, so you're losing small pieces of the system, rather than the whole system at once. Also, each store is simpler to administer."
Today, Cassandra is the preferred database for almost all database needs at Netflix. The company stores 95 percent of all its data in Cassandra, including customer account information, movie ratings, movie metadata, movie bookmarks, and logs.
Netflix operates more than 50 Cassandra clusters with over 750 nodes. At peak they process more than 50,000 reads per second and 100,000 writes per second across all their clusters. On average they process more than 2.1 billion reads in one day and more than 4.3 billion writes in one day.
Cassandra is has provided a new, more solid foundation for Netflix. As Cockcroft says, "It makes it possible for us to deliver the service our members rely on."
This article, "Big movies, big data: Netflix embraces NoSQL in the cloud," was originally published at InfoWorld.com. Read more of Andrew Lampitt's Think Big Data blog, and keep up on the latest developments in big data at InfoWorld.com For the latest business technology news, follow InfoWorld.com on Twitter.