Horizontal Scaling with HiveDB

At the MySQL Conference & Expo 2008, Britt Crawford and Justin McCarthy, both from Cafepress.com, gave us a very interesting talk on scaling with HiveDB. I took a few notes (pasted below), their slides are online (warning: 6.1MB PDF), and if you’re after their abstract its available as well.

I also took a video of them (refer to Slide 12, for the IRC conversation):

The quick notes:

  • OLTP optimised (as it serves cafepress.com)
  • Cannot lock tables, or take it offline
  • Constant response time is more important than low latency (little slower query is ok, just not exponentially slower)
  • Queries run might return wildly sized result sets.
  • There can be growth and usage hotspots. You cannot predict this at all.
  • Partition by key (the set of all partition keys is the partition dimension)
  • Partitioned Hibernate from Google (Hibernate Shards). HiveDB is now married up with shards.
  • Thought about MySQL Proxy to support high availability components, but it was dismissed

Related posts:

  1. Scaling MySQL presentations
  2. EXPLAIN Demystified
  3. Federation at Flickr: A tour of the Flickr Architecture
  4. Scaling Twitter: “Is Twitter is UDP or TCP? Its definitely UDP.”
  5. Fedora News Updates #7

i