Posts Tagged ‘spider’


One thing you might have missed from the MariaDB 10.0.4 Alpha release announcement is that SPIDER is finally in-tree, inside MariaDB. I’ve been watching the SPIDER engine for quite some years and fondly remember the time when Kentoku Shiba dressed up as spiderman at a MySQL Conference.

There are many factors that plays to an engines success. Distribution is one of them. Having a company backing it with services is another, so it is with great pleasure that I also show you Spiral Arms Ltd by Kentoku and Goto.

Overall, a good time to celebrate and I’m very happy for Kentoku and Goto. Now to promote its use. There’s a lot of dated resources that we need to get back up to speed with, so I expect that in time the Knowledgebase entry will well populated.

Sharding for the masses: Introducing the SPIDER storage engine (OpenSQLCamp @ FrOSCon)

This is the Sharding for the masses: Introducing the SPIDER storage engine by Giuseppe Maxia, given at OpenSQLCamp, at FrOSCon, in August 2009. These are somewhat live notes, and the slides are available too.

Why sharding? Scaling, of course. The MySQL way to solve this, is replication (even Yahoo! and Google use this).

When the master doesn’t have enough resources to cope with what you do (i.e. large data sets), replication chokes.

You can use proxies for sharding. There exists MySQL Proxy (can be programmed using a scripting language – Lua), HSCALE (built on top of MySQL Proxy), SpockProxy (a fork of MySQL Proxy, without LUA scripting, specialised for sharding), in the market these days. This however, is the single point of failure – everything has to pass through one proxy.

Enter SPIDER – a MySQL storage engine, built on top of the partitions engine. It associates a partition with a remote server, and is transparent to the user. Its developed by Kentoku Shiba.

Installation: Get 5.1.37 sources, then get the source code for Spider 1.0, and then get the patch for condition pushdown.

Why the condition pushdown patch? Remote server works less, by receiving the condition. The SPIDER engine without the condition pushdown patch is still fast, but it can be more than 10x faster with condition pushdowns. (works with NDBCLUSTER), (works with MyISAM). The patch by Kentoku, will add cond_push and cond_pop, to ha_partition – so now, every storage engine that uses table partitioning can get condition pushdown through ha_partition.

You need to setup the engine first: (the SQL is also available in the DOCS).

spider_remote_employees.sql – use this in conjunction with – a good example of how to use the SPIDER storage engine.