{"id":534,"date":"2007-04-25T14:16:36","date_gmt":"2007-04-25T19:16:36","guid":{"rendered":"http:\/\/www.bytebot.net\/blog\/archives\/2007\/04\/25\/a-storage-engine-for-amazon-s3"},"modified":"2007-04-26T12:34:02","modified_gmt":"2007-04-26T17:34:02","slug":"a-storage-engine-for-amazon-s3","status":"publish","type":"post","link":"http:\/\/www.bytebot.net\/blog\/archives\/2007\/04\/25\/a-storage-engine-for-amazon-s3","title":{"rendered":"A Storage Engine for Amazon S3"},"content":{"rendered":"<p><strong>A Storage Engine for Amazon S3, Mark Atwood<\/strong><\/p>\n<p>It looks mighty interesting, as transfers to Amazon S3 are free. I think it&#8217;ll work well in America and places where bandwidth rocks, but I don&#8217;t see this working too well in Australia. Oh how I wish the Internets will improve.<\/p>\n<p>Mark has got all his stuff online at <a href=\"http:\/\/fallenpegasus.com\/code\/mysql-awss3\/\">A MySQL Storage Engine for AWS S3<\/a>. He was also kind enough to upload most of the <a href=\"http:\/\/fallenpegasus.com\/code\/mysql-awss3\/presentations\/MySQLCon2007-handout.pdf\">notes<\/a>, which made my reporting easier, and don&#8217;t forget to view the <a href=\"http:\/\/fallenpegasus.com\/code\/mysql-awss3\/presentations\/\">presentation<\/a>.<\/p>\n<p>Traditional storage engines use the local disk.<\/p>\n<p>Networked engines: Federated, ODBC, HTTP, MemCacheD and S3 storage engine.<\/p>\n<p>What is S3?<br \/>\nContents can have 1 byte to 5GB. Amazon has several petabytes of storage for you to use :-)<\/p>\n<p>Owning your own disks kind of suck. Pay for storage, even before you use it.<\/p>\n<p><strong>&#8220;An empty disk costs the same as a full one&#8221;<\/strong> &#8211; pay a lot of money to put disks in data centers. RAID isn&#8217;t &#8220;excellent&#8221;, then what about disaster recovery?<\/p>\n<p>Can&#8217;t move existing database over, and the S3 storage engine isn&#8217;t ready for a full schema yet. There are hacks that allows this, but maybe it will be available next year.<\/p>\n<p>Over a billion items in a bucket, and they all come back in good time.<\/p>\n<p>A bucket is fully virtually hosted, you get a SQL CMS in the MySQL server. Save your EC2 work.<\/p>\n<p>S3 is very distributed (geographically) and asynchronous. Writes are replicated, so your data may be reordered (and delayed). So there are no temporal guarantees.<\/p>\n<p>Use the WHERE clause &#8211; otherwise it will do a full table scan, and you&#8217;ll be paying Amazon lots of money :-)<\/p>\n<p>The talk ended with about 20 minutes to spare, and I do certainly hope he hacks on it more for the next year. He&#8217;s also soliciting feedback, so try it out if you can. And now, to run to the remainder of the talk on Highly Available MySQL Cluster on Amazon EC2! Two Amazon talks, with emerging technology goodness, at the same time? Pfft.<\/p>\n<p>Technorati Tags: <a href=\"http:\/\/technorati.com\/tag\/amazon\" class=\"performancingtags\" rel=\"tag\">amazon<\/a>, <a href=\"http:\/\/technorati.com\/tag\/amazon%20s3\" class=\"performancingtags\" rel=\"tag\">amazon s3<\/a>, <a href=\"http:\/\/technorati.com\/tag\/mysql\" class=\"performancingtags\" rel=\"tag\">mysql<\/a>, <a href=\"http:\/\/technorati.com\/tag\/storage%20engine\" class=\"performancingtags\" rel=\"tag\">storage engine<\/a>, <a href=\"http:\/\/technorati.com\/tag\/networked%20engine\" class=\"performancingtags\" rel=\"tag\">networked engine<\/a>, <a href=\"http:\/\/technorati.com\/tag\/s3\" class=\"performancingtags\" rel=\"tag\">s3<\/a>, <a href=\"http:\/\/technorati.com\/tag\/mysqlconf\" class=\"performancingtags\" rel=\"tag\">mysqlconf<\/a>, <a href=\"http:\/\/technorati.com\/tag\/mysqlconf07\" class=\"performancingtags\" rel=\"tag\">mysqlconf07<\/a><\/p>\n<div class=\"sharedaddy sd-sharing-enabled\"><div class=\"robots-nocontent sd-block sd-social sd-social-icon-text sd-sharing\"><h3 class=\"sd-title\">Share this:<\/h3><div class=\"sd-content\"><ul><li class=\"share-email\"><a rel=\"nofollow noopener noreferrer\" data-shared=\"\" class=\"share-email sd-button share-icon\" href=\"mailto:?subject=%5BShared%20Post%5D%20A%20Storage%20Engine%20for%20Amazon%20S3&body=http%3A%2F%2Fwww.bytebot.net%2Fblog%2Farchives%2F2007%2F04%2F25%2Fa-storage-engine-for-amazon-s3&share=email\" target=\"_blank\" title=\"Click to email a link to a friend\" data-email-share-error-title=\"Do you have email set up?\" data-email-share-error-text=\"If you&#039;re having problems sharing via email, you might not have email set up for your browser. You may need to create a new email yourself.\" data-email-share-nonce=\"bbc3da7ee1\" data-email-share-track-url=\"http:\/\/www.bytebot.net\/blog\/archives\/2007\/04\/25\/a-storage-engine-for-amazon-s3?share=email\"><span>Email<\/span><\/a><\/li><li class=\"share-facebook\"><a rel=\"nofollow noopener noreferrer\" data-shared=\"sharing-facebook-534\" class=\"share-facebook sd-button share-icon\" href=\"http:\/\/www.bytebot.net\/blog\/archives\/2007\/04\/25\/a-storage-engine-for-amazon-s3?share=facebook\" target=\"_blank\" title=\"Click to share on Facebook\" ><span>Facebook<\/span><\/a><\/li><li class=\"share-linkedin\"><a rel=\"nofollow noopener noreferrer\" data-shared=\"sharing-linkedin-534\" class=\"share-linkedin sd-button share-icon\" href=\"http:\/\/www.bytebot.net\/blog\/archives\/2007\/04\/25\/a-storage-engine-for-amazon-s3?share=linkedin\" target=\"_blank\" title=\"Click to share on LinkedIn\" ><span>LinkedIn<\/span><\/a><\/li><li class=\"share-twitter\"><a rel=\"nofollow noopener noreferrer\" data-shared=\"sharing-twitter-534\" class=\"share-twitter sd-button share-icon\" href=\"http:\/\/www.bytebot.net\/blog\/archives\/2007\/04\/25\/a-storage-engine-for-amazon-s3?share=twitter\" target=\"_blank\" title=\"Click to share on Twitter\" ><span>Twitter<\/span><\/a><\/li><li class=\"share-end\"><\/li><\/ul><\/div><\/div><\/div>","protected":false},"excerpt":{"rendered":"<p>A Storage Engine for Amazon S3, Mark Atwood It looks mighty interesting, as transfers to Amazon S3 are free. I think it&#8217;ll work well in America and places where bandwidth rocks, but I don&#8217;t see this working too well in Australia. Oh how I wish the Internets will improve. Mark has got all his stuff [&hellip;]<\/p>\n<div class=\"sharedaddy sd-sharing-enabled\"><div class=\"robots-nocontent sd-block sd-social sd-social-icon-text sd-sharing\"><h3 class=\"sd-title\">Share this:<\/h3><div class=\"sd-content\"><ul><li class=\"share-email\"><a rel=\"nofollow noopener noreferrer\" data-shared=\"\" class=\"share-email sd-button share-icon\" href=\"mailto:?subject=%5BShared%20Post%5D%20A%20Storage%20Engine%20for%20Amazon%20S3&body=http%3A%2F%2Fwww.bytebot.net%2Fblog%2Farchives%2F2007%2F04%2F25%2Fa-storage-engine-for-amazon-s3&share=email\" target=\"_blank\" title=\"Click to email a link to a friend\" data-email-share-error-title=\"Do you have email set up?\" data-email-share-error-text=\"If you&#039;re having problems sharing via email, you might not have email set up for your browser. You may need to create a new email yourself.\" data-email-share-nonce=\"bbc3da7ee1\" data-email-share-track-url=\"http:\/\/www.bytebot.net\/blog\/archives\/2007\/04\/25\/a-storage-engine-for-amazon-s3?share=email\"><span>Email<\/span><\/a><\/li><li class=\"share-facebook\"><a rel=\"nofollow noopener noreferrer\" data-shared=\"sharing-facebook-534\" class=\"share-facebook sd-button share-icon\" href=\"http:\/\/www.bytebot.net\/blog\/archives\/2007\/04\/25\/a-storage-engine-for-amazon-s3?share=facebook\" target=\"_blank\" title=\"Click to share on Facebook\" ><span>Facebook<\/span><\/a><\/li><li class=\"share-linkedin\"><a rel=\"nofollow noopener noreferrer\" data-shared=\"sharing-linkedin-534\" class=\"share-linkedin sd-button share-icon\" href=\"http:\/\/www.bytebot.net\/blog\/archives\/2007\/04\/25\/a-storage-engine-for-amazon-s3?share=linkedin\" target=\"_blank\" title=\"Click to share on LinkedIn\" ><span>LinkedIn<\/span><\/a><\/li><li class=\"share-twitter\"><a rel=\"nofollow noopener noreferrer\" data-shared=\"sharing-twitter-534\" class=\"share-twitter sd-button share-icon\" href=\"http:\/\/www.bytebot.net\/blog\/archives\/2007\/04\/25\/a-storage-engine-for-amazon-s3?share=twitter\" target=\"_blank\" title=\"Click to share on Twitter\" ><span>Twitter<\/span><\/a><\/li><li class=\"share-end\"><\/li><\/ul><\/div><\/div><\/div>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true,"jetpack_social_options":[]},"categories":[23],"tags":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p4vJD-8C","jetpack_sharing_enabled":true,"jetpack-related-posts":[{"id":790,"url":"http:\/\/www.bytebot.net\/blog\/archives\/2008\/04\/15\/ahead-in-the-cloud-by-verner-vogels","url_meta":{"origin":534,"position":0},"title":"Ahead in the Cloud by Werner Vogels","date":"15\/4\/2008","format":false,"excerpt":"Ahead in the Cloud - The power of Infrastructure as a Service CTO Amazon.com, Dr. Werner Vogels Pretty much everyone in the audience uses Amazon! Announced: Persistent Storage for Amazon EC2. Hitting one page, might actually go to 250 different services, before the page is generated for you. Shows the\u2026","rel":"","context":"In &quot;General&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1451,"url":"http:\/\/www.bytebot.net\/blog\/archives\/2009\/04\/19\/its-a-storage-engine-world-after-all","url_meta":{"origin":534,"position":1},"title":"Its a storage engine world, after all&#8230;","date":"19\/4\/2009","format":false,"excerpt":"While Zack covered the storage engine and appliances sessions pretty well, I feel he's missed out on a few important new engines (or engine related talks): DDEngine will be represented, speaking about Automated data versioning with a storage engine Long time MySQL Community Member Beat Vontobel, will be presenting on\u2026","rel":"","context":"In &quot;MySQL&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2122,"url":"http:\/\/www.bytebot.net\/blog\/archives\/2011\/04\/15\/plugins-storage-engines-summit-for-mysqlmariadb","url_meta":{"origin":534,"position":2},"title":"Plugins &#038; Storage Engines Summit for MySQL\/MariaDB","date":"15\/4\/2011","format":false,"excerpt":"As is tradition after the O'Reilly MySQL Conference & Expo, there tends to be a storage engine summit right afterwards. This year it was expanded to also include plugins. I must graciously thank Facebook for hosting us at their campus, and giving us a rather healthy lunch, plus fueling us\u2026","rel":"","context":"In &quot;MariaDB&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2832,"url":"http:\/\/www.bytebot.net\/blog\/archives\/2013\/11\/30\/groonga-fulltext-search-library-for-cloud-web","url_meta":{"origin":534,"position":3},"title":"groonga &#8211; fulltext search library for cloud &#038; web","date":"30\/11\/2013","format":false,"excerpt":"This is an incomplete fragment from 2011. Figure its worth publishing this now, considering MariaDB is likely to get groonga in the near future. The groonga team have released MariaDB 10.0.6 binaries as well. This is all part of the mroonga\u00a0project. These were my quick notes from the groonga talk\u2026","rel":"","context":"In &quot;MariaDB&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2985,"url":"http:\/\/www.bytebot.net\/blog\/archives\/2014\/11\/03\/osquery-is-neat","url_meta":{"origin":534,"position":4},"title":"osquery is neat","date":"3\/11\/2014","format":false,"excerpt":"Facebook recently made opensource, osquery. It gives you operating system data via SQL queries! Its very neat, and you can test this even on MacOSX (it works on that platform & Linux). It is by far the project with the most advanced functionality, linked here in this post. I noticed\u2026","rel":"","context":"In &quot;MariaDB&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2816,"url":"http:\/\/www.bytebot.net\/blog\/archives\/2013\/11\/11\/mariadb-10-0-5-storage-engines-check-the-linux-packages","url_meta":{"origin":534,"position":5},"title":"MariaDB 10.0.5 storage engines &#8211; check the Linux packages","date":"11\/11\/2013","format":false,"excerpt":"Today before Ivan's tutorial, he told me that in the 10.0.5 virtual machine images he created, he couldn't find the Cassandra storage engine. I told him it had to be installed separately, and this is true - you have to install some engines separately! When you do a yum install\u2026","rel":"","context":"In &quot;MariaDB&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"amp_enabled":true,"_links":{"self":[{"href":"http:\/\/www.bytebot.net\/blog\/wp-json\/wp\/v2\/posts\/534"}],"collection":[{"href":"http:\/\/www.bytebot.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.bytebot.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.bytebot.net\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.bytebot.net\/blog\/wp-json\/wp\/v2\/comments?post=534"}],"version-history":[{"count":0,"href":"http:\/\/www.bytebot.net\/blog\/wp-json\/wp\/v2\/posts\/534\/revisions"}],"wp:attachment":[{"href":"http:\/\/www.bytebot.net\/blog\/wp-json\/wp\/v2\/media?parent=534"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.bytebot.net\/blog\/wp-json\/wp\/v2\/categories?post=534"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.bytebot.net\/blog\/wp-json\/wp\/v2\/tags?post=534"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}