Posts Tagged ‘flickr’

Compression algorithms, archival

My strategy for photos is to place them in my Dropbox folder, and sync them to one box in particular but they also live on the Dropbox cloud (it also used to be that they end up on Flickr as they sometimes still do; Google Photos does backup in regular resolution stuff from my iPhone). All this gets backed up via various backup services (Backblaze, CrashPlan).

Lately, I’ve been wondering if that makes sense since Dropbox applies compression to my JPGs. I missed the announcement of Dropbox using Lepton image compression, but did read with great interest that Flickr is using it too – A Year Without a Byte.

Lepton “provides lossless, bit-exact storage for any type of photo, whether it be for archival purposes, or for serving live.” It is open source.

I understand it saves Dropbox countless petabytes. I just don’t like the idea that my images are being re-compressed without me knowing about it. It may be that other services do this too. But one thing is for sure – the moment its in my Dropbox folder, they reduce in size, and my backups are also getting said reduction. Sure it’s lossless, bit-exact storage. But I’m thinking about archival forever!

P/S: I’m going to guess this is why Dropbox doesn’t support Live Photos in Camera Uploads yet. They haven’t figured out how to save space…

Backing up or taking out your data from Flickr with flickrdownloadr

I’ve been a Flickr Pro subscriber since 2005. I consume 75.36GB of storage, spread across 25,765 photos. I spent a lot of time on it since 2005, creating albums, joining groups, etc. I love the product and I’m paid up till 2018 and do not intend to stop subscribing to Flickr Pro. But I’m worried about Yahoo! and their future.

I read the Wired piece on how to get your photos off Flickr and thought it might be nice to have a backup of photos using their new Camera Roll feature. Simply put, I can get my data out but it involves me clicking “Select all” quite a lot. This is not a good solution naturally.

I found flickrdownloadr, a Mono app. It installed quickly on the Mac, it pulled in a whole bunch of Mono libraries, but after an initial login with Flickr, I would get an unhandled exception (you can’t tell from the GUI, it just crashes silently; I manually ran the app from /Applications/flickrdownloadr).

Unhandled Exception:
System.Net.WebException: Error: SendFailure (Error writing headers) ---> System.Net.WebException: Error writing headers ---> System.IO.IOException: The authentication or decryption has failed. ---> Mono.Security.Protocol.Tls.TlsException: The authentication or decryption has failed.
at Mono.Security.Protocol.Tls.RecordProtocol.ProcessAlert (AlertLevel alertLevel, AlertDescription alertDesc) [0x00000] in <filename unknown>:0
at Mono.Security.Protocol.Tls.RecordProtocol.InternalReceiveRecordCallback (IAsyncResult asyncResult) [0x00000] in <filename unknown>:0
--- End of inner exception stack trace ---
at Mono.Security.Protocol.Tls.SslClientStream.EndNegotiateHandshake (IAsyncResult result) [0x00000] in <filename unknown>:0
at Mono.Security.Protocol.Tls.SslStreamBase.AsyncHandshakeCallback (IAsyncResult asyncResult) [0x00000] in <filename unknown>:0
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult) [0x00000] in <filename unknown>:0
at System.Net.HttpWebRequest.GetResponse () [0x00000] in <filename unknown>:0
at FloydPink.Flickr.Downloadr.Logic.UpdateCheckLogic.UpdateAvailable (FloydPink.Flickr.Downloadr.Model.Preferences preferences) [0x00000] in <filename unknown>:0
at FloydPink.Flickr.Downloadr.Presentation.LoginPresenter.ApplyUser (FloydPink.Flickr.Downloadr.Model.User user) [0x00000] in <filename unknown>:0
at FloydPink.Flickr.Downloadr.Logic.LoginLogic+<CallApplyUser>c__async0.MoveNext () [0x00000] in <filename unknown>:0

It looked like the fix might have been related to this github issue and the best way to fix it was: mozroots --import --ask-remove. I did that. I tried mozroots --import --sync, especially after reading the Mono Security FAQ, but to no avail.

Turns out, the fix was cleaning out Flickr’s Sharing & Extending — there were accounts that had access for companies that had closed down (remember Qoop anyone?). But weirdly, there were two varying “flickr downloadr” authorisations (one had a version string attached to it).

Glad that was fixed but it seems like an app that would take a long time to actually get going. My next plan is to install this on a Linux box and sync it to an always on box. Presumably all this will “just work”, since Mono/GTK looks the same everywhere.

Its a shame that Flickr doesn’t provide an easy exit path. I mean I cancelled my Basecamp account not long ago, and it was not the refund bit that impressed me the most — it was the fact that I could take all my data out, and it would be a generated ZIP file that I could unzip and consume in my browser. Things just worked.

Here’s to Flickr’s long future. And I guess, Yahoo!’s…

Abusing MySQL (& thoughts on NoSQL)

The NoSQL/relational database debate has been going on for quite some time. MariaDB, like MySQL is relational. And if you read these series of blog posts, you’ll realise that if you use MySQL correctly, you can achieve quite a lot.

  1. It all starts with Kellan Elliott-McCrea with his introductory post on Using, Abusing and Scaling MySQL at Flickr. Follow the entire series.
  2. He starts of the series with Ticket Servers: Distributed Unique Primary Keys on the Cheap. Flickr scales using shards, and ticket servers give unique integers to serve as PKs.
  3. Richard Crowley talks about OpenDNS MySQL abuses. Nothing too out of the ordinary, but it shows MySQL getting the job done.
  4. Mikhail Panchenko talks about using The Federated engine for his series.

If you’re using the Federated engine, know that MySQL disables FEDERATED by default. In MariaDB 5.1.42, you get FederatedX, which is a maintained fork of FEDERATED, by the author himself! Bugs are fixed, and this is a supported engine, so if you’re using the FEDERATED engine, it might be wise to try out FederatedX.

I’d also like to bring to attention, an interesting essay by Dennis Forbes: Getting Real about NoSQL and the SQL-Isn’t-Scalable Lie. Monty says: “NoSQL is for very smart people who need a very sharp knife. People who are not capable of mastering SQL should not even attempt to try out NoSQL.”

Online photo printing comes to Malaysia – why not harness the Flickr API?

Found a cool Malaysian company, eoe. They apparently have physical stores, but what’s cool about them is online printing of photos — something which I believe is totally new in Malaysia.

They’re cheap – RM0.30/4R print. They’re even trying the viral marketing thing – get bloggers to write about them (no, I am not participating) in exchange for free photos.

I might try there services at some stage, but not today. Why? Because I have to upload photos through my browser. Their “Easy Upload” app displays a silly message saying they don’t support Firefox (so I don’t get the fancy pants editing of images, et al.).

<free advice>If they want to be a smarter Malaysian company, they would partner with Flickr, and harness the Flickr API, so that people can make easy prints from Flickr. After all, advertising for Flickr is already done for free, thanks to Maxis (they love advertising Flickr and how easy it is to use on their phone network). Printing images that are already there, rather than re-uploading (with slow, unreliable Streamyx) will make a whole lot more sense. Besides, each photo coming out of my camera, is probably 5MB in size — so I’ll pass.</free advice>

Here’s wishing Eoe much success, and hope they partner with the likes of Flickr, or just harness their darned API, for easy printing.

Some tabs – Marten interview, Facebook, Flickr

I’ve been collecting a bunch of tabs, MySQL related, that I think people might have missed during the holiday period.

Contrarian Minds: Marten Mickos – this is a great interview with former CEO of MySQL, now SVP of the Database Group, at Sun Microsystems. Its got a bit of interesting history, and thoughts about the future. There’s also some interesting photography.

Facebook is now at 150 million users. They grew quite a bit recently, it was just 140 million about a month or two ago.

Flickr has seen traffic reaching ten terabytes. As you know, Flickr runs MySQL, and they make use of InnoDB. Recently, Chief Operations Officer, John Allspaw, showed how fragmented one of their databases was. Reminds us all, that running optimize from time to time, is useful. Flickr is also using MySQL 5.0.51 currently.

Flickr make extensive use of Ganglia. I found that there are Ganglia graphs for MySQL metrics available now. Interesting stuff.

On fearing the continuity of online services

Today I read that co.mments bit the dust. Another web service (who remembers the I Want Sandy discussion a while back), ceasing to exist (though from what I see, a lot of folk are using Disqus more).

It got me a little worried. I rely quite a bit on online services.

  • Bookmarking, once previously living in my bookmark.htm file, now is shared on delicious. It has proven to be invaluable, storing 3,108 bookmarks. They are a Yahoo! run company.
  • Photo storage and sharing, once previously sitting in directories on my web server, are now kept on my Flickr account. Flickr is great, because I can share photos with just friends, family, or participate in a vibrant community of photo enthusiasts. I currently have 16,813 items stored there, with backups on various media sitting in my various homes. They are a Yahoo! run company, and I happily pay them for a Pro account.
  • I depend on Google Reader (read my shared items) to read RSS feeds. In fact, I have been sharing items as a form of bookmarking them. Ditto with adding stars to items. Don’t say Google doesn’t close services – they have.
  • I use Google Calendar, because it simply rocks. I also use Google Docs, and I also use GMail (hosted, and regular).
  • I use Twitter, who has no business model, as of yet. I like it over FriendFeed for one minor detail – I can update via SMS.

Most of these services have ways for you to get your data out of them, assuming they don’t exist in due time. But what will replace them?

Sure there are desktop applications. But with the variety of devices I utilise, I’m trying to cut down from using desktop applications and just focus on working online. In fact, all that is open now is Firefox, Adium, iTunes, TextMate (where I carve this text out), Terminal, Skype and twhirl. On my work laptop, its just Thunderbird, Firefox, Terminal, and Skype that’s open.

So maybe I need less desktop applications. It’s good, because that’s the hope of online services – live right in your browser.

But in tough(er) times, what do you do if the online service you use, disappears? Where’s the continuity (i.e. will my grandkids be able to browse my Flickr photo albums?)

I do wonder, if this will lead to more open source, peer-to-peer/federated run, online services. Like if Twitter folds up, who’s to say its excellent community won’t move to identi.ca ? (till then though, the latter probably doesn’t stand a chance, besides the very geeky top-of-the-trend open source folk…)


i