Colin Charles Agenda

I attended Clinton Roy’s excellent session titled An introduction to ANTLR: A parser toolkit for problems large and small. Now that the slides, and the video (1, 2) are online, I don’t know if my bits of notes are of any use (I made them while the tutorial was in progress), but they’re sitting on my desktop, and really should just get published. The files referenced below, you get via checking out the Antlr Tutorial Preparation wiki page.

Why use Antlr?
To parse configuration files, syntax highlighting, Domain Specific Languages (DSL), interpreters, translation/transformation.

Generates easy to follow code, LL(*) parsing algorithm. Bison is more powerful than Antlr. Compined lexer, and parser generator.

fun (int a, char b); <– as you do LL, till you hit the “;”, you have no idea if you’re dealing with a function or a declaration. Of course, there are look-ahead LL parsers too. An LL3 parser, which can see 3 tokens ahead, you still can’t see ahead enough, till you hit the ;. This is why, there exists an LL(*) - pick the smallest look-ahead, your grammar would need

Antlr, will help you get rid of using regular expressions.

Island grammars - one language, inside another (like HTML, inside PHP, or Doxygen inside C) are supported by Antlr.

Antlr Wiki is good, but hard to find things. Mailing list is great. The book by Terrence Parr is good, but out-dated, so go ahead, and get the online PDF version. A new cookbook/recipe list is coming out soon.

Using AntlrWorks. java -jar antlrworks.jar

conffile.g parses a = 1, b = foo.

IDENT   :       (’_'|’a’..’z'|’A’..’Z')(’_'|’a’..’z'|’A’..’Z'|’0′..’9′)*;
NUMBER  :       ‘0′..’9′+;
WS      :       ‘\r’ | ‘\n’ | ‘ ‘ {$channel=HIDDEN;};

The above are lexer rules. WS = whitespace. It reads from bottom up. White space, a number (0-9). IDENT will match foo, foo, foo1, but not 1foo (identifiers don’t start with numbers).

{$channel=HIDDEN;} <– IDENTs and NUMBERs get through the channel get through the parse. The whitespace, the parser sees them, but it will ignore them (i.e. hide them).

[-(byte@hermione)-(pts/6)-(11:21am:31/01/2008)-]
[-(/tmp/antlrworks)> l
total 248
drwxrwxr-x  2 byte byte   4096 2008-01-31 11:20 ./
drwxrwxrwt 44 root root 192512 2008-01-31 11:20 ../
-rw-rw-r–  1 byte byte    340 2008-01-31 11:20 conffile__.g
-rw-rw-r–  1 byte byte   8492 2008-01-31 11:20 conffileLexer.java
-rw-rw-r–  1 byte byte   4180 2008-01-31 11:20 conffileParser.java
-rw-rw-r–  1 byte byte     28 2008-01-31 11:20 conffile.tokens

conffile__.g - lexer file
conffile.tokes - tokens

CMinus.g takes input, which is a C program. Go to the interpreter, and you can then see the entire parse tree. Very impressive!

Technorati Tags: lca08, antlr, antlrworks

If CentOS (and by the same vein, Oracle Enterprise Linux) claims compatibility with RHEL, why is Zimbra saying that the issue with ZCS 5.0, Scalar::Util, and Perl, is caused by CentOS?

QA’ing against RHEL, and not CentOS is expected, but saying there’s no compatibility between CentOS and RHEL, sounds like a bit of a fib, don’t you think?

Even better, the recommendation to use Ubuntu. Will there be a LTS release, at some stage soon? It looks like Canonical are behind schedule for another LTS release…

Technorati Tags: zimbra, zimbra collaboration suite, zcs, zcs5, qa, scalar:util, centos, perl, compatibility, rhel

I took the opportunity today evening to get myself upgraded (from 4.5.3_GA_733) to the latest (5.0.0_GA_1869) open source version of Zimbra - ZCS 5.0 GA. The database migration took about the longest, mainly due to some schema changes. Lots of starts and stops to the database. Its now running MySQL 5.0.45 Community.

What prompted the upgrade? A few days ago, I got a bunch of new packages, and rebooted the server (new kernel). To my dismay, Zimbra started to have issues - amavisd wouldn’t start. This meant that there was a large amount of mail, sitting in the queue, not being delivered. Things you don’t normally check for, immediately, anyway.

Turns out Compress::Zlib was too old. Well, not the system provided Compress::Zlib, but the Zimbra provided Compress::Zlib. Kind of annoying when there are two packages of software, sitting on your system, right? However, the benefits of having an easy-to-administer and use mail system, somehow I think outstrips all the pain associated.

I found the web interface in ZCS 4.5.3 to be a bit limited, even when logged in as an administrator. There was absolutely no way to restart, failed services. For this, I actually needed to login via SSH, and use zmcontrol. Running SSH on a non-standard port, and not having your laptop nearby (or remembering the non-standard port) can allow you to have some fun :)

So after fixing ZCS 4.5.3, and realising that it had some gaping holes, I decided to upgrade. The upgrade process went on pretty smoothly, till I saw:
Updating from 5.0.0_RC3
5 is only avaliable with the XS version at /opt/zimbra/zimbramon/lib/IO/Socket/SSL.pm line 30
BEGIN failed–compilation aborted at /opt/zimbra/zimbramon/lib/IO/Socket/SSL.pm line 30, <DBCONFIG> line 21.
Compilation failed in require at /opt/zimbra/zimbramon/lib/Net/LDAP.pm line 970, <DBCONFIG> line 21.

This has largely got to do with the RHEL4 supplied Perl, as referenced by zimbra bug #22466. However, it seems that it was fixed in 5.0.0_GA_1809. Problem still seems to be around in 5.0.0_GA_1869. Verified that it existed - /opt/zimbra/zimbramon/lib/i386-linux-thread-multi/Scalar/Util.pm (and was newer than the version on the system). Verified that Zimbra saw it too - check out .bashrc in /opt/zimbra (the home directory for the zimbra user) for the various PATHs that Zimbra sees/requires. However, I was running this install, not as the zimbra user, so the Perl PATHs had to be specified.

Specifying the Perl PATH, also didn’t help. The forums mentioned just installing from cpan, Scalar::Util and letting the install progress. It still failed.

I thought I’d try a clean install. By golly, it failed on RHEL4. An upgrade of a clean install from ZCS 4.5.10 also failed. I’m almost convinced that Zimbra spent very little time QA’ing ZCS on RHEL4. Sure, RHEL5 probably works a charm, but the drive of enterprise software is not upgrading the OS too often. This is where I can so see, FreeBSD succeeding - pity there isn’t an official Zimbra/FreeBSD port.

For fun reading, check out their forums: [SOLVED] Big Fubar on 5 FOSS GA Upgrade (how was it solved?), Upgrade 4.5.7 -> 5.0 GA Failed, centos4 upgrade to 5.0 errors. I’m sure this magical list can go on and on. All purported solutions generally, do not work.

Moral of the story? Even with backups, don’t try upgrading Zimbra on a production box. Be prepared to cry, a lot.

Technorati Tags: zimbra, zimbra collaboration suite, zcs, zcs5, qa, ga, upgrades

Been playing around with The GIMP (about time), and one of the most useful features that might need to be drummed in, is for creating transparent images. The much touted Colour-to-Alpha plugin, simply just rocks!

Layer -> Transparency -> Colour to Alpha

Now, I don’t need to do dodgy hacks to get an image, made transparent.

Technorati Tags: graphics, gimp, colour to alpha, transparency, transparent images

Today, I was playing around with Linux, and RAW management. Not wanting to carry a laptop just for photo editing (i.e. I plan on travel without my MacBook Pro, and beautiful Lightroom, Aperture, iView Media Pro [okay, Microsoft Expression Media now], and Photoshop CS2), I figured I should make Linux up to scratch.

My favourite photo browser, is GQview. Its not standard software any longer, but its pretty old, and it works fairly well. It however, doesn’t support RAW. Today I discovered f-spot (ok, I discovered it ages ago, but I didn’t like its iPhoto-ness, where it wants to import stuff for you). Never fear, that’s what

    f-spot –view <path>

is for. Shortcut wise, its a little different, so I just use the up/down arrow keys to browse.

Now to convert the RAW files to JPGs, with preservation of EXIF data. I played around with the commands manually after reading the man page for dcraw, and figured there must be a better way. So I hopped on over to Yahoo!, did a little search, and came up with Jamie Zawinski’s mvpix. After changing the script a tad bit to suit my environment, I have images! RAW, JPEG, and copies of the JPEGs in an EDIT folder. mvpix also works on OS X.

The options being passed to dcraw seem to be simple: -w for using camera supplied white balance, -t 0 for no tilting, and -c for writing decoded images to stdout. Its passed to cjpeg, and run at 95% quality. However, the images seem to be a tad different, and I have no idea why.

On left, is when the camera was shooting in RAW+JPEG, and on right is the JPEG generated from the RAW file (click for larger image). These are 100% zooms.

The colour differences befuddles me. Why does the camera come out with “brighter” RAW, and dcraw come up with paler tones?

On left, is what the in-camera JPEG gives, when you’re shooting in RAW+JPEG, in the middle is what UFRaw sees from the RAW file, and on the right is what the converted JPEG looks like, after being parsed through dcraw+cjpeg (click for larger image).

What gives? Are there better options in dcraw to give me an as-close-to-in-camera JPEG experience? Are the converted images better/more true coloured? I’m not sure which to pick, and I’m not sure what’s the correct setting, to be honest.

Now about the photos. Apologies to the girl pictured here, all I know is that she’s probably a student at the Caulfield Campus of Monash University (so no, I don’t know her name). She was randomly picked for some quick studio photos, which was really just a bunch of us playing with a few studio strobes, and having the power of wireless triggers. These photos all rolled off an EOS 350D digital back, with a 50mm/1.4 lens attached to it, shot at f/1.6, at 1/25s. Sure, this should have really been made at f8, but there was no time to coax the girl.

Technorati Tags: linux, raw, cr2, gqview, f-spot, jpg, dcraw, mvpix, cjpeg, colour correction

There didn’t seem like a clear way to make a copy (or clone) of a virtual machine with VMWare Server. Not with the 1.0.3 build-44356 which comes standard with the Ubuntu Commercial repository (one of the good virtues of Ubuntu).

So, I fired up the console, and did:
cp -ax Centos\ 5/ Centos\ 5-new

It took 4m5.643s to copy this on my laptop hard disk (only 1.5GB). I loaded it into VMWare, via their Open a virtual machine option, renamed it in the inventory, powered my new virtual machine on and was asked if I’d like to create a new UUID for it. I’d advise you to create one, and once that was done, my virtual machine is ready to go.

Create a new UUID, and you’re set

Exactly what I like. Not re-installing CentOS everytime. Just create a “golden” image and start cloning from there. Maybe with some slack/puppet integration in the future, if I end up using a lot of VMs.

Technorati Tags: linux, virtualization, vmware, vmware server, cloning an image, image, clone, testing, testbed, test environment

Standardisation is important.

Tagging in Uploadr involves writing tags in the format such that its:
    australia victoria melbourne “notting hill” clayton

Tagging in ScribeFire, involves writing tags that are parsed in a different way (for Technorati):
    australia, victoria, melbourne, notting hill, clayton

Notice the commas (”,”)? Without them, your tags are all lumped together. I’m wondering if I should change Uploadr to similar behaviour as ScribeFire (or vice versa)? What do other applications do for tagging in a field?

It should be trivial to make this change, the question is if my patch will be accepted upstream. I’m already using a patched version of Uploadr, as I await the author to implement my patch (which adds a description field, which the Flickr API supports). Incidentally, PyGTK is pretty easy to get around with, with superb documentation making it easy for anyone to get on the bandwagon. More on pygtk programming later…

Technorati Tags: tagging, uploadr, scribefire, standardisation, tags, flickr, pygtk, firefox, gnome

MySQL has had this Enterprise Monitor around for a while, but I’ve not had the chance to try it. Well, all this changed briefly in July, when I was doing my APAC tour, and folk wanted to see Enterprise Monitor at work. Back then, I showed them some canned screenshots, and in Singapore, Kim Seong (famous on #mysql) from our partners Global Link, had some kind of special login, so they demoed it. It wasn’t until this week that I actually tried it (besides, it wasn’t until the UC-J that it was re-launched with new features), and found some time to play with it. And I was blown away.

An excessive table scan, you say?

With new features for replication monitoring, personally what I find most useful are the pretty graphs (you may not find it useful with one or two machines, but in an enterprise-level install base of databases, graphs are not only pretty, they are useful). On top of the graphs, there’s also advisors, which save you time, so that you can avoid reading the documentation, or finding pain points, yourself. I think this is what Brian really referred to on saving your Jedi’s time. Advisors, are crucial, and I think that’s what differentiates this product.

How to fix table scans… and links to the manual!

The UI is impressive. Its all very modern, looking very Web 2.0. Its Tomcat based on the backend, so the install file is a tad large, and the usual requirements that come with Java applications are present, but I’m pleased to say the server I’ve had access to play with is an under-powered AMD machine. You can set it to auto-refresh in your browser, which can be useful for maybe displaying it on a dedicated screen (think of being in a air traffic control tower, maybe?).

Shiny! Pretty graphs… showing useful information

Installation is somewhat easy (it could be easier, like maybe via apt-get or yum). Support diagnostics are interesting enough, it really does remind me of the sysreport utility in Red Hat/Fedora. It talks to 5.0.x and 5.1.x seamlessly, it doesn’t care what OS they’re on, and I’m unsure if the agent runs against say an older MySQL (though I don’t see why not).

Replication monitor (click for larger image)

The marketing materials on the Web don’t do enough justice for this useful tool. I wish there was an online demo of sorts or maybe the user manual was placed online, or just way more screenshots. Failing which, give the 30-day Enterprise Trial a go. Yes, Enterprise Monitor is not free :( I signed up a couple of days ago, spent about five minutes signing up (three email exchanges! Highly complicated), and got access to everything Enterprise-y.

Technorati Tags: mysql enterprise monitor replication