Note that this doesn’t really provide full emulation or proxying of a site, you need to take into account cookies, etc. to do that correctly.

Here is how I most recently cracked it (in C# .NET).  This isn’t fool proof; however it should work with most HTML:

Okay, so I’ve written a CMS.

Not a huge deal I know, but there are a couple of things about this CMS that make it a bit different.  Check it out.

The reasoning behind creating it was I had recently moved to London and was looking for work.  I thought it might be a good way to up-skill a little and have something to show a little bit of Java experience when it comes to web.

I’ve also kinda wanted to do some work with the Google Application Engine since it was released, and I figured that this was a good opportunity.

Turns out I got a job doing .NET!  Come to think about it, I’ve actually got more Java here than PHP… hmmm.  Just to clarify, I have done more PHP than anything else, but I do like to have experience across a range of platforms.

So, I decided to finish it off and publish it.  It’s based on the Google App Engine, which gives it a number of advantages (with a couple of drawbacks).  Working with a No-SQL database was an interesting experience, but given how difficult some of the simple things can be (aggregates anyone?) I think there would need to be some strong reasoning behind the decision to use them.

Anyway, check out the website, have a play, give me feedback.

I like the huge storage, the reliability (I know they’ve had the occasional issue, but it’s nothing compared to some other services), the interface with the vim-like shortcuts and I simply love the searching capabilities.

When Google finally enabled IMAP support for Gmail I decided it might be worthwhile to move my work email over to it (people may recall my love for IMAP).  This meant I could still use Outlook with IMAP and get all of the benefits such as being able to read calendar invitations, RTF formatted emails as well as the horrendous HTML generated by client with Outlook, but it also gave me the added features of Gmail.

Unfortunately I had about 4GB of email stored in my old Outlook PST.  I really wanted to migrate this over to Gmail as well to take advantage of those sweet searching features to help me navigate through the vast minefield which is my email archive.

So! I tried drag-dropping the emails from my PST file over to the Gmail IMAP store through Outlook, and started the migration.  My Outlook almost instantly hung while each email was uploaded to Gmail at what seemed a glacial pace.  Indeed, I left it for a weekend and managed to get a couple of gig up, but my outlook had completely crashed by the time I walked in on Monday.

So, I thought to myself “there has to be a better way”, and started searching for a migration tool.  Unfortunately I couldn’t find one.  No probs I thought, I’ll just write my own… only there wasn’t much in the way of libraries that could read PST files.

First I tried libpst, which is a cute little C library.  I hadn’t really done any development in straight C before, so I decided I’d try to make a little ncurses app to read my PST files to test out the library.  I proceeded with moderate levels of success.  Unfortunately I ran into a few issues, the first being my deep-seated and intense hatred for C as a language.  The second and slightly more fatal one was that the library didn’t work particularly well for me; it would crash when reading any emails under the Inbox and it would simply hang when trying to read my larger 4GB PST file.

I then came across libpff.  This is another C library that was built from libpst. By this time however, I had had a gutful of programming in C.  I also suspected it may have some of the scalability issues present in the original libpst.  It was also a little difficult for me to work out how exactly to use the library as C isn’t one of my strong points and there isn’t much in the way of usage / API documentation.

So, I did the only reasonable thing that a self-respecting geek would do: started to write my own library in Java.  Thankfully, Joachim Metz of Hoffmann Investigations who was responsible for the vast majority of the work in libpff also studiously documented the file format as he went and it looked like there was more than enough there to make a start.

The library (java-libpst):

Okay, so I’ll admit that I was biting off a bit more than I could effectively chew, let alone swallow.  It turns out that PST Files are all little-endian, and Java is, of course, big endian.  Oh, did I mention that PST files are also encrypted by default using a substitution cipher? But… not completely encrypted… just enough of them to make it difficult, oh! all the RTF items are compressed as well.

There is also the matter of the reasonably important unknown bits of the file format which were not yet documented.  For example, there is this B-Tree in the file format that allows you to quickly locate an item within the file.  Which is great! only, there didn’t seem to be a way to find out what child items may be attached to a parent item (like a folder).  Oh! you can find out who your parent in easy enough, the id of the parent is sitting right there… but the child items… no sir!  This was a bit of a problem you see because as with most tree structured files, you start from the top and work your way down.

I managed to work around this by reading the entire contents of the B-Tree and creating my own inverted map of parent to child objects.  It turns out that this wasn’t really that bad of a solution and only takes a few seconds to read through them all and create the map even on huge PST files.

Frustratingly, soon after I had managed to get this part of the library to work, a new section of the file format was documented which describes “related” nodes, which contains, among other things, a list of all the child items.  Sigh.

I also had a few difficulties understanding some of the finer points of the documentation, like when data tables fall across different blocks with external lookup tables.  There was also a fun point when I started to hit “named items”, which are basically keys that change meaning from file to file.  The identifier that denotes a recipient’s email address may, for example, be a different identifier in a different PST file!  Once again, this is only in some instances (yay for consistency!).

After much head scratching, reading, and staring at massive amounts of Hex encoded data, I managed to get the library working well enough to navigate the PST structure and extract items including emails, attachments, contacts, calendar items, etc. relatively quickly without crashing too horribly.

The result is available on Google code here:  http://code.google.com/p/java-libpst/

But my work was not yet done.

The migration app (pst2gmail):

Of course, the real goal was to create an application that could migrate all of your email from your legacy PST file to Gmail.

With the library working, this part was a lot more straightforward, and a mostly working alpha can be downloaded from here: http://code.google.com/p/pst2gmail/

I did have grand plans of having the system migrate your contacts and calendar items as well, which wouldn’t actually be that difficult to do, however my personal development focus has once again shifted and this functionality will probably remain disabled and unimplemented until people start to howl for it to be implemented.

Until then, please enjoy, the emails are uploaded through IMAP, flagged messages are passed through as starred items, folders are automatically created as tags, attachments and all that work, you can even resume your upload through the generated log files if you need to stop the process midway through!

I’m yet to check it out, but it looks promising: http://www.microsoft.com/downloads/details.aspx?FamilyID=61bf87e0-d031-466b-b09a-6597c21a2e2a&DisplayLang=en

Unfortunately I didn’t keep backups of my posts.  It’s strange, you don’t really count these sorts of things as important and therefore it was overlooked with my normal backups.  Thankfully, I was saved by the Way Back Machine, which had a copy of my Jasper post which was destroyed.

Google is currently in the process of removing me off their blacklist.  Apparently it can take a bit of time.

A few reasons for the lack of posts and generally not being around.  Firstly I went through Europe, and caught up with Nick, Haley, Havard and Alex, which was awesome.  Secondly, I have been completely swamped at work and it is a little difficult to be geeky at home when you have had a rather full-on day.

Hopefully this need to update my blog will prompt a bit of a resurgence in posting.

I’ve done a bit more work with Jasper using PHP and I can say that a few simple steps can make life infinitely easier.

First of all: Setting up and using the PHP/Java Bridge.

You don’t need to use the extension, there is a library written in pure PHP that you can use as a fall-back. When populating a report with Jasper, you don’t use that many Java calls so the reduced performance should not impact on your application too much. If you are calling a fair bit of Java from PHP, I would attempt to get the extension working as the performance is increased dramatically.

Here is a nice little function from the PHP/Java Bridge examples that detects if the extension is loaded and falls back to the native bridge if it isn’t available:

/** * see if the java extension was loaded. */ function checkJavaExtension() { if(!extension_loaded('java')) { $sapi_type = php_sapi_name(); $port = (isset($_SERVER['SERVER_PORT']) && (($_SERVER['SERVER_PORT'])>1024)) ? $_SERVER['SERVER_PORT'] : '8080'; if ($sapi_type == "cgi" || $sapi_type == "cgi-fcgi" || $sapi_type == "cli") { if(!(PHP_SHLIB_SUFFIX=="so" && @dl('java.so'))&&!(PHP_SHLIB_SUFFIX=="dll" && @dl('php_java.dll'))&&!(@include_once("java/Java.inc"))&&!(require_once("http://127.0.0.1:$port/java/Java.inc"))) { return "java extension not installed."; } } else { if(!(@include_once("java/Java.inc"))) { require_once("http://127.0.0.1:$port/java/Java.inc"); } } } if(!function_exists("java_get_server_name")) { return "The loaded java extension is not the PHP/Java Bridge"; } return true; }

It assumes that you have the PHP implementation in the “java” directory. See the guide below for downloading this.

Second: Classpath is still screwy.

There is something about the bridge that I cannot figure out. Regardless of environment variables, they way you use java_require, settings in php.ini or any other setting this still seems to stuff up. There is one solution that I attempted to steer away from as I did not see it as ideal, however please believe me when I say that this single step will save you hours, if not days of frustration.

Put all the Jasper libraries into your jre lib/ext directory. Trust me.

I have got things tantalisingly close to working with other methods of making the libraries available to the bridge, and even succeeded on the odd occasion. However this is the only reproducible guaranteed way to get rid of those class woes once and for all.

By using the PHP implementation of the bridge and putting your jars into the lib/ext directory, you should be able to get this to work on any platform. I have got it working on Windows and Linux, but by removing any reliance on external settings this should work on pretty much any platform.

Third: Use the JDT compiler.

This little tid-bit of gold was dropped on my blog by Lucian, one of the primary contributors to Jasper’s continued development (thank you).

If you have the jdt-compiler jar in your lib/ext folder as well, the problem of java having a different running directory to that of PHP becomes a non-issue. Most importantly however, this removes the need to set environment variables!

With my previous approach to getting jasper working, I resorted to writing a wrapper class primarily because you can not set the “jasper.reports.compile.temp” and “jasper.reports.compile.class.path” through the bridge (it doesn’t work). Setting these gave the javac compiler the information it needed to compile the report, without them it failed horribly. The JDT compiler does all this stuff in memory, it is hot.

Lastly, the easiest way to get all the libraries required for running Jasper is to download and install iReport then grab them from the lib directory. Please pay attention to the licensing! Some of these libraries like the mysql connector are under the GPL so distributing it with a non-GPL product will be frowned upon.

Putting it all together.

After this work, compiling and running a report from PHP becomes a simple exercise in translating from the Java examples to PHP:

$compileManager = new JavaClass("net.sf.jasperreports.engine.JasperCompileManager"); $report = $compileManager->compileReport(realpath("test.jrxml")); $fillManager = new JavaClass("net.sf.jasperreports.engine.JasperFillManager"); $params = new Java("java.util.HashMap"); $emptyDataSource = new Java("net.sf.jasperreports.engine.JREmptyDataSource"); $jasperPrint = $fillManager->fillReport($report, $params, $emptyDataSource); $exportManager = new JavaClass("net.sf.jasperreports.engine.JasperExportManager"); $outputPath = realpath(".")."/"."output.pdf"; $exportManager->exportReportToPdfFile($jasperPrint, $outputPath);

Note that you still need to use complete class names, otherwise you will get Class Not Found problems.

Okay, a quick guide to running Jasper from PHP:

1. Setup the environment as per normal with PHP and Java (I use WAMP on Windows);
2. Download the latest iReport software and install it on your workstation;
3. Grab all of the files from “C:\Program Files\JasperSoft\iReport-2.0.2\lib” (version number will change) and copy them into your java’s lib/ext directory;
4. Download the latest version of the PHP/Java Bridge with source and unzip it;
5. Unzip the src.zip file (should be included in the download) and copy the contents of the “java” directory to your PHP app.
6. If you want to use the stand-alone Java Server (easiest), grab the JavaBridge.jar from the ext directory (in the download) and copy it into the “java” directory you just placed in your PHP app.For production, you will probably want to use the Java extension and configure it to come up with your web server. There are a few configuration options here (including J2EE back-ends and the like), and the PHP/Java Bridge website explains how to get working with them.
7. Start the Java backend (java –jar JavaBridge.jar). This will usually bring up a dialog asking what port to use, however you have command-line options as well if you are running headless.
8. Try it out!!

Here is an example report: test.jrxml
And some example PHP code to compile, run it and send it to your browser: index.php.txt

Other things to note:

• Please cache the compiled version of your reports; otherwise you will be waiting for the report to compile every time you run one. I have some example code for doing this in my ReportGenerator class I used in my previous post.

Hopefully people will find this approach a little more straight-forward.

Enjoy!

One problem however: Enter Apple Mail, exit attachments.

Instead of the nicely revised document, you get a nice, anonymous, useless attachment. It’s frustrating to know that this “noname” attachment contains all of the precious work, just out of reach.

So, on opening this attachment (in vim, of course) I found that it was basically a base-64 encoded version of the document and some other stuff, with normalish MIME headers. Sort of like an attachment in an attachment. So by grabbing the most important (biggest) bit of encoded text, stripping out the new lines and un-encoding it, I could successfully get the document back and everyone was happy.

Having needed to use this script on a few occasions, I decided that it might be useful for some others, so here you go:<?php if (!empty($_FILES["test"]["tmp_name"])) { $data = file_get_contents($_FILES["test"]["tmp_name"]); $bits = explode("\n\n", $data); // filename is the last line of the 3rd bit $lines = explode("\n", $bits[2]); $filename = trim($lines[sizeof($lines)-1]); $filename = str_replace("filename=", "", $filename); header('Content-Disposition: attachment; filename="'.$filename.'"'); header("Content-Type: application/octet-stream"); header("Expires: 0"); $data = $bits[3]; $data = str_replace("\n", "", $data); echo base64_decode($data); die(); } ?> <html> <head> <title>Apple Mail Attachment Decoder</title> </head> <body> Select the document and hit submit:<br /> <form action="" enctype="multipart/form-data" method="post"> <input type="file" name="test" /><br /> <input type="submit" name="submit" value="Submit" /> </form> </body> </html>

Of course I can’t be held responsible for the script not working, destroying priceless documents, killing your cat or anything else.

I have also put the script up on the server for quick reference: http://www.rjohnson.id.au/attachment_decoder/
Hope this is useful to someone out there…

The first and most important thing I noticed was that the News was actually there! For example, I heard on the radio this morning that there was an incident involving our opposition leader and a strip club in New York while he was a little shin dig for the UN (wont someone think of the children??). Apparently a statement had been released but there wasn’t any explanation about what the substance of that statement was, jump on to the website and there is the article, posted a whole 24min ago.

I also wanted to know what happened to that passenger jet from Turkey that was hijacked. A quick click on World and then the fantastic tagging system at the top of the page and I found the article. Once in the article I could see all of the related stories so I could back-track through each of the updates.

There are also a bunch of Web2.0 (if I can use that term) features that include a personal list of tags for new stories that are applicable to you, and a personal stash of Articles (like internal bookmarks). Combine that with a lack of advertising, a smattering of new media and the tagging system for navigation and we have a winning site. Of course this is all government funded so check it out and subscribe to the RSS feed, public interest will help ensure the future of this great service and the ABC in general.

News Home: http://abc.net.au/news/

First impressions are really quite good, here are a few of the highlights:

Quartz font rendering.

It just looks sweet, using the same font rendering as available on OS X. I love my fonts, and I love seeing them displayed without the hinting (that modifies the shape of the letters to better fit the pixels) combined with the secret sauce that manages to make things not too blury.

Aqua widgets

Rather than go through the rather lengthy process of porting the application to an alternate toolkit, Apple ported their toolkit to Windows, which is good “˜cause it means us Microsoft ppl can get a bit of a taste of the Mac world.

Thin Borders

I know it sounds lame, but one of the things I love about Macs is the way that everything is presented on the screen. The application just sits there with no borders; you just use the resize option at the bottom-left of the window to do the needful. Vista on the other hand looks terrible, with great chunky borders that attempt to show off how well the glass theme works. As a result, the shadows don’t seem to make the windows float like they should, it’s a small thing I know, but it’s the small things that make a big difference.

We can test!

Being a fan of accessibility and cross-browser compatibility, I am really really happy that this is happening. I can actually see how the site will look on a Mac! That is complete with the same font rendering, DPI settings, etc. It’s really handy.

The frustrating things:

I can’t get it to work on my machine at work.

I think I must have had some dodgy version of the “Lucida Grande” font that has caused Safari to start off without any text being displayed. Kind of a downer. I have attempted to replace these problematic fonts with the correct ones, but I have still had no success. I hope Apple will be able to do something about this “˜cause I am keen to use it.

My laptop touchpad scroll doesn’t work

Which is rather annoying. I assume it has something to do with the lack of integration with the underlying OS, both a good and a bad thing I guess.

We will potentially have another browser that we need to support.

Yeah, I know that I said it’s great to able to test on a new browser, but it will be a pain when we can no longer say “well, it’s not perfect on Safari, but that’s only 1.5% of the market so that’s acceptable”. I do think that the browser share will increase considerably with this move to be something more significant, and that will mean another Javascript DOM to accommodate for and another set of CSS curiosities to consider.

Download the beta here: http://www.apple.com/safari/

Everyone has been watching every drop that goes down the drain. We have buckets in the shower so that we can still water our herbs and chilli plants. The car has gone unwashed for months and months. We are trying to run the dishwasher only once a week. We have been encouraged to keep our water consumption down to 140L / day, sounds like a lot, but go a minute or two over the allocated 3 in the shower and it starts to become rather difficult.

So, water was a major election issue for Premier Beattie and it will be for the Federal government when it comes time for little Johnny to defend his position. The Federal government has already stepped in, trying to get the Northern NSW government to setup pipes to take water from some of the better off areas to help support Brisbane. There is also a national water grid going in that will take water from some of the other dams in the surrounding regions for Brisbane, and some desalination plants going up.

All sounds good eh? Well, not really, I’m not overly convinced that it will be in here enough. We only have about 15 months left of water if we keep going at the current rate. Our population is also increasing at about 1700 people a month, which will put further strain on our resources.

The rain is just divine, nothing nicer than listening to it while going to sleep and I just love it when it is cold (not really cold, Brisbane cold). Unfortunately this particular fall is only enough to extend the 15 month deadline by 2 days.

Some more links about the situation:

• SEQ Brisbane Dam Levels
• Brisbane City Council drought information
• Details on Level 5 restrictions, Level 6 kick in as we approach September.