In case you were wondering...

Well, I managed to put together a todo list.

It is several pages long.  And, I have almost certainly missed half of the things that I need to do.

But, it is a start.  I still need to find some single place to keep/update it.

Until now, I have kept several concurrent todo lists.  One (at least) in my head and several on paper that I usually misplace before finishing everything on them.

One thing that I was on my list as a recurring task was updating this blog: Once a week - on Tuesdays.

I missed out on last week - partly because I was trying to more completely list out the rest of my todo list and partly because I had a thousand other things to do (some on the list and many that slipped through the cracks).

So, I am going to try try to cut myself a little slack and allow myself to make that Tuesday post anytime before the next Tuesday.  This is the first such 'late but not missed' post.

Hopefully, there will be more tomorrow (Tuesday).


But, I guess we'll see.

The first step...

The first step to really getting organized is to figure out everything that you need to do.

The second step is to order the list in terms of what should be done first, second, etc.

Unfortunately, the first step to becoming totally overwhelmed with what you need to do is the same as the first step to getting organized.

A todo list is supposed to have an end, right?

It will be nice when I find it.

Drupal simmering...

I had hoped that I would be able to spend a good chunk of time learning how Drupal works and building something useful with it.

But, real life got in the way - so that has not happened (yet).

And, the reason for that is probably that on my todo list, everything has a priority of 'Should be done Yesterday!'

I think I will try to get my list to have a little more structure.


This is going to be interesting.

Handing over the keys...

I'm generally not a fan of systems where the computer tries to figure out what you 'really want to do'.

But, I have gotten to a point in my life where I might be open to the idea that it is possible for a development environment to guess correctly.

And so, I am beginning to try out Drupal.

If things work out, then it will make my development of sites easier (I'll still hand code my web -apps- at least for the time being).

Taking a deep breath....

And off we go.

Doc, it hurts when I...

Normally, I would say that if something hurts - you should stop doing it.

But, I didn't know how badly I was hurting myself when I created multiple versions of all of my JPA entities depending on the depth of data that I wanted to have fetched.

So, fixing my app to use FetchGroups (an OpenJPA feature) to do what I had hacked together has been (and still) is a fairly painful process.

I first have to find each place that needs to have a custom fetch group defined and then add to the fetch group until it contains everything that I need.

Once I have it set up correctly though...It is a beautiful thing.

Fast, clean, and obvious what is going on.

That was painful, but worth it...

Last week I found out about a feature that the OpenJPA folks put into their implementation of JPA. But, it is not part of the standard.

That feature is custom fetch plans - And, I think I can safely say that they are my newest favorite thing.

The reason that these are exciting for me personally is that I use JPA to pull data from my back end database that is then converted to XML using JAXB and sent to a browser for processing/display. If I were to transform a completely populated 'top-level' entity, then I would be creating an XML document that could be several Mb.

Over the past year, JavaScript engines have gotten faster - and continue to do so. But, trying to make JavaScript parse and manipulate blocks of XML data that are that big is not a nice thing to do to the browser (or the user).

Until this past week, I thought that I would need to create tailored versions of my JPA entities in order to send back just the part of the XML that I actually needed. Then, I found out about (Cue choir of angels) dynamic fetch plans.

What dynamic fetch plans do is allow you to specify which fields and relations are eagerly fetched at the time that you execute the query. That may not sound particularly earth shattering - but give it time to sink in.

You are able to specify down to the individual database column level exactly what will be pulled from the database (and/or specify the fetch depth). When doing JAXB processing, this allows me (and you if you need it) to tailor the exact data that will be turned into XML.

One poignant example of how this can clean up the XML sent to the browser is to send a list of top level entities for listing in a drop down. The only data that really needs to be sent back is the entity key and description. Without custom fetch groups, JAXB would try to build the entire fully populated entity tree for each entity. The amount of data being sent back to the browser would be insane! My previous solution of creating an array of JPA entity beans in order to define every possible grouping of desired fields was also insane. I picked a middle ground of an overly populous graph that small enough/big enough for most uses and a sparse graph that would be super quick but only useful in one or two cases.

Now, I have been able to remove all of the extra Entity beans. I have simplified the definition of my entity relationships and removed the possibility of missing changes if I change the actual table structures. Plus, I am able to exactly specify what I want to send back in each situation (anywhere from fully populated entities to single fields).

I just wish I had known about them two years ago when I first started using OpenJPA. It would have saved me a lot of refactoring and testing (at a time that I really can't afford to spend the time). But I am glad that I managed to 'stumble onto' them. I didn't really go looking for an EJB 3 entity manager - I just used the one that came with Geronimo. If I had shopped around, I might not have given this the weight it deserves.

And it is -huge-. Not only can you pare down an overly generous eager fetch - you can also expand an excessively lazy fetch - at runtime with very little programming cost.

By the way, the painful part is trying to undo two years worth of hacks to accomplish something that was included in OpenJPA - in five days.

I think I used to sleep - didn't I?

Can we talk...

I get to spend a lot of my time (most of my time actually) working on web apps.

Nowadays I use Geronimo for my server, Firefox for my browser, and Dojo for the two to talk to each other.

Last time (actually, the time before that) I showed how I parse the XML that I am sending from the browser to the server. But, I did not show how that XML gets created.

Well, since I like to make things simple for myself, I wrote a couple of wrapper functions to abstract the Dojo functions so that I would not have to rewrite large blocks of code if the syntax changes in Dojo (which it has done since I started). Also, having a single way to send messages to my server means that I can have a consistent way of parsing those messages.

I broke the sending of messages into two parts:

1. Building the message
2. Sending the message and specifying the callback

And go figure, there is a JavaScript function (that I wrote) to do each of those things.

For building the message, I created a function called 'queueCommand'. The idea was that you could create several message queues that all needed to be sent to the same servlet (and whose output would be consumed by the same callback function). I actually set up a number of functions and made it possible to build several message queues - but for simplicity, we'll pretend there is only one at a time.

Here is queueCommand:
function queueCommand(command) {
var commandXML = encapsulateCommand(command);

if (window.commandQueue === undefined) {
window.commandQueue = [];
}

window.commandQueue[window.commandQueue.length] = commandXML;
}

Pretty simple right? Ooops! You caught me. There is a third method that I neglected to mention. The encapsulateCommand function tries to remove characters that would cause the generated XML to be invalid as it puts together a simple (and standardized for my purposes) XML snippet.

Here is encapsulateCommand (with a helper function):
function encapsulateCommand(command) {
var nodes = "";

for (i in command) {
nodes += "<" + i + ">" + escapeString(command[i]) + "";
}

var commandXML = "" + nodes + "";

return commandXML;
}

function escapeString(inputData) {
var outputData = inputData;

if (typeof inputData == 'string') {
if (inputData != null && inputData != "") {
outputData = inputData.replaceAll("& ", "& ");
outputData = outputData.replaceAll("<", "<"); outputData = outputData.replaceAll(">", ">");
outputData = outputData.replaceAll("\"", """);
outputData = outputData.replaceAll("\\", "~1~");
outputData = outputData.replaceAll("%", "~2~");
outputData = outputData.replaceAll("\'", "~3~");
outputData = outputData.replaceAll("\n", "~4~");
}
}

return outputData;
}

And here is an example of how you would call it:
var pushCommand = queueCommand({
action: "doSomething",
fieldValue: dojo.byId('fieldID').value
});

I don't actually send back a return value. But, I do capture the result in a variable because if something goes wrong - receiving the result into a variable prevents the error from stopping program execution. If I were being more diligent, I would send back an actual result status (or maybe even the assembled command) - But I didn't do that.

So, when the above command (var pushCommand = ...) is executed, the following XML snipped gets added to the queue (shown here 'pretty'):

<command>
<action>doSomething</action>
<fieldValue>SomeValue</fieldValue>
</command>

The pushQueue function bundles up all of the commands that have been placed into the queue inside of a proper XML header and a 'envelope'.

Here is pushQueue:
function pushQueue(url, onLoad) {
var queue = window.commandQueue;

if (queue == undefined) {
window.commandQueue = [];
queue = window.commandQueue;
}

var xmlDoc = "";

xmlDoc += "";
xmlDoc += "";

for (var i = 0; i <>";

window.commandQueue = [];

var parser = new DOMParser();

var xmlDocument = parser.parseFromString(xmlDoc, "text/xml");

var ajax = dojo.rawXhrPost({
url: url,
postData: xmlDocument,
load: onLoad,
headers: {
"Content-Type": "application/xml"
},
handleAs: "xml"
});
}

You would call pushQueue with the URL of the servlet (or whatever resource is going to handle the message) and the callback function that should get the result.

Here is an example of calling pushQueue:
var sendTheMessage = pushQueue("/Handler", callBackFunction);

After that call, Dojo would send the following XML document to '/Handler':

<?xml version='1.0' encoding='utf-8' ?>
<commands>
<command>
<action>doSomething</action>
<fieldValue>SomeValue</fieldValue>
<command>
<commands>

Now, just in case you were wondering, there is nothing special about the 'command' and 'commands' tags that I used. They are really entirely arbitrary, but they make sense for what I am sending (a list of commands, each or which is a command).

And, when the result is sent back from 'Handler', it will send the result to my JavaScript function called 'callBackFunction'.

So, with a couple of functions placed in a JavaScript file that is included on all of my pages, I am able to have a standard way of sending messages to my server. And because the format of the messages is the same every time, I can use the same XML parsing code (shown on a previous post) to extract the information.