That was painful, but worth it...

Last week I found out about a feature that the OpenJPA folks put into their implementation of JPA. But, it is not part of the standard.

That feature is custom fetch plans - And, I think I can safely say that they are my newest favorite thing.

The reason that these are exciting for me personally is that I use JPA to pull data from my back end database that is then converted to XML using JAXB and sent to a browser for processing/display. If I were to transform a completely populated 'top-level' entity, then I would be creating an XML document that could be several Mb.

Over the past year, JavaScript engines have gotten faster - and continue to do so. But, trying to make JavaScript parse and manipulate blocks of XML data that are that big is not a nice thing to do to the browser (or the user).

Until this past week, I thought that I would need to create tailored versions of my JPA entities in order to send back just the part of the XML that I actually needed. Then, I found out about (Cue choir of angels) dynamic fetch plans.

What dynamic fetch plans do is allow you to specify which fields and relations are eagerly fetched at the time that you execute the query. That may not sound particularly earth shattering - but give it time to sink in.

You are able to specify down to the individual database column level exactly what will be pulled from the database (and/or specify the fetch depth). When doing JAXB processing, this allows me (and you if you need it) to tailor the exact data that will be turned into XML.

One poignant example of how this can clean up the XML sent to the browser is to send a list of top level entities for listing in a drop down. The only data that really needs to be sent back is the entity key and description. Without custom fetch groups, JAXB would try to build the entire fully populated entity tree for each entity. The amount of data being sent back to the browser would be insane! My previous solution of creating an array of JPA entity beans in order to define every possible grouping of desired fields was also insane. I picked a middle ground of an overly populous graph that small enough/big enough for most uses and a sparse graph that would be super quick but only useful in one or two cases.

Now, I have been able to remove all of the extra Entity beans. I have simplified the definition of my entity relationships and removed the possibility of missing changes if I change the actual table structures. Plus, I am able to exactly specify what I want to send back in each situation (anywhere from fully populated entities to single fields).

I just wish I had known about them two years ago when I first started using OpenJPA. It would have saved me a lot of refactoring and testing (at a time that I really can't afford to spend the time). But I am glad that I managed to 'stumble onto' them. I didn't really go looking for an EJB 3 entity manager - I just used the one that came with Geronimo. If I had shopped around, I might not have given this the weight it deserves.

And it is -huge-. Not only can you pare down an overly generous eager fetch - you can also expand an excessively lazy fetch - at runtime with very little programming cost.

By the way, the painful part is trying to undo two years worth of hacks to accomplish something that was included in OpenJPA - in five days.

I think I used to sleep - didn't I?

Can we talk...

I get to spend a lot of my time (most of my time actually) working on web apps.

Nowadays I use Geronimo for my server, Firefox for my browser, and Dojo for the two to talk to each other.

Last time (actually, the time before that) I showed how I parse the XML that I am sending from the browser to the server. But, I did not show how that XML gets created.

Well, since I like to make things simple for myself, I wrote a couple of wrapper functions to abstract the Dojo functions so that I would not have to rewrite large blocks of code if the syntax changes in Dojo (which it has done since I started). Also, having a single way to send messages to my server means that I can have a consistent way of parsing those messages.

I broke the sending of messages into two parts:

1. Building the message
2. Sending the message and specifying the callback

And go figure, there is a JavaScript function (that I wrote) to do each of those things.

For building the message, I created a function called 'queueCommand'. The idea was that you could create several message queues that all needed to be sent to the same servlet (and whose output would be consumed by the same callback function). I actually set up a number of functions and made it possible to build several message queues - but for simplicity, we'll pretend there is only one at a time.

Here is queueCommand:
function queueCommand(command) {
var commandXML = encapsulateCommand(command);

if (window.commandQueue === undefined) {
window.commandQueue = [];
}

window.commandQueue[window.commandQueue.length] = commandXML;
}

Pretty simple right? Ooops! You caught me. There is a third method that I neglected to mention. The encapsulateCommand function tries to remove characters that would cause the generated XML to be invalid as it puts together a simple (and standardized for my purposes) XML snippet.

Here is encapsulateCommand (with a helper function):
function encapsulateCommand(command) {
var nodes = "";

for (i in command) {
nodes += "<" + i + ">" + escapeString(command[i]) + "";
}

var commandXML = "" + nodes + "";

return commandXML;
}

function escapeString(inputData) {
var outputData = inputData;

if (typeof inputData == 'string') {
if (inputData != null && inputData != "") {
outputData = inputData.replaceAll("& ", "& ");
outputData = outputData.replaceAll("<", "<"); outputData = outputData.replaceAll(">", ">");
outputData = outputData.replaceAll("\"", """);
outputData = outputData.replaceAll("\\", "~1~");
outputData = outputData.replaceAll("%", "~2~");
outputData = outputData.replaceAll("\'", "~3~");
outputData = outputData.replaceAll("\n", "~4~");
}
}

return outputData;
}

And here is an example of how you would call it:
var pushCommand = queueCommand({
action: "doSomething",
fieldValue: dojo.byId('fieldID').value
});

I don't actually send back a return value. But, I do capture the result in a variable because if something goes wrong - receiving the result into a variable prevents the error from stopping program execution. If I were being more diligent, I would send back an actual result status (or maybe even the assembled command) - But I didn't do that.

So, when the above command (var pushCommand = ...) is executed, the following XML snipped gets added to the queue (shown here 'pretty'):

<command>
<action>doSomething</action>
<fieldValue>SomeValue</fieldValue>
</command>

The pushQueue function bundles up all of the commands that have been placed into the queue inside of a proper XML header and a 'envelope'.

Here is pushQueue:
function pushQueue(url, onLoad) {
var queue = window.commandQueue;

if (queue == undefined) {
window.commandQueue = [];
queue = window.commandQueue;
}

var xmlDoc = "";

xmlDoc += "";
xmlDoc += "";

for (var i = 0; i <>";

window.commandQueue = [];

var parser = new DOMParser();

var xmlDocument = parser.parseFromString(xmlDoc, "text/xml");

var ajax = dojo.rawXhrPost({
url: url,
postData: xmlDocument,
load: onLoad,
headers: {
"Content-Type": "application/xml"
},
handleAs: "xml"
});
}

You would call pushQueue with the URL of the servlet (or whatever resource is going to handle the message) and the callback function that should get the result.

Here is an example of calling pushQueue:
var sendTheMessage = pushQueue("/Handler", callBackFunction);

After that call, Dojo would send the following XML document to '/Handler':

<?xml version='1.0' encoding='utf-8' ?>
<commands>
<command>
<action>doSomething</action>
<fieldValue>SomeValue</fieldValue>
<command>
<commands>

Now, just in case you were wondering, there is nothing special about the 'command' and 'commands' tags that I used. They are really entirely arbitrary, but they make sense for what I am sending (a list of commands, each or which is a command).

And, when the result is sent back from 'Handler', it will send the result to my JavaScript function called 'callBackFunction'.

So, with a couple of functions placed in a JavaScript file that is included on all of my pages, I am able to have a standard way of sending messages to my server. And because the format of the messages is the same every time, I can use the same XML parsing code (shown on a previous post) to extract the information.

Days go by...

It doesn't seem like it has been over a month since the last time I posted.

But, the fact is that it has been (ack!).

So, back to the grindstone.

Goodbye to the old...

When I first started using Geronimo (the Apache JEE server), it was on version 'point something'. And, I needed to parse XML messages being sent from a browser to servlets.

So, I used JDOM. It was fairly simple to use and the JDOM libraries were included with Geronimo.

That made things very simple. All I needed to do to parse the XML messages was put something like this in the doPost method:

Reader reader = request.getReader();

try {
SAXBuilder builder = new SAXBuilder(false);
Document doc = builder.build(reader);
Element root = doc.getRootElement();


Then, to get values out of the document, I wrote a number of wrapper functions like getString, getLong, getX - to convert the values in the XML document into something useful (something other than strings).

But, when Geronimo went from version 1.x to 2.x, they dropped the JDOM library from the assembly - And I started to have to include it myself.

Recently, I finally got tired of having to add the library myself and started to look for alternatives to using JDOM.

Enter W3C...

So, now I get to change a bunch of servlets from the code above to this new version:

Document document = getXML(request, dbf);

Ok, so that wasn't quite all that I did. That 'getXML' function isn't included with the W3C's DOM classes. I had to write it myself. And, here it is:

protected Document getXML(HttpServletRequest request, DocumentBuilderFactory dbf) {
Document document = null;

try {
DocumentBuilder db = dbf.newDocumentBuilder();

BufferedReader in = request.getReader();

String input = "";
String xmlText = "";

while((input = in.readLine()) != null) {
xmlText = xmlText + input;
}

System.out.println("Received: " + xmlText);

InputSource source = new InputSource();
source.setCharacterStream(new StringReader(xmlText));

document = db.parse(source);
} catch (Exception e) {
System.out.println("Exception: " + e.getMessage());

e.printStackTrace();
}

return document;
}

Now, that is much bigger (as far as lines of code) than the old way. But, by putting it into a function - you can't tell. And, once I am done with all of the change over, I will be able to get rid of that extra JDOM dependency.

Plus, along with finding how to change over to use the W3C's DOM, I also found the XPath libraries. So, I was able to change all of my XML to use real, properly formed documents. What I had been doing before was storing all of the real information in attributes of a single element.

Bonus!

Take a deep breath...

Well - I have been slowly trying to get myself accustomed to the idea of social networking over the internet.

This blog was my first tentative step.

Then, I let myself get a twitter account - I don't plan on posting anything just yet. That may change.

But, I finally got around to reading other people's blogs.

And I realized something.

I have a long way to go to get my blog to be what I want it to be.

So hopefully this is a changing point. I hope that from now on, this will become a blog that someone else would actually follow.

Wish me luck.

Tools holding you back...

I have gotten too used to working with tools and skipped over learning how to build java artifacts by hand.
And now, I am starting from scratch on my new project.

Which isn't such a big deal. Using Eclipse, I started creating my WAR file - no problem.

But, I want to use Maven to manage the project parts. Including creating the database pools.

So, I finally -have- to learn how to use Maven my self.

Yay!

(No, really - Yay. Seriously).

Wait, I think I saw it move...

There actually has been (a very small) step forward.

I have one web service and one entity written.

(One down and how many to go?)

Now I have to put it into a maven project and start setting up unit testing.

And then write the rest of the General Ledger.

Piece of cake.

(Maybe pie - pie is a little trickier to make than cake).