Seaview, a Developers Perspective

Rigid definition of done considered harmful

2009-08-04T22:58:00.001+02:00

Most people that has been doing iterative development in general and scrum in particular are probably familiar with the concept of discussing a definition of done. I have more than once been on projects where the concept of done has been defined in one form or another, and where project members have been trying to deliver running parts of a system that is compliant with such a definition. Even more I have witnessed scrum-masters proudly announcing that their definition of done is really done, done, done. The concept of done, done, done will of course be different from appliance to appliance, but it usually ends containing some of these:

Code checked into SCM
Code passed QA
Tests running green in the Continuous build system
Running part of the code in a test system ready to be accessed by functional experts and product-owners
Possible discussed and assessed by product-owner before staged to the demonstration environment
Possibly accompanied by customer driven tests like fitnesse
Ready to be put into production

My claim goes as follows; forcing your team members to adhere to a definition of done that is to rigid, and using that same definition as a criteria before moving it to the done category on the scrum board (complete in the current iteration), will cause your team to deliver less functionality than they could, and with properties that will make that functionality more difficult to change or even remove. I will try pinpoint why following such a rigid definition along with some indicators on going in the wrong direction.
Just to clarify one thing; for me, writing code with accompanying tests that are checked in and runs green in the CI system is all about the same thing. Make no mistake, I do it and I encourage my team-members, and everybody else for that matter, to do the same.

Team-members are reluctant to pushing functionality out

All those parts that need to be completed to get the "done, done, done" badge might not be necessary or even productive work to deliver the functionality ready for test. Maybe the criteria that says that all functional tests should be supported by a FIT test or some test specified in some BDD-flavor just isn't the way to go if the functionality doesn't lend itself to such a definition easily. Maybe only programmers are going write tests and specifications for that part of the system, in which case JUnit will probably be the best choice for a Java-based system. Not easily being able to comply with the done-criteria will probably cause some developers to go that extra mile to comply even if it is probably not worth the effort for that particular piece of functionality.

Consistency is will of course matter here. Using the same techniques for every bit and piece of the system will make it more consistent. But the strive for consistency will also manifest in supporting test-code or a greater number of test artifacts. These bi-products will have to be maintained and refactored together with everything else, and may actually make it more difficult to change the system later on. It takes experience to know when to put emphasis on consistency over throughput.

Somebody is questioning why we need to work further on something that we have moved off the board

Does this question sound familiar: "I thought you said that this task was done, done, done. We really doesn't have time to take this task back in again". I have heard it quite a few times and although it tends to be accepted, it still surprises me that this is not perceived as the rule, not the exception. If you get all the functionality right most of the time, my claim is that your lead-time to production is probably much longer than it should have been. If you develop build and deploy fast, like in hourly or daily increments, the likelihood of getting it right the first time is probably lower, but the basis for distilling the functional requirement is the running system itself, not some temporary artifact used just for aiding the specification process. Do as much of the specification work as possible in the production code.

The irony of all this is that it is the people evangelizing on how to eliminate waste that are most prone to push forward a framework to assist in the specification-process where that framework is not part of the code running in the production system. The observant reader may have understood that I am being intentionally vague here, and I have chosen not to prove my point with an example.

It tends to propagate into rigid definitions of other areas of development

The work to be put down to comply to a rigid definition of done to sign off on a user story or a task has the tendency to propagate into all areas in development. I think this is partially caused by the fact that you tend to move the some finalization-criteria out of the common-sense and ad-hoc part of decision making and into an (often) static set of rules. We need some statement on what the rules are, if not for anything else, at least to get some transparency related to the development process so that it may be explained to architects, management and other stakeholders and decision makers.
I believe that the urge to push rigid definitions of compliance rules forward is partially caused by the strive for consistency. The expectation is as follows; if you have rules that govern the development process, in the case the definition of done to move something off the scrum-board, then you will end up having delivering functionality in a consistent and predictable manner. That may be, but what if the functionality delivered is delivered slower than it could be and, and much more hardened in the source-code in terms of difficult to refactor tests and Fit tables.

The same goes for code-qa (pair programming is not always how code comes to light). Way too often this boils down to developers commenting on each-other code that may have been developed another way. Some times it identifies and prevents serious design-flaws, which is great. Developers produce at different levels of quality, use this input in the QA-process as well, it is nothing wrong with checking A's code with a great level of scrutiny while you may browse through B's commit-log in a split second. That's life, learn and educate instead, don't force the same compliance requirements on everyone.

What to do and where to go?

"There is no sense being exact about something if you don´t even know what you´re talking about." (John von Neumann).

It is very important that you observe and take notes of undesired effects of certain rules that your team is using to guide their day-to-day operation. This goes for code and architecture-centric rules as well as for rules that are meant to support the day to day process of harvesting requirements. When undesired effects show up, maybe it is time to remove or adjust the rule accordingly. Maybe this particular web-module wont need the Selenium-harness?, or maybe it does.

All in all it is about delivering functionality in the running system, supported by tests so that you may change code when new knowledge and requirement tells you to. A rumor says that Thomas Edison did not hire employees if the person on the interview salted his/her soup without tasting it. Use your ruleset the same way, don't always strive for consistency when applying your rules, it may lead to code manifesting in a way that makes it harder to implement your functionality fast and with less effort.

Laptop retrospective

2009-07-31T23:00:00.001+02:00

Inspired by ma latest post on different cell-phones I have had, I thought I would give it a second go, but now replace cellphones with laptops . I have been using laptops on a regular basis since the 80's because my father had a genuine interests for computers in combination with spreadsheets. I have decided to include these as well since it reflects my overall exposure to these kind of devices, even though they not reflect laptops I have owned, but merely laptops that I have used.

IBM Portable 5155

To call the first laptop a laptop would be to classify the infamous but very effective German Tiger tank of the WW2 light artillery. The IBM Portable 5155 was an awesome computer at the time and the model my father had was equipped with 256Kb of memory. I was at the time barely able to lift the computer; in its heavy-duty plastic case it weighed in at almost 15Kgs!! I had access to this computer from 1985-1987, it was my first experience with a computer apart from using the Commodore 64 at some friends, because I newer actually had one myself.

Power Book 150

The next time I got to use a portable computer on a regular basis after those that my father brought home from work in the late 80's wasn't before 1997 when I had a part time job for SINTEF. The (at the time outdated) laptop was given to me so that I could do some work while doing my military service as a conscript in the Royal Norwegian Airforce. I used the computer to write user manuals and documentation for a system I was working on at the time. I has no connectivity and was offline while doing the work. This was my first experience using a Mac, but I found it quite easy to master.

IBM Thinkpad 380

I got this laptop while I was stationed as a conscript at Kjeller airfield just outside Oslo. I was lucky enough to part of a project programme that was part of a cooperation between Det Norske Veritas, Aerospatiale Missile and Royal Norwegian Airforce. I used it to program a reporting application in Visual Basic for Applications and MS Access. This was the first really nice laptop that I got to work with and the quality and robustness of the Thinkpad series has appealed to me ever since, even though I never got to own one myself. I also used it for a fair amount of gaming, especially Command and Conquer (one of the earlier versions I believe), excellent game BTW. I think that the keyboard on the Thinkpad models have been in a league of their own.

Compaq (unknown)

This was the first laptop that was handed to me for my personal use after I started my first full-time job as a consultant in Sybase, Copenhagen Denmark. The laptop was not new at the time that I got it, but had been used by a consultant that had stopped working there. It had an SVGA screen and a 150MHz CPU. Before started working there I had really been looking forward to the unpacking of my work-laptop, and all I got was this second hand Compaq. To all you guys hiring staff working as consultants. Please give them new equipment when they start working there. This way you will retain loyal staff for a much longer time. I guess however that unpacking a brand new laptop is not much of a big deal now as it was in the 90's.

I haven't been able to retrieve an image of this laptop, and I dond't have have a picture of myself using it so, sorry...

Compaq Armada 7400

Fortunately I got my hand on a much better laptop after complaining about my first laptop for almost a year. This time it was equipped with a 300Mhz cpu and a neater casing. I cannot completely recall but I think it was equipped with 128MB RAM. Decent laptop as far as I remeber, but nothing to be remember as special after I turned it in when I left the company that I was working for at the time, that must have been in 1999.

Dell Latitude (unknown model)

I got my first Dell as my home-office computer while working for Nokia. I did not really need one because I worked so many late hours there that I could not possibly have squeezed in some more office time after I got home from work. It was nothing special with this laptop. I actually didn't use it much as a laptop, it kind of had its home in the docking station at work. It was however good enough to persuade me to get a Dell as my next laptop. I don't remember the exact spec, but I seem to recall that my stationary computer at the time was a 600Mhz, 512Mb something, so I it was along those lines. I really cannot remember how it looked, it was stuck in the docking most of the time. For the same reason I was not able to find a picture.

Dell Inspiron 8100

Equipped with a 1,1GhZ processor, 30GB harddisk, 512MB memory, DVD, WIFI and a 15" 1600X1200 screen this laptop is the most expensive piece of equipment that I have ever bought. Bought in 2001, it was a true kick-ass laptop at the time, and I still have it although I don't use it. Staying at the top of the throne is costly, and you don't stay there for very long. I actually kept this laptop for a record-long period of 3,5 years, so it paid itself in the end. The lesson learned was however to use another long-time strategy; buy cheaper equipment and upgrade more often. This was the first laptop I bought after founding Zenior, also the place where I still work. I could choose whatever laptop I wanted, I guess that is one of the reasons it ended up to be a bit too expensive.

Dell Inspiron 8600

I had been quite happy with my previous Dells so I wouldn't change a winning team. I actually believe that Dell being the only vendor having a decent site allowing for massive customization of the equipment was one of their reasons behind their success. The laptop had a Intel mobile 1,6GHz CPU, 1GB RAM and a 60GB Hdd running at 5400. The display 1600x1200 screen which was good. Apart from the CPU and double up of memory, 2Gb, , it's specifications wasn't that much different from the previous Dell, but the price was about one third. This was to be my last PC, possibly ever.

MacBook Pro

After using PC-based laptops (apart from the brief encounter with a PowerBook in 1997) for more than a decade, I finally decided to give in and get myself a MacBook Pro. I must say, apart for some annoying issues when running Mac-OSX I am very happy with the Mac. My Macbook has a 2,33 GHz CPU, 2Gb RAM and a 15" monitor. I have never regretted to not getting the 17" screen, it is simply too big. I believe that even my graphics-card is the same as in the 17" model so why bother. I got this laptop in 2007, and given my adopted two-year replacement strategy, I am now opting for a new MacBook Pro. I am planning on using that computer as a desktop replacement for my workstation as well. The MacBook is one of the more pleasing products I have ever bought, regardless of category. I am not an Apple-addict, I just find joy working with pleasant technology in an appealing wrapping; the MacBokk fits these attributes nicely.

Cellphone retrospective

2009-07-27T00:26:00.001+02:00

A couple of days ago when I used my iPhone to post a Twitter kind-of status, I was contemplating on how many different cellphones I've had, some excellent, good, some mediocre and some just very bad. I must say after buying an iPhone and being on a couple of trips this summer, for the first time i didn't regret bringing my laptop. The maps application in combination with Safari on the iPhone did the job just fine. Cellphones has come a long way since I bought my first Ericsson phone back in 1995. Some of the phones have been lost on the way, others has been borrowed (and probably never returned) and given away. The first phone I bought I actually managed to sell for 50Dkr in 1999. I have attempted to rate the phones according to a relative rating-scheme; what did I think of the phone when I had it back in whenever. I actually still have quite a few of these phones in my possession still, but they are scattered in different boxes in the attic or maybe in different storage bins in my home office,....sigh.

Ericsson GA 318

This is the cellphone I bought back in 1995 when i still went to university. The phone was nice, and leaner than most other models that was on the market at the time. The display a very simple black and white (no color lcds at the time i'm afraid) LCD that barely had enough real-estate for an international phone number. Netcom did not charge for SMS at the time, and they also provided a gateway that enabled you to send sms-messages directly to the cellphone using mail on the form: MSISDN@gw.netcom.no. I actually forwarded alarms from a production system to the phone using this gateway.

Verdict: 4.

Nokia 6150

I bought this phone when I worked as a consultant in Denmark back in 1998. My needs were really simple. I basically used this phone as a replacement for the fixed line that I terminated back then, and being a consultant, I had to be available. This is one of the most durable phones I have ever had, and I actually got this as a work-phone when I started working for Nokia Networks back in 1999. Later I have used that particular instance as a replacement phone for other phones that have ceased to work. However this was the first time I encountered the "it just shut itself of" syndrome that seems to have been haunting Nokia phones ever since.

Verdict: 4

Nokia 7110

This was a truly amazing phone at the time. It was, I believe, the
first Nokia model to support WAP and T9. It also had this neat little
keyboard-cover that you could release by pressing a small button on the
side of the phone. It made this "wsssh-click" that was very easy
to recognize. This phone was actually engineered and developed at the
research-facility in Copenhagen where I worked at the time. The early
prototypes had some flaws and one of them that I remember particularly
well was the problem with the ejecting spring for the cover being too
strong so that the cover would actually separate from the phone and be
ejected several meters. I more than once had to pick such a cover out
from my lunch (not my cover) being ejected from one of my coworkers
trying to enjoy their lunch while receiving a call, and sitting at my right side.

Verdict:5

Nokia 3310

I got this phone as a prototype back in 1999, 4 months before it went into full production. At the Nokia facilities at Fredrikskaj they handed out this model to all employees. It was nothing special with this model, and it was ugly. It had, however, a sms-chat function that was pretty neat that I haven't seen on any phone models since this one. This phone was one of the first ones with widely available snap-on covers, i had only one cover, grey.....

Verdict:3

Nokia 6210

I got this phone as a replacement after my 7110 was lost on a business-trip to Finland. The phone had many of the same features as in the 7110 but had a much better mechanical design and no antennas pointing out. The screen was nice and it had a small lean design so that it would easily fit in to any pocket. It also turned out to be one of the more robust models I have had.

Verdict:4

Nokia 9110i

Wow, I felt like a kid the first time a got my hands on one of these. I first got it in 2000 and given the functionality it had at the time still amazes me. It is one of the reasons I loved (and still do) Nokia as a company, being truly innovative. When the first Communicator was launched, virtually nobody knew, before the lightning came out of the sky. This was one of the predecessors, only with even more features. I had work-mail using a VPN connection, and I more than once used the built-in telnet terminal to log into test and development-servers to launch new builds, check build-status and so on. The excellent QWERTY keyboard was one of the reasons I really had a hard time to switch back to more regular type phone models.
The only problem with this phone was its limited durability. I had to send it away on repair twice before it finally gave in in 2002. The backbone was an AMD 486, I had a Cyrix 486 as a refrigerator magnet several years erlier :-)

Despite it's mechanical weaknesses I have to give this phone a verdict:6

In the period after the decease of the legendary 9110i but before the 2003 i used my old 6150 and 6210 before finally giving in ans buying my first Sony Ericsson phone with MMS.

Sony Ericsson T610

I just wanted one of these phones to try something else than Nokia. It also was to my knowledge one of the first Sony Ericsson mobiles to come around apart from the legendary T68i and of course the P800. The phone was my first MMS-phone. A lot of service providers campaigned this as part of their MMS launch. The phone was OK, nothing more. Its dimensions in combination with a lousy keyboard made it very difficult to write sms-messages using only one hand, the bottom part of the keyboard was simply to close to the edge.

In fall of 2004, a scratching disturbing background noise made it difficult for the people I was talking to too hear what I was saying. That made an end to a not very exiting relationship.

Verdict:3,5

Nokia 9210i

My brief encounter with what we referred to as "those other guys" when I worked for Nokia up to February 2001, the Ericsson part of Sony Ericsson, made me want to go back to Nokia. During the last couple of years I hadn't been able to get the Communicator out of my head. I will admit to possibly being a bit nostalgic, but it was no other way, I had to get back to the life I had with the Communicator. So I went down up to Telekiosken at Storo and got myself a 9210i. The form factor of this phone was better than the previous Communicator models, and the foldable antenna was all gone which was about time. The mechanical flaws from the previous models had vanished and it now had a color TFT which today I still remember as one of the best cellphone-displays ever. Even in broad daylight the screen was completely unaffected, as pristine as a piece of paper. Some of the magic feeling I had from the first model was gone, that together with a battery with limited quality made the re-encounter shorter than I hoped for. After the phone shutting down when I was about to agree on a rendezvous point with my off-piste friends in Hemsedal, for doing some serious downhill just before I was able to communicate my location, aided me lin shutting it down. I decided to get a more robust and less amp-hungry friend.

Verdict:4,5

Sony Ericsson W950i

A good friend of mine (and co-worker) that was much more knowledgeable about cellphones than I was, became all worked up about this new model. It had a decent size, heaps of functionality and a touch screen that was usable together with the pen that accompanied the phone. No fuzz, I got one immediately. This became my first mp3 player as well I am embarrassed to say. The phone was decent and even had a radio which I have missed on most of the other models I have had (and still miss). But it did not take long before I re-discovered the flaw that I had seen on other Sony Ericsson phones as well, a keypad that relly sucks. The keyboard is one of those plastic ones where a thick film of plastic covers a keypad with mechanical devices sensing keypresses underneath. The problem with this model was that they had messed up the placement of those sensors completely so that you actually had to press slightly above the marked digit to get the desired result. I wonder what kind of substance the engineers designing such a piece of ***** had been consuming before they hardened the design....

Verdict:3

Nokia 6500 Classic

After been doing my fair share of smartphone-hype, carrying around power hungry and feature-bulky phone for almost a decade, I wanted to go for a small, feature stripped phone with an appealing and urban design (!!?). I went for this simple and not very expensive model from Nokia. It had Java as well so together with the small but very nice screen it made a good company for the J2ME version of GMail. All in all I was quite happy with this phone that I got in the summer of 2007.

Verdict:5

iPhone 3G

I finally gave in in in 28'th of march 2009, my wife convinced me to finally get myself an iPhone. I gave in because I was very pleased with my acquisition of an iPod touch a couple of weeks earlier. After the 3.0 upgrade which arrived fashionably late in my opinion i think this is probably the best device with a cellphone I have ever had. Even though it does not matter it kind of pisses me of that it does not run Java.

Verdict:6

5 things I wish for in the near and not so near future

2009-07-24T22:34:00.001+02:00

In no particular order or priority.

I am currently on vacation, spending time with family and friends, and approaching things a little bit differently. By doing something else than going to work and not going about the normal routine (which I really enjoy, don´t get me wrong), you get new ideas on how to approach things, which is great. The small deviation in how to approach a problem can get you into new areas of unexplored ground. Here another day, inspired by Thinkertoys, I took another way to work. I drove passed a building I had never seen before with a name on it. I started thinking and it gave me an idea of a company name dealing with a business idea I have been puzzling with for a while. Small change may yield inspiring side-effects.

Speaking of business idea, I thought I would like to share the tings that I would like to have, of which some exists, but most don't. Some of the products or solutions are things I want and will probably get in the near future, some are products where the technology is and may never become available. Some of the things may address a small and not very important part of everday life, others may affect the way we live. Some of these things that I suggest may already exist, then excuse me for my incompetence and inadequate investigation of prior art. To avoid stepping on to many toes, I have chosen not to go into areas that may offend groups of people committing to a specific religion or political direction, maybe with the exception of public transportation :-)

Wireless everything

Everytime I want to install a new piece of equipment in my house, the thing that caused me the most headache is wiring. Coax for television, cat 5e for networking (no, Gigabit WiFi is not available outside laboratories, at least not to my knowledge), wiring for speakers, and of course, the infamous houshold power. And for crying out loud, when are we going to standardize on power transformers for DC devices, and I am not just talking of cellphones here (where they, believe it or not have agreed on a common standard). I think i have at least 50 different transformers in the house, where at least 15 of them are in use regularly, most of them ranging between 5 and 12 volts and 0.1 to 1.2 amps. None of them has a common physical interface. All these transformers make parts of my office space into a mess. By not being constrained by cabling, you can probably mount heating devices, lighting, stereo systems and everything else so that it will be more energy efficient, but also more pleasing from an aesthetic point of view. Giving what I save on my electrical bill I may no spend on buying more television sets, because I may now mount them on the wall without anything else, right? But then, what about the set-top box...well I don't think that there is enough storage space available in the world to actually store a persistent version of that topic, so that will have to wait for later.

Choose my own hardware when working at client site

Why is it that all businesses standardize on a workstation platform that is supposed to accommodate occasional needs for creating office documents and e-mail running on MS-Windows, often bundled with a 19"-inch monitor, insufficient amounts of RAM and a CPU that was at best mediocre two years ago. This is about as far as you can get from the specification of a decent development platform. In addition to that you usually have to go through the regular process of getting administrative rights, and to disable virus-control. You security people don't really get it, do you. Believe it or not, developers may need privileges to actually install applications on their workstation other than those approved and already installed by support. Why not trying to save some money, and let contractors and consultants use their own equipment instead, most possibly a laptop that they probably have already, but maybe also an external display as well. If networking is properly configured this should not be a problem neither from a maintenance nor a security perspective. And in order to avoid any potential envy or conflict, please provide your employees with the best possible equipment.

As for software, I am usually allowed to work with whatever tools that I want to as long as I don't expect my clients to provide them. My primary concern is hardware.

An Amazon Kindle or other e-book device

I am a regular Amazon customer, I buy mostly science and programming books, but recently I have also been buying my fair share of books on parenting, productivity, entrepreneurship and creative thinking. Books take up physical space and is heavy to carry around. More than once have I needed a book that I have bought without having it where and when I need it. I gudd this is partially caused by the ambulant nature of my work. With a decent e-book device I could possibly bring as many titled that I want in a package about the size of a regular book. Most titles would always be in stock and I would never-ever again be concerned again that the delivery service would deliver the package while I was away. I really need one of these things. The only little hitch about this device that made me quite worried may be found in this article. Words are not sufficient to describe the irony and hopelessness of such an act.

Seamless and effortless public transportation

If you do not live in Oslo or another provincial Nordic town, parts of this section may not apply to you. I have been using my car for work for years to go now. It is very convenient and has made it possible to cut down transportation-time to around 15 minutes each way. In addition to that, getting groceries or other things on the way home is the easiest of things. After getting my first child some time ago, me and my wife now share the car more and more, so I use the subway for work maybe twice a week. With good timing and no delays, I am able to get to work in around 45 minutes each way. This is from what I hear pretty decent, but still almost the triple of what I use today. I just don't have an extra hour a day to waste. I guess that without having to do a transfer, I could be down to around 20 minutes which is comparable to what I use in a car. Taking transfers out of the equation as long as both home and work is nearby a station for public transportation, or making transfers transparent will make this a competitive alternative for a much larger group of potential commuters.

The kind of system I had in mind consists of a network of autonomous small cubicles that may accommodate up to four passengers each. The backbone of the infrastructure is s grid of rails that may transfer a large set of these cubicles at high speeds and to their programmed destination. The grid may be powered by small electric motors or, preferably by magnetic leviation. A google search on "fremtidsrettet kollektivtransport" gave me 101 hits. When I get this few number of hits it is usually because I have mis-spelled something or that it doesn't exist. When it takes +7 years to get a non-working ticketing system for Ruter to go into production, i don't see this alternative being available anytime soon.

Being able to attribute the electrical consumption down to individual devices

In Norway, we are fortunate enough to cover most of our electrical needs from hydropower. From an emission perspective this is probably the cleanest, currently available source of energy that exists, but it cannot be exploited without significant consequences for the wildlife and environment that surrounds the power plant and any dams possibly built to provide for sufficient flow of water. The electrical consumption in Norwegian households have been increasing significantly and will continue to do so for the years to come. During periods with limited water supply, we are dependent upon supply from Denmark, that are more than willing to crank up their old coal-based power plants in Jutland to provide frozen Norwegians with stand-by power for their brand new LED tv's. Don't get me wrong, I am not a saint, I use my fair share of stand-by power. My point here is that most of our daily activities are assisted by some kind of device powered by electricity, and the electrical bill being a major part of the household budget, it is just fair that we should be able to have some more transparency utilization down to individual devices.

The power meter in my house has no means of even exposing the aggregated consumption of the household, it is one of those arcane, antiquated devices with a rotating disk. I guess that the likelihood of getting one of these equipped with some kind of interface that would enable me to offload some data is close to nothing. If I had one of the more recent meters I could do something like this but I don't and it will not provide me with all the information that I need.

If being able to attribute the power consumption down to individual devices, it would be much easier to convince people how to actually be more conscious on how they use their household devices and implicitly giving them the ability to understand how to reduce or improve their use. I think that getting this level of transparency is the key to get people to really understand how to spend the household-budget in a more environment friendly way. So are there any takers for developing this kind of infrastructure? I guess that the only thing it will take is for each device that consumes power to emit a signal containing an identifier of the device for every miliamp consumed.

2008-09-18T12:11:00.000+02:00

Continuously translating requirements into code in production fulfilling user expectations

This is a long phrase, but that is the term that I use when I refer to what I am doing as a developer through consultancy. Being a Senior Consultant, my workday consists of much more than just writing code. Although I like to refer to myself as a programmer, just coding won't do it. It is all about coding the right stuff. Avoid coding to much of the wrong stuff, and get the code into production as fast as possible. This is my mission, that is what i want to be good at.

Mary Poppendieck is talking about waste. Waste is pretty much anything that you do in your production line (aka project :-) that does not provide value for customers either directly or indirectly. I just came from a talk where she was talking about how to maximize throughout and how to reduce utilization. It is funny how many metrics not directly related to providing value for customers we have been measured against. It has been a long time since I have have been measured against how many lines of code I produce. Now we are measured using Story Points. This is better but in most cases, far from measuring the right thing.
The right thing is value for end-users delivered in production, this is what we should measure, or, we should at least know when we provide it :-)

2008-09-17T14:25:00.000+02:00

Nordås on domain-driven design

I must say, very nice introduction on Andres Nordås' talk on Doing Domain driven design better at JavaZone. He has really raised the bar with respect to getting someones attention to start with. Unfortunately It was no space left in Lab III for me and my co-worker. But now we are chilling out in the Overflow area, with really good audio and video so this is really great. Star Wars, get outa heeee....good work Anders.

Being able to provide infinite flexibility in the borderline between your domain and your infrastructure and to be more specific, repositories. When you want infinite flexibility, you also need to be sure that your coupling is loose so that you may adjust implementation strategies without afecting your domain, e.g what an order looks like or what it means to place an order.
Being able to refer to a repository in form of a generic repository interface is a very nice thing. It is even better that use you generically typed interface without knowing if it is an in-memory repository, or a Hibernate reposiory. Dealing with persistent lifecycle is in concept trivial, you store/delete and retrieve objects by query or by id. This is how all interaction with repositories should be stated.

Anders presens several examples that shows how easy it is to achieve loose coupling using generic types a container and pluggable strategies. You should really take a look to see how our code may be improved. Eirik Maus and myself will touch briefly into the same topics in our talk on Dealing with one to too many relations in Hibernate. I beleive that if you are able to correctly model your repositories as Anders does, all the other bits and pieces will fall into place.

2008-09-17T13:03:00.000+02:00

Talk about dynamic languages using Glassfish.

Just came from a talk of using dynamic languages in Glassfish. I would actually have expected some more in-depth information about practical appliances of dynamic languages, but the examples was pretty basic. I am really fond of the idea being able to pick the language of choice when confronted with a problem. If you are encountering a problem that would call for a functional language, pick Scala. If you want to leverage your existing investments in Rails-development, pick Ruby or Jruby. I don't thing that the strength of actually being able to do this cross-language is something that we are going to see much of. However, being able to use a hug range of languages in the same runtime is a very good thing.

Bringing web into all this is a problem. The package and deploy-step is a mess, and one of the reasons why I am looking into actually moving away from java for web. War-files is a mess, deployment is a mess and app-servers and standards (and deviations of them) is a mess. Why cant we just get some mod_java stuff for Apache and just use apt-get to install it so we can get faster into production. This is where I want to go.

2008-09-17T10:28:00.000+02:00

Javazone 2008.

It is finally time for doing JavaZone 2008 which I have been looking forward to quite some time. This year, I will try to go to as many hardcore presentations as possible. The reasons for that are pretty simple. Usually when I read articles and watch conference videos online, it is usually late in the evening after a long days work. My ability to dig into hardcore stuff is limited. Whereas, when I attend conferences, which I really enjoy, it something I can do fully rested, therefore I want to dive into technical stuff, closely related to what I do the most, programming.

The program this year looks OK, a good mix of technical and non-technical sessions. I think you will find something to your taste. What kind of sessions to you prefer yourself?.

2008-08-29T23:12:00.000+02:00

Testing code or testing requirements?

I have been in several projects where multiple attempts to classify tests into different categories has been attempted. Unit-tests, Web-tests, Behaviour-tests, Fitnesse-tests, Requirement-tests, Integration-tests, all of them written in Java in some ways or another. As you will see, the typical categories refers to different properties of a test. In some cases related to which layers of the code it touches. A typical unit test typically tests the contract of a given class, but will often test intentionally or unintentionally also test adjacent classes unless the rule is strictly enforced. A typical Web test could possibly be Selenium test that verifies various steps of a wizard with or without the back-end systems or databases to drive the data in place.
Some definitions of tests refers to the layer of code it refers to and how many interconnected classes they may include such as a unit-test or an integration test. Others may be more technology specific like Fitnesse-tests or Seleniums test that refers to the technology being used when implementing the test.

I believe that the core property of any test should be to capture and verify the expectation you have related to how the class, module, layer or application should work. That is in most cases why you actually write a test to start with. This is a very code-centric endpoint of the spectrum of continuously transforming user expectations and needs into a running system that provides value. Actually verifying the user-requirements and expectations is in the other end of this spectrum. There are other (possibly subordinate) properties of the process of writing tests also. If you use the same language for both tests and running code or supportive libraries, you will also verify if a class, interface or module is easy to use, and as part of the process you may actually improve it while writing the test. That makes that process of writing the test itself into something that validates more subtle and less quantifiable properties of the code. If doing TDD, writing a test is a starting point for writing code.

There are some bits and pieces missing in this picture. Very often I read about or hear about developers doing various kinds of testing in their project while applying some kind of label to what they are doing. The label may vary but terms like Requirement driven tests are becoming more and more widespread. I believe that this is good, because verification of requirements is probably the most difficult and important thing we are dealing with as system developers. But I have also seen this urge to classify everything as requirement test lead us in the wrong direction. Requirements means different things depending on who you are talking to.

I will try to address that using a brief description taken from a User Story document in one of my previous engagements:

"It should be possible to search for customers by first-name, last-name and birth-date. If multiple customers are returned a list of customers with the current address of residence should be presented along with the result"

We can discuss if this is a good definition of a requirement or not. This is not the point here. The point is, it captures a fragment of a user-expectation, a requirement. We may think of multiple tests in many of the categories above. But when does a written test really capture a requirement? I will say that depends, because requirement means different things depending on who you talk to. For me, as a coder, currently developing a jsp-based solution, one requirement applicable to the actual code I am writing may be that the web-controller returns the customer from a search as an array-type rather than a Collection based type (even though we have generics) simply because there is no way to refer to such a type using the useBean directive in JSP 2.0. This will therefore be a non-functional requirement in my code. It does not mean that I will be writing a test for that requirement in particular, but the unit test of my controller will most definitely be aware of this property.
The part of the code I am writing may also need a utility that merges two arrays into one. The SDK does not provide a utility that does that for us, but if a create one myself, i will most certainly update ArrayUtilsTest to provide a test for the new merge method in ArayUtills. It this a test of a requirement? most definitely, if verifies my expectation. Is it a requirement of the application itself that the system provides such a function? not necessarily, the primary property of the system is to provide customer search by the means necessary as defined by lead-developers and programmers on the project.

The primary conclusion put forward is that tests addresses the expectation of a whole spectrum of what we refer to as requirements. Quite a few of these requirements manifested as code in tests may not be easily translated into functional requirements owned and understood by the customers. Trying to be dogmatic about what kind of tests we write and to say that every line of code should be directly translated into a customer requirements may turn out to be a side-track. Very often when writing tests, the limitations of the test-framework and intrinsic properties of the interfaces or libraries you are using will force you to simplify and limit the number of assertions you apply to you test harness. It is important to understand that the requirement coverage provided by such tools will never be perfect. Strict tests will almost always end up being to brittle.

Getting your automated builds running the right set of tests is difficult. In my experience it ends up with a mix of unit tests, integration tests, web-tests and some flavor of requirement tests like fitnesse. These tests will test customer requirements and quite a bit of other properties. These tests tends to be something owned and maintained by developers more than by domain experts and customers, those that actually understand the problem domain.

What are your experiences?

2008-01-05T18:13:00.000+01:00

Using Macintosh keyboard in Ubuntu

I use various computers during a normal working day. I use a Macbook-pro as my laptop, a windows workstation running XP at the client-site where I am currently contracting, and at home I use (probably my favorite computer) an AMD workstation running Ubuntu Gutsy. Since I am primarily a Java developer, I use IntelliJ as my primary workspace, which is of course available on all platforms, the same goes with firefox and quite a few of the other applications I work with as well. One thing that has been annoying me lately, though is that I have not been able to find a really nice keyboard. The key bindings for shortcut keys in some of my favorite applications also varies slightly when using the different platforms. One thing i do know is that it is quite unlikely that I will change the keyboard on my Macbook, which i quite good anyway. I like way the keys both sounds and works.

Some years ago I used to really enjoy typing with those old IBM-style PC-keyboards when using OS/2 workstations at the university. They had really hard keys and made this really load "click" sound. Awfully noisy of course, but I really liked both the way they sounded and behaved. These are anyway not possible to get a hold of anymore.

I really like the looks of the new mac 105-keys keyboards in aluminum, so I got one ow those. The cool ting about this is that it will be much easier for me to switch between the macbook and my buntu workstation. I had a little trouble making the keyboard working as it is supposed to, but after some tweaking, I got it working.

What I had to do

Selecting the correct keyboard layout in Ubuntu is really simple. Just open the "Keyboard" application and you are all set to go. Select the appropriate keyboard layout, which in my case a PC-keyboard with the Norway Macintosh as the variant as shown in the figure.

The next thing you will have to do is enabling the Command-key as a third level chooser (this is usually the windows key on a regular PC-keyboard). Just set it like this:

Now I thought I was almost set to go, having the ()[]{} thing working for me, which is pretty damned important since I am a programmer, and I use these braces all the time. The only thing I discovered was that the keycode=49 was not mapped according to my keyboard. According to my layout it was supposed to be apostrophe bu it was bar. This does not correspond with my current layout, fortunately there are simple ways around this. I didn't know that the keykode was 49 for this key, so then I used "xev" to find it.
After finding the correct keykode, I just launched:

sudo xmodmap -e "keycode 49=apostrophe bar"

to change the mapping to get the apostrophe in the right place. Now I am all set to go. I am probably going to buy myself one of these keyboards for the windows-workstation I am using at the client-site as well.

2007-09-13T17:20:00.000+02:00

My tracks for the second day

It is only two sessions left for day one at JavaZone and I have to make planning arrangements for day III lready. Hopefully, I will also be able to update my first blog entry with comments on my previous encounters at JavaZone 2007 as well.

Ok, we have a lot of nice sessions tomorrow as well, actually we are doing BOFS' until 21PM on Thursday, not bad!!
This is my schedule for September 13.

*Time*	*Talk*	*Comment*
9.00-10:00	Java Persistence 2.0 - What's New, Linda DeMichiel
10:15-11.15	Open source Enterprise Integration with Mule, Tijs Rademakers og Jos Dirksen
11:45-12:45	Deploying Maven in the Enterprise, Kristoffer Moum
13:00-14:00	None
14:15-15:15	How Do I Invoke a Method, and Yes I Am A Senior Developer, Ole Andre Ranvik, Bjørn Bjerkeli	This is my own talk actually, I really recommend it, puts some things in perspective (I might be biased here :-)
15:45-16:45	Teaching computers How to Drive, Mary Poppendieck	Mind blowing
17:00-18:00	WS Duck Typing, Arjen Poutsma

2007-09-12T15:18:00.000+02:00

Jim Coplien talking about "Agile Stuff"

The Agile track is covered pretty well at JavaZone 2007, we have a lot of good speakers doing agile-talks. This is very much thanks to Jessica Hildrum that seems to be very well connected. Jim has been looking into agile issues for the past 12 years and works for a Danish company called Nordija.
Most of the projects that I have been working on lately has been organized in an Agile matter, Jim goes into the some details on explaining some common myths on agile processes and what works and whatnot. The five Myths discussed are:

On-Site customer
User Stories
TDD
YAGNI
DLS's

Before Jim starts to talk out, I will poin out the ones that I wold tend to focus on, from a developers perspective will be YAGNI and TDD. Jim tells us that it doesn't work, he's go me curious.
Jim clains that TDD will destroy your system, which is a pretty bald claim, nonetheless interesting. He claims that you will end-up building an unmaintainable and procedural architecture, which does not lend itself to refactoring. Well, he is getting my attention. I simply do not agree. I think what Jim thinks about here is that Unit testing at the JUnit level is not really testing of customer requirements, and he is perfectly right. It is tests testing parts of the system implemented using classes to build an object oriented system. This means requirements translated into something computionally feasible. Maybe I am not using TDD properly, but I do not expect that my JUnit tests maps one-to-one to functional-requirements.

What Jim misses out on here, especially with respect to saying that YAGNI will kill you is what Mary Poppendieck states about eliminating waste. I am a strong believer in up-front design, but I am not a strong beleiver in building anticipated structures, especially without tests. I think you should really eliminate waste.

2007-09-12T12:46:00.000+02:00

My track for the first day at JavaZone 2007

I have just came home from speakers dinner after talking to a lot of different people that are giving talks at javazone.no. I really didn't have time to figure out what tracks I will visit tomorrow, there are a lot of options, but I am trying to prepare i bit now. I finally was able to meet Tor Nordbye, Carl Quinn and Dick Wall from the JavaPosse, I am really looking forward to be present at the live podcast tomorrow evening.

Here are the shortlist of talks (selected from 7 parallell tracks) that I will probably attend the first day:

*Time*	*Talk*
9.00-10.0	Web 2.5 development with Java, Stephan Janssen
10.15-11.15	The State of Ajax, Dion Almaer and Ben Galbraith
13.00-14-00	Five practical solutions to Agile myths, James Coplien
14.15-15.15	Latest in Java Technology From Sun Microsystems , Simon Ritter og Angela M. Caicedo
15.45-16.45	Have to think about this one not sure actually
17.00-18.00	Scripting in Java 6 , Per Mengshoel og Thomas Heiberg
18.15-19.15	JavaPosse RoundUp, Live podcasting from Oslo (You would not want to miss this one)

2007-09-12T12:26:00.000+02:00

Kicking of JavaZone 2007

Finaly it has launched, JavaZone 2007 is running with more than 120 speakers from all over the world, and more than 2200 attendants, make this a pretty large conference, at least according to Scandinavian standards. I have been participating the program-commitee myself and it is very fun to see everything come to a start. The program this year is pretty well balanced, Enterprise, Web , Agile and Hard-Core Java speakers from all over the world is here in large numbers.

There are definitely a lot of focus on Agile this year. On other conferences I have been attending during the last couple of years, there is always one or two subjects that stands out, but I don't find it to be as clear this year. There are definitely bias do use less heavyweight infrastructure in web-like deployment-scenarios that it used to be. There is also a lot of focus on Maven. We actually had to reject as much as five talks on Maven, because it was already so well-covered.

I spoke with the Javaposse crew-members yesterday, and my impression is that Maven has a strong position in Scandinavia, and Less in the US.

Regardless if you are interested in Maven or Ajax there should definitely be something for every taste in this years program, hope you will enjoy it, we enjoyed putting it together!

2007-06-25T01:30:00.000+02:00

Continuous integration infrastructure, which OS to use

I have been on several projects lately where we have been using continuous integration-servers like CruiseControl and Continuum. This part of the development infrastructure is crucial, unless you are all by yourself (unlikely) you will need some server that will continuously integrate, build and deploy your code. This has been pretty much been a de-facto standard for quite a while now, and is not what I am trying to address here. What I am trying to address is, does the OS used for continuous integration-servers (CI servers) really matter? Yes it does and it should not run on Windows for reasons that should become obvious after reading this article.

It really matters, it is really not the question of which operating-system to use, but which operation system not to use. For some reason, it is not uncommon to configure and set-up CI-servers on Windows early in early development stages. I know several companies that only supports W2K/XP as part of their central operations, or Unix of course. But in case of Unix, the servers are often quite costly, cannot be dedicated for running CI, needs to be ordered and configured centrally and so on. Lately more and more companies are also supporting Linux as part of their infrastructure base, but it is not as widely supported as W2K/XP (I don't know anyone running Vista yet).

It is also quite common to use Windows as development desktop, even in environments that does not rely on windows-specific features. I am primarily developing in Java and for the most I use Windows desktops for development at client sites, not because I want to, but becose this is the company standard at most of my clients where I do contracting. At my home office I run Linux (Ubuntu) and Laptop runs Mac OSX. Given the fact that you will most likely end-up developing code using a Windows box when you are at work, it is also quite possible that it is easier to get access to a spare box for running one or several instances of your preferred CI-server if you go for Windows.

Well, so you go about and install CruiseControl, Continuum or whatever other software that you want to use, on Windows. Well, I have tried the same quite a few times now, and I can tell you, it simply doesn't work on Windows. Windows cannot and should not be used ever to run CI-server, because you will most certainly end up in a situation that some file is locked and that the CI server cannot run its goal because it cannot delete a file locked by some other process.
Another thing is long pathnames. Windows doesn't support pathnames that are longer than 255 characters, and I guarantee you, if you have a deep project structure you will almost certainly run into this problem. A third issue that may become a problem is access. When accessing a windows-box, people has this mental image that they will need to use explorer to actually view and edit contents of directories and files, which is just stupid. People will need to access the box when the test fails on the CI-sever and not locally, which is a quite common case (hard coded paths, locale-issues and network access in integration tests are common things). This means that developers will need to use RemoteDesktop to access the server. Well, only a limited amount on connections are available, and it is heavy on the server to support many simultaneous connections so this will quickly bring you into trouble as well. All these issues makes Windows an no-go for CI servers, it is simply not possible to maintain stable CI servers without file-locking, pathname-problems and locking because of simultaneous access. You will need a full time employee to do the maintenance or go with Unix or Linux.

What should I do?

Well, an option is to take any Unix/Linux box available and use that for continuous integration instead. It will make all the problems mentioned in the previous paragraph go away. File locking is not a problem, pathnames can be as long as you want (almost true) and since everyone is using ssh, you could support a high number of simultaneous users viewing the build-log files using terminal access (which is very lightweight) instead of RemoteDesktop. The short answer is go with Unix based system. But what if that is not possible?
That was the case at previous engagement. Going for Linux was a no-op. And if we wanted to go for Unix, Solaris was the way to go. But we would never get root-access to that box and they have to order it first....*sigh*. Well, first of all.

My experience is that configuring CI-servers are generally best left as a task to the development team, preferably someone that has done this a couple of times before. I don't think it is possible to put these kind of things on order at your system-administration department, those responsible for test and production environments. It is probably outsourced anyway, so you will have to fill out a form before you are even allowed to all them. So much for agile.
Well, so what did we do. We went for Virtualization using VMWare. VMWare gives you the ability to run any number for operating systems, called the guest OS, by using a player. The OS running the player is referred to as the the host OS. Here we were able to use a Windows-box running Ubuntu-Linux (my choice of course :-) as a guest operating system. After the guest operating system has been started it behaves looks an performs like a full-blown standalone OS without all the quirks mentioned earlier. The Virtual image was created with VMWare Workstation and may be executed with either VMWare Workstation or VMWare Player which is free. The process of installing the operation system (Ubuntu Feisty) is identical to what you would do when you install it from scratch. I will not cover the installation details here, but briefly reflect on some additional benefit that may be provided when using VMware when running CI-servers (these issues may also apply to running appliances in general). Note that this also makes Windows boxes a viable alternative as a host OS.

Benefits of running a virtualized CI-server

In my experience, CI servers are not treated as production-goods my most organizations, although they should, This often results in decisions like, hey we need to use that box we gave you last week, can't you just use this one instead? The point is, it should be really easy to change hardware. In VMWare your entire installation is basically just a couple of files in a directory. If you copy and transfer those files to a new box, you can start the guest-OS using the player there instead. The only thing you will have to install on the new box is the VMWare Player.

Staging or upgrading such environments are often also problematic. This is actually production software for your development team. If you want to upgrade the CI server with lets' say a new version of Continuum, you can do so my taking a snapshot of your current image. Copy that image and start it up on your desktop. Here you can do whatever changes that you want to the guest operating system, including upgrading to the newest version of Maven and Continuum. You can even dry test the build on your local machine. If it work, you are all set to offload an updated snapshot to the CI-production server. By using VMWare in this way, it really gives the term "staging" true meaning.
Based upon experience for several years running large projects with CI servers, this is really the best approach. Having a build CI-server going to a stop caused by reasons not related to code or test uncovering actual problems over and over a again will quickly render your CI infrastructure useless, people will stop to trust it, and this is not what you want.

2007-05-04T23:51:00.000+02:00

Upgrading Ubuntu from Edgy to Feisty

Last week I upgraded my my Ubuntu installation from Edgy to Feisty. I had been playing around quite a bit to get Compiz and Beryl to work whithout much luck on Edgy. Some time ago I also tried to install the beta of Feisty, but i could not get my Radeon X1800 card working properly, the desktop was a mess. Anyways, I am occasionally using the Synaptic package manager, and not to long ago it asked me if I wanted to upgrade from Edgy Eft, which is 6.10 to Feisty running on 7.0.4. Apart from a few new tools, more adequate support for restricted drivers, like my ATI -card and build-in support for Compiz and Beryl, I was tempted to give it a go.

I have had my Dual screen desktop running 2 x 22" screen monitors running with pristine colors for some time, but withould 3D effects. After upgrading however, I could just see that it was not working properly. It did not render fonts properly in the dual heda mode, but as working fine when using just one monitor. Disappointing, I was really looking forward to getting Compiz and Beryl up and running, instead I was actuallu set back to start. Now I am waiting for the system to download all updates after I reinstalled the entire system from scratch.

So if you are using an ATI Radon X1800/1900 card, do not upgrade to Feisty if you are using a dual-head configuration on you displays. Now I have been wise enough to configure a separate partition where I can engage myself in a feasibility study before I actually see if it is working.

Anyways, after switching from XP 64, which did not work at all!! I am very satisfied with my Ubuntu distro, I really want to recommend it.

2007-03-19T00:20:00.000+01:00

Unit testing, a lot of developers don't get it

As a developer I quite often stumble across code that I would like to rewrite. The reasons why I want to rewrite the piece of code, may differ from case to case, reduce coupling and remove duplication are well known reasons. Lately I have been doing a lot of refactoring and one thing that stands out is how much terrible test code it is out there. It seems as if developers think that if they just have written a test for it, and the code coverage if good, then everything is good. This is dead wrong. I have seen people backing away from changing bad code just because they were backed by brittle and hard to understand tests.

Doing refactoring requires that tests are present. In systems using code written less than two or maybe three years ago, it seems to be quite rare that the code is not been backed by tests to some extension. But the way tests are written varies greatly. Having tests is a necessary but not sufficient prerequisite to do effective refactoring. In the latter cases where I have been doing requirements it strikes me as if a lot of developers do not understand that the same principles that they apply when writing system-code, should also be applied when writing test-code. It is all code, and should be treated as such. I have seen by far too much duplication, lengthy pieces of code iterating over collections just to add a conditional and assert for the particular element they where looking for, and by far to much cut-copy paste, ecpecialy in test-fixtures.

I am am doing a Lightening Talk on how to simplify unit testing on Oslo XP Meetup on 26 March. I will discuss small things you can use to write more fluent tests, tests that are easier to read and that requires less code. When I was reading up on different aspects on Unit testing, I came over this great article that was linked from unit.org. It really explains in sufficient details code smells in test-code and how to refactor them. Read it and distribute it to your peer developers.

2007-03-04T19:53:00.000+01:00

Effective pagination using Hibernate

This article presents with an solution on how to page through large amounts of data effectively without taking down your database and without keeping it all in memory.

Last major update 07.09.2008

I have now added references that describes how to retrieve the source-code for the examples. All source-code is released under the Creative Commons License as can be seen from the Licensing details, the source code can be retrieved from http://svitjod.rymdweb.com/repository/com/zenior/sandbox/ which contains the source-code in maven form. To include a dependency to the source code just include:

<id>internal</id>
<name>Svitjod</name>
<layout>default</layout>
<url>http://svitjod.rymdweb.com/repository</url>
<snapshots>
<enabled>false</enabled>
</snapshots>

</repository>

and

<dependency>
<groupId>com.zenior.sandbox</groupId>
<artifactId>example-domain</artifactId>
<version>1.0-SNAPSHOT</version>
<scope>compile</scope>
</dependency>

<dependency>
<groupId>com.zenior.sandbox</groupId>
<artifactId>hibernate-sandbox</artifactId>
<version>1.0-SNAPSHOT</version>
<scope>compile</scope>
</dependency>

To your project pom.xml.

If do not use maven, you may just download the source code by looking for files ending with -sources in the maven repository mentioned above.

Abstract

Dealing effectively with large resultsets is important. Batch applications may require to process millions of rows as part of a transaction. The brute force approach would be read everything up in memory before iterating over theelements. This approach will effectively kill performance and use far to much memory. One common approach when using Hibernate is to use options in the Criteria interface called setFirstResult/setMaxResult. This will generate SQL (at least for Oracle) where the database will actually have to read and throw away data before returning them to the client. Another approach is to use ScrollableResults, but here you will have to keep the connection open during the entire pagination process. By retrieving a collection of all the primary keys for the matching object, you will need to perform an extra query before actually retrieving the data, but it is very easy to generate effective SQL to retrieve using this approach. This is by far the best solution in terms of database-load and memory footprint, and it is easy to implement in a portable manner.

The problem

In order to have a rich domain model, we want relationships between domain objects to be implemented as relationships between the Java classes. If we are going to be able to do that without killing performance, we have be smart in how we deal with relationships having high cardinality.

Most importantly we must iterate through a large resultset without keeping all data in memory. We also want the approach to generate simple SQL, and avoid using esoteric options that make the SQL difficult to optimize or difficult to port. We might also need adjustments to the mapping strategy and relationships to retrieve data effectively.

It is assumed that the reader is familiar with Hibernate and with common ORM issues like n+1 selects.

The example

An order application processing an order with more than 100.000 order lines.We will see how different solutions differ with respect to memory consumption, resource usage in general, effectiveness in terms of execution and differences in how the SQL is generated. The solution will be using Java 6, Hibernate3.2.0GA with annotations, the Spring framework and Oracle XE.

I will test three different approaches that don't use custom code and for each approach say something about execution time, memory consumption and explain the inner details.
As a domain model for the test I have been using a domain-model representing a generic order-domain as shown in figure 1.

Figure 1
The model is simplified but representative.

Test fixture

I have used Hibernate and Springs AbstractTransactionalSpringContextTeststopopulate the test fixture that has been inserted into the database. Idon't want the unit test to include timing for actually setting up the data.Therefore I have executed the fixture-setup by adjusting the test-case slightly so that the fixture is actually committed, by using the following approach:

  1 protected void onSetUpInTransaction() throws Exception {
  2         if (!isTestDataPresent()) {
  3             createOrderWithOrderLines(1000 * 1000);
  4             setComplete();
  5             endTransaction();
  6         }
  7 }

Listing 1
By using the AbstractTransactionalSpringContextTests I can actually force the test itself to commit the data, which may sometimes be useful when inspecting persistent side-effects to data when executing tests. The methods setComplete() and endTransaction() will force the execution to commit after the test is finished. To make it idempotent, I have added a method that checks f the test-data is already there.

The brute-force approach yields the following test-results on my workstation, which is a custom built AMD Athlon 64 X2 4600, 4GB Ram, Dual Raptor 150GB SATA in a Raid 0 setup. The tests have been launched from IntelliJ 6.0.4 and with the Oracle XE instance running on the same machine. The test runner is launched in a spawned process using a maximum of 1 GB RAM (-Xmx1G).

Brute force

In the first approach, we process the entire order without thinking about memory consumption when we retrieve OrderLine instances from Hibernate.
This solution basically uses the one-to-many relational mapping and leaves it up to Hibernate to retrieve the data.

The mapping is defined as follows:

  1   @OneToMany(mappedBy = "order", cascade = CascadeType.ALL, fetch = FetchType.LAZY)
  2     public Set getOrderLines() {
  3         return orderLines;
  4     }

Listing 2
And on the OrderLine side of the bi-directional relationship, we have:

  1  @ManyToOne(fetch = FetchType.EAGER)
  2     public Order getOrder() {
  3         return order;
  4     }

Listing 3

The test that gathers timing and resource consumption information is a simple JUnit-test method:

  1 public void testRetrieveOrderBruteForce() {
  2         Order order = (Order) hibernateTemplate.get(Order.class, orderId);
  3         ResourceMonitor monitor = new ResourceMonitor("order.getOrderLines()");
  4         assertEquals(1000*1000,order.getOrderLines().size());
  5         System.out.println(monitor.stop());
  6 }

Listing 4

Time elapsed	Memory usage delta
79,5s	501000kb

Table 1 Test results brute-force approach

As we can see, it is quite fast to retrieve all the data in just a single select. We have retrieved 1000000 OrderLine instances in close to 80-seconds. The problem with this approach is that it is memory intensive. On most 32-bit systems, there is a practical upper limit of around 2GB heap, which will make it difficult to have more than three such queries running simultaneously [1].
In addition to this, when using a StatefulSession in Hibernate to keep track of updated fields, keeping this much data in memory at the same time will eventually halt your system or make it very slow [2]. Note that making the relationship lazy (which is the default behavior in Hibernate) will not change that fact that everything is fetched using one single select.

Using Criteria search limited by firstResult and maxResult

In order to retrieve the OrderLine side of the relationship using a Criteria query, we clearly cannot have a Hibernate-managed relationship between Order and OrderLine, we will have to manage this relationship ourselves. The most visible consequence of this will be that the Set<OrderLine> property of Order will have to be removed.This is unfortunate because it removes an important aspect of the domain-model, which is the explicit composition that constitutes the relationship.

When dealing with cases like this, I would like to be explicit about the relationship also in the domain of the Order-class, so other than removing the getOrderLines()method from the Order class, I would prefer changing it to something like this shown in listing 5.

  1 public Set<OrderLine> getOrderLines(HibernateTemplate template) {
  2         ....
  3 }

Listing 5

Here it is still quite clear that Order has a relationship to OrderLine. Still, however, we will have to implement the paging mechanism using a Colllection-implementation, which preserves a pretty decent domain-model with respect to an explicit relationship between Order and OrderLine. This might turn out to be a good thing, because it will probably easier to refactor the getOrderLines method to be extended to take a HibernateTemplate as an argument instead of in lining some other implementation whenever we need to retrieve OrderLine instances belonging to an Order.
An even better approach here would be to de-couple the dependency to Hibernate completely.

I discovered this after implementing the getOrderLines() - method, taking hibernateTemplate as a parameter. The example-domain module that contains the order-domain related classes, did not have a dependency to Hibernate, which I did not want to introduce because I wanted the example-domain to be as lean as possible. In order to compare multiple strategies for paging using the same domain-classes I had to introduce an interface that made plugging different strategies as in listing 6.

 1 public Collection getOrderLines(CollectionBuilder collectionBuilder) {
 2         return collectionBuilder.build(this);
 3 }


 1 public interface CollectionBuilder {
 2     /**
 3      * Build the collection using information from the given instance.
 4      * @param instance
 5      * @return
 6      */
 7     Collection build(Object owner);
 8 }

Listing 6

Using the Collection-builder interface, I could easily provide a Hibernate based implementation of the different strategies without introducing a Hibernate dependency into the domain-classes. Here it would be even more visible that the one-to-many collection is owned by an order instance, signified by the owner parameter of the build-method.

To support paging by restricting the underlying resultset Hibernate uses a mechanism by using the Dialect interface. This mechanism varies greatly between different RDBMS'es. The Orcale9Dialect uses a construct in oracle called rownum restrictions, which may be found in the getLimitString-method of the Dialect implementation given in listing 7 (listed only in part).

  1 public String getLimitString(String sql, boolean hasOffset) {
  2 ....
 10 		StringBuffer pagingSelect = new StringBuffer( sql.length()+100 );
 11 if (hasOffset) {
 12 			pagingSelect.append("select * from ( select row_.*, rownum rownum_ from ( ");
 13 		}
 14 else {
 15 			pagingSelect.append("select * from ( ");
 16 		}
 17 		pagingSelect.append(sql);
         ........
 30 }
 31

Listing 7

Without careful indexing and dependent upon the underlying RDBMS you might end up with a situation that may force the RDBMS to scan all the data for the entire set of OrderLine items each time in order to return the subset limited by setFirstResult and setMaxResult.
Here is an implementation of a Collection implementation that may page the underlying data based upon limiting the page size in the way mentioned above:

  1 public class PagingCollection extends AbstractCollection {
  2     private HibernateTemplate hibernateTemplate;
  3     private HibernateCriteriaSpecification specificationCriteria;
  4     private int pageSize = 2000;
  5     private Integer collectionSize;
  6     private Criteria queryCriteria;
  7 
  8 
  9     public PagingCollection(HibernateTemplate hibernateTemplate, HibernateCriteriaSpecifiaa criteria, int pageSize)
 10         this.specificationCriteria = criteria;
 11         this.hibernateTemplate = hibernateTemplate;
 12         this.pageSize = pageSize;
 13         this.queryCriteria = criteria.populate(createCriteria());
 14     }
 15 
 16     public Iterator iterator() {
 17         return new PagedIterator(size());
 18     }
 19 
 20     public int size() {
 21         if (collectionSize == null) {
 22             Criteria criteria = specificationCriteria.populate(createCriteria());
 23             criteria.setProjection(Projections.countDistinct("id"));
 24             collectionSize = (Integer) criteria.uniqueResult();
 25         }
 26         return collectionSize;
 27     }
 28 
 29     private Criteria createCriteria() {
 30         Criteria criteria = (Criteria) hibernateTemplate.execute(new HibernateCallback() {
 31             public Object doInHibernate(Session session) throws HibernateException, SQLException {
 32                 return session.createCriteria(specificationCriteria.getPersistentClass());
 33             }
 34         });
 35         return criteria;
 36     }
 37 
 38     private class PagedIterator implements Iterator {
 39         public boolean lastPage = false;
 40         private int offset;
 41         private List currentCollection;
 42         private Iterator currentIterator;
 43         private int size;
 44 
 45 
 46         public PagedIterator(int size) {
 47             this.size = size;
 48             this.currentCollection = loadPage(0);
 49             this.currentIterator = currentCollection.iterator();
 50         }
 51 
 52         protected List loadPage(int offset) {
 53             if (offset > size) {
 54                 throw new NoSuchElementException();
 55             }
 56             this.offset = offset;
 57             queryCriteria.setFirstResult(offset);
 58             queryCriteria.setMaxResults(pageSize);
 59             currentCollection = queryCriteria.list();
 60             if ( (this.offset + pageSize )>= size) {
 61                 lastPage = true;
 62             }
 63             return currentCollection;
 64         }
 65 
 66         public boolean hasNext() {
 67             return size() > 0 && (currentIterator.hasNext() || !lastPage);
 68         }
 69 
 70         public Object next() {
 71             if (!currentIterator.hasNext()) {
 72                 unloadPage(currentCollection);
 73                 currentCollection = loadPage(offset + pageSize);
 74                 currentIterator = currentCollection.iterator();
 75             }
 76             return currentIterator.next();
 77         }
 78 
 79         protected void unloadPage(List currentCollection) {
 80             for (Object o : currentCollection) {
 81                 hibernateTemplate.evict(o);
 82             }
 83         }
 84 
 85         public void remove() {
 86             throw new UnsupportedOperationException();
 87         }
 88     }
 89 }

Listing 8

The test case looks as follows, note that we have to iterate the entire set if we want to poll everything out from the database. The size() method just returns an aggregated value by issuing a count.

  1  public void testRetrieveOrderLinesUsingPaging() {
  2         Order order = (Order) hibernateTemplate.get(Order.class, orderId);
  3         Collection orderLines = order.getOrderLines(hibernateTemplate, 50000);
  4 
  5         ResourceMonitor monitor = new ResourceMonitor("order.getOrderLines()");
  6         for (Object orderLine : orderLines) {
  7             //
  8             OrderLine line = (OrderLine) orderLine;
  9         }
 10         assertEquals(1000 * 1000, orderLines.size());
 11         monitor.stop();
 12         System.out.println(monitor);
 13 
 14  }

Listing 9

Without tuning the underlying database with respect to query optimization, the test result looks like this:

Time elapsed	Memory usage delta
381s	29791kb

Table 2 Test result using setFirstResult/setMaxResult

Note that the footprint is reduced to below one tenth, but the response-time is not satisfactory. In this case a new query will be issued across each page boundary, in this case each page being 50.000 elements. It turns out that executing the SQL to retrieve one single page is almost as expensive as retrieving all rows in a single select. To understand why, we have to examine the SQL.

 1 SELECT *
 2 FROM
 3   (SELECT row_.*,
 4      rownum rownum_
 5    FROM
 6     (SELECT * 
 7        FROM orderline o
 8        WHERE o.order_id = order2_.id)
 9      FROM orderline this_ LEFT
10      OUTER JOIN customer_order order2_ ON this_.order_id = order2_.id LEFT
11      WHERE this_.order_id = 10)
12   row_
13    WHERE rownum <= 100000)
14 WHERE rownum_ > 50000;

Listing 10

The SQL in listing 10 has been simplified from what Hibernate generates, but the most important attribute is the nested select with the rownum-restriction, which is shown. The query above will retrieve the second page in the select, the rows ranging from 50000 to 100000. Note the nested query where the first (innermost) select will return all rows with a rownum less than 100000. This is followed by a filter on the inner result, returning only the rows counting from 50000 and above. This means that the RDBMS will actually read and throw away most of the rows for each query. For this to be implemented consistently, we also need to make sure that the same select will return the same rows repeatedly. This can only be done if we add an order by clause to the select above (which is not included in this case). See [3] for a discussion on how rownum works in Oracle.

Using this approach for large results will effectively use up all your database resources.

Using ScrollableResults

The approach that uses ScrollableReults relies on the underlying JDBC-driver providing support for some kind of cursor that may allow you to retrieve data in an incremental manner without keeping everything in memory. A ScrollableResults can beretrieved from a Criteria implementation by using one of the Criteria.scroll(..) methods.Support for this may vary. Old versions of the JDBC-drivers in Oracle has implemented this as a client side cache that keeps everything in memory.Whereas newer-versions implements this using server-side cursors.

The implementation that uses the ScrollableResult is straightforward. The CollectionBuilder implementation is shown in Listing 11 and the actual Collection implementation in Listing 12.

 1 public class OrderLineScrolleableResultSetCollectionBuilder implements CollectionBuilder {
 2     private HibernateTemplate hibernateTemplate;
 3     private int pageSize;
 4 
 5     public OrderLineScrolleableResultSetCollectionBuilder(HibernateTemplate hibernateTemplate, int pageSize) {
 6         this.hibernateTemplate = hibernateTemplate;
 7         this.pageSize = pageSize;
 8     }
 9 
10     public Collection build(Object instance) {
11         Order order = (Order) instance;
12         return new ScrollableResultsCollection(hibernateTemplate, new HibernateCriteriaSpecification(OrderLine.class, order, "order"), pageSize);
13     }
14 }
15

Listing 11

  1 public class ScrollableResultsCollection extends AbstractCollection {
  2     private HibernateTemplate hibernateTemplate;
  3     private HibernateCriteriaSpecification specificationCriteria;
  4     private int pageSize = 2000;
  5     private Integer collectionSize = null;
  6     private Criteria queryCriteria;
  7 
  8 
  9     public ScrollableResultsCollection(HibernateTemplate hibernateTemplate, HibernateCriteriaSpecification criteria, int pageSize) {
 10         this.specificationCriteria = criteria;
 11         this.hibernateTemplate = hibernateTemplate;
 12         this.pageSize = pageSize;
 13         this.queryCriteria = criteria.populate(createCriteria());
 14     }
 15 
 16     public Iterator iterator() {
 17         return new PagedIterator(size());
 18     }
 19 
 20     public synchronized int size() {
 21         if (collectionSize == null) {
 22             Criteria criteria = specificationCriteria.populate(createCriteria());
 23             criteria.setProjection(Projections.countDistinct("id"));
 24             collectionSize = (Integer) criteria.uniqueResult();
 25 
 26         }
 27         return collectionSize;
 28     }
 29 
 30     private Criteria createCriteria() {
 31         Criteria criteria = (Criteria) hibernateTemplate.execute(new HibernateCallback() {
 32             public Object doInHibernate(Session session) throws HibernateException, SQLException {
 33                 return session.createCriteria(specificationCriteria.getPersistentClass());
 34             }
 35         });
 36         return criteria;
 37     }
 38 
 39     private class PagedIterator implements Iterator {
 40         public boolean lastPage = false;
 41         private int offset;
 42         private List currentCollection;
 43         private Iterator currentIterator;
 44         private int size;
 45         private ScrollableResults scrollebleResults;
 46 
 47 
 48         public PagedIterator(int size) {
 49             this.size = size;
 50             this.scrollebleResults = queryCriteria.scroll(ScrollMode.FORWARD_ONLY);
 51             this.currentCollection = loadPage(0);
 52             this.currentIterator = currentCollection.iterator();
 53 
 54         }
 55 
 56         protected List loadPage(int offset) {
 57             if (offset > size) {
 58                 throw new NoSuchElementException();
 59             }
 60             this.offset = offset;
 61             currentCollection = new ArrayList(pageSize);
 62             System.out.println("Loading offset ["+offset+"]");
 63             if ((this.offset + pageSize) >= size) {
 64                 lastPage = true;
 65             }
 66             int scrollSize = lastPage ? size - offset: pageSize;
 67             for (int i = 0; i < scrollSize; i++) {
 68                 if (!scrollebleResults.next()) {
 69                     throw new NoSuchElementException();
 70                 }
 71                 currentCollection.add(scrollebleResults.get(0));
 72             }
 73 
 74             currentIterator = currentCollection.iterator();
 75             return currentCollection;
 76         }
 77 
 78         public boolean hasNext() {
 79             return size() > 0 && (currentIterator.hasNext() || !lastPage);
 80         }
 81 
 82         public Object next() {
 83             if (!currentIterator.hasNext()) {
 84                 unloadPage(currentCollection);
 85                 currentCollection = loadPage(offset + pageSize);
 86                 currentIterator = currentCollection.iterator();
 87             }
 88             return currentIterator.next();
 89         }
 90 
 91         protected void unloadPage(List currentCollection) {
 92             for (Object o : currentCollection) {
 93                 hibernateTemplate.evict(o);
 94             }
 95         }
 96 
 97         public void remove() {
 98             throw new UnsupportedOperationException();
 99         }
100     }
101 }
102

Listing 12
The ScrollableResults -interface is basically just a low-level approach in mapping row-based data into domain-objects. The instantiated and mapped objects are retrieved using the ScrollableResult.get(..)method. The get-method retrieves objects index from 0. It may be possible to retrieve several objects from a result-set, because hibernate generates queries that provides data for several objects when using outer-join fetching to avoid multiple roundtrips to the database.

Let us say that we extend the model presented in Figure 1 with a one-to-many relation relation to OrderLineDetail as shown in Figure 2.

Figure 2

In tis case, to get all the data from OrderDetail using one select and outer-join fetching, will make the returned select return four rows in total with the data for OrderLine repeated once for each instance caused by the outer-join with OrderDetail. This means that we lack a layer of abstraction here. We cannot implement this in a generic manner. We will actually have to alter the implementation of the strategy implemented to retrieve OrderLine instances just because we did a small change to the domain-model, adding the OrderDetail -relation.

The ScrollableResults is backed by a ResultSet and needs the connection to be open during the iteration of the query. This will pose some challenges when implementing paging spanning multiple web-requests, typically paging in a web-page, because the ScrollableResult will need to be valid and usable across requests.

Even though the approach is fast and lean on memory as can be seen by looking at the test-results in table 3, these limitations will probably get you in to problems except from in the simplest cases.

Time elapsed	Memory usage delta
89s	26009

Table 3

The solution

The most effective solution is to use the primary key as a paging criterion.This enables us to rely of first class constructs like a between range query which is simple for the RDBMS to optimize, the primary key of the queried entity will most likely be indexed already.
Retrieving data using a range query on the primary key is a two-step process. First one have to retrieve the collection of primary-keys, followed by a step to generate the intervals to properly identify a proper subset of the data,followed by the actual queries against the data.

  1 package com.zenior.sandbox.hibernate.pagination;
  2 
  3 import no.zenior.domain.HibernateCriteriaSpecification;
  4 import org.springframework.orm.hibernate3.HibernateTemplate;
  5 
  6 import java.util.AbstractCollection;
  7 import java.util.ArrayList;
  8 import java.util.Collection;
  9 import java.util.Iterator;
 10 
 11 /**
 12  * Collection implementation that iterates using a range query for available identifiers
 13  * for the queried collection.
 14  * </br>
 15  *
 16  * @author Bjørn Bjerkeli
 17  */
 18 public class IdIntervalCollection extends AbstractCollection {
 19     private static final int DEFAULT_PAGESIZE = 2000;
 20     private HibernateTemplate hibernateTemplate;
 21     private HibernateCriteriaSpecification finderSpecification;
 22     private int pageSize = DEFAULT_PAGESIZE;
 23     private IdIntervalCollectionBuilder pager;
 24     private int size;
 25 
 26     /**
 27      * @param hibernateTemplate
 28      * @param finderSpecification
 29      */
 30     public IdIntervalCollection(HibernateTemplate hibernateTemplate,
 31                                 HibernateCriteriaSpecification finderSpecification) {
 32         this(hibernateTemplate, finderSpecification, DEFAULT_PAGESIZE);
 33     }
 34 
 35     /**
 36      * @param hibernateTemplate
 37      * @param finderSpecification
 38      * @param pageSize
 39      */
 40     public IdIntervalCollection(HibernateTemplate hibernateTemplate,
 41                                 HibernateCriteriaSpecification finderSpecification, int pageSize) {
 42         this.hibernateTemplate = hibernateTemplate;
 43         this.finderSpecification = finderSpecification;
 44         this.pageSize = pageSize;
 45         initialize();
 46     }
 47 
 48     /**
 49      * Initialize the instance of BucketIdIntervalPager.
 50      */
 51     private void initialize() {
 52         Collection identifiers = HibernateRepositoryTool.idProjection(hibernateTemplate, finderSpecification);
 53         this.pager = new IdIntervalCollectionBuilder(pageSize);
 54         for (Object identifier : identifiers) {
 55             pager.add((Long) identifier);
 56         }
 57         this.size = identifiers.size();
 58         hibernateTemplate.clear();
 59     }
 60 
 61 
 62 
 63     public int size() {
 64         return size;
 65     }
 66 
 67     public Iterator iterator() {
 68         return new Iterator() {
 69             private Collection currentPage = new ArrayList();
 70             private Iterator currentIterator = currentPage.iterator();
 71             private Iterator currentIdIntervalIterator = new ArrayList(pager.buildIdIntervals()).iterator();
 72             private IdInterval currentIdInterval;
 73 
 74             public void remove() {
 75                 throw new UnsupportedOperationException();
 76             }
 77 
 78             public boolean hasNext() {
 79                 return ((size > 0) && currentIdIntervalIterator.hasNext())
 80                         || currentIterator.hasNext();
 81             }
 82 
 83             public Object next() {
 84                 if (!currentIterator.hasNext()) {
 85                     unloadPage(currentPage);
 86                     currentIdInterval = (IdInterval) currentIdIntervalIterator.next();
 87                     loadPage(currentIdInterval);
 88                 }
 89 
 90                 return currentIterator.next();
 91             }
 92 
 93             protected void unloadPage(Collection currentPage) {
 94                 for (Object o : currentPage) {
 95                     hibernateTemplate.evict(o);
 96                 }
 97             }
 98 
 99             /**
100              * Load the data corresponding to the idInterval.
101              */
102             protected void loadPage(IdInterval currentIdInterval) {
103                 finderSpecification.setIdInterval(currentIdInterval);
104                 currentPage = HibernateRepositoryTool.find(hibernateTemplate, finderSpecification);
105                 currentIterator = currentPage.iterator();
106             }
107         };
108     }
109 }
110 
111

Listing 13

The key here is basically to implement the iteration process in two steps, one to retrieve identifiers, and one to actually provide the iterator with an id-interval that is used tospecify a between range-query. Retrieving the set of identifiers is easy, the HibernateCriteriaSpecification that uses Criteria for search has a Projections.id()factory that returns all identifiers for a given Criteria-query. Basedupon the entire set of identifiers we are able to construct buckets of identifiers each representing a logical and proper page of data.

  1  public void testRetrieveOrderLinesUsingPagingIdInterval() {
  2         final Order order = (Order) hibernateTemplate.get(Order.class, orderId);
  3         Collection orderLines = order.getOrderLines(new IdIntervalPagedCollectionBuilder(hibernateTemplate, 50000));
  4 
  5         ResourceMonitor monitor = new ResourceMonitor("order.getOrderLines()");
  6         for (Object orderLine : orderLines) {
  7             //
  8             OrderLine line = (OrderLine) orderLine;
  9         }
 10         assertEquals(1000 * 1000, orderLines.size());
 11         monitor.stop();
 12         System.out.println(monitor);
 13         
 14 }

Listing 14

The IdIntervalPagedCollectionBuilder is the pluggable implementation of the CollectionBuilder interface described in listing 6.

 1 public class IdIntervalPagedCollectionBuilder implements CollectionBuilder {
 2     private HibernateTemplate hibernateTemplate;
 3     private int pageSize;
 4 
 5     public IdIntervalPagedCollectionBuilder(HibernateTemplate hibernateTemplate, int pageSize) {
 6         this.hibernateTemplate = hibernateTemplate;
 7         this.pageSize = pageSize;
 8     }
 9 
10     public Collection build(Object instance) {
11         return new IdIntervalCollection(hibernateTemplate, 
12                 new HibernateCriteriaSpecification(OrderLine.class, instance, "order"), pageSize);
13     }
14 }

Listing 15

The three examples have almost the same amount of code, but rely onquite different approaches. The test-result for the last test is remarkable.

Time elapsed	Memory Usage Delta
99,5s	29402

This approach is almost as fast as the brute-force version. The memory consumption is about one tenth. By selecting the appropriate page-size for this implementation, you may alter the ratio between execution time and memory consumption. This version is also stateless, it does not keep references to resources like the ScrollableResult version does, nor does it strain the database like the version using setFirstResult/setMaxResult.

Limitations

We need a primary key, preferably a surrogate key, that has a natural ordering so that we can implement the paging concept by using a between search for the identifiers. Another limitation is the need to actually retrieve all the identifiers before performing the search, which means two select-statements instead of possibly only one.

The Author

The Author is working as a senior consultant in a Norwegian consulting company, Zenior AS, and is an active member of the Norwegian JUG, javaBin.

Licensing

Hibernate paging using id-intervals by Bjørn Bjerkeli is licensed under a Creative Commons Attribution-Share Alike 3.0 Norway License.
Based on a work at svitjod.rymdweb.com.

References

[1] Memory limitation on 32 bit systems, http://www.theserverside.com/discussions/thread.tss?thread_id=19442
[2] Hibernate batch processing http://www.hibernate.org/hib_docs/v3/reference/en/html/batch.html
[3] Oracle rownum http://www.oracle.com/technology/oramag/oracle/06-sep/o56asktom.html

Sourceforge download problems

2006-10-22T13:00:00.000+02:00

Just wanted to download the updated version of TortoiseSVN which I like a lot better than the command-.line svn tool when importing projects from different folders. The only problem was that for some reason, the only available download site was a server in kent, that for some reason was down.
I inspected the download url, that is on the form http://prdownloads.sourceforge.net/tortoisesvn/TortoiseSVN-1.4.0.7501-x64-svn-1.4.0.msi?use_mirror=kent
Well i assumed that the request parameter use_mirror = could probably be replaced by something else. I tried google to see if I could get at list of all these alternate servers, without luck. I saw that the download-server had a dns-name on the form http://kent.dl.sourceforge.net/, then i turned to netcraft.com and entered *.dl.sourceforge.net and it immediately gave me back a list of servers matching the dns-query. Voila, i replaced use_mirror=kent with use_mirror=belnet and there my binary dumped down on disk :-)

Saturday night in

2006-10-21T13:42:00.000+02:00

It's Saturday night and I think I will spend most of it at home together with my computer.
blogger.com has gone live with a new version of blogging software, so I have just downloaded the new Firefox 2.0 RC3 bowser and will give both the face lifted browser and the new blogger a try.

The first thing I noticed was that my del.icio.us plugin doesn't work anymore. I have really been used to adding sites and bookmarks to del.icio.us using the extension, especially since I am using multiple computers, and being able to centralize the management of my bookmarks is really a good thing. Fortunately, I found a replacement, del.icio.us post that works fine with 2.0.
The 2.0 browser now comes with a built-in spell-checker, which is nice. Text that you enter in form field can be validated on the client side, and does not rely on external services. I have used it already.

Other things that I have been using and spending my time on lately is:
* IDEA IntelliJ 6.0 that was released last week
* Google Web Toolkit, GWT
* Directory Opus a commercial alternative to Total Commander

And if you haven't moved to the new beta-version on blogger.com, do it, its better than before, I guess that the Google-guys has been busy lately.

JavaZone closeup and end-comments

2006-09-14T23:58:00.000+02:00

The last day of JavaZone has been just as exiting as the first one. I tried to add a nice mix of high level process-oriented stuff with hard-core technical sessions to keep me from falling asleep today. Didn't have much sleep tonight, huh.
Mary Poppendieck had a great session on Lean SW-development. I am a great fan of the Poppendiecks and especially their first book on Lean Software Development:An Agile Toolkit.
Conventiently she announced that her new book is shipping as we speak. I am also in the process of entering my credit card number on amazon to get a copy.
Thereafter I dived into a session on DTrace, a feature of Solaris systems that enables full tracing of the entire software stack from a thread in the java-vm and down to system calls in the kernel. This approach to tracing will make searching for bottlenecks in the system much easier. The speach was accompanied with some nice live demos.
Simon Ritter another speaker that made JavaZone 2006 worth visiting. He has been experimenting with SPOT and robots, talking about RTSJ and esoteric VM-options on the Solaris VM. Did you know that there are more than 400 different -XXoptions available in the Solaris VM as from java 6? Well , now you do. A I am currently back to work (in the middle of the night) because we are going in to production with a system for a client. Suddenly I realized that it is actually a year until next time I can spend two great days@JavaZone.

Have something on your mind? post-it!

What is new at JavaZone 2006

2006-09-14T10:35:00.000+02:00

I am currently attending a session at JavaZone 2006 called Are you a plumber? How to avoid plumbing for business-enriched remote clients. It is a co-talk by Ole Andre Ranvik which is a co-worker of mine and Bjlørn Nordmoen from Western Gecko. The are basically speaking about a project they have been doing together for the last year and of they solved some of the issues they encountered.

The thing about the program this year as far as I can tell is that we do not have to much stunning news, new technology, new releases and all new specs. Much more of the material this year is based upon how we actually use our technology in day-to day work. Agile tracks and and technology driven tracks share a common property with respect to this, they cover what developers have been working with, not what someone think they should work with. With some exceptions, EJB 3.0 being one of them.

Backwards compatibility, why?

2006-09-13T18:08:00.000+02:00

This is quite an interesting topic. I am currently attending a session on EJB 3.0 and JPA given by Patrick Linskey which has been participating in the spec work. An issue related to backwards compatibility related to some things introduced in JPA (global named queries). And I asked if Patrick sees it as much of a problem to break backwards compatibility. And he said, "we did it with EJB 3.0 and we do not want to do it again". Well I have never ever heard of anyone actually complain about this compatibility break. In my opinion in many cases the effort spent not to break backwards compatibility is just a waste. We would be much better of getting new specs and implementations that improves faster as opposed to versions that does not break backwards compatibility.

Meeting the speakers and authors

2006-09-13T15:54:00.000+02:00

I am currently staffing the javaBin stand at JavaZone 2006 so I cannot attend any sessions as of now. However staffing the stand gives me a perfect oppurtunuty to talk to attendees, partners and speakers. Seems like if everybody is having a good time. Some of the sessions has been so popular that we had to schedule them twice, like Patrick Linskeys EJB3 Java Persistence API: The Good, The Bad, and The Ugly.

So far the conference has been running smoothly. I am soon of the the next one when releaved off duty. Think I will head along to a REST vs. Soap smackdown to see what all the fuzz is about :-).

Second Session at JavaZone at 2006

2006-09-13T10:21:00.000+02:00

The first session I attended to was the Java Effective Reloaded with Joshua Block. It was packed, so I had to stay in the back. Anyways, I got a picture of what it was all about.

I have allays been fond of GUI development, both web and standalone, although I have never done to much work on GUI's during projects, mostly on my spare time.

I am currently attending the First Aid for Swing by the founder of JGoodies, Karsten Lentzcsh. Karsten is guiding us through the process of doing a project where you have inexperienced developers, no time for usability design and the customer want pay for anything. Cool what a pragmatic attitude, this is how it works in most cases :-)

There are a lot of dos and dont's in developing swing applications. One if to use consistent colors and allays to use native fonts, simply because the render better. Karsten often does a test on applications with excessive use of borders by imagining that the GUI is a physical three dimensional artifact and by drawing the fingernails over this artifact, does it make any sound? "Rattcchhhhhhhhh" not good, to many borders, good tip? :-)

Are your GUI application symmetric? It should be, use gradients and weights to achieve symmetry across the interface, subtle but effective!. Use to three, maybe five fonts at a maximum as a guideline. I guess a lot of this concepts applies to GUI's in general, not only to swing applications.

Depot Lineup at JavaZone

2006-09-13T07:53:00.000+02:00

At the javaBin boardroom we are currently in the process of getting the last small details in place for the conference. Yesterday we had the speakers dinner with almost all the speakers for this years venue. I finally got to meet Mary Poppendieck whom I had been emailing for the last six months regarding her speach on the conference, at the Table was also Tom Poppendieck and Gregor Hophe. We all had a nice chat and vere looking forward to a couple of hectic and nice days at javaZone in Oslo.

I will try to get out up to date new published along the way.

Heading to JavaZone 2006

2006-09-11T23:26:00.000+02:00

Currently planning the various sessions to attend at JavaZone 2006 in Oslo 13-14 September. The lineup and two-day agenda is impressive I must say.
I have myself been part of the program committee have been working for almost a year now to get the program in place. Hope all of you developers out there look forward to joining up at JavaZone 2006 as much as I do. A mighty combination of Well known names like Rod Johnson and Mary Poppendieck and local heroes like Eirik Torske and Ole Andre Ranvik are some of the speakers this year.

Please also spend some time looking through the agenda and don't join the crowd. Maybe you are in for a surprise when attending the not so well-known speakers?

See you there!

TSSJs Barcelona closeup

2006-06-24T04:36:00.000+02:00

Well, all good things come to an end, including this conference. I will use the spare time tomorrow to reflect upon what I liked and what could be improved about the sessions in TSSJS in Barcelona, which all in all was a pleasant experience, hola!

Model generation and visualization

2006-06-23T14:49:00.000+02:00

With MDA being totally out of the picture, Gregor Hophe (energetic and excellent speaker) and Erik Doemenburg is getting down to the point of what is really important point when it comes to modeling, not do create pictures that you can generate code from, but to generate a graphical presentation from an existing system and an existing codebase.

I write a blog entry called "Feeding the architect" that dealt with this topic. In that case using maven, dotuml and GrapViz to generate diagrams (static and dynamic) from an existing codebase. The main problem is to combine data from static code analysis and from a running system, exactly what Gregor and Erik is going into presenting.
Three different models are suggested:

DAGS
Metrcics
Petri nets
Trees

Try to extract data from static and dynamic analysis in a simple form, preferably in a text file and then extract data from the text files and map into whatever model you seem fit for the given analysis. A good example of this may be to get a representation of message flow through a workflow system.

The first example being using a 95 line XSLT to generate a dot file put into graphviz to generate a dependency diagram from a spring config file. Well, since I am lazy, I would rather just use BeanDoc. Remember that lazy is good here, do more with less kind of thing :-) But I think the audience get the point here.

The next demo is much more interesting. By doing some modification of the jar packaging and adding some metainformation, the build step create metainformation in text files that makes us able to create a graphical presentation of dependencies between jar - files, components and to create clickable images drilling down to classes and source code. This may be a very effective means to get an overview of the system and to see how well it really aligns with the architecture that you anticipated. This is good stuff, and it should be added to the tool portfolio of every architect and developer.

Gregor now gives a presentation of how he has generated an svg from a running messaging system by instrumenting his own messaging library.

Process mining is interesting in the sense that we do not only present a model that can be generated from a simple dynamic and static analysis, but a heuristic approach to get a picture of what kind of processes that executes and in what order. Gregor presents a really nice example with using ProM to visualize processes using petrinets. This is a tools that accepts XML files as an example that may be nicely represented in a swing gui, really cool.

References:

JCP Panel discussion

2006-06-23T14:05:00.000+02:00

OnnoKluyt is leading the panel with the following guys:

Jon BostrÃ¸m
Tom Baeyens
Mike Keith
Cameron Purdy
Gavin King

With such a lineup where most (all?) of the representatives on the panel is also either a JSR member or spec lead. Makes it little bit difficult to get some interesting and heated arguments but anyway. Having the experts in the field or at least someone profiles within the community lead the work is to start with a good thing. This is not sufficient to get the right technology out to the people, bit it might be a start.

The time to market problem, especially with the Java Platform releases is beeing discussed piched up with a comment from Gavin.

Companies like Microsoft probably implements similar processes internallyalthoughh they are most definitely not public!
Kirk Pepperdine claims that you pretty much need a lawyer to get through the initial process of signing up and getting up to speed. This is something quite different from entering an Open Source community where you are judged on what you have (or are) achieving other that being able to signoff legal documents.

Cameronencouragess the members of the audience to participate andcontributee instead of, as Gavin King says, just complain instead of just complaining about stuff thatdoesn'tt work. Some (around 4) of the JSR's currently running are led by individual members, with Groovy (scripting for java) and Concurrency as good examples. Surprisingly enough more that 50% of the members of the JCP are actually individual members.

Grid based banking

2006-06-23T12:07:00.000+02:00

John Davies from C24 has been of of the profiles at this convention. I missed the keynote so I wanted to drop in at the banking and grid session to see what the fuzz was all about.
A lot of cases where we deal with applications we don't really need databases as we know them John claims. More than viewing the database as the central repository of all information you can view the grid as the repository of the information.

Typical cost is 1$ per CPU per hour. Interestingly when several of the banks that John is working for has as many as 10K CPUS. Utilizing these effectively is important with respect to ROI.

C24 want to avoid using XML though the wire when transferring information is not to use XML because it is not efficient enough, but use java databinding and efficient object serialization instead. They do this through something they call Integration Objects. The fact that John discourages the use of XML in the grid computing space is that is not efficient enough. This is also an important moment in the SOA space discussions.

How do you do this when you don't have any data stored in a central repository only in a grid where you don't have a query language.
John is referring to a case study where everything is stored and manipulated in a grid space in two different grid spaces, one that receives the feeds and organize them and one replicated feed that is optimized and used solely for querying.

The technology is based upon JavaSpaces and has some quite interesting attributes. One of them being that querying is done in javaspaces based on what kind of interface(s) that are implemented. Then some matching logic. The thing here is that the searching seems to be quite procedurally oriented, like searching for some course grained items and then doing a new query on that subset. I would prefer this to be more declaratively oriented, don't tell the system how to do it tell it what to get!. This is what has been working well with RDBMS'es for the last 15 (maybe even more) years. Maybe it just because I didn't understand the example?

Richard Ãberg in the audience asks what are the maingoctchass. One of the gotchas with JavaSpaces and the way that C24 applies the technology is classloading, understanding the nature of the distribution and multithreading issues in grid environments.

What kind of knowledge will you need to implement this technology effectively. Knowledge of grid computing and products technologies that may have been around for a while without having to much momentum is definitely an issue. This might be a major challenge in implementing grid computing effectively.

Productive coding

2006-06-23T10:27:00.000+02:00

Dr. Heinz Kabutz, author of the Java Specialist Newsletter is talking to us about coding productivity and about using the IDE ad your tools effectively. This is also something I have been thinking a lot about since I am often in a position where I do pair programming and coaching. Then you tend to notice how other people work, if they use keyboard shortcuts, how they navigate and so on.

being able to use the IDE effectively, especially using keyboard shortcuts is really a differentiating factor in productivity. How fast you type, and if you use touch or not may also be a factor, because we still think faster than we type.

Important to actually make sure that you know what code you actually need, which code gets executed in production. Important, since most money goes into maintenance and you don't want to maintain unused code.

Heinz also talks about how he has mate this doclet that finds this and that like unused methods fields and so on. I use IDEA to locate stuff like this intead of running a doclet.
More discussions on exception handling, increasing tghe use of private/protected scoping and so on. Many developers should read up about this, resonable scoping will make your class easier to use and less confusing and actually reduce the chance of anything being duplicated if the api uses a reasonable naming strategy and relevant javadoc where needed. Still Heninz has not told me anything that I am not allready aware of, except for for a couple of funny comments in the J2SE core awt code and how the actual enforcements access to final fields as changed back and forth from 1.1 to 1.6.

Development environment of the future

2006-06-23T08:51:00.000+02:00

This is a really interesting topic, the topic of the discussion is development environments. IDEs has been one of the most important factors related to developer productivity and is often the starting points for religious debates amongst developers.

Panelists

Wayne Beaton (eclipse community)
Erik Dornenburg (Thoughtworks)
Bruce Tate
Cedric Beust (Google)

Cameron Purdy is currently mediating the Panel and he is collecting questions from the audience in writing. I think the fact that he collects questions in writing prevents some effective dialog. People want to talk, not to write so that someone else can talk :-)
Johannes Brodwall (sitting next to me) took up a question related to refactoring and if we are going to see continuing support for refactoring. The panel is not in consensus, Erik claims that we will not see to many new refactorings being developed whereas Bruce points out the fact that the diversity of languages that we are actually using, especially in the AJAX space will require IDE support and continuous effort refactoring support.

Large team support and tools support though collaboration is a very important aspect. Improved integration with communication tools and XP planning tools might me something that can keep us focused in what we do. This a major problem on projects I have been working on. Unfocused developers that are align with fine grained priorities is something that puzzles me. Continuous effort in coaching development teams are something that might be facilitated through the use of an IDE integrated with planning tools (I am not sure if I want a project manager in my IDE .....).

AOP support and support for auxilliary functionality that is not represented as java classfiles like scripting languages, navigation, debugging and profiling is missing. Getting first class support for multiple languages been integrated more easily would be a neat feature.

Cedric points out how neat it is to use a debugger now and then. He urges people that are not using debuggers in java to start doing that instead of using system out printlines. Not much has happened in the debugging space for the last 10 years.

As it seems now, the Java community with IDEA and eclipse is now by far ahead of MS Devstudio while this was quite different only a few years back. This seems to be swinging back and forth, MS DevStudio now getting refactoring support and so on. Myself I am a strong believer of competition and I really think that this will only improve the quality of the tools we are working with.

One reason why Java has been successful is because backwards compatibility has been supported to great length. This means that leveraging existing systems, migrating from one JDK is easier that it might have been. There are a lot of language issues that need to be fixed if we should be able to support stuff like continuations. Thus JVM support for running new languages might be a nice feature.

For those of you that didnt know Cameron is using (as myself) IDEA.

Erik Doernenburg from Thoughtworks answers a question related to BPEL support and workflow support in IDE's by saying that SQL was supposed to be the language that would enable business analysts to write code. Now not even developers use SQL anymore (a pity in my optinion, because the declarative approch in SQL adresses problems, especially in batch applications, that cannot be solved in middleware), so to beleive that enhanced support for business modelling language approaches is a dead end, at least according to Erik.

Closing up the panel, the ultimate question is thrown out, in which direction are we heading:

java is still going to stand strong
tools that enable us to work with multiple languages in an integrated manner
quick feedback related to catching error-situations to reduce roundtrip time
higher abstractions like AOP and annotation, maybe also metaporgramming (like GWT) will be even more widely supported
domain specific languages
lessen the need for code that has to be written, which means less code, even less generated code because generated code will also have to be managed

Quest for effective Web frameworks

2006-06-22T16:14:00.000+02:00

We had a discussion during lunch here related to web frameworks, how many there are, how they compare and how web development in the Java space is actually quite difficult. Creating a nice looking web interface with good interaction capabilities with dynamic content is difficult and costly, and I am not particularly happy with what I have seen so far. I have done some work using spring MVC and JSP 2.0 and JSTL lately, and it works fairly well, but for the presentation part there is still a lot missing. Spring 2.0 will have a new set of tags that will make life easier, but anyway. Wicket seems appealing.

The underlying component model and the fact that you may are able to work with the page itself as a first class object, and being able to have a simple databinding approach.

JPA and caveats

2006-06-22T15:07:00.000+02:00

Patrick Lindksey is giving his talk "The good bad and the ugly" related to EJB 3.0 persistence. Interesting to get a more detailed picture of the JPA specification. I really find it more interesting today compared to last year. One of the reasons is that we have more or less nothing else but runtime dependencies and some wrapper code around a generic repository implementation in my current project. This may make it feasible to actually swich persistence providers as long as we have several good open source ones available already. The only thing that will require some work is getting rid of the implementations we have related to searching that uses the Hibernate Criteria API. Maybe JQL will do the job? I don't know I have never been using this part of the API.

One thing Patrick went into depths explaining related to POJO based persistence is really interesting. It really hides some subtleties related to designing API's and access to fields and properties. This applies to most mapping frameworks including Hibernate and is relevant when it comes to discussing domain models and where to put the actual business logic.

Consider the following code fragment representing the setter for the age property in the Customer class:

public void setAge(int age) {
    if (age < 18) {
       throw new JuvenileException("We dont accept young customers");
    }
    this.age = age;
}

This might seem like a business rule to implement in the Customer class, why? Because principles of cohesion should be the rule and the business logic should for reasons of clarity be placed in the Customer class implementation. But what if this is a new Rule that did not previously exist? You have provided this rule such that the API will prevent you from entering Illegal data. But that also means that the persistence framework that will use the setter for populating objects from a search will get the same ,exception as a side effect on data that was valid last year. Hence using setters as a mechanism for managing state in persistent entities is dead wrong. It is a good way to enforce integrity for new (transient entities) but not for dealing with persistence lifecycle. After all APIS and business rules tend to evolve, but historic data tends to stay the same!
The Irony is that you will probably be pretty happy with the first release of the system but after the system is going into maintenance errors will start coming in after the first upgrade. Consultants will probably not even notice, they are on the way implementing setX and setY with heaps of business logic in new domain objects for new customers and by habit use property (instead of field) based access in entities :-)

Geronimo lineup

2006-06-22T13:41:00.000+02:00

Who is interested in Geronimo? Obviously a lot of people, I don't know too many people actually using it apart from using it as maven dependencies for the J2EE API's.

I attended Bruce Snyders presentation on ServiceMix and found that one interesting because I have been reading up on Mule a little bit so I attended to get a comparison. I am currently also working on a project where we are on a good way to actually implement small parts of an ESB so the topic is really interesting. With that said, I am not necessarily an advocate of adding a dependency to a third party library just because it contains an out of the box implementation of something I can do with 10 lines of javacode.
Managing 100 different and possible incompatibilities might be just much of a nightmare as writing the 1000 lines of code that those frameworks help me to replace.

The panel lineup of the Geronimo Panel Discussion are:

Bruce Snyder
Matt Hogstrom
James Strachan
Aaron Mulder
David Jencks

panel is moderated by Ted Neward, a great speaker in my opinion. First thing discussed is the role of OSGI and XBean and how Geronimo is may be used to complement each other.
although OSGI is not supported in Geronimo, one of the reasons is complexity the panel claims.

Why should we select a commercial vendor instead of going for an Open Source alternative, and actually end up paying for a J2EE server, here are some of the points from the panel:

Commercial vendors often provides entire suites of products that may complement the J2EE server
Support and product roadmaps
Performance and scalability
Wheter the product has ben subject to rigorous testing (is this a way of saying that the testsuite accompanying Geronimo could be improved?)
Service pack/fixpack approach, most commercial vendors provide support for incremental updates

There are a couple fo guys from IBM up on the panel, and they are currently debating how developing and embracing Geronimo may affect Websphere and vice versa. An IBM representative says that they will try to get to a point where changes are merged and updated bidirecionally. Nothing is said related to if they want to estabilsh a common codebase for Geronimo and Webshere.

Spring 2.0 Highlights

2006-06-22T10:48:00.000+02:00

Rod starts the presentation with saying that the final 2.0 release will probably be scheduled at the 5'th of July this year. This is good because enable a lot of projects, during the summer to conclude the upgrade to Spring 2.0. A lot of development environments enforces summer freeze periods which makes the summer an attractive period for experimenting with such things. Currently I am busy with getting thins production ready for a big client so I will have to do this on my spare time when I am not hanging out with the guys at PhatMC.com

There are a lot of nice new features that are of particular interest to me as we are using spring at the project I am currently working on. Especially the support for Asynch Messaging without MDBS's with is really good. By trying to replace the message activation infrastructure we are currently using by POJO based messaging we will probably be able to

simplify packaging and deployment, no longer any need to package EJB's (we only use MDBS)
Move some of the testing in from the container based integration environments into the continues testing environment and JUnit tests

Also simplified configuration, and XML-schema support will probably enable us to reduce the amount of spring config we are using. The support for dynamic languages will probably also enable us to experiment with new ways of handling spring configuration and the proliferation of configuration files. And yeah, its fully backwards compatible.

One of the reasons why Spring 2.0 has been delayed has been the support for JPA, which seems to gain momentum and supported by Toplink, Hibernate and Kodo.

Understanding open source

2006-06-22T08:50:00.000+02:00

A guy from Sun responsible for Suns managing http://www.sunsource.net/ and participating in developing Suns Open Source strategy is here to explain to us different strategies and approaches for dealing with open source. He starts explaining to us the shift from to the consumer age to the participation age. The key is to understand that certain events changes the way we think about stuff. I think the point he is getting to is the way that Open Source has changed the way we think about software. Various patent models and how large corporations think regarding licensing and patents is also interesting.

Simon Phipps also believes that we will stop paying for using software in advance, and shift towards paying when it adds value. It would be interesting to see how such policies might be enforced. As a consultant I play a role in providing business value through adapting and creating software, not through the software itself, so this is an interesting thing.

Open source communities is based on a casual allignment of interests Phipps concludes.

Would be interesting to discuss further how it is possible to actually charge for software when it adds value and beyond simple scenarios like the number of customers or the number of employees in the company that bought the software. Neither of these marketing models are sufficient to define market value.

Day II TSSJS

2006-06-22T08:44:00.000+02:00

Looking through the agenda briefly, regarding Web Application frameworks, there are only two presentations on webframeworks this year, Wicket and RIFE. Not to many people had even heard of those last year. The proliferation of frameworks in the web space really says something about the maturity or lack of such in this space. Rife has continuations which enables a really interesting programming model the reaches far beyond being able to support using the back button in the browser.

Enabling productivity, effective development environments and a component based model like JSF, maybe where everything is expressed in java will be the way to go I beleive. I have been doing a bit of JSP programming and JSTL 2.0 lately. In combination with clever use of CSS like Maxdesign makes webdevelopment much more simple, event not as simple as I would like to.

Gregor on SOA

2006-06-21T14:35:00.000+02:00

I attended Gregor Hophes session on SOA in Las Vegas I little over a year ago and liked it. I wanted to drop into his session in Barcelona to see if the presentation has been updated.

Gregor now works for Google, let us see if that has changed his attitude and presentation form.

Gregor gives the following definition of a service:

Self contained
Independent of consumer context (makes few assumptions on use)
universally accessible

Seems as if Gregor is pretty much doing the same presentation as he did at TSSJS in Vegas 2005, he is even cracking the same jokes. If he had put up the warning sign, I could be listening to Ross Mason talking about using Mule as an ESB instead.

Achitectural intent. Gregor talks about architectural styles and what architectural artifacts and specifics that defines SOA as a concept and architectural style. To define SOA he claims that it is just as important to state what SOA is not in order to give a proper definition of SOA.

Gregor uses Starbucks as an example of his discussions in the SOA space. At Starbucks (at least in the U.S) the staff performs a highly specialized task, one that collects the order, one that makes the coffee and one that processes the payment. This is throughput optimization, but requires a lot of staff, at least three people in this case. If thoughput it not the case, you might me just as well of having one effective employee that does a good job, it is simpler, more general purpose and less expensive. The point is, maybe you can stick with simpler. Is loose coupling and high co hesion allays for the best, even if you can implement the everything in one small class instead of three? I think that the problem with metaphors like the one with Starbucks is that I find it rare that people actually ask questions about what is the reason for having this design, and why not make it simpler. Developers tend to embrace new technology and paradigms regardless of their appliance. This may be good or bad. Bad in the sense that we may choose the wrong technology for a specific task. Good in the sense that we build knowledge on what works and what doesn't.

For written definitions of what SOA is that you can memorize before you attend JavaZone2006 and want to ask questions, Gregor recommends Orchestrationpatterns.

Q & A
The difference between REST and SOAP is brifly discussed. Gregor points out that the important part of SOA is really the architectural style not implementation specific issues.

Where is the coffe and WIFI

2006-06-21T12:02:00.000+02:00

Hey, we have to pay for WIFI here at Fira Palace. I stay at Vincci Arena nearby where WIFI is free. I miss free WIFI, well, but to feed the community with information I am more than happy to pay 17 Euros for 24 hour access.

We didnt get coffe when we arrived, I would like to get coffe when I want and as musch as I want.

Anyway the conference is actually quite intimate. 250 Participants, 275 with speakers.
My current session crasshed with Gavin King's session about Seam, so I went to listen to Jonas Boner instead.

Clustering in Spring

2006-06-21T11:35:00.000+02:00

Jonas Boner is talking about clustering in spring, a topic of interest since I am currently developing applications that are deployed into a clustered envrionment, and I want the fact thta applications are deployed into a clustered environment to be transparent. It should be possible to deploy applications that are clusterable also in to non clustered environments such as an embedded Jetty container because that facilitates effective unit testing.

Jonas comes up with a problem definition, scaling and that we need to share data throughout multiple JVM's. I think that in most cases, this should not be neccessary. In stateless applications you might just as well keep multiple copies of reference data (which constitutes what Martin Fowler refers to as the knowledege tier), which is most likely read-only anyway.

Jonas starts go into an example, a Store domain class that is subject to crud operations, and how to replicate state changes across a cluster. Hmmmm, this sounds more like how to replicate state across a cluster, well before you do that, ask the question do you need to? If you want to store the data, put it in the database, you will have to put it there eventually anyway.
Now he continues to discuss aspects related to track changes, well, I have been using Hibernate for that some time, so this is for the most just problems from an OR mapping perspective.

Now some stuff that actually relates to the spring clustering stuff and Teracotta of course, which is the product you will need to support directly what Jonas is referring to. Hani has some viewpoints on this.

In my perception try to design your application so that you have uniform access to data, either read mostly/only reference data and operational data. By designing a data layer effectively I see no reason whatsoever to associate spring with clustering throuch teracotta. If clustering relates to sharing data in a cluster, use isolate this in the data layer. You will most probably be using an OR-mapping tool anyway. Seems as Jonas is taking areas related to data synchronization from the data layer and moving it higher up in the application tiers where tehy do not belong. Synchronizing data using the database is probably the most effective solution anyway.

Continuing the TSSJS venue

2006-06-21T11:15:00.000+02:00

Ive just been relaxing in the back on a session with Geert Bevin, the founder of the RIFE framework and he has been explaining to us about continuations and how it is beeing used in RIFE, a webframework that uses continuations heavily.
I was missing some informations related to how continuations may be used decoupled from RIFE by apart from that it is a very powerfult concept. I am not really sure how it will affect the programming model that I am currently using on my projects since all the classes that may contain state in continuations must be cloneable, but I will have to dig into that.

Preparing for TSSJS-Europe

2006-06-18T17:07:00.000+02:00

Just been browsning through the agenda and preparing myself for the Barcelona trip. Been planning the agenda online as well using the online agenda planner available at http://javasymposium-europe.techtarget.com/ . The slots available in the online tool are not the same as in the online agenda. It does not present the slots side by side when they are overlapping either. Guess I will stick to a printed version and just put circles (with a pen) around the slots I want to join.

The agenda is not overly impressive. I counted 43 sessions plus some keynotes and some bofs. But given the lineup, I will not have any problems attending something interesting.

Compared to what we are planning for JavaZone 2006 the venue in Barcelona is small compared to the +80 session programme we are currently concluding, with an even more impressive lineup.

Seaview is Back in business

2006-06-02T01:34:00.000+02:00

After the hacker havok at svitjod.rymdweb.com, we never got our b2evolution installation up again. A pity, but thats life. I might as well find somewhere else. I am soon heading for TSSJS In Barcelona as welll, and I want to get ready for some hardcore blogging from the conference, so get ready, at least I am (will?)