Last time, I talked about the problems I had working on an important project that had not only to integrate into our existing systems, but to tie them together in a way they weren’t originally designed. These existing systems had seemed to work alright on a day-to-day basis and due to manpower and finances we made do fixing errors manually and repairing bugs on a ‘when really needed’ basis. There were plans for a second phase to deal with the problems (mainly caused by data integrity problems from an external source) and to add more functionality, but these were put on hold; new applications introduced by our parent company offered better data integrity and greater integration into their systems but with the downside being that we had other things to think about than the programs we had just written.

The new project was going to build on top of these systems, linking together pieces of the business that had previously required the expertise of a shipping manager, a logistics department 200 miles away, and an accounts department that had better things to do than manually maintain a 20 year old paper-based system. Unfortunately, we soon began to find that the previously small errors that we had been fixing manually were causing all sorts of problems to our attempts to tie the whole thing together. We were nearly two thirds of the way into the schedule and we appeared to have two systems that couldn’t be connected. What could we do?

Luckily, however, we managed to sort out the data integrity problems from our external source. A pre-processing routine was drafted and implemented to clean our data up and shift responsibility for the data integrity from the IT department back to the shipping company. This was refined over a few integrations until we reached the point where we had almost complete control again. The data we were relying on was almost 100% accurate and we had procedures in place to limit the number of potential errors, which also had the benefit of allowing us to pass this control to our logistics department, thus freeing up IT staff for more important work. We eventually got the links we required to tie the old to the new, and we’ve had a highly successful first few months. Only time will tell how much benefit it will have to the business, but from early reports, the effects are starting to be seen.

Now that we’re passing control on to others, I am in a position where I would like to look back on what happened, and analyse what we did right, where we went wrong, and what we as a department (and myself personally) could have done differently. I am a relative newcomer to group project management; the departments I have worked in have all been fairly similar, sharing a lot of similarities in terms of the size, structure and tasks performed. Most project management is handled by the IT manager (for the bigger, more important tasks) and after that, or for smaller individual projects, responsibility is placed in the hands of the developer.

There are no demarcations between designer, analyst, programmer and unit tester. You need to be strong in all areas in order to take your projects from the initial user request to final handover and the solution may require a broad range of skills that can’t be tied down to a single job description covering more than the three words just do it’. If I were to look for a new position, how would I categorise my skill set, other than a jack of all trades? (I wouldn’t particularly want to be labelled a master of none!)

I began this analysis by examining the Java projects I had worked on and looking at some of the things I had learnt from them. By the way, I know there is a recurrent theme in my articles. I firmly believe I learnt a lot more about programming in general, and about RPG in particular, through working on our J2EE-based intra/extranet site. Within the Java/OO environment, there appears to be a lot more emphasis placed on project management than there is in the RPG world. Maybe we as RPG developers feel we don’t need to learn anything from these newcomers to the software table. I mean, we’ve programmed in this language for over thirty years -- we were punching cards while OO was reserved for bearded geeks developing ways to find out if the coffee machine in the next room was empty. Our users had few preconceptions of how the finished product should look or behave (the screen was green and that was that).

On the other hand, maybe I’ve just never been formally introduced to proper project management. Within the OO development world, there appear to be two slightly different approaches to software development methodologies. The first I was introduced to was the Rational Unified Process (RUP). This is an iterative procedure, whereby four stages (inception, elaboration, construction and transition) are passed through; each stage consisting of one or more iterations of between two and six weeks, and each iteration or pass being used to provide more and more functionality.

At the end of each pass, a fully tested and working piece of software was produced. The main emphasis within RUP is that it is a methodology that was more business than technology-oriented. It is a process by which IT staffs are able to apply a fixed process to a project and on scientific grounds be able to present the risks and analysis of the project in a way that even company directors are able to understand.

It must be said that the modelling and design tools used with RUP are not the most important items in the process, but the main priorities are in risk assessment and iteration to produce a working piece of software. When I look back on how our project progressed, it was clear that we had used an iterative procedure, though this was not formally laid out. There were a few of the artefacts that one would expect to find from an iterative process (such as documentation and design plans) but there didn’t appear to be any real risk analysis -- we just started working towards a goal, got there and then decided what to do based on the results we had achieved.

The second approach to project management I found was one called agile model-driven development. It is really an umbrella term to describe where ‘project management and development methods…encourage continual re-alignment of development goals with the needs and expectations of the customer’, according to John Amans in the July 2005 issue of iSeries News UK magazine). It seems to me that all of the IT departments I have worked with have, in one way or another, been involved in agile development. We have a change control system in place, but we aren’t bound by any formalised procedure. We have regular input from our users, regular discussions on how the project is evolving and, most importantly, we are regularly putting tested software live and can add more functionality and patches without having to go through an entire reworking procedure.

This methodology allows the shape and focus of the project to change from one day to the next. It not only encourages collaboration between the developers and the customer but almost requires it, which in turn can only be a good thing. Combined with extreme programming (XP), it allows development groups to produce some of the best tested, most reliable software available. Again, to mention John Amans article, he points out that when you compare Windows NT to Windows XP, the BSoD (blue screen of death) used to appear regularly in NT, but nowhere near as much in XP. Yet XP contains around 40 million lines of code (compared with NT’s 20 million lines of much more unreliable code).

It appears that Microsoft has adopted some form of agile methodology in order to churn out that massive amount of code (around 10,600 man-years of development). One of the strengths of agile or extreme programming lies in the ability to first define a test for the software you want to write, and then (and only then) do you write the code itself. There is a package called JUnit that enables Java developers to work like this and this, combined with paired programming (where two developers work side-by-side, monitoring each other) ensures that the code that comes out is much more robust. And added to that, you build up a test suite for your software as you go along.

Since a lot of the time on my project had been spent fixing bugs that had been present in the previous systems, I have come to the conclusion that had I been aware of such a method of working a few years ago I may have saved myself a lot of time over the last few months. When we were developing our website, I worked a lot with a fellow developer and together we seemed to come up with a lot more ideas than when we worked as individuals. It appears, then, that agile programming has a lot of relevance to how I work at the moment, and I will be spending a lot more time investigating how I could put more into practise.

There are a lot of sites on the internet dealing with both agile methods and extreme programming. Have a look at http://www.martinfowler.com/articles/agileStory.htmlfor some information on how it started, and http://www.agilemodeling.com for more information on what it’s about.

Looking further back though, one of the oldest forms of formalised project management appears to be one referred to now as the ‘waterfall’ process. This is probably one that a lot of IT departments (and possibly finance departments) are most familiar with. It is a predictive procedure that was first laid out in the 1970s by one W. W. Royce and basically stated that there was a flow to project management. It was initiated by a formalised gathering of all requirements necessary within the project. Once (and only when) this was complete, could you move onto the next stage, which was the analysis of the requirements and a design of the blueprints from which the coders would use in the next stage.

Formal specifications would be drawn up and these were to be the bibles that the coders were to follow when they started their stage of the process. The implementation stage covered how the final code moved from test to live, and the final stage of the waterfall process was the maintenance of this code, including any bug fixing, adding additional functionality and reworking to achieve targets set after the process had started. Bear in mind that these stages were meant to be set in stone -- once one milestone had been passed, you couldn’t really return to alter it.

From looking at the number of major government projects reported as failing over the last few years, it’s clear that although the waterfall procedure has its merits (there is a great article on the Wikipedia site discussing them), it also has severe drawbacks. The advantages that the RUP and other iterative processes have is that there is no ‘Big Design Up Front’ (another project managers acronym to go with RUP is BDUF – I wonder if there is a project maintained for these somewhere?) Looking back to my project, we didn’t really subscribe to the Big Design Up Front (however small our project was). Our only big design was to get it working within a specific time-frame, which we managed to do, but we didn’t have any of the artefacts one would associate with a waterfall-based project.

To be fair, Mr Royce did come to the conclusion that there were a lot of flaws with the waterfall process, and his papers also included work on iterative development. But these were written a long time ago, and given how technology has moved on even in the last few years (you need only look at the developments in RPG programming in the last few releases of OS/400 or i5/OS), I feel it is time to look at how we approach project management, as follows. Be customer focused. Embrace change. Develop iteratively.

Work with motivated people. Communicate regularly. Use software as a measure of progress. Always look for elegance with simplicity and efficiency.

These are some of the values underlying various agile methods. I’m going to see if I can try and implement them in my working life. What about you?