The False Seductiveness of the Priority List as a Planning Tool

Priority lists: can't escape them, can't trust them. As developers, we all make our lists: a list of issues to be triaged, a list of bugs to be fixed, a list of features to be implemented, but which to do first?

It seems so easy to just go down the list and assign a priority to each item, then to start work on one of those you have assigned the highest priority to.  BUT:

• We often confuse importance with urgency - an urgency-based priority list will often conflict with an importance-based priority.  And both can be valid.

 

• Whose priority? Yours, your client's, your co-worker's, your boss's?  A client or colleague bashing down your door may in the end be just one voice among many demanding your attention.

• Priority/Urgency/Importance may not be visible at first glance.  A live production system bug may need to be triaged urgently, but may reveal an issue of low urgency.

• Issues are often related to one another: often a single fix can hit several problems at once, each of which is of low importance but which taken together are a good use of effort.

• Often a low-priority job may be tacked on to a high-priority job in a related area and so be addressed at a very low effort cost.

• Priorities change over time: a task that may be useful to carry out today may have no value tomorrow: for example, a task needed by the month-end will increase in priority as the month progresses.

This is bad enough when you are in control.  When others (e.g. clients or managers) are in charge of the prioritisation process, things can become very frustrating as irrational choices are made on the basis of indadequate information.

So, is there a more rational way of choosing which job to work on next? The two extremes are the obvious approach: highest priority first; and on the other extreme, a random choice. In practice, nobody would run their daily lives according to either these extremes: most of us would arrange things so that low priority tasks simply took a smaller slice of our attention through the day.  And so it should be with work priorities: it makes sense that even low priority jobs should receive SOME attention.

To make good choices here, you need good triage (and that deserves a write-up of its own): good decision-making needs good inputs, and it makes sense to think about what these inputs should be: a single crude "priority" rating is not enough.

I think that what is then needed is some sort of calculus of priority, whereby all the above considerations are included to work out the relative value of carrying out various jobs at different times, and an algorithm (formal or informal) that selects what items, performed when, would give the best overall result.  

Good developers have good intuitions in this regard, have some knowledge of the target system, and would unconsciously weigh these factors in comparing candidate tasks, thereby getting a “sense” of when the time was right to tackle particular jobs.   It's interesting to speculate whether this process could be assisted by smart "prioritising" tools.

Craft, not Manufacturing

The book “The Pragmatic Programmer” has as its subtitle “from journeyman to master” and on the cover a picture of a hand plane.  (The book itself is highly recommended).  The point is quietly made: good software development is more akin to craftwork than to manufacturing.  

Software developers are smart people, they are employed for their ability to create and engage with the logics that lie within our systems, and good software developers are at their best when they are being creative, and can exercise creativity.

This means:

• Developers who engage with what they are doing take on a feeling of personal ownership, of responsibility. They don't need to be held to timesheets or working hours, but will make their work part of their lives, their self-identity.
• Developers will want to take chances to look a little further than just the immediate problems they are dealing with. This allows opportunities for refactoring (which improves design and may increase the capabilities of the software), and for sweeping up of related problems, etc.
• Developers should be recognised for the work they do, they way they do it, and the unique contributions they make.
• Software development becomes enjoyable, and (speaking personally, but I think this is generally true) what this means is doing work that is useful, clever and efficient: “neat” in all senses of the word
• The developer who is proud of their work will spread the word. If they have just extended the capabilities of a service in some way, they will encourage its use and re-use.
• Likewise, working in a happy shop (and this may be a virtual workplace) will inspire and stimulate innovation, creativity and quality.
• Reduced staff turnover. If our working environment is highly rewarding, and great fun, and in particular if it allows us to do good work that makes us feel good about ourselves, who would look elsewhere?

Everything Changes ... but some things change faster than others

Some Requirements are Enduring - Others are Volatile

Enduring Requirements are Coded, Volatile Requirements are supported as Configuration Models

When dealing with requirements and change the "best practice approach" could be summarised as:

• Gather current requirements
• Establish known points of variation
• Investigate likely future requirements
• Build a highly parameterised application to cope with this anticipated level of change

This is all well and good: difficulties arise when:

• Unanticipated change occurs (often driven by regulatory change)
• Anticipated change does NOT occur (the system is left with unused and confusing configuration mechanisms)
• There is an attempt to extend the scope of the existing system to areas not covered by initial analysis (new lines of business, new territories)

Very often, despite the best efforts of talented developers, systems will not have adequately been prepared for the change that comes. Chaos ensues.

How can we do things differently? We can take a very wide definition of which requirements should be regarded as volatile, and plan to support this in a structured way:

• Anything driven by regulation is volatile
• Anything driven by “products” is volatile
• Anything driven by markets and competition is volatile
• Anything unique to the line of business, market sector, or territory should be regarded as volatile.

We can then analyse these volatile requirements and capture their variation in open-ended and structurally sophisticated configuration.  We call this process modelling, and the configuration definitions thus obtained are models.  Every effort goes into making modelling easy, so that as far as is possible, model change becomes a non-programming skill.

Enduring requirements are those aspects which are unlikely to change over a period of decades. “In a contract, we need to know who the contracting parties are”, “We will need to keep track of financial movements” are examples of enduring requirements.  Enduring requirements are supported by software applications that we call engines.

Engines provide raw, generic capability.

Models shape that capability into purpose-specific services.

When volatile requirements change, as we expect them to do, these changes are managed without change to the engines, purely as model change.

This is not to say that engines do not change: they do. But engines change only to extend generic capability.  This means that engine change is exclusively a positive thing. This is hugely important. It means that as time progresses, we expect the engines to improve: extended capabilities, better performance, more flexibility in deployment.

Engines + Models = Services

THINK

Meetings, Planning, and Corporate Process Can’t Build Better Systems

IBM used to make extensive use of the simple catchphrase “THINK”.  

Wikipedia : At an uninspiring sales meeting Watson interrupted, saying The trouble with everyone of us is that we don't think enough. We don't get paid for working with our feet — we get paid for working with our heads. Watson then wrote THINK on the easel.

Too often we act as if process is a substitute for thinking.   The response to a tricky problem is often to call a meeting.  While discussion is essential, the best discussions are preceded by and followed by periods of thought.  

This blog will strive to redress this balance. The posts here will represent principles, attitudes and practices that are the result of experience, followed by analysis, reflection, thoughtful design, implementation and testing.

No claims are made that these will all be final answers, or beyond the scope of change: after all, these are "Working Propositions". I will write about things that on balance, on the basis of experience and THINKing, seem to work.

Building Enterprise Systems that Last, that Improve With Age

I come from a background of financial services and specifically insurance systems.  It is a world of frightening levels of compromise and muddle.  Once (50 or so years ago) it was this industry that led computerisation. Now after generations of mismanaged solutions, bewildering requirements change, and growth in technological possibility it is struggling to even achieve stability in a changing world.     It is a world where every decade or so, a new layer of technology gets overlaid on what went before, a layer that typically is out of date before it is fully implemented.

The plight of the industry is made even more apparent when compared to the astonishing rush of new consumer computer technologies that put the equivalent of a supercomputer in everyone's phone or tablet, which has capabilities that Science Fiction writers would have scarcely dreamed of a couple of decades ago.

The purpose of this blog is to explore why enterprise systems in financial services companies is in such a poor state, and to offer some alternatives.

Let's do a thought experiment.  Cast your mind ahead 20, 30, even 50 years ahead.  What will the financial services architecture of the mid 21st Century look like?

On a bad day I can imagine it looks a lot like today's, just with more layers of failed solutions, new languages, hugely faster computers running the same bodged set of applications, messing things up more thoroughly than ever before.

On a good day I can believe that today's mess is a sign of immaturity:  that 50 years is an awfully short time to progress from Assembler through COBOL and PL/1 to Java to {insert whizzy language of choice here}. We have been learning, all this time.  Maybe it would be useful to see what we have learned.  Maybe it would be useful to think about what system design / architecture principles we can articulate that we can believe will still be valid in 50 years time.

As computerisation / automation / data processing / information technology reaches maturity, I can hope that we can stop reinventing the wheel.  This is not to say that the pace of change will slow: rather, as the ground firms beneath our feet we can hope to run faster and faster.

This blog will focus on the principles, attitudes and practices that I think stand a chance at the 50-year survival criterion.  The lessons that we think will stand the test of time.  Some of them will be wrong: that's why I call the blog "Working Propositions".  True for now, but not guaranteed for all time. Still, solid enough to build on.