Why software projects fail: Matthew blames the Real Programmers. Not being one, but heavily surrounded by them now for many years, let me explain why I fail to find one single valid point in his reasoning. Obviously, Matthew is talking about software development inside commercial projects, so you'll have to scale his and my reasoning to your particular situation. Even more obviously, I can only speak from my own experience, which seems very different from his. There must be different types of Real Programmers, so it seems.
I'll start with collapsing his first two points:
There's still some big truth in these points, in that projects often fail because of a lack of a clear vision, and the requirements contained by this vision. These requirements often never materialize because of sheer laziness: both the customer and the programmer are too lazy to descend into one another's mindset, in order to really understand enough of each other's work to come up with a set of effective requirements.
Computers are dumb things, and it requires skill to flex them towards a resolution of your problem. By not being able (or willing) to precisely state your exact problem, in a way that it fits in the limited execution context of a computer, you will decrease the chances for a potential solution to effectively happen. Since one cannot expect the customer to have full knowledge of the execution environment, it is the task of the programmer to make him aware of enough context to be able for the customer to state the problem in a way that both parties are communicating in a common language, and are stating a precise and minute set of requirements.
Customers are not expected to go a lot into implementation details - they should even be advised not to state many implementation requirements. What I have seen many times in my small career, were customers who wanted to hear all the fancy buzzwords, who wanted to feel part of the programming crew, yet who were not able to exactly state what they wanted, and ended up demanding immense flexibility: a recipe for disaster. "I can't make up my mind, yet I want you to do it."
Also, programmers love precise specs. They might be frightened by the complexity of them, or become aware of inconsistencies which will make them ask annoying questions to the customer, but in the end, their job becomes much easier if they know what to do.
In companies I used to be working for, this communication problem was often solved by some middleman: an analyst. One could expect from analysts to be able to talk to customers and make abstraction of the implementation noise the customer was trying to inject into the requirements, and also them to be able to go into enough technical detail to not come up with inconsistent problem domain models to explain the task at hand to the developers. A good analyst could make or break the project. Another recipe for disaster was putting a project manager or sales person in the role of the analyst. For obvious reasons, these roles would always walk the customer's walk, rather then trying to help the developer along. The developer would feel bad because this would simply mean an increase of the number of customers in the project to him, rather than a communication bridge to the customer. So don't staff your failing projects with more infrastructural roles, but try to hire good analysts on beforehand. I know some, but they tend to be a scarce resource.
For larger projects, programmers also can't take up the analyst role, since their brain will simply melt because of dimensional overload: when you have to look at a problem from too many sides at once, while also worrying about minute implementation details, your perception will become heavily distorted and you'll suffer from neurological collapse. Being able to decompose large problems into composable solutions is the main skill of a developer, but you cannot add the task of defining the problem space to his job description - that would be God-like. In the end, writing down requirements should be nothing more than that: a thorough write-up of the problem space. The developer can then focus on designing the solution space.
- Asking a Real Programmer how long he/she needs to write a component/application causes the same brain cells to fire that would be involved if you asked them (in a bar) how often they have sex. Albeit in the opposite direction numerically speaking
- Whatever the reply to the previous point, management will think they are exaggerating and reduce the numbers
- Decomposition is your friend. Make sure you have a clear vision (see above) of what needs to be achieved, and think of the different building blocks this would require. Try to make identically-sized blocks if possible, and reduce each block until you come up with week estimates for each of them: one week, two weeks, never more than a few of them. Building blocks cannot be specified using manmonths: that's too big a unit.
- Design to budget. By doing your own homework like this, you can budget easy or obvious things easily, and make difficult things optional. You can explain to your customer that he will get 80% of the project for a fixed (and mutually safe) fee, but that you're unable (yet) to come up with precise guestimates for the remaining 20%. Sell those separately. Explain the problematic blocks so that he understands what they will buy him (or not). Also, keep an eye on the total amount of days/weeks/months, both in terms of full-time equivalents and lapse time. Your team size and their exhaustion rate is part of your budget, and you can easily deplete it. You cannot expect someone to be focussed on building a single building block for more than a couple of weeks, and you need a variety of varying blocks to keep someone focussed on a single project for more than three or four months. After that period, it's death row marching unless the variety helps in pushing fresh (and sufficiently isolated) problems into the developer's mind working area.
- Don't expect someone to decide when his own work will be finished. We are all proud people, and live to love what we do. Parting from our child is difficult, since there's always little things we know that we still want to tweak, or where some refactoring would be applicable, if only to find out what the result of this refactoring would be. In a sense, the journey is much more interesting than the destination, and we want the journey to last as long as possible. Programmers however do like planning, since it gives them some organisational context for their work, and also since planning very often decides on interfacing moments, where people are to be confronted with their work or its progress. They need feedback, and deadlines are a way to look forward to that feedback. By deciding on a planning, a project organizational structure is established, and people will feel effectively pulled into the project team, knowing they can depend on people to deliver them raw material in due time, and also knowing they have to deliver somewhere and sometime. Planning is a great motivational and teambuilding tool, if correctly executed, and based on feasible guestimates. See above.
(to be continued - off to sleep now)