If D and fixed time & price outsourcing are not one and the same they are very similar.

Last time I looked changes to the scope of fixed time & price contract require change requests which are generally for most costly than agile blabla.

I understand that you consider D something else but for me it's all blurry. Fixed time & price is clearly giving up responsibility. You may "hedge" for P but in the end you acknowledge that you know up front what you want to build and I don't see how that's different from D.

Fixed time & price contracts from the contrator side is another story.
 

Dear anonymous:

Here you are writing and trying to tell me what I mean when I say "Theory D." That is all very well and good, but may I humbly suggest that instead of focusing on what ought to be written in the dictionary, you answer this question:

Do you personally believe that software development is inherently deterministic?

Ignore contracts and dates and in-sourcing and out-sourcing. Pay no attention to what your boss or your client or your employees want. Disregard what sells books or lectures or training contracts.

I don't just mean do you believe you can pick a set of requirements at the beginning of a project and hit them at the end: I explained in the original post that airplanes do this every flight: they choose a destination and a time and they hit both destination and time far more often than they miss.

But airplanes are still probabilistic: they still must be managed in flight with course corrections as they encounter weather.

But I am asking you if software development is essentially fully deterministic, not like an airplane, but like throwing a ball, that if you make enough calculations you can make the perfect plan in advance and all people have to do is follow the plan.

Do you personally believe that we ought to be able to make a perfect plan, picking the technology, the architecture, the people, the tasks, the GANTT chart, the whole ball of wax, and then as long as people don't screw up, we'll arrive at the end date with working software that meets the stated requirements?

What do you think?
 

You make a convincing case to no longer post comments on your blog. Maybe you're tired.
 

It’s an easy enough proposition: buying a computer application – or program – to meet a certain end.

The combinations of all potential benefits and problems that may ensue from such a transaction are called the economics of software development. It’s a vast topic that can impossibly be covered by one book or even a series of books.

The only way to cover it is to describe some of the dynamics at play. Software development, its dynamics, limitations and potential solutions have been studied for decades. Some attempts are successful where others fail yet no one seems to be able cover the true nature of the economy of software development.

Software is abstract: it exists yet cannot be touched or seen. What can be seen – and read – are the computer programs that tell computers what to do. Alan J. Perlis talks about the three phenomena that are combined in software development: the human mind, a collection of computer programs and the computer.

Computer programs are modeled based on real or mental processes. These models never function as we expect them to due to our changing perception of what they should do. A computer program that at one time works exactly as we expect it to may continue to satisfy our needs as long as our perception about the processes it models doesn’t change.

In economical terms investing in software development in part means investing in processes as we understand them. As our perception changes we have to evaluate if our commitments will still bear the fruits we once thought they would.

Our understanding of processes may be blurred, incomplete or suffer from bad or weak consensuses. Yet based on them new software is being commissioned every day. If investments in software causes frustration it is often because of the realization that our perception of processes may have changed while the software remained static.

Many citizens for example complain about hugely expensive software development projects commissioned by their governments. To add to their fury these software development projects have a tendency to fail more often than they succeed.

We seldom look at the reasons for success. This is offset by our intolerance and fear of failure. Software development requires team work, human interactions over extended periods of time, understanding of processes, commitments, skills in many disciplines, consensus, agreement, control, evaluation and review.

In economical terms investing in software development may look like a shady deal. There are just too many unknowns and failure is never far away. It’s the inherit competitive nature of software development that makes us humans push the barriers and invest our time, resources and money in it anyway.

People that invest and succeed in software development have leverage over people who don’t. In the market this creates advantage which creates margins and profit. It is software development that became the backbone of our 21st century markets exactly because both are highly competitive and complementary.

People may have lost their religions but they are holding on strong to their faith-based dependency on mental models. Our economies are completely based on mental models. Money has value because of our mental models and it requires constant work and adjustments to keep this model intact.

We can drive our cars thanks to our mental models of how a car works. Thanks to them we can accelerate, steer, break and get from point A to B. It’s the ability to predict the future combined with the parallelism of our brains that make our mental models so powerful and capable.

The opposite – and foo – of our mental models is the physical world. Just as hitting your car against a wall hurts – despite any mental models we may hold – so are computers limited in what they can and cannot do due to the law of physics.

In a fantasy world computers can do anything – think Star Trek. In our physical world computers and their designers are facing enormous limitations that they have to overcome in order to keep up the pace of technological innovation.

In software development too there exist fantasy worlds that are made impossible by certain physical limitations. There’s a universal dream of being able to transfer any data from anywhere to anywhere. The Internet has evolved enormously over the past years to make some important steps towards making this dream come true.

However there are still many bridges to cross before we will be able to transfer any data we want without a glitch. Part of the problem is that data has to be interpreted – understood – before it can be transferred without limitation. Another problem is the vast amount of storage and bandwidth that would be required to make universal data transfer possible for everybody.

Other problems are more subtle and thus more easily overlooked, ignored or misunderstood. This however does nothing to ease the limitations they impose. One example is the different models to represent data – hierarchical, composite, tree-based – or the different model to hold (or store) it – in memory, on disks, on a network.

These models may sound similar but they are actually not at all. Bridging these gaps – for example converting data from one representation model to another while keeping semantic meaning intact – causes problems and costs money every day for millions of people all over the world.

One might think that the dramatic proportions of these gaps – they are also called impedance mismatches – should have encouraged people to solve them by now. They have been solved up to a certain point but a lot of work still has to be done – and money to be made – before we can say data can flow, be processed and be stored flawlessly and unhindered.

The competition between mental models, software models on the one hand and the physical world on the other hand defines the economics of software development more than anything else. Some people and companies have been very successful at solving problems for themselves and their customers.

Huge amounts of money are to be made from successful software development for the world and the market is evermore hungry for suitable software solutions. Innovation cannot be stopped, not by the value of the dollar or the price of oil, the state of the economy or world politics.

In fact software has had and will continue to have a tremendous influence on the full spectrums of economical, political and social endeavors. Operational costs of a construction site for example are influenced by how suitable and practical the software is that is being used. The construction industry has many customs and models that date back centuries, back when software did not exist.

Yet today software makes a difference on cost and margins as do new compound building materials. Fraud detection, tax collection, payment tracking, cash flow predictions and many more trades can and will evolve as our mental models and the software that goes with them continue to evolve.

The accounting profession and the many accounting rules and practices are purely based on mental models. Yet the software that helps accountants and companies has to adhere to the limitations set by physical rules that are translated by computers into real limitations for accounting software.

One example is the limitation that accounting data cannot be kept in volatile computer memory without risk. After all, in case of a blackout or technical failure the computer memory would be wiped and all data would be lost. The solution is to store data on disks to protect it from accidental erasure.

This however only replaces one problem with a set of other problems for software works best with data held in memory. Data sitting on disks is accessible to software but only at a price to be paid in slower response times and crossing the bridge between the world of volatile memory and disk memory.

But the economy of software development isn’t only defined by the clash between mental models and the physical world.

It is also influenced by clashes between mental models. After all, the economy itself consists of a very large set of mental models that are also under influence by the physical world. We have to eat every day and it’s the economy, manufacturing, transportation and agriculture that bring food close to our homes. The price of our food is affected by the price of oil and the demand for food across the world. But we can’t witness this process directly; these effects are imposed on us by the markets and the economy in general.

As with the economy in software development mental models clash as well and very often due to differing ideas held by people. Developing software requires collaboration between people which unavoidably leads to different beliefs, ideas, opinions and eventually conflict.

The Responsibility Virus is an excellent study on this subject. I like to call it “the missing manual for human interactions”.

Yet human interactions affect not only our mental models about processes that are to be implemented by software. Human interactions also affect the processes and mental models by which we develop software.

This often leads to highly problematic situations in software development projects. Not only do projects have to deal with the uncertainty and dynamics of the mental models that have to implemented, they also have to tame the processes and dynamics that go on within the project.

These dynamics with a project can best be described as tensions and conflicts between: internal, external, control and flexibility. A successful software development project requires a balance between all of them yet this is very often very hard to achieve.

Some customers are hard or even impossible. Yet some teams are able to tame even those. It helps if the customer and the team members all have read The Responsibility Virus.

Customers generally cover the external and flexibility part and to some extend control. The internals of a project and the control that has to be exerted over its progress are generally related to the actual software development.

Software development projects often not only deal with software development but also with hardware and network installation, business analysis, technical design, testing and documentation.

All of these responsibilities combined tend to become a lot for any team. This is also the economics of software development: to which extend can the scope of a software development project grow?

It is generally accepted that project management is 5% administration and 95% human interaction. The balance is different for software development and it depends on the kind of software that is being developed.

Software can generally be divided in one of two categories: software that is technically challenging and software that has to implement complicated mental models. Software that is complicated in both areas exists – look at what NASA does – but is extreme and not defining for the industry.

The people that have to work on any software development projects better should have skills that are relevant for the kind of software that is being implemented.

A typical example of clashes between mental models is the strategy or methodology towards software development. There exist two extremes and a lot of common ground: rigid design up front versus agile software development.

The strategy is often determined by the legal agreement between customer and software development contract. If the customer wants to see his or her software development under a fixed time and price contract the contractor has the right to stick to pre-agreed requirements and specifications.

If the customer however wants to pay per hour for as long as develop goes on – time and material – it’s in the customers interest to work closely together with the contractor and share the responsibility to come up with the correct mental models to be implemented.

The market tends to go for the fixed time and price model by a large margin. Software developers however prefer close collaboration with customers based on the experience that mental models tend to change frequently.

Some customers have a hard time to make up their minds and an open-ended contract would probably not get them anywhere. Fixed time and price contracts are probably the least risky for them given that the initial requirements and specifications make some kind of sense.

Governments are notorious for not being able to make up their minds on requirements and therefore usually do pre-studies to define clear and agreed upon requirements and specifications for big projects.

The market however still tends to put too much value on fixed time and price values, mainly for budget and accrual reasons. It is indeed nice to know up front how much a software development project will cost you.

However, fixed time and price contracts come with change requests that have to be signed every time the customer changes his or her mind. The advantages or disadvantages of these are very hard to add up and depend on the situation at hand.

It is however clear that if changes in the understanding of models can immediately be reflected in a software product the costs of the change will be much lower. Fixed time and price contracts are a good remedy against constantly changing your mind.

Some customers however would be better off with the time and material approach if customer and contractor can practice shared responsibility on keeping the mental models sane and on the mark.

As a software developer I tend to prefer the time and material approach but that’s a wholly subjective stance. I do believe the market isn’t correct for the simple reason that software development is a highly complicated business that cannot be easily translated into an all-saying statistic.

The best reason however for going for time and material contracts with shared responsibility is that customers have a much better chance of getting what they need if they can keep control over the process of changing mental models.

Companies and governments go for fixed time and price contracts without a second thought because they want to reduce the risk of failure. Having been on quite a number of fixed time and price contracts on the contractor side I’m well aware how the stress create by fear of failure feels.

Contractors very often have to produce bank warranties to provide financial guarantees for the customer in case the contract goes bad. Apart from the financial consequences for the contractor in case of failure there is also the loss of face in the community as whole.

I’ve watched one of the biggest IT consultancy companies in the world go down on their knees in a fixed time and price contract with a government department that got cancelled after six years of work. Although I’m not an insider in the case it seemed clear to me based on testimonials that both parties were to blamed for the failure, not just the contractor.

The customer has a stronger legal case when a fixed time and price contract is cancelled which unfortunately leads to under-responsibility from their side during the course of the contract.

Fixed time and price contracts unavoidably lead to under-responsibility from the customer side and over-responsibility from the contractor side. It shouldn’t have to be like this. However, it requires change requests to make important changes to the scope of fixed time and price contracts, unavoidable leading to slowed down progress and potentially underperformance for the invested funds.

Companies and governments don’t like time and material contracts because they feel like it sets them up for under-responsibility and under-performance from the contractor side. Under-responsibility and under-performance by the contractor are however very typical for the fixed time and price contracts, specifically because the contractor knows the customer has to stick to the contract throughout.

Time and material engagements are obviously risky but it is a fallacy to think fixed time and price contracts are less risky. This false sense of security that seems to be offered by fixed price and time contracts stems from the believe that software development is not competitive.

Customers look at the contractor from their higher ground and feel they are safe. After all, in case the contract goes wrong it can be blamed on the contractor and that will be the end to it.

Customers don’t seem to recognize the risk of loss of time, sub-optimal technical solutions, sub-optimal process implementations and many other defects – either visible or hidden – that can be the result of even “successful” fixed time and price contracts.

By not assuming responsibility over the course of a software development project customers set themselves up for failure. I have yet to come across software produced by fixed time and price projects that impresses me by its functionality.

And even if some contractors can offer this kind of software through fixed time and price contracts it is unlikely these qualities can or will be taken into account in the bidding phase.

Fixed time and price contracts give the impression they share certain qualities with fixed price contracts (or just buying software off the self).

This again is a fallacy since you can’t make any assumptions about software that has yet to be written. You can try out off-the-shelf software and decide whether you like it or not. You can’t do the same with software that doesn’t exist yet and assigning any qualities good or bad to none-existing software is like describing what happens after we die. You need to have a lot of faith.

Unfortunately faith doesn’t work on the market where out-guessing your competitors rarely works out to your advantage. As in the economy we have to learn that ignoring the competitiveness of software development is a recipe for failure.

If you emerge yourself in the software development community you’ll find the same kind of competition and collaboration you’ll find in the economy at large and in life itself. Platforms, operating systems, web browsers, word processors, programming languages, databases are all played out against each other to determine their strength and weaknesses.

Inside the community such competitive language is often bad mouthed yet is it the most obvious thing to do. People are in competition and want to find the best tools. If they like a product but want more features they’ll ask its vendors to add them so they can use its new version to compete better.

Software development always has and always will battle on the edge where competition is the harshest. Opening new markets with software innovations always has been and always will be considered as the equivalent of finding a new vein of gold.

Considering software development as a commodity like cleaning toilets sets you up for failure and disadvantages in the market. New competitors that can write better software than you will have a significant advantage over you.

The best example of this in recent years is probably Google. When everybody thought the search function on the Internet had been exploited for what it’s worth Google stepped in and took over the entire market.

Another example is Tesco, a UK based grocery chain, which got a significant advantage over its competitors by managing the stock per store based on what customers tended to buy. This lower overhead allowed them to get much further with their capital and resources than their competitors.

Allowing mental model – ideas – to transform as easily and painlessly as possible in software models will define tomorrow’s markets. Today we believe computers and software development are things of the 70’s that by now are reaching maturity.

From inside the software development community I can testify this couldn’t be more wrong. Software development is highly competitive in all its perspectives. The apparent trouble software development projects are going through today are signs of failure on the part of the customers that commissioned them.

Actually, the requirement that says we have to transform our mental models in software in the first place is slows down software development considerably. One day soon computer software will literally read our mind and construct software for us. But more importantly this disadvantage goes for everybody so the competition can go on.

The day the mind plug becomes a reality is the day we either fear or long for, only showing how strongly we feel about this potential competitive advantage.

Until then we have to find the best software developers, let them function instead of dysfunction in groups and let them write software for us.

As I said before, companies and governments continue to invest in software development despite the failure label that sticks on such undertakings. This is due to the market which demands system to become ever smarter and better.

There is significant money to be made is methods that produce better software. Better is obviously a subjective measurement but so is the competitive market. Some aspects of better can be defined as measurable, sensible, useful parameters. Other aspects of better will remain subjective and in the eye of the beholder.

People have tried to come up with parameters and metrics that are supposed to distinguish good software from bad software. Despite their obvious shortcomings they are still being used. The fact that some of them are now defined as ISO standards gives them more weight but unfortunately does them make not more correct.
Most software metrics are only meaningful in a very limited context. Others are far more useful as they can unearth aspects of software that can genuinely make predictions about its competitiveness.

But again our blurred view of software development as not competitive stops us cold in our tracks. A lot of money can be made by offering better alternatives to existing software that is inherently not competitive.

My favorite example if Hibernate, a tool that facilitates database access for Java applications. Hibernate is very popular in the Java community, in part because it was the first open-source tool in its kind. Another reason is the reasonable state of its documentation.

However, the people behind Hibernate are in a state of denial when it comes to certain qualities of their cherished product. As an example they regularly chastise people that critize Hibernate on public Internet forums.

Hibernate’s dirty little secret is that fact that its software quality is terrible. This has disallowed Hibernate to make any significant progress in terms of new features over recent years, something which becomes obvious to more people as time goes by.

Hibernate’s real victory has been a political one, namely it’s been used as a template for the database part of the recent Enterprise JavaBean 3 specifications.

Hibernate has competitors but they are not aggressive enough. On top of that they are now in the same bed with Hibernate since they all comply with the EJB3 specifications. Their compliance with mediocre specifications is apparently more important than offering competitive advantages to their customers.

Anybody with sufficient time and knowledge could step in and offer a replacement for Hibernate. Hibernate’s inability to follow suit would become apparent and the market would go for the more competitive offering. Giving up on the EJB3 specifications and the Hibernate way of doing things would be a powerful signal to the market and the community that won’t be appreciated by all but that’s not the goal.

Yet we don’t look at software in this way. We see Hibernate as an important part of how we develop software. If we would replace Hibernate for a competitor, and knowing that we would do this for competitive reasons, would require us to admit that we too have become too lazy and too satisfied with our own performance over recent years.

Hibernate does do a number of things well. We tell ourselves that Hibernate does what it’s supposed to do. In doing so we admit that Hibernate’s shortcomings are really our own. We accept these shortcomings as annoyances because apparently everybody else does so too.

To step up and stop this nonsense requires courage and goes against the tendency to minimize anxiety. For the benefit of competitive advantage this will have to happen anyway.