Wasabi cannot cure rotten fish There is only one problem with development methodologies. Just one. It affects all methodologies, agile and otherwise: No methodology can 'fix' projects that are staffed with underperforming people. People are more important than process, period.

This is the underlying issue with the language 'wars' as well. No magical combination of JSON, static typing, design patterns, IDE whizzies, and frameworks will somehow help a million monkeys produce any of Shakespeare's works.

If you want to build a ship, don't drum up people to collect wood and don't assign them tasks and work, but rather teach them to long for the endless immensity of the sea.

Steve Yegge has suggested that if 90% of the projects adopting a methodology fail, you have to stop blaming the people. You have to stop saying "you're not doing it right." You have to say there's something wrong with the methodology itself if 90% of the projects fail to use it properly.

Sounds reasonable. But why would this be true for methodologies but not true for programming languages?

99.999% of the software written in almost any language (including Lisp, Python, and Ruby) is buggy. Yet we readily say that the problem is the programmer. We think that some languages are better than others, that some languages help eliminate certain classes of bugs, but we take as a given that good tools are not sufficient unto themselvesfor good results. We know that the quality of the result is almost entirely driven by the quality of the programmer.

I have the uncomfortable feeling that others are making a religion out of it, as if the conceptual problems of programming could be solved by a single trick, by a simple form of coding discipline!

Home Depot might suggest that buying a new table saw will cause beautiful woodwork to appear in your home. Yet just because a duffer with a table saw cannot make built-in closets all by himself, we don't abandon table saws outright. We assume that good woodworking is better with good tools. And if we are blessed with the skill to make a built-in closet with a hand saw, we know that we can also make the closet with a table saw, and we judge the table saw on whether and how it can improve the results we could have obtained with another tool.

This is my criteria for judging a methodology. Can it improve a team that is already fundamentally qualified to write software?

If you want to stop reading here, my statement is that you can only judge a methodology on the basis of the results it delivers for a team that has already proven it can ship software. You judge it on the basis of incremental value. You don't compare its results to the results some other team gets, or to your fantasy of what results the team 'ought' to get.

The mythical marginal

What I've written seems obvious. Yet, billions of dollars are paid every year to people who are selling a different perspective. What is it that causes companies to buy silver bullet after silver bullet? Why do companies lurch from consultant to methodology to programming language like infomercial junkies looking for something that will flatten their tummies without sweaty exercise or unpleasant diets?

I bifurcate teams into those that are fundamentally qualified and those that aren't. And I believe that to fix an unqualified team, you start with the people, not the process or the tools. So what could the rest of the IT industry possibly believe?

I have observed a belief in a marginal team. A team that is somehow straddling the fence between incompetence and competence. They can ship software, but only with help from their process and tools. If 'marginal' is too perjorative a term, you could think of them as having unactualized potential.

In truth, I have worked with several such teams, so I believe they exist. However, I believe that such teams are quite rare, while proponents of silver bullets make a living convincing customers that marginal teams are commonplace.

The primary example I have observed of a marginal team is a team composed of individuals who are not working together well, yet they have achieved success on other teams in prior roles. Something like the Los Angeles Lakers before Phil Jackson arrived. There is a lot of latent championship potential in the individuals, but the team isn't performing.

Shipping software is not an emergant property of competent programming, nor is it a consequence of management technique.

Why does the IT industry believe that perhaps most teams that fail are marginal teams? They have to believe this, otherwise they wouldn't waste time and money trying new silver bullets ("everything should be a stored procedure," "tests are the only documentation that matter").

It always comes back to hiring

My broken-record assertion is that the industry as a whole embraces the above model of a marginal team: latent potential in the individuals. The difference between the industry and I is that I have an objective measure of latent potential: has the individual demonstrated success shipping software in the past.

True happiness comes from the joy of deeds well done, the zest of creating things new.

The industry doesn't apply this test rigorously, if at all. A typical interview with a developer focuses on patterns and archiecture, on talking about past achievements. Developer's don't juggle. Developer's don't design software. And especially, managers don't hire developers with super-strong references from strong employees that have worked with the candidate in the past.

Basically, managers ask developers if they can ship software and then take the candidate's word for it. If you take only one thing from this blog post, take this: shipping software is not an emergant property of competent programming, nor is it a consequence of management technique. Someone can demonstrate the ability to write software but lack the ability to actually ship software.

(Update: nothing in this post should be construed to suggest that people cannot become better developers through study, mentorships, or training. However, training is to hiring as tools are to skill. And they are not mutually exclusive, any more than good programmers and good tools are mutually exclusive.)

I believe that the reason why the IT industry believes that most failing teams are marginal rather than outright incompetent is that they believe that these teams are composed of individuals who can write software and managers who can direct work. All they need are the right tools to write software "better" and better software will emerge. All they need is a methodology to manage software development better and software will ship.

If you have a marginal team, a team composed of individuals with proven ability to ship working software, you might have something there. But if a team isn't even marginal, no methodology will help. And that's why a useful methodology can still fail to help 90% of the teams that try it.

For the same reason that Wasabi makes Sushi divine.

Follow-up: Three questions and three answers about Wasabi and Rotten Fish

Postscript: The difference between Sashimi and Nigiri

If we're judging success or failure of a development methodology, we have to judge the results on whether the development itself was successful. We can't judge whether the software made money, or whether the company's stock soared. Such things can be studied, however those are product management problems, not software development problems.

People have criticised Google, saying that althought it has a track record for shipping products, only one, AdWords, is financially successful. If you want to compare product management methodologies, that's fine, but you aren't talking about software development. You have to compare similar kinds of projects, for example you can't compare an in-house IT project with a known ROI (reduce the time spent entering a new Blort Ticket by 50%, saving $47.32 per week per Fizbang team member) to speculative product development like a new Web 2.0 calendar.

Who else is trying to create new products? Apple? Microsoft? Dan Bricklin? Compare them to Google, and if they are doing a better job of making money with their projects, post a critique of Google's product management on your blog.

-r.b.

Labels: agile, popular

  ¶ 8:13 AM

  People want holes, not drills, and especially not drill bits

DSLs are a hard way to sell your language. They haven't worked for Lisp yet. Before you can sell Ruby based on DSLs, you need to tell people why they need DSLs.

My sales background tells me that people want holes, not drills, and especially not drill bits. For that reason, you have to sell the holes themselves, not the technology for making them.

It's a mistake to 'sell' DSLs, just as it's a mistake to sell metaprogramming. Wherever programmers hang out, you'll find people arguing the merits of high signal-to-noise techniques like DSLs. It's fun and instructive to have this debate, but it isn't selling. It's coffee-shop banter about drill bits, and whether diamond tips are worth the price, and are all carpenters equal or are some better than others, and should all drills have safeties so any monkey can make holes, and can you do concurrent drilling with a drill press framework?

Lots of people love the way Ruby on Rails makes certain things declarative and makes their code smaller than whatever else they used to use. They love the holes, and that's as it should be. DSLs and metaprogramming are implementation details for framework builders, not selling features for framework users.

At least, not up front. Later, after they've gotten hooked on drilling holes, they'll be interested in drill technology. :-)  ¶ 11:46 AM

  Surf's Up!

Every decade or so we have a major revolution in the way we develop software and the environments we develop for. But, unlike the object and web revolutions, we can see the concurrency revolution coming.

Awesome.

I've watched and participated on the periphery of several "waves" of computing progress. Yes, I wrote my very first SNOBOL programs on punch cards. Yes, I wrote a networked database application in MP/M. Yes, I wrote commercial software for Macintosh and marvelled at its Toolbox. And to learn Java, I used it to write a Scheme interpreter. (And yes, it did optimize tail calls and support continuations).

So I've lived on and in this ocean we call the computing industry, or computer science, or programming, for a while. I've seen it roil, I've seen it's placid moments. And I've seen its thundering, express train waves.

And I can tell you, it feels like another big wave is coming. A monster wave. A wave of tow-in surfing epic proportions. It's damn exciting. It feels like if we paddle like hell and try to stand on our boards we could be sucked under and crushed.

Or we could have the ride of our life, teeth bared in the most insanely happy grin we've ever worn.

I know that after the wave hits the shore, it will obliterate half of the vending stalls and the beach blankets and the hordes of sun tanners. And when the water subsides the industry will rebuild the beach houses and the industry will reopen the vending stalls, and the people who are rich today will probably still be rich then.

It's tempting to just wait, wait for the wave to hit and for the water to suck back into the ocean. Wait for others to solve the problems, to harness the energy. Wait until Microsoft and Sun and Apple and Linux and Apache and whoever else decide what we should do. Wait until they give us the frameworks, until they tell us where to swim.

Wait until there's no risk, like many people did with the Web and with Java and with their whole lives.

We can wait, just hunker on the shore and not take any chances out there in the water on our own.

But where's the fun in that?

Labels: passion

  ¶ 1:58 PM

  Ruby Metaprogramming: A generic controller for REST-ful services on Rails Disclaimer:

This is very raw code, and I've only been using it for a few days. It isn't in production. It hasn't been peer-reviewed. There are far fewer lines of test code than lines of functional code.

In short, this is really a peek over my shoulder at what's running on my MacBook.



It has shortcomings. I recently read a blog post where the author complained that many of the naked people at Burning Man weren't pleasing to his eyes. Fair enough. But... is nudism always about exhibitionism? I think in many cases, it's about being perfectly aware of one's lack of physical beauty but being comfortable enough with oneself that you don't need to hide inside your clothing.

I am posting this code in that spirit. Warts and all, I am trying to get my head out of the "don't show it until it has been polished to perfection" mind set.

If you don't find it attractive, I hope you at least find it interesting.

What I needed:

I'm doing some work with bridging multiple Rails applications. In the back of my mind I thought I'd want to use ActiveWebServices in the production code, but when I got to the point of integrating my databases, I followed the principle of YAGNI and decided I would use the simplest thing that could possibly work.

So I figured I would build one of the Rails applications around a REST-ful interface. The other applications could wrestle with REXML and Net::HTTP to talk to it. The joke on me is that this isn't the simplest thing that could possibly work. In my ignorance of all things Rails, I didn't know that I could have one Rails application use different databases for different models.

Rails and REST:

Rails 1.1 makes REST controllers as easy as:

class WardersController < ApplicationController

 def index
     @warders = Warder.find :all
     respond_to do |wants|
         wants.html
         wants.xml   { render :xml => @warders.to_xml }
     end
 end

 # ...

end

I also needed to implement the show action. Once again, it’s fairly easy to handle the default case:

def show
 begin
   @warder = Warder.find params[:id]
   respond_to do |wants|
     wants.html
     wants.xml { render :xml => @warder.to_xml }
   end
 rescue ActiveRecord::RecordNotFound
   render :nothing => true, :status => 404
 end
end

Shortly after I had this working, it was time to implement filters. My first need was to be able to find the first Inmate that exactly matched something other than an id. I believe that it is far better to extract infrastructure through refactoring common code than to over-engineer up front. But it seemed obvious that I needed a generic “find on any column” feature.

I did cobble something together, and it worked: you could search for an inmate using a GET such as http://thevillage.org/inmates/show?number=6. Shortly after I had it working for the show action I discovered that I needed the exact same functionality for the index action as well. And it was obvious it ought to work for any model controller without resorting to copy and paste.

Rails Controllers and Filters

This is a common concern, and Rails provides Aspect-Oriented Programming hooks for controllers called filters just for this kind of purpose (they’re also handy for security). I decided to write a filter. Since I wanted to use it for all of my models, I placed the filter in application.rb, the common superclass of all controllers.

Let’s start with a look at what a typical controller now looks like:

class InmatesController < ApplicationController

before_filter :first_inmate, :only => [ :show, :update, :edit, :destroy ]
before_filter :all_inmates, :only => [ :index ]

def index
  respond_to do |wants|
    wants.html
    wants.xml   { render :xml => @inmates.to_xml }
  end
end

def show
  if @event.nil?
    render :nothing => true, :status => 404
  else
    respond_to do |wants|
      wants.html
      wants.xml { render :xml => @inmate.to_xml }
    end
  end
end

# ...

end

As you can see, I’ve added before_filters to our actions. These filters set the instance variables @inmate or @inmates for the action’s use. You can also see that the filter methods are customized for each model class. As you’ll see in a moment, if you add a new model class, say, Rover, you’ll get first_rover and all_rovers methods for free.

Here’s the code for the methods:

class ApplicationController < ActionController::Base

 class << self
   def before_filter *args
     method_name = args[0].to_s
     if method_name =~ /^first_(.*)$/
       class_name = Inflector.singularize((Inflector.camelize $1))
       instance_name = '@' + Inflector.singularize((Inflector.underscore $1))
       load "#{Inflector.singularize(Inflector.underscore $1)}.rb"
       clazz = Kernel.const_get(class_name)
       define_method method_name do
         self.instance_variable_set instance_name, (find_filter :class => clazz, :find => :first)
       end
     elsif method_name =~ /^all_(.*)$/
       class_name = Inflector.singularize((Inflector.camelize $1))
       instance_name = '@' + Inflector.pluralize((Inflector.underscore $1))
       load "#{Inflector.singularize(Inflector.underscore $1)}.rb"
       clazz = Kernel.const_get(class_name)
       define_method method_name do
         self.instance_variable_set instance_name, (find_filter :class => clazz, :find => :all)
       end
     end
     super
   end
 end

private

def find_filter options
  options[:find] ||= :first
  if params[:id]
    begin
      (options[:find] == :first && (Event.find params[:id])) || [ (Event.find params[:id]) ]
    rescue ActiveRecord::RecordNotFound
      (options[:find] == :all && []) || nil
    end
  else
    column_names = options[:class].column_names
    clauses = (column_names.inject([]) { |cond_clauses, column_name| params[column_name] && cond_clauses << "#{column_name} = ?"; cond_clauses })
    full_conditions = (!clauses.empty? && (column_names.collect { |column_name| params[column_name] }).compact.unshift((clauses.join ' AND '))) || nil
    if params[:order_by]
      words = params[:order_by].split
      order_column_name = (column_names.include? words.first) && words.first
      if order_column_name
        direction = (words.last && (words.last.upcase == 'DESC') && ' DESC') || ''
        (full_conditions && (options[:class].find options[:find], :conditions => full_conditions, :order => "#{order_column_name}#{direction}")) || (options[:class].find options[:find], :order => "#{order_column_name}#{direction}")
      elsif full_conditions
        options[:class].find options[:find], :conditions => full_conditions
      else
        options[:class].find options[:find]
      end
    elsif full_conditions
      options[:class].find options[:find], :conditions => full_conditions
    else
      options[:class].find options[:find]
    end
  end
end
end

As you can see, I intercept calls to before_filter and define a small wrapper method around find_filter. I was going to introspect on the controller name to get the class, however I wanted to allow the posibility that I would create controllers with arbitrary names. And after looking at the way the filters look, the method names seem to have documentation value.

It also seemed possible to eliminate the filter calls in the controller, but they are so short that it didn't seem like a big code win to abstract those up into ApplicationController.

So... there you have it. In summary, all of the controllers in this rails application now support simple filtering on column names like http://thevillage.org/escapes/outcome=failure or http://thevillage.org/butlers/stature=short&umbrella=yes.

My thoughts on metaprogramming:

I had to stop learning Lisp. I realized that at the rate I was going I would quickly automate myself right out of a job.

I like dynamic metaprogramming. Obviously. On the other hand, it’s hard to write an IDE that knows where to find the definition for all_rovers. I think that as Ruby matures, people will look for ways to assist tool vendors, perhaps by extending reflection.

OO theorists sometimes complain that even this mild form of dynamic metaprogramming violates the purity of defining all of the verbs for a noun up front. At the risk of arguing with a straw man, I feel that this is well within the spirit of OOP. What we’re really saying is all objects that extend ApplicationController have two extra methods, first_something and all_somethings, we just don’t know the name of something yet.

Defining the methods using method_missing:

My first cut at the implementation used method_missing to handle the new methods instead of define_method. Stefan Tilkov's comment (below) inspired me to try defining the methods dynamically. Here's the original code:

 def method_missing method
  method_name = method.id2name
  if method_name =~ /^first_(.*)$/
    class_name = Inflector.singularize((Inflector.camelize $1))
    instance_name = '@' + Inflector.singularize((Inflector.underscore $1))
    load "#{Inflector.singularize(Inflector.underscore $1)}.rb"
    clazz = Kernel.const_get(class_name)
    self.instance_variable_set instance_name, (find_filter :class => clazz, :find => :first)
  elsif method_name =~ /^all_(.*)$/
    class_name = Inflector.singularize((Inflector.camelize $1))
    instance_name = '@' + Inflector.pluralize((Inflector.underscore $1))
    load "#{Inflector.singularize(Inflector.underscore $1)}.rb"
    clazz = Kernel.const_get(class_name)
    self.instance_variable_set instance_name, (find_filter :class => clazz, :find => :all)
  else
    super
  end
end

While I embrace the spirit of dynamic metaprogramming, I don’t care for the aesthetics of implementing methods with method_missing. In my ideal world, we’d combine pattern matching with our existing syntactic forms of defining a method so that it is much more obvious that ApplicationController defines those two methods. That would be a win for programmers and for tools. And we could do that today as syntactic sugar by rewriting pattern matching definitions using method_missing.  ¶ 10:00 PM

  Scientists Announce Empirical Evidence for Greenspun's Tenth Rule It's Friday, so let's have some fun.

Sinclair Schuller posted an interesting article, How to Rewrite Standard Recursion through a State Stack & Iteration (the original page has been removed). In his own words:

Recursion, as a construct, is quite beautiful. It offers an elegant means of achieving an algorithmic goal and is used in everything from mathematics to text processing and data structure manipulation. The problem is, using it in practice through today's popular languages (such as my favorite, C#) can prove to be a disaster.

The problem, quite simply, is that many flavours of Blub popular languages have a limitation on how deeply you can make function calls before returning. That limit is far smaller than the limitations placed on the size of data set you can manipulate.


Recursive functions often have the property that they require storage On. Therefore, if the number of function calls you can make before returning is fixed, the size of data set you can manipulate is fixed.

Yes, I said data set. That's very interesting, especially when you consider that Sinclair's example, the factorial function, doesn't really manipulate what most people would think of as a data set. But it really is, because if you were to think of 100! as a macro instead of a function, you would end up with 1 x 2 x 3 x 4 x 5... x 99 x 100. So there really is a set of 100 datums to manipulate.

Now back to Sinclair's example. He shows how to emulate a programming language's call stack. This is Greenspunning at its finest:

Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp.

Sinclair's transformation implements a better programming language (like Lisp) that does not have an arbitrary limit on the size of the stack it uses for function calls. This used to be a neat programming challenge. Old-timers will remember when certain languages (BASIC, FORTRAN) did not support re-entrant functions or procedures. So if you wanted to do something recursively, you had to Greenspun. (re-)Discovering this technique in order to write a "Towers of Hanoi" program was an early "Aha!" moment for me.

I call such a transformation a pseudo-iteration. It isn't really iterative, as the presence of a stack shows. Instead, it is emulating or interpreting a recursive algorithm. Sinclair also shows how to optimize the algorithm so that it always executes in constant space (O1).

I consider this a variation on Greenspunning where instead of implementing half of Common Lisp, we implement half of Scheme. Scheme is a member of the Lisp family that can optimize tail-recursive functions. (So can most Common Lisp implementations. Maybe. As Wikipedia says: The Scheme standards documents require tail-call optimization, which the CL standard does not. Most CL implementations do offer tail-call optimization, although often only when the programmer uses an optimization directive.)

Let's look at what's going on with some Ruby. The first thing we need is a factorial function:

def factorial n # whole numbers only
 (([0,1].include? n) && 1) || (n * (factorial (n - 1)))
end

(If you aren’t familiar with Ruby, what you need to know is that the && operator returns its last argument if all of its arguments are truthy (neither nil nor false) and the || operator returns its first truthy argument.)

Okay, so the first thing we do is convert this factorial function into tail recursive form:

def factorial n, acc = 1
 ((0 == n) && acc) || (factorial (n - 1), (n * acc))
end

As you already know (or have learned by following the links to an explanation of tail call optimization), this version of factorial has its call in tail position. That means that every time it calls itself (just once), it returns the result of the call directly. The first version of the function didn't return the result directly; it called itself and then performed a multiply on the result.

Because it does no further work with the result of the recursive call, there is no need to store the calling parameters or the result on a stack. What you do is this. You set up a loop. Inside each iteration you perturb the parameters you have received. When the loop is done, you return some expression on the final set of parameters.

As it happens, factorial is easy in this respect because the final return is simply the accumulator we had been carrying around as a parameter in the tail recursive algorithm.

Let's perform the conversion, using this approach to omit the stack:

def factorial n
    acc = 1
    until 0 == n
     acc *= n
     n -= 1
 end
 acc
end

We have progressed beyond half of Common Lisp into half of Scheme. Now, like a properly optimizing environment, we have greater performance and we require constant memory.

Only we have done by hand the work that a machine (you know, a labour-saving device) could have done for us, had we used a programming language of 1970s vintage.

How quaint.

Update: C# and Java Leak

Why is this kind of manual transformation necessary? In the end, it is because the so-called modern programming languages are so obsessed with copying popular languages that came before them (like C and C++) that they copied the bugs as well as the features.

The present an abstraction, function calling, but the abstraction leaks badly: you can't actually use it for non-trivial data sets. To me, this is kith and kin with the leaky abstraction known as primitive data types.

The only way to deal with the leaks competently is to learn about how the abstractions work and what they are abstracting. So the abstractions save us time working, but they don't save us time learning.

And all this means that paradoxically, even as we have higher and higher level programming tools with better and better abstractions, becoming a proficient programmer is getting harder and harder.

I am deeply dissappointed that programmers need to do this kind of work by hand in this day and age. We have known how to release function calls from the tyranny of a fixed stack for decades. We have known how to optimize tail calls for decades.

No matter how simple and obvious these transformations are, every time we do this by hand we are sidetracked from ading real value through innovation. Every time we do this by hand, we add some risk of defect to our software. Every time we do this by hand, we obfuscate the core expression of our intent behind the leaky abstraction.  ¶ 11:43 AM

  The Revolution will not be on YouTube

You will not be able to stay home, hacker.
You will not be able to plug in, turn on and cop out.
You will not be able to lose yourself on MySpace and skip,
Skip out for espresso during reloads,
Because the revolution will not be on YouTube.

The revolution will not be brought to you by the
Microsoft Research Group and will not star James
Gosling and Guy Steele or Homer and Marge.
The revolution will not give your startup sex appeal.
The revolution will not get rid of the bugs.
The revolution will not make your code look
better, because the revolution will not be on YouTube, Brother.

There will be no pictures of you and The Woz
dumpster diving for access codes behind a featureless building,
or trying to carry those motherboards on a stolen SegWay.
Fox News will not be able predict the winner at 8:32
or report from 29 districts.
The revolution will not be on YouTube.

There will be no pictures of pigs arresting
hackers in the instant replay.
There will be no pictures of pigs arresting
hackers in the instant replay.
There will be no pictures of ESR being
run off the Firing Range on a rail with a brand new process.
There will be no slow motion or still life of Richard
Stallman strolling through Redmond in a Free
Software tee shirt that he had been saving
For just the proper occasion.

C++, J2EE, and PL
SQL will no longer be so damned relevant, and
women will not care if Gil finally gets down with
Catherine on Crime Scene Investigation because hackers
will be in the Net looking for a brighter day.
The revolution will not be on YouTube.

There will be no highlights on the Yahoo
news and no pictures of Afghanistan
liberationists and Angelina Jolie blowing her nose.
The interfaces will not be designed by Ray Ozzie,
Bill Gates, nor written by Charles Simonyi,
Sergey Brin, Larry Page, Peter Norvig, or Kleiner Perkins
The revolution will not be on YouTube.

The revolution will not appear on ad banners
about enterprise software, enterprise integration, or enterprise sales.
You will not have to worry about a Paperclip in your
window, a Duke in your server, or the Prophet in your database.
The revolution will not go better with FUD.
The revolution will not dumb down the code that may confuse a monkey.
The revolution will put you in the driver’s seat.

The revolution will not be on YouTube, will not be on YouTube,
will not be on YouTube, will not be on YouTube.
The revolution will be no MPEG, hackers;
The revolution will be live.

Labels: passion

  ¶ 10:35 AM

  Off Topic: Bad Artists Copy. Great Artists Steal.

I just want to know what’s up with all of the retarded images the guy puts in every single one of his god damned posts.

He’s probably trying to imitate Phillip Greenspun. Unfortunately he has neither the taste nor the skill to a decent job of it.

  ¶ 1:24 PM

  Business programming simply isn't that hard Business programming simply isn't that hard, right?

Seriously, how hard can it be to wire a SQL database to some basic business rules ("employee HAS-A manager, employee.manager_id != employee.id") and add some AJAXy goodness on top like edit_in_place? Business applications are a simple pyramid with the frameworks doing the heavy lifting at the bottom and the AJAXy goodness on the top providing the sizzle that makes users say "wow." Right?



And needless to say, business applications are borrrring! Nothing about the programming is stimulating. It's all a question of managing requirements, priorities, and figuring out how to cram ninety hours worth of work into a seventy hour week to stave off the outsourcing.

But... Could there be interesting problems to seek out in business? Try a couple of these on for size:

Real time risk management

You have an online banking application. Your goal is to produce a system that can alert a customer service rep when suspicious activity is detected so that they can telephone the customer before the customer has left the web site. How can you detect suspicious activities in real time?

For an example of how this works, consider credit cards. I often make very large computer purchases, and they never bat an eyelash. But when I purchase jewelry for my wife, I get a call. A friend swears she gets the same pattern, but they called her when she purchased an iPod because she doesn't usually purchase electronics.

Your system will notice weird patterns, like a customer who never banks internationally suddenly trying to transfer money to Russia. But take heed: you don't want to be bothering the fur importer who regularly does business in the "Wild Wild East." You need to look for suspicious patterns. And you need to explain it simply enough that a customer service rep knows how to verify the transaction with the customer.

Did I meantion in real time? Visa normally calls me within a few hours. Your goal is to make the call while the customer is still on line.

Let's get our hands dirty

Concrete is funny stuff. From the moment you start mixing it, you have a limited amount of time it can sit in the mixer truck before it has to get poured. Construction sites need so much an hour, no more and no less. A cement company can't send a customer one hundred trucks simultaneously and have them idling on site, they have to arrive just as they're needed.

In major cities, traffic is a problem. So build a system that can dispatch and direct the trucks in real time. Your input will be demand from customers plus real time traffic and road construction reports, which your system will use to update a graph representation of the road network. Naturally, it will search the network in real time for apropriate paths for the trucks. They're outfitted with GPS units, so you will have real time locations for your trucks.

Don't forget to manage the return of the empty trucks to your site. Concrete is a low margin business, so operational efficiency is a competitive advantage. Do a great job, and your company could become the dominant supplier.

update: This isn't just finding the fastest way to get trucks to their destinations. That is not a simple problem, but you must also solve the flow problem: when do the trucks have to leave your yard in order to arrive at each customer's site on schedule? If a customer is ninety minutes away and traffic is building towards ruch hour, you have to leave earlier. How much earlier? Do you send two trucks by different paths at certain times and reroute one of the trucks if the other arrives ahead of schedule? When an accident suddenly clogs the freeway, is your system able to predict how long the freeway will be closed and what the effects will be on alternate routes?

To do this job well, you're going to have to have a really good model of traffic in the city you serve. Don't forget to build the baseball, football, hockey, and basketball schedules into your system, along with big ticket concerts :-)

...his lips are moving, that's how

Consider any large organization that makes big-ticket sales to walk-in prospects. Let's choose car sales. Some prospects will walk in and buy right away, others will never buy. Some will buy, but only if you follow up with a call after they've been in for a test drive.

When's the right time to call them back? If salespeople call too soon, they waste time, make the prospects feel pressured, and their quote ends up being fodder for the competition to beat and steal the business. But of they call too late, the prospect will have bought from someone else.

Build a system that learns over time when to call each prospect. For starters, you'll have access to a survey team that will call a sample of prospects that didn't buy. They will fill out a form about the prospect (age, gender, household income, zip code, car they owned, car they bought, anything else you can gather), along with when they bought.

This becomes your training corpus. Now build a system that the salespeople use to manage callbacks. Your system must suggest when to call the prospect in order to have the best chance of closing the deal.

Yes, your system must be able to train itself. Yes, your system must do "meta-training": if the zip code isn't significant but the age of their existing car is significant, your system must recommend dropping the former from the survey and promoting the latter to the top of the form. And you ought to add new questions from time to time and test them for relevance.

Oh yes, you'll have to infer distributions like the Erlang, Normal, or Binomial for purchase times: it should figure out the most appropriate model for lag between test drive and purchase.

Did I mention the scale? This system should be rolled out to every dealer in North America, and every salesperson on duty will be using it. How many dealers are there? I think you answered that question in your job interview, didn't you?

Well?

Does business programming have to suck?

Labels: popular

  ¶ 4:13 PM

  Just in time for "labour" day

I strongly oppose the false dichotomy of either you program for fun or you program for business. That split is not only artificial, but seems more like a self-serving fairy tale you can tell yourself to why your current environment is making you unhappy. Well, I'm doing Serious Business Stuff, so it has to suck. That's just the way the world spins. Bullocks.

Labels: passion

  ¶ 10:38 PM