raganwald
(This is a snapshot of my old weblog. New posts and selected republished essays can be found at raganwald.com.)

Wednesday, September 12, 2007
  We have lost control of the apparatus


I am writing to you as a fellow programmer and software developer. I write in friendship and brotherhood. My heart is heavy, and the news I impart is not good: We have lost control of the apparatus.

I know, the IT department soldiers on grimly. They lost a great battle to the PCs more than twenty years ago, and it took years of struggle with NT Domain Controllers and web proxies before control could be wrestled back from the users. In some places I hear battles are still being waged over USB ports and Bluetooth. IT was lucky to find a new ally against the users in MSFT just when their ancient supporter IBM’s star was fading.

We’ve taken care of everything
The words you hear the songs you sing
The pictures that give pleasure to your eyes
It’s one for all and all for one
We work together common sons
Never need to wonder how or why

But we programmers have lost and we must be realistic about things. The fact of the matter is this: people own their own computers, and our applications are no longer the primary way they learn how computers ought to work.

I know, I know, they stare at our work for eight, ten, or twelve hours a day. So you would think that we would set the standard for how computers ought to be. But the Good Old Days when most of users had never seen a computer before work have gone. Some of our users, fresh out of school, have already been using computers for ten years!

As if that wasn’t enough, the really bad news is, when our users go home they have this thing called the Internet. I know, IT locked that down in the office. But we can’t stop them from getting on it at home, on their mobiles, and now even on those insidious Apple iPods! And when people use the Internet, they are actually using other people’s applications.

I’m not kidding. Our users are being exposed to applications we don’t control. And it messes things up. You see, the users get exposed to other ways of doing things, ways that are more convenient for users, ways that make them more productive, and they incorrectly think we ought to do things that way for them.

computers

This business of users buying their own computers is really troublesome. For one thing, they can buy a computer in the store for a few hundred dollars that is twice as good as the warmed-up left-over IT put on their desk. Until recently, we could shrug our shoulders. We weren’t the ones that had to explain that to get an up-to-date model, they needed to fill out a twenty-seven-b-stroke-six. Which needed approval. Against budget. And then IT would order it from DELL. Who don’t have the CPUs in stock.




Blatant plug: Pre-order your Apple 16 GB iPod Touch with this affiliate link. You’ll get the most revolutionary device ever made and you’ll support ragawald, a worthy a pretty-good ok, a drivellous weblog but at least it’s entertaining. Please. And thank you, I really do appreciate the clicks!

And meanwhile, the very same users could walk across the street and buy themselves a much better PC for less money than we pay and take it home the same day. And until now, we didn’t worry about it. We’re the programmers.

But now it’s a problem. Here’s the thing: those PCs they buy at home? The ones that are two, three, or even five times better than the ones on their desk? With huge drives, and lots of memory? The thing is, they run much richer applications than the warmed-over TTY stuff we’ve been feeding them under the guise of “being hip to the Internet revolution.”

For example, my Mother uses Skype to talk to her friends. She thinks it’s normal to see all of your voice mail messages in a list on the screen. If I tried to give her a CRM application for managing contacts, the very first question she would ask would be, “Why can’t I listen to all of the voice mails from that contact in the application?”

Do you think she would have patience for my explanation that the company’s phone systems are complex and proprietary and that we can’t install Asterix just for her? She would grab me by the ear and drag me to my desk to get cracking on it!

You laugh, but users are used to a lot more functionality than a web page with a form on it these days, and if we don’t give it to them, the noise from all the whining is going to drive us insane.

I tried telling them that databases were not designed for multi-megabyte Word files and PowerPoint presentations, and they just goggle at me like I’m a circus freak and ask me why I’m using a database if it doesn’t do what they want for them?

Here’s another thing. How many monitors do your users have? One, right? There’s no point in giving them more, because we build our applications with session management that basically goes insane if they try to open two windows at once, right? Well guess what? Users have multiple monitors at home for games, and they want them at work.

And when they have them at work and you explain that no, they can’t be in the middle of doing something for XYZCorp in one window, and open a new window when a call comes in from ABCLimited, because that screws up the session, they are going to bitch and moan, because they can do two things at once in their mail application and their social network and their game. Sucks to be us.

And don’t get me started about had drives. Would you believe that users now expect—as if anyone gave them permission to have expectations—what was I saying? Oh yes, users now expect that we should store everything and anything. That’s right, they think we should be able to handle arbitrarily large text fields, with styling, and pictures, and even sound or video, all stashed away where they can find it again. They want to be able to store everything to do with an account or a customer or a project or whatever right in our applications.

I tried telling them that databases were not designed for multi-megabyte Word files and PowerPoint presentations, and they just goggle at me like I’m a circus freak and ask me why I’m using a database if it doesn’t do what they want for them? I try to tell them that we don’t have a budget for buying bigger drives, and the poor, deluded fools pull out their credit cards and offer to buy a terabyte drive over the Internet for less money than our vendors charge us per phone call.

I tell you, users refuse to sit and be trained like puppies. And the bottom line is this: we can’t keep putting HTML lipstick on a 1960s pig forever. We have to get out of the office and look at what a modern PC looks like—drive, speed, RAM, monitors, everything—and write applications that can take advantage of it.

fields

You know how all of our applications have a first-name and a last-name? This has worked for decades, you would think people would understand the value of standards, of consistency. But out there in the Internet, where there is no Adult Supervision, some of those applications have just one field for a name. You put spaces in it to separate first and last, just like writing a letter, I guess.

When I told one of our users, a business analyst, that using just one field for the name meant a huge amount of work for programmers, she actually asked me what our job was, if not to do the work that makes users productive.

Please, stop laughing. I had the same reaction. How could that possibly work? But the rogues who build that kind of heresy put in a lot of clever little rules and things so that the single field can handle case like Braithwaite, Reginald or Geroge Smithers, Esq. or even Doctor Wu properly.

When I told one of our users, a business analyst, that using just one field for the name meant a huge amount of work for programmers, she actually asked me what our job was, if not to do the work that makes users productive. I’m very much afraid that things are out of hand. I tried to explain how our database schema works, and so on, but she impatiently insisted that it was our job to make things work.

She then started lecturing me—lecturing ME!—about copy and paste, of all things. She said that it should be convenient to copy and paste between our application and other applications like Word, Excel, and even Outlook. With separate fields, you need to do multiple steps to copy a name between our applications and a letter.

I felt betrayed by Microsoft. Whatever happened to the days when people used just one application, and if they needed another we would give them an export routine? Now she wants to be able to copy a name out of an email and paste it into our application without carefully selecting the last name, first name, and honorific separately.

When I left her office, she was mumbling something about SaaS or some such. I don’t remember what she meant, but “sass” just about sums up her attitude.

If she was just one lone uppity user, I could handle it. But they’re popping up like toadstools. Just the other day another user was asking why he couldn’t paste an address as one field, including zip code. I told him I didn’t have time to explain how database columns worked, but again he muttered something disrespectful and later I saw him hefting his swingline rather menacingly while looking at our application on his screen.

Brothers and sisters, I know this is hard to take, but we’re losing this war. Our DBA cousins have brainwashed Corporate into believing that they are the custodians of the data, and of the sacred Stored Procedure that Controlleth Access. We have very little choice about how things work.

But still, the users expect us to make applications every bit as useful for them as the applications they use on the Internet, and I fear that they will rise up and revolt very soon if we don’t find a way to make the database invisible and make user applications conform to these horribly user-centric heresies.

I know, I see the pitchforks in your hands, and your desire to maul, hang, and burn the messenger is understandable. But that won’t fix things, so please put them down, ok?

stories

And don’t give me that “User Stories” flim-flam, please. I practically invented bamboozling users into doing what we want by pretending to put them in charge of an Agile Process. Agile Process indeed, anybody ought to know that if it’s a Process, it sure as heck can’t be Agile.

So you think you can do what we’ve always done, hunh? When they complain about copy and paste, write it up as a story, put it in the backlog, and then—look at that—there’s always a higher priority story to do. There’s always some new functionality that offers a greater ROI than polishing an old feature.

Well, sister, where that goes wrong is that this isn’t polish: it’s what our users expect as basic functionality. You might think it is new and improved and doesn’t add value, but what our users think is that our applications are old and broken and waste their time.

So while on paper a new feature is more important than making paste work, in practice it looks like we build software that slows the organization down.

So save your breath and stop using Agile as an excuse for slapping the crudest crap together and putting fast in front of finished.

search

You would things couldn’t get any worse. But they are worse, much worse. I’ll just say one word. Google. Those bastards are practically the home page of the Internet. Which means, to a close approximation, they are the most popular application in the world.




The authors of Prototype and Scriptaculous in Action are the people who brought you the incredible Prototype and script.aculo.us Javascript libraries. This book explains how to use them to build reusable, literate Javascript, how to build dynamic, Web 2.0 applications, and best of all, how to write web applications without tearing your hair out in frustration with Javascript.

With this book and these libraries, you’ll learn how to write better Javascript in a Lisp-like functional style, and as a bonus you’ll also learn how to write better Javascript in a conventional OO style.

And what have they taught our users? Full-text search wins. Please, don’t lecture me, we had this discussion way back when we talked about fields. Users know how to use Google. If you give them a search page with a field for searching the account number and a field for searching the SSN and a field for searching the zip code and a field for searching the phone number, they want to know why they can’t just type 4165558734 and find Reg by phone number? (And right after we make that work for them, those greedy and ungrateful sods’ll want to type (416) 555-8734 and have it work too. Bastards.)

I have tried explaining that there’s an ambiguity if an account number is also 4165558734. But those damn users just give me that “Boy, you are stupid” look that made Samuel Jackson famous. They think we should just show them what we find and let them sort it out. They’re idiots, obviously, but they’re our idiots and I’m pretty sure that if we fire them all we’ll have to clean our own desks out the following day.

They don’t even get that search results should always show stuff from the same table. Would you believe, if they type a phone number, they want us to search companies and persons. They have no respect for our careful husbanding of hardware resources. The profligate spendthrifts think that just because they have a two gigahertz PC at home that can search their entire hard drive for a phone number as fast as they can type it—thanks again, Google—we should make search as fast and as easy to use in our applications.

We can use tools like Nutch and what-not for full-text search. But users want to search everything, everywhere, just like Google. And try as we might to get them to use Sharepoint, they just deride it as a heap of junk. We are going to wind up ceding control of our data to Google sooner or later. I hate to be the one to tell you this, but you might as well hear it from a friend:

You need to start coding your applications so that an external search engine can search them. That’s right, you need to work with a desktop search tool and a network search tool so that people can type 4165558734 and see everything, mail, word docs, and records in your database, in one place.

It’s the future. A miserable, groveling future where our applications work for the users instead of the users working for our applications, but it’s our future.

Suck it up and roll with it.


Psst! Are you Smart? Do you Get Things Done? Do you want to work in New York City using Rails and Java? Mobile Commons is hiring developers!

Labels:

 

Comments on “We have lost control of the apparatus:
And here I was, just about to ask the IT department at my college to install Scheme48 and to update Emacs. Thank god I didn't click send!
 
Are you pitching for CouchDB in some wierd attractive way? I am all for it man. Best post of the day :D
 
Only that even Google uses Oracle and MySQL.

Nice try, it waſn’t the firſt time people try to kill order in IT nor will it be the laſt.
 
computer users should get a license, just like a car license.
 
The idea behind this post seems to be that sometime in the near future a lot of software will suck less which is a fucking retarded idea.
 
The idea behind this post seems to be that sometime in the near future a lot of software will suck less which is a fucking retarded idea.

Welcome to my dream. As they say, think globally, act locally.
 
computer users should get a license, just like a car license.

I'll play along. What are the requirements for licensing?
 
Fantastic stuff. Hopefully there will be a place in CIA rendition facilities for the current IT types after the revolution.
 
Google sells the Google Mini. That way you can get the benefit of google search without opening your intranet site up to the world.
 
Sean:

I was thinking of the Google Appliance. But it can't do much with your proprietary application data unless you code for the 'bot.

In the simplest case, you can build REST-ful applications that expose an index to the Intranet Spider.

It can then index the data in your application against stable URLS. You then code your log on process such that when you click an URL in the search results, you sign in and then are taken to the page you requested.

This requires a lot of thought about what is exposed in search previews and so on, but designing for full text search is now part of the gig.
 
I would go even farther. Not only are they going to expect full text search, they are going to expect matching, collation and correlation based on that full text search in a fuzzy matching sort of way.

This whole thing comes out of actually looking at what people want to do instead of what the technology allows us to do. The problem is that the vast majority of programmers don't have the skills to implement some of the things that would make corporate programs easier to use. Things like:
- full text search
- pattern matching
- probability and producing partial results based on it
- intelligent data recombination
- evolutionary processes
- multiprocessor programming (since we need that horse power in order to do the above things quickly)

I would guess that most of us weren't trained to develop that kind of application, nor have we been practicing it in our drone CRUD corporate jobs.

Actually, this might be a good start-up opportunity.
 
computer users should get a license, just like a car license.

This thought has been pondering me for the last couple of years. Before committing to programming full-time, I was a freelancer and my full-time job was system administration. To make the pain go further, we had about 12 lawyers and law consultants on the network, so I have been exposed to the equivalent of 120 arrogant, braindead but talkative idiots. Daily.

Normally, I would be against any "license" that would disallow anyone to have free access to information. Like, you don't need a license to read the newspaper.

However, when you read the newspaper you are at least expected to be able to read. Those who read newspapers also understand a couple of basic facts about mass-media -- like biased articles, hiding some facts and so on -- so that they can make a clear picture by themselves.

I've seen this happening way too often with computer users. They don't know how to use X. They want their report, but they don't want to learn how to use X either, and when it doesn't work, X is difficult to use, buggy, unstable, bloated or bad, it's not them who don't want to spare one hour of their time to learn how to use it.

This is a reason for concern when it starts affecting other users. It happened to me with a macro virus. It was a "warm" one, not yet in the virus signatures, and when the guy saw his Excel sheet was not very well-behaved, he tried it on every computer on the network instead of asking me. Needless to say, it ended up in a huge mess.

As far as I'm concerned, users should get a license for working with a piece of software in corporate environment. If they can't use a piece of software, they can't work there. How the hell are you expected to work as a taxi driver when you can't drive?

The only issue is with people actually proving they can use their tools. I've seen countless people claiming they can use Word, but barely knew anything except how to type. But that can be arranged with standardization :-P.

I know, I'm antisocial and the BOFH.
 
Blog post of the year. Seriously. You have a gift, Reg.
 
IT software and hardware just isn't going to ever catch up to the standards that users are used to with their desktop PCs at home (especially if they're using Mac OS X and are used to the quality of Mac applications). The simple reason is: the compeitition just isn't as strong in the IT world.

If you get hired by Acme company and hate the system that IT throws at you - so what? You're going to leave your job just because the system sucks? No. You just take it, and then move along and do your job (while grunting with tones of displeasure every now and again).

If The Management decides to buy a certain system that costs A Lot Of Money and you complain about how the address field works - do you expect The Management to go shopping for another system? No - they're commited already.

At home with the desktop, its a completely different landscape. If you don't like a certain solution, there are lots of other competing products and solutions that you can go for.

This competition, coupled with the relative low cost of the products (which makes switching from one product to another product relatively easier) drives up the quality of the products.

I used to work for a company that created custom hardware/software solutions to government agencies. The UI of the product is absolutely atrocious - but so what? The employees of the government agencies can't complain - it all goes to deaf ears. Their employer has literally paid millions for the system already, and they're not going to pay anymore money "just" for UI changes.

[Note: This is a reproduction of a comment I posted to programming.reddit.com]
 
I whole-heartedly support the goal of making software as usable as possible, but there are two limitations that your satirical piece ignores: cost and purpose-specific optimization.

Last I heard, Google had a bit of cash, with concomitant hardware and brain-power to throw at its problems; not every customer would be willing to pony that up just to get Google-search of their very large data.

Also, the use case that Google is solving with their flag-ship app is fairly simple: partition an unmanageable amount of data down to a siftable pile of data.

Again, not all purposes will tolerate this kind of fuzzy output. Some problems call for fairly exact data as output.

If we get better at focusing in on both these factors and analyzing them for our specific problems, I think we CAN build better software, but just saying "Google does it" or "Apple does it" so I should be able to too will only get us in trouble.
 
Marc:

Full-text search of corporate data is ridiculously easy. For example, if you expose your database records REST-fully with stable URLs, you can then use an off-the-shelf search appliance like the Google Mini to do the rest.

But of course, if some companies wake up in the morning thinking it is a very difficult problem, and they insist on building instead of buying, there will always be an excuse why it cannot be done.
 

For example, if you expose your database records REST-fully with stable URLs


I think this is where the complexity is hiding.

This presupposes that your data groups straight-forwardly into suitably REST-ful groupings that are universally useful.

The web is pre-partitioned for Google in this regard, but figuring out the variations that make sense for your enterprise can be quite challenging.

Not impossibly hard, of course, but perhaps involving more effort than the organization is willing to spend for the functionality.

Also consider the plight of the ISV. He can certainly make the pages of the app spider-able, but I doubt too many ISVs are going to want to have to distribute a Google Mini (or equivalent thereof) with each copy of the app.
 
Requiring a license to use computers or the internet is like requiring a license to drink water.
 
Hehehe...

Great stuff Reginald :-)

Love to read something as insightful as this :)
 
This is EXACTLY what the smug IT "Nick Burns" people need to hear! but they won't that's why they call it arrogance. This was very funny, Thanks!
 
I think this post is wrong. See, whenever I have requests like that, first I issue a quote. If you actually have to pay for it, it roots out half of them. Then I ask them to assign priority. It will be something like 4 or 5 in the 1-5 scale. So I'll stick it at the end of the list of bugs and change requests. And I'm pretty sure I'll never get there as there will always be more and more priority 1-2 requests. Or sometimes bugs. And if I do get there sometime in the far future, what's wrong with that? It just means making more money of a client whose system is already working quite well.
 
Please tell me I'm not the only one reading that's old enough to notice the Rush reference...

Fine post BTW.
 
Spot on. A fun read. Welcome to the Temples of Syrinx
 
Hey Now Reg,
Just saw this from a link on Coding Horror. Nice post very interesting.
Thx,
Catto
 
That "one search field" is a great idea. Maybe with advanced operators like cust:42 when you really need to tell the system "I want customer #42, not invoice #42 or 42 Peachtree Ave."

As odd as it sounds to admit, with the skill and experience I think I have, this could be paradigm shifting. Thank you.
 
First of all, great post Reg.

Donkey, regarding your proposal that computer users be licensed, I respectfully disagree with the second part of your analogy:

However, when you read the newspaper you are at least expected to be able to read. Those who read newspapers also understand a couple of basic facts about mass-media -- like biased articles, hiding some facts and so on -- so that they can make a clear picture by themselves.

Newspaper readers definitely have to be able to read, but I believe you are being over-generous in assuming everyone who reads possesses the ability to think critically about whatever is on the page.

This is the same situation with most computer users. They don't know how computers work, and for the most part, they don't care. The guy who attempted to run his infected spreadsheet on several computers is no different than the person who repeatedly clicks their mouse when things aren't happening fast enough on the screen. Come to think of it, this is basically the same behavior as the person who repeatedly clicks the elevator button.

My point is that the area where humans interact with technology is a huge landmine of ambiguity. There will always be this desire for a given device to "just do what I want." And, unfortunately/fortunately, it's up to the people who design the things we use to make that a reality.
 




<< Home
Reg Braithwaite


Recent Writing
Homoiconic Technical Writing / raganwald.posterous.com

Books
What I‘ve Learned From Failure / Kestrels, Quirky Birds, and Hopeless Egocentricity

Share
rewrite_rails / andand / unfold.rb / string_to_proc.rb / dsl_and_let.rb / comprehension.rb / lazy_lists.rb

Beauty
IS-STRICTLY-EQUIVALENT-TO-A / Spaghetti-Western Coding / Golf is a good program spoiled / Programming conventions as signals / Not all functions should be object methods

The Not So Big Software Design / Writing programs for people to read / Why Why Functional Programming Matters Matters / But Y would I want to do a thing like this?

Work
The single most important thing you must do to improve your programming career / The Naïve Approach to Hiring People / No Disrespect / Take control of your interview / Three tips for getting a job through a recruiter / My favourite interview question

Management
Exception Handling in Software Development / What if powerful languages and idioms only work for small teams? / Bricks / Which theory fits the evidence? / Still failing, still learning / What I’ve learned from failure

Notation
The unary ampersand in Ruby / (1..100).inject(&:+) / The challenge of teaching yourself a programming language / The significance of the meta-circular interpreter / Block-Structured Javascript / Haskell, Ruby and Infinity / Closures and Higher-Order Functions

Opinion
Why Apple is more expensive than Amazon / Why we are the biggest obstacles to our own growth / Is software the documentation of business process mistakes? / We have lost control of the apparatus / What I’ve Learned From Sales I, II, III

Whimsey
The Narcissism of Small Code Differences / Billy Martin’s Technique for Managing his Manager / Three stories about The Tao / Programming Language Stories / Why You Need a Degree to Work For BigCo

History
06/04 / 07/04 / 08/04 / 09/04 / 10/04 / 11/04 / 12/04 / 01/05 / 02/05 / 03/05 / 04/05 / 06/05 / 07/05 / 08/05 / 09/05 / 10/05 / 11/05 / 01/06 / 02/06 / 03/06 / 04/06 / 05/06 / 06/06 / 07/06 / 08/06 / 09/06 / 10/06 / 11/06 / 12/06 / 01/07 / 02/07 / 03/07 / 04/07 / 05/07 / 06/07 / 07/07 / 08/07 / 09/07 / 10/07 / 11/07 / 12/07 / 01/08 / 02/08 / 03/08 / 04/08 / 05/08 / 06/08 / 07/08 /