Wil Shipley on writing Not So Big Applications Actually, he was talking about not making your code more flexible than needed. And he gives a really great argument for writing the briefest possible code and then only making it faster, more robust, more featured, or anything else after you establish a need by testing it.

Here’s his take on writing classes for libraries when you ought to be writing application software:

If you find yourself writing a class for your “library,” then:

• You’re not writing your application, which is where you make your money,


• You’re writing something that you’re hoping Apple¹ will someday replace, which is a sucker’s game,


• You’re writing code you are going to have to test SEPARATELY from your app, because BY DEFINITION you’ve added functionality you didn’t need,


• You’re never going to really know which methods in your library work and which ones don’t (eg, which ones are used in shipping programs) because you don’t have user base that a company like Apple² does (and witness how buggy even their under-used frameworks are),


• You’re writing code that is going to need documenting (or some other way to comprehend it), so you’re requiring yourself and everyone at your company to understand not JUST all of Apple’s APIs³ (which are, at least, SOMETIMES documented) but also yours, and, possibly worst of all,


• You are attempting to predict how your application’s needs will change in the future, and spending time NOW on your guess, instead of shipping the damn application, getting feedback, and THEN making changes.

---

1. Or Apache. Or Microsoft. Or BEA. Or Linus Torvalds. Or whomever. The point is, if your existing OS or framework or language or tool doesn’t do it, don’t extend it unless you really need it, preferably in two different places.
2. Or Apache. Or Microsoft. Or BEA. Or Free Software. or…
3. See one and two above.

  ¶ 5:37 PM

  Icebreakers One of the problems with all interviews is that the formality and importance of the meeting forces people into rote, scripted behaviour. This makes it very difficult to determine whether someone is a good fit.

If someone tells you they are seeking “an opportunity to apply their dynamic skills and team-orientation to furthering the company’s mission in an upwardly mobile career position,” are they a snot-nosed corporate drone? Or are they saying what they think interviewers want to hear because they have invested far more time improving their development skills than time improving their interviewing skills?

Your job as an interviewer is to talk to the person, to get them talking about who they really are, not trying to guess what you might want to hear. “Icebreakers” are interview questions designed to put you both at ease and break out of the scripted, stilted talk that results in bad hires and missing good hires.

(That being said, I do not suggest that they be used to determine whether someone is a good candidate or not: with all of these questions, there really is no “right” answer, and if you try to read too much into the answers, you run the risk of evaluating the interviewee on the basis of whether they are your clone.^{1, 2})

So don’t go overboard: an interview should not be dominated by questions like these, any more than it should be dominated by computational complexity questions, algorithm discussions, design explorations, review of past achievements, or any other single factor.

I suggest you use one or two of these in an interview as a way of loosening things up. There’s anecdotal evidence that even very good candidates may react poorly to the pressure of a job interview, generating “false negatives.” Anything that lowers the pressure raises the chance of identifying great people for your team.

Bootstrapping

• What was the first computer you used? What was the first computer you programmed?


• What was your first programming language? What did you like about programming at the time?


• How did you get into this industry? Why? Do you still think that reason is/those reasons are important to you?

Programs

• What’s the most incredible program you’ve ever seen? Why?


• What’s the best software you’ve ever written? Why? Can I see it/try it?


• If you had a million dollars (cliché, but still) and you already did everything else you wanted to do, and now you could write any program you wanted, what would it be?

People

• Who’s the best programmer you’ve ever worked with? What made them great? What did you learn from them?


• What’s the best team you’ve ever seen? Why were they the best: Quality? Speed? Awesome Software? What’s the most important reason for their greatness?


• What’s the best company in the world? Why?

Languages

• What’s your favourite programming language (it doesn’t have to be work-related)? Why?


• Of all the languages you’ve used seriously, what is your least favourite? Why?


• I see from your résumé that you have experience with both Scala and SNOBOL (Pick any two, obviously). If you could pick any one feature from Scala and add it to SNOBOL, what would it be? if you could pick any one feature from SNOBOL and add it to Scala, what would it be?

Sharpening the Saw

• What’s the most recent serious thing you’ve learned? Why is it important?


• What’s the best way for you to learn something new? Formal education? Books? Hands on experimentation? Mentoring? A combination?


• Who’s the best teacher/mentor you’ve ever had? What made them great?

Final words

One final tip about icebreakers: Try sharing your own answers to the questions. You might not want to answer them first: some candidates will feel pressured to echo your responses, but there’s a lot of value in turning a question into a conversation. Don’t turn the conversation into a monologue: this isn’t your interview, and these are icebreakers, not some sort of secret mechanism for identifying talent.

Did you ever take that test yourself? Deckard?

As soon as you feel the conversation relaxing, move on to something important, be it asking the candidate to solve a problem or working through their experience in a more structured manner. Return to an icebreaker if thing stiffen up too much. Learn a little. Share a little. Laugh.

Remember, you are looking for someone who can make or break your team. You might want to talk with them first :-)

---

1. In my own case, I have spent a lot of time thinking about tools and languages. So I am always tempted to like programmers who have also spent a lot of time thinking about tools and languages. However, someone who is not particularly reflective about tools and languages might be a great contributor in certain roles that don’t have a heavy “sharpen the saw” emphasis. Or, they may be delighted to learn some new tricks if placed in a supportive environment.
   
   So the lesson for me is that if someone says the kind of thing I like to hear, great. If they don’t, I need to try another icebreaker, not evaluate them based on their response.


2. Here’s a dissenting view

Labels: jobs

  ¶ 10:20 AM

  A caveat about job interviewing

Never hire an engineer on the basis of questions any idiot could answer.

If you ignore the above maxim, the result of your negligence may one day wind up becoming your boss.

(Follows: A caveat about job interviews)

Labels: jobs

  ¶ 5:13 PM

  A caveat about job interviews

If you take a job with a company that asks you questions any idiot could answer, sooner or later they’re going to put you on a project with some idiot who answered them.

(Follow up: A caveat about job interviewing)

Labels: jobs

  ¶ 9:36 PM

  We really have a lot in common Tell me what you like about static typing, and I’ll tell you what I like about functional programming.

Tell me what you like about automated refactoring in your IDE, and I’ll tell you what I like about referential transparency.

Tell me what you like about imperative programming, and I’ll tell you what I like about dynamic typing.

Tell me what you like about objects that execute methods, and I’ll tell you what I like about components that execute domain-specific languages.

Tell me what you like about configuring your programs with XML, and I’ll tell you what I like about writing literate software.

Tell me what you like about Inversion of Control, and I’ll tell you what I like about first-class functions.

And hey, if you tell me why dynamically extending and changing classes bothers you, I’ll tell you why mutable local variables bother me.

See? We have a lot in common. Is there anything else I’ve missed?  ¶ 10:22 PM

  Moonlighting and Ruff Riders Hasan Luongo has a nice post about The Dangers of Moonlighting. In his case, the problem is that he was fully employed at the time he founded his business. There’s some nit-picking to be done about how much of his business activities occurred during “work hours” and how many did not, and so forth, but let’s hand wave all of that.

The issue is this: employers want every last penny they can squeeze out of you. That’s how our capitalist system works: The corporation’s business is to maximize return on capital, and you are “human capital.” No surprise there.

Some people have pointed out that you ought to obtain a release from your employer before starting a business on the side. I think this is excellent advice. I will go further: before starting employment with a firm, ask them how they would handle the “hypothetical” case of you coming up with an idea.

Some employers will decide not to hire you just because you asked this question. What does that say? Some employers will tell you about how they make room for entrepreneurship within the company, and will back it up with stories about how some of their best new lines of businesses were created by individuals with ideas, and not by management committees. What does that say? And some will just shrug their shoulders.

I have two actual experiences I can share. The first was when I was employed by KL Group. I was working on the JProbe line of Java development tools. That line of business was created because they were working on Java components (a business that didn’t really pan out), and a fellow named Richard Fogel noticed that there were no decent Java performance profilers.

So he wrote one. And so the company ran with the idea. Jumping straight to the happy ending, Quest bought the company largely to get the product line and the development team that produced it.

Another time I was interviewing with a firm I shall call the “Ruff Riders” after a tee shirt I once saw wrapped tightly around the very muscular chest of a diver on vacation in Roatan. Any ways, I was fiddling about with my ongoing (and so far futile) attempts to predict the outcome of software development projects based on empirical evidence.

So I mentioned this in the interview and asked how the rights to things I invented in my own time and on my own computers would work. The president was direct. Ruff Riders would own it all. And just to be clear, their draconian policy was not intended to steal my ideas, it was to discourage their employees from thinking about anything except furthering their business. His perspective was that when you became a Rider, you dedicated yourself to Ruffness 24/7/365.

I can’t say I think he was being unreasonable, but I can tell you that as I was driving away from his office, I placed a call to my Estate Agent and put my house up for sale. There was absolutely no way that I ever wanted to be in a position where I had no alternative but to give my every creative thought away in exchange for a fixed salary.  ¶ 3:39 PM

  Scott Hanselman's Diabetes Walk 2007

Blogger Scott Hanselman is raising $50,000 to fight diabetes. Well, he only needs $49,980 because I just gave $20.00.

Please read his story, and then donate.

Thank you!  ¶ 4:05 PM

  The Not So Big Software Design A little less than a decade ago, Sarah Susanka wrote a very provocative book, The Not So Big House. I found out about it one evening while watching PBS. I switched to Channel 17, and there was an interview with her in progress. My partner and I were enthralled. We had been struggling to purchase a new home from “tract” or “subdivision” builders, and we simply couldn’t find anything that spoke to us. In a few short moments, Sarah articulated exactly why we were so frustrated by the builders.

Sarah spoke about a culture of building homes to impress. Of cookie-cutter McMansions, where everything was big, but nothing was warm and inviting. I can give you a very practical example of this syndrome: drive through any subdivision these days. Measure the space between the houses. It’s pitifully small! The reason is that the builders are building the largest homes the possibly can on each lot.

That means that very little light can get into the sides of the houses, and you see this when you look at the floor plans: everything is organized around large picture windows in the front and rear of the house. And no wonder: there is nothing to see to either side except the brick or siding of your neighbour’s house just a few feet away.

Sarah’s solution to the problems of poorly designed homes is to take a given budget, and instead of buying the largest home for that price, purchase a smaller home but invest in features and details that customize the home for your needs.

Applying this “not so big” thinking to the problem of houses squeezed together in a subdivision, you can try to place a smaller home on the lot and invest the construction savings in windows on three sides instead of having nearly all the windows on just the front and the back.

Everything in Sarah’s philosophy is driven by the owners’ actual lifestyle, not some imagined lifestyle that never comes to pass. So… unless you are a competitive dancer, Sarah is not going to design an impressive ballroom for your home. On a more practical note, she spends quite a bit of time discussing the merits of doing away with the formal dining room.

Very few people want to have company over for dinner in their kitchen, so Sarah often designs an eating area separated from the kitchen by sliding doors. You have an eat-in for everyday dining and a formal spot when you need to throw a dinner party.

This kind of thing is not free: sliding doors are expensive, and that’s why very few “tract” homes have them, even very expensive tract homes. But if you want a home that works, you make the choice to have fewer square feet but make those square feet work for you every day.

Lemons

The problem with tract houses can be summed up in a phrase: the builders are selling you lemons. I hope Bruce Schneier forgives me quoting wholesale from his excellent article about security problems:

In 1970, American economist George Akerlof wrote a paper called “The Market for Lemons,” which established asymmetrical information theory. He eventually won a Nobel Prize for his work, which looks at markets where the seller knows a lot more about the product than the buyer.

Akerlof illustrated his ideas with a used car market. A used car market includes both good cars and lousy ones (lemons). The seller knows which is which, but the buyer can’t tell the difference — at least until he’s made his purchase. I’ll spare you the math, but what ends up happening is that the buyer bases his purchase price on the value of a used car of average quality.

This means that the best cars don’t get sold; their prices are too high. Which means that the owners of these best cars don’t put their cars on the market. And then this starts spiraling. The removal of the good cars from the market reduces the average price buyers are willing to pay, and then the very good cars no longer sell, and disappear from the market. And then the good cars, and so on until only the lemons are left.

In a market where the seller has more information about the product than the buyer, bad products can drive the good ones out of the market.

Now don’t think about the house builders as bad people trying to sell you a bad house.

In the case of new homes, the bottom line is that if most buyers of homes cannot tell the difference between a home that will suit their lifestyle and one that will not, the builder has very little choice but to offer homes that offer the superficial features (like gross square footage) that will sway people into buying.

The entire problem is centered around the fact that the average home buyer is unable to tell the difference between a good house and a bad house, so they settle for superficial distinctions.

Building Better, Not Buzzwordier

Does this sound familiar? There are two obvious blog posts here, one about the fact that the average employer cannot tell the difference between good programmers and bad. The other about the average buyer of custom software. Since someone has already noted the similarity between used cars and programmers, let’s look at the similarity between houses and custom software projects.

I recently had a chance to review an architecture design for a custom software project. The designer was given a telephone book sized specification written by the client and asked to put together a high-level architecture plan.

Now right away, I want to say this is a tough spot to be in: it’s all well and good to talk about customizing things for clients, but you really need to talk to them if you want a shot at doing a good job. Whether you are a proponent of Agile or of BDUF, I think you will agree that no amount of documentation can replace communication, ever.

Any ways, I saw right way that the document was… What is the phrase?… Oh yes, as Richard Feynman would say, it was no damn good. It was a lemon.

It is very poor form to criticize this person’s work after establishing that they had very little chance of doing well. So this is my disclaimer: I am writing to talk about why these circumstances conspire to produce a lemon of an architecture plan. Got it? Good person, bad circumstances.

So what was wrong with the design? Quite simply, there was no client in it.

There was a technology stack, there were buzzwords, there was a very popular programming language, there even were some quasi-open source components. Lots to like, and difficult to criticize. Think about how such a conversation might go between two lemon sellers: “Why are you specifying Java, C# is the best language!” Or perhaps, “BizTalk?!?! No way, you want open standards, not lock-in!”

But there was no client in it. Tract houses are designed for the features that all inexperienced clients want to buy, making owners and tract houses interchangeable.¹

And this design articulated the features that inexperienced clients (and inexperienced software designers) like to think about. These kinds of designs and clients are equally interchangeable.

Where is the client?

What I saw was a design with such broad strokes (“Database,” “ORM,” “Workflow,” “Templates”) that it could have been presented to hundreds of different clients without change. Now obviously, hundreds of clients need databases and what-not. So the design wasn’t wrong in the sense that none of the decisions it articulated were bad decisions.

But let’s stop for a moment and compare that architecture design to a home design. Imagine you hire an architect. They put together a preliminary design, a kind of sketch, for your consideration. They call you into their office for a presentation. The lights are lowered and the presentation begins.

The Consulting Engineer speaks. “Concrete foundation!” She says. Next slide. “Wood frame.” Next slide. “Brick exterior.” You are getting the same treatment as the clients looking at a design that goes into detail about the technology stack (“Java,” “Oracle,” “Hibernate,” “BizTalk”).

Or maybe the Consulting Engineer sits down and the Junior Architect takes over “Four bedrooms. Maybe five.” Next slide. “En suite bathroom.” Next slide. These things are all decisions that must be made, but they have little or no connection to the client’s needs.

Isn't it obvious that a well-designed home with vinyl siding is a better choice than a poorly designed home clad in brick? But in the absence of better information, clients are forced to pick the brick over siding instead of choosing whether to have a formal dining room or whether to separate the eat-in from the kitchen with sliding doors.

And obviously two parents and three children want at least four bedrooms. But there is no talk of whether the master bedroom is on the main floor, or whether the architect has chosen to place the play room adjacent to the children’s bedrooms upstairs where the children can play without disturbing the adults or whether to place it downstairs where it can be seen from the kitchen.

It’s easy to see that the exterior of the house and the number of bedrooms are superficialities: To get at the important details, you have to ask a simple question: How is this different than what every other client is getting?

The really important architectural decisions are the ones that address how each client is unique, not what all clients have in common.

Better Software Architecture

Designing software is not easy. And truthfully, our environment makes it difficult, because our clients are not knowledgeable enough to distinguish the not-so-big applications (“Domain-specific languages,” “Agile development”) from the McMansion applications (“Industry-standard platform!” “Detailed Specifications!!”)

In the context of software developed for clients, good software architecture is, at its heart, architecture that is specific, not general. It isn’t all high-level abstractions that could apply to any client, it’s specific choices that address specific problems for that specific client.

It is easy to say that the cure for the general architecture is to add detail. If the lemon design requires five slides, flesh the design out into fifteen slides. If that isn’t specific enough, triple the length again and go to forty-five slides.

This would be the equivalent of taking the builder’s floor plan of a McMansion and filling in the exact dimensions. Or perhaps selecting the kitchen finishes and whether the shower fixture will be pressure-balanced or not.

Adding detail makes a design more specific, but it only makes it specific for a client if the choices expressed address the most important needs of the client. Naturally every new home buyer has a preference with respect to kitchen cabinetry. But does expressing that decision really reflect a deep understanding of the client’s lifestyle?

When you look at a high-level design for a client, it should be obvious at a glance that the design addresses specific needs. Someone who doesn’t know the client may need an explanation—if you looked at a home design with the master bedroom on the ground floor, would you know instantly that the clients have teen-aged children?—but if you know a little something about the client, you ought to be able to literally see the client in the design.

This should be true at each level of detail. It should never be necessary to drill down into the details to understand how the design solves the client’s specific problems. If you are looking at a five-slide high-level design, it should convey the one or two most important ways the software will solve the client’s most important and pervasive needs.

When you drill down to detail requiring forty-five slides, you should see solutions to problems that are a ninth as important as the solutions evident in the five-slide presentation.

Like Sarah’s approach, this type of design has a cost. When you only have five slides, using one slide to address a client’s specific problem means foregoing a slide full of buzzwords that impress the less-knowledgeable client.

I wish I could tell you that this will outshine the McMansion presentation from the big consulting firm full of buzzwords and no attention to the client. But it will not: most clients will buy the idea that their needs are not-so-unique, and if what they need doesn’t fit the architecture, they must change to adopt “best practices.”

But for the serious practitioner, good design is more important than technology stacks and buzzwords. More important than size and impressiveness. It may be “not so big.”

But it is better.

---

1. In all my looking at tract houses, I saw just two departures from the norm: one builder offered a two storey model with a master bedroom on the ground floor so that when they children moved out and the owners aged they wouldn’t need to go upstairs (bungalows solve that problem as well, but you need a much larger lot). Another style, “New Urbanism,” put garages back behind the house where they belong. But 99% of them were just variations on the same theme.

Labels: popular

  ¶ 5:18 PM

  A Programmer's Story To reduce a wonderful movie to a few lines, A Soldier’s Story is a tale about a murder investigation, set on a US Army Base in the South near the end of WWI. A Negro (this is the historically accurate term used in the film) Sergeant is murdered while returning home from leave.

The prime suspects are two white MPs. Howard Rollins, Jr. is masterful as the Negro investigator sent by the army from Washington to investigate the case. Rollins is educated, a senior officer, and is the classic “stranger coming to town.”

Howard’s performance, as wonderful as it is, is eclipsed by Adolph Caesar’s Oscar-nominated performance as the victim Sergeant Waters, seen through flashbacks. He is bitter, mean, and full of contempt for his fellow Negroes. He especially hates those that go along with the system and accept their subservient role to the Whites. He is a man who hates himself and takes it out on others.

The only person he respects in the movie is Denzel Washington’s character, Peterson. Peterson is young, angry, and stands up for himself against Sergeant Waters. Waters beats him brutally, but in his own way he respects Peterson for not taking subservience lying down.

The movie is so much more than a mystery investigation. Each scene painfully peels away a layer of the facade covering up the painful system of oppression. It never settles for simple White-bad, Black-stolidly suffering. It dares to examine the collaborators, those who actively manipulate the system for their own ends and those who stand by and do nothing to change it. It dares to show the divisions within the ranks of the oppressed.

Why shouldn’t we feel sympathy for those who just want to get along, to play baseball, to make music, to be liked without dedicating themselves to revolt? Why shouldn’t we question those who turn on their fellow man for not sharing their outrage and hurt?

In the end, Rollins’ investigator has the most poignant lines in the movie. He asks:

Who gave you…
…the right to judge?
To decide who is fit…
…to be a Negro…
…and who is not?
Who?

A programmer’s story

I see another round of chatter about someone who has drawn a connection between his programming and art. I have seen this idea expressed many times. Sometimes it is accepted, sometimes applauded, sometimes reviled.

Personally, I think it is fine to believe this about your work or the work of others you have seen. I also think it is fine to believe that your work is not art, that it shouldn’t be art. I know lots of people who think that programming should be mundane, that aesthetic beauty in code is suspect, as if it is necessarily inefficient.

The arguments remind me of the arguments about Brutalist Architecture, but even making that comparison brands me as one who thinks art, architecture, music, dance, and programs have something in common, so I accept that I may not really understand the arguments against aesthetics in programs at a deep level.

But you know what I do not believe? I do not believe that someone who thinks programs are not art is not a programmer. I do not believe that they are somehow barred from some mythical inner circle of super-hackers. I do not believe that they are less- or more-worthy of respect.

My greatest disappointment with some of the criticism I have read in response to Joe Marshall’s thoughts are those that are blatantly Ad Hominem, those that accuse him of pomposity and self-importance. Even if his words cut those programmers who do not see art in their work, even if his words celebrate those programmers who create artful or clever works, let me ask all of those people this question:

Who gave you…
…the right to judge?
To decide who is fit…
…to be a Programmer…
…and who is not?
Who?

People search for meaning in their lives. I do. You do as well. Talking about that search and what we find is something very deep to all of humanity. I cannot judge who is fit to philosophize. The beauty of the Internet is that there is no barrier to entry. No corporation controls the right to preach your beliefs. There is no requirement that you have a degree of a certain sort before you can write about the meaning of code.

Of course we can disagree about what we believe, about what we preach. We can even disagree with each other about what I am writing here. But honestly, aren’t we all fit to decide for ourselves what meaning our work has for us?  ¶ 5:09 PM

  Trickster amongst the Samoleans One hot summer Trickster decided that farming was far too hot and tiring, so he summoned his sons and daughters and told them, “Tend to my farm, I have to leave for important business in a far off village.” And they tended to his farm, and he left for a far off village.

Now Trickster’s only business in his travels was trying to meet as many pretty ladies as possible and get by with as little work as possible. And Trickster considered this very important business indeed.

So Trickster wandered and wandered, and eventually found himself in the land of the Samoleans. Let me tell you a little about the Samoleans:

The Samoleans were great thinkers and valued truth, insight, and honesty above all other virtues. They maintained a great library of knowledge and travelled far and wide seeking out the greatest works on mathematics, astrology, engineering, history, and many other subjects. As a result of their great knowledge, the Samoleans were sought after as advisors on projects and ventures of all kinds. Even mighty Kings would consult the Samoleans before waging war, such was their perspicuity and wisdom.

To maintain their tremendous reputation, the King of the Samoleans enforced strict rules about consulting. By his law, Samoleans had to attend school and further their studies in the great libraries until they were married and had children to take care of. Any Samolean wishing to offer consultative services had to pass rigorous tests designed to ensure that they were qualified to provide advice on their chosen subject. Samoleans themselves sought each other out and happily paid for advice and consultations before making any decision.

Now Trickster found this very interesting. He also found the beauty of the Samolean women very interesting. How could he meet these women? He hit upon a plan.

One morning, just outside the Great Samolean Library, Trickster sat himself cross legged in his finest robes and announced to all that passed that he was offering consultation on matters of the heart.

“Come one, come all. Accurate predictions of your happiness! Will you meet the one you love? Will you find happiness in marriage? Trickster knows all, Trickster sees all. Modest fees!”

Such was the confidence in his voice that he soon had Samoleans lined up to seek his advice. They were so eager, none thought to question his credentials. After all, this was the land of the Samoleans! All who gave advice were wise!

Trickster had very little difficulty telling the Samoleans what they wanted to hear. To an old women, he promised a youthful suitor. To a young and studious man, he promised a breathtaking bride who was longing for a man of intelligence. To a lonely widower, he promised a youthful bride eager to start a family. But to each comely woman who sought him out, he took especial care to tell them they would meet a mysterious sage from a far off land. By the end of the day, he had six promising leads on single women!

As he was preparing to finish up his first day, he was practically dancing with excitement. Why, after a few days, a week at the most, he would have twenty or thirty women to court. And more than enough money for gifts and high living. What a fantastic scheme, and how lucky to be in a land where people paid good money for “advice”, all because they thought that anyone who gives advice must give good advice.

Trickster was so busy congratulating himself that he did not notice the King’s Soldiers until they had firmly grasped him by his limbs and started to carry him off.

Despite his cries, entreaties, and scolding, he was taken to the King’s palace and chained to a large stone. As they finished chaining him, a robed functionary read a series of arcane and unintelligible legal dictates from an ornate tome. “The King of the Samoleans will render justice in a week’s time” was the only phrase Trickster understood.

Trickster was in a bad fix. The Samoleans fed him scraps and water, but for an entire week he remained chained to the stone. When he was finally taken to the King, he was hungry and sore from crouching.

The King’s Court

The King of the Samoleans presided over a court in a wing of his palace. Every surface of wall was adorned with writings in languages from around the world. Tables were piled high with legal tomes.

Functionaries and officers of the court wore robes and head cloths adorned with symbols and characters. The King sat on a stool on a raised dais, his slippered feet resting on rich carpets. He gestured with a horse tail as he spoke.

“So!” thundered the King, “you took advantage of my people and offered fraudulent forecasts and worthless advice!” Trickster stammered a protest, but the King silenced him with a shake of the tail. “Not another lying word, miserable Sophist! You foul our presence with your prevarications!” Trickster swallowed nervously.

“Clearly,” continued the King, “you should be put to death, painfully and publicly, as a sign of our displeasure with those who would take short cuts with knowledge. This would be to the benefit of our great and learned society, as it would remind our subjects of the importance of honesty and accuracy in all things.”

“We are a fair judge. Have you any counter suggestion to offer? Be brief!” Trickster summoned his wits, he knew this was his chance.

“Wise King, Omniscient Monarch,” he flattered, “Death? Punishment? There has been some mistake (although surely not by your bibliophilatilic self). I admit I am not an artium magister, like your most Learned Majesty, but that does not mean my advice was not sound or that my predictions were not accurate. Only a week has passed since my imprisonment. Can any of these multarum literatti prove that my predictions are necessarily false?”

The Verdict

The King considered this for a moment, then handed down his ruling. “Your point is not sufficient to earn your release. Your cleverly worded ‘predictions’ may never come true, yet are vague enough to be difficult to prove false until such time as all in this room are long dead (especially yourself!), however I will consider it in mitigation. You have the opportunity to demonstrate your power of prediction.”

“You are to make a prediction that can be checked within the hour. If you cannot prove your prediction is correct, you will be executed immediately. If you can prove your prediction is correct to my satisfaction, I will mete some lesser punishment of my choice, such as branding, flogging, or a lifetime quarrying heavy rocks for the construction of our new library. I will make my choice by the time the hour has passed. I have spoken!”

Trickster looked around the room, but there was no sign of pity, no opportunity to escape. He sweltered in the heat of the afternoon. The King leaned forward.

“Well? Nothing to say? Would you prefer we get it over with? You can go with my men right now if you prefer to avoid an hour of unhappiness!”

Trickster stood up straight and looked the King directly in the eye. “No thank you, your flatulence!” At the insult, a soldier struck Trickster in the back of the knees, driving him to the floor. Trickster struggled back to his feet, and again faced the King.

The Prediction

“You asked me for a prediction, here is my prediction. I predict either you will sentence me to Death, or you will release me immediately, give me as much gold as myself and six maidens can carry, and furthermore give over to me the following six women as my wives. That will be your ‘punishment’.” Trickster then read the names of six women to the court. There was silence, then a roar of laughter.

“That’s no prediction!” said the King. “Take him away and flog him to death for his lack of respect. Be sure to take your time about it. And since he is so attached to the women of our country, have those six women throw salt on his back. I have spoken.” At this, the soldiers seized Trickster.

“Wait!” shouted Trickster. “If you flog me to death my prediction will come true. You will have sentenced me to death! And you said you would not sentence me to death if my prediction were to be true!” The soldiers did not pause, and the King turned to another matter. “What’s the matter?” Screamed Trickster. “Where is the vaunted Honesty of the Samoleans?” At this final insult, the King looked up and raised his hand. The soldiers paused.

“Very well,” said the King balefully, “flog him as I said, but do not let him die. Flog him daily, but see to it he lives.” He turned, but Trickster was ready with his response. “No, if you flog me my prediction was surely false, and you promised to execute me if my prediction turned out to be false within the hour!”

The King considered this twist. As he deliberated, a functionary whispered in his ear. “Aha!” Yelled the King. “Take him away! I will adjourn this court until tomorrow!” “With all due respect, I fear not!” retorted Trickster. “You promised to rule within the hour!”

The Final Verdict

Well, they argued back and forth until the hour was nearly up.

Finally, the King relented. “You are a Trickster, but clearly you are a powerful logician and I have learned that knowledge and power can be found in the most unlikely places, not just in books. In reward, I shall suspend your death sentence and provide you with gold and the six maidens. But be careful not to return to my land, for I may discover a new methods of logic and I would consider it a pleasure to try them on your case.”

And so it was that Trickster departed the land of the Samoleans with gold and six comely maidens.  ¶ 9:12 AM

  Some times, free costs too much

If you choose a platform that needs tools, if you give up the viral soft collaboration of View Source and copy-and-paste mashups and being able to jam jQuery in the hole that used to have Prototype in it, you lose what gave the web its distributed evolution and incrementalism.

You lose what made the web great, and what made the web win. If someone tells you that their platform is the web, only better, there is a very easy test that you can use:

When the tool spits out some bundle of shining Deployment-Ready Code Artifact, do you get something that can be mashed up, styled, scripted, indexed by search engines, read aloud by screen readers, read by humans, customized with greasemonkey, reformatted for mobile devices, machine-translated, excerpted, transcluded, edited live with tools like Firebug?

Or do you get a chunk of dead code with some scripted frills about the edges, frozen in time and space, until you need to update it later and have to figure out how to get the same tool setup you had before, and hope that the platform is still getting security and feature updates?

  ¶ 9:27 PM

  Hard-Core Concurrency Considerations Kevin Greer responded to What Haskell teaches us about writing Enterprise-scale software with some insightful emails about the pros and cons of using purely functional data structures for managing concurrency in a multi-processor environment. I’ve reproduced them (with his permission) below.

Now, your first reaction might be, “Ah, Reg is not nearly as smart as he thinks he is.”

If you feel like giving me some credit, you can keep in mind that I was not writing the definitive essay on designing concurrent software, just pointing out some interesting overlaps between what I consider to be the most academic programming language around and the most practical requirements in business applications.

But there’s something more important to take from this.

The original post was a response to someone asking whether there was any value to the stuff you read on programming.reddit.com. His query was whether reading about Haskell was practical. My response was, yes it is, and here are some examples of where functional data structures have an analogue in concurrent data structures. My thesis (that’s a two dollar word for “point”) was that many “impractical” things have a lot to teach us about things we will encounter in the pragmatic “real world.”

Of course, most people read the post and thought, “Cool, some neat stuff about concurrency.” Nothing wrong with that. If you value tips on concurrent programming (and I certainly do), you’ll find Kevin’s emails very insightful.

And if you are still wondering whether you should look at “impractical” and “academic” material like Haskell, Erlang, or other things not being promoted by MegloSoft, consider that one of the papers Kevin cites describes a data structure running on a 768 CPU system. Is this in a University lab somewhere? No, it is in a company that promotes itself as an Enterprise-Scale Java company.

You simply can’t assume that everything the industry tells you to learn is everything you need. Or that any one source (Cool! Raganwald has a new post on Lazy Evaluation) has the definitive answer.

You need to commit to life-long learning to be a software developer. Some of that learning is straightforward MSDN Magazine stuff, simple updates to things you already know. And some of that learning is a little further out on the curve.

Without further ado, here is one of the most comprehensive technical replies to a post on my blog I’ve received to date.

Concurrency Considerations

Hi Reg,

I was just reading your article on concurrent collections and have a few comments:

1. Just because readers do not update the structure doesn’t mean that they don’t need to synchronize. This belief is a common source of concurrency bugs on multi-processor systems.
   
   Unless you synchronize on the root of your data-structure (or declare it as volatile), you can’t be sure that your cache doesn’t have a stale version of the data (which may have been updated by another processor). You don’t need to synchronize for the entire life of the method, as you would by declaring the method synchronized, but you still need to synchronize on the memory access. You hold the lock for a shorter period of time, thus allowing for better concurrency, but you still have to sync.
   
   If you fail to synchronize your memory (or declare it as volatile), then your code will work correctly on a single CPU machine but will fail when you add more CPU’s (it will work on multi-core machines provided that the cores share the same cache). If you have a recursive datastructure (like a tree) then you will need to actually synchronize on each level/node, unless of course you use a purely functional data-structure, in which case, you’ll only need to synchronize on the root.
   
   Given that you need to make a quick sync anyway, this approach is not much better than just using a ReadWrite-lock (it is slightly better because you can start updating before the readers finish reading (not a big deal for get()’s but potentially a big deal for iterators()), but then again updates are more expensive because of the copying).


2. I don’t think that you should be using Haskell as a model of “Enterprise-scale” anything. “Enterprise-scale software” usually entails “Enterprise-scale hardware” which means, among other things, multiple-CPU’s. The problem is that Haskell purely-functional model doesn’t support multiple-CPU’s (it’s true, check the literature (except for specialized pipelined architectures, but not in the general case)).
   
   The whole processing “state” is essentially one large purely-functional data-structure. The problem of synchronizing your data-structure appears to be eliminated, but it has really just been moved up to the top-level “state” structure (monad), where the problem is actually compounded. This is worse, because not only would you need to lock your structure in order to make an update, but you would in fact need to lock “the whole world”.
   
   Some people will advocate launching one process per CPU and then using message passing to communicate between them. This is just a very inefficient (many orders of magnitude slower) way of pushing the synchronization issue off onto the OS/Network-stack. (Q: What do multi-core systems mean for the future viability of Haskell?)


3. You forgot to mention the common technique of using “striping” to improve concurrency. Basically, what you do is create multiple sub-datastructures which each contain only a sub-set of the data. You can partition the data based on the hash of the key being stored. You then wrap the whole thing up so that it still has the same interface as the single data-structure.
   
   The advantage of this approach is that when you use a ReadWrite lock you only need to lock a small portion of the data-structure, rather than the whole thing. This allows multiple updates to be performed in parallel. This is how Java’s concurrent collections work. See: ConcurrentHashMap: Java creates 16 sub-structures by default but you can increase the number when even more concurrency is required.


4. Have a look at Azul’s non-blocking HashMap implementation. They can do 1.2 billion hashMap operations per second (with 1% of 12 million/sec of those being updates) on a 768 cpu machine (the standard Java ConcurrentHashMap still does half a billion ops/sec which isn’t bad either) . This shows how scalable non-functional methods can be.
   
   I’ve never read of any Haskell or other purely-functional implementation being able to scale to anywhere near these numbers.

Summary: Using a purely-functional data-structure does make it easier to support multiple readers, but you still need to sync briefly at the “top” of the structure. Striping has the added advantage of also supporting multiple-writers as well, and in practice, this is a much bigger win. Haskell is inherently limited to a single CPU, which doesn’t match modern hardware; especially of the “enterprise” class. Shared-memory models of concurrency have demonstrated much better performance than purely-functional models.

Best Regards,

Kevin Greer

p.s. I've actually implemented a b-tree structure for very large data-sets and chose to use a purely-functional data-structure myself. My reason for doing so was that I have some very long-running read operations (~8 hours) and I obviously can't afford a ReadWrite lock that's going to starve writer threads for that long.

Another nice advantage of using purely-functional data-structures is that they make it easy to implement temporal-databases that let you perform time-dependent queries.

I just wanted to point out that if all you have is quick-reads then purely-functional is no different than a simple ReadWrite lock and that a super-pure implementation, as with Haskell, doesn't scale to multiple-CPU's or to highly concurrent updates. However, it can be used to good effect in combination with striping or other techniques.

(A tangential note: Java's GC is getting so good in recent releases that P-F data-structures are becoming much more efficient (given that P-F generates more garbage).)

p.p.s. One more advantage of functional data-structures (the largest advantage for me actually):

They map well to log(or journal)-based storage. Functional data-structures never update old data, but instead, just create new versions. If your data-structure uses disk-based storage then this means that you never need to go back and overwrite your file; you only need to append data to the end of the file. This has two advantages:

1. This works well with the physical characteristics of hard-disks. They have incredibly high transfer rates (10’s of Megs/sec) but very slow seek times (~ 200 seeks/sec if 5ms access time). If you are adding say 1k objects to a data-structure which requires 2 updates on average to a file then you’re looking at about 100 updates per second. If on the other-hand you write all of these updates to the end of the file then you’re looking at say 20,000 updates per second!


2. You can’t corrupt old data with interrupted writes. The old data is always still there. The only place that a corruption occur is at the end of the file, in which case you just scan backwards until you find the previous good head of your data-structure.

---

This is fantastic stuff to share, thanks, Kevin!

What other tips can readers contribute for people building highly concurrent software (besides the frequent use of JProbe Threadalyzer, of course)? What online resources (papers, articles, books) do readers recommend for the professional developer?

Labels: java, passion, popular

  ¶ 2:17 PM

  Alternate Universes

J_____ leads the dev team for Vault and Fortress. In the course of our discussion, I suddenly realized that none of our marketing efforts would reach J_____. He doesn’t go to trade shows or conferences. He doesn’t read magazines. He doesn’t read blogs. He doesn’t go to user group meetings.

Now I don’t know J_____. But I do respect Eric, so I assume he has hired a very capable individual. But I have to admit, I’m scratching my head. I am obviously living in a bubble, surrounded by hackers and bloggers and other refugees from Normalia.

I simply can’t imagine anyone leading a development team who doesn’t attend trade shows or conferences, ever. Who doesn’t read professional publications (perhaps more online than dead tree, but still). Who is too busy to go to user group meetings, or (a remote possibility) has tried them but found there was nothing of interest, nobody of value to his life. And especially who doesn’t read any blogs. Are blogs that much of a time waster that there are none that add value to his career?

My colleagues at BigCo read blogs and go to Java conferences. I was recently part of a team discussing a new business opportunity with a very buttoned-down, penny-loafer organization. Very smart, capable people. And they read my blog. I suspect they go to conferences and read magazines. I can’t imagine any of them being so isolated from the rest of the industry. Yet here is this capable individual made from a different bolt of cloth.

What is he reading? MSDN magazine? I assume he is not a complete outlier. There must be a whole universe of people out there who are writing software who simply never intersect with my world, not even briefly. This situation is like two particles that are so far away in space-time that their four-dimensional cones of influence never interact.

Amazing.

And as Eric explained, I need to get out there and meet some of these people, learn how to talk to them, learn from them. My Universe suddenly feels very small and isolated.  ¶ 12:47 PM

  A parable illustrating the importance of the differences between Scheme and Common Lisp in a world where Java is the Lingua Franca of Programmers An encounter on a bridge.

Labels: java, lispy

  ¶ 1:13 PM

  Style Matters

Art Matters

Giles Bowkett has written an entertaining opinion piece, wherein he argues that the gulf between sophisticated programmer and fundamentalist programmer is far, far wider than the hairline between fundamentalist programmers that use different programming languages.

You know, blog posts with sufficient depth (anything more than “How to call a COM+ object on Windows 2000 Server using CORBA over SOAP from IE 7”) are Rorschach Blots. If you’re predisposed to affinity, you read your own thoughts in them. And indeed, I immediately liked the article, because I thought he was saying that fundamentalist programmers use Blub, and that when he talks about the limitations of languages he was saying that people matter more than tools.

Of course, he’s actually saying a little of that but also a little of something else, and what’s really important to me are the ways in which what he says are different than the things I was already thinking. A mirror is a dangerous tool for experiencing the universe!

And then I read Andrew Norris’ reply, and what struck me was that Andrew may have had the opposite reaction to Giles’ opinions: he seemed to fixate on the differences between what Giles was saying and what Andrew was thinking. But closely reading Andrew’s impassioned response, I recognize so many similarities to the two points of view that I’m shocked they aren’t both the same person.

As a colleague used to say, these two men are in violent agreement. Giles is talking about the importance of the programmer’s deep understanding, and Andrew is talking about the importance of the programmer’s deep understanding as well!

They both seem to agree without reservation on the utter dead-endedness of the fundamentalist, “all innovation stopped with the invention of language X” thinking, for all values of X (including Lisp!). You can sort out the differences for yourself, they pale in comparison.

Styling

What I find most valuable (in the sense of expanding my own thinking, not just confirming what I already thought I knew) about Giles’ post is his assertion on the value of a personal style, of having a voice, on finding something unique to contribute, something that shines through the specific language and platform you use.

Isn’t that related to the entire meta-programming/DSL thingie? This idea that the language is a platform for a program so personal it deserves its own programming language?

Isn’t that exactly what both Giles and Andrew are promoting? Giles says “develop your personal style that is independent of the programming language.” Andrew says “use the tool that fits the job.” Well, what could be more personal than bending a tool to become a new tool customized for the problem? What could be more “polyphonic?”

And Giles’ example of Avi Bryant brings up an important component of developing a personal style. Giles pointed out the common thread in Avi’s work. he suggests it is Avi’s style, and it is. But Avi can’t use that style on any arbitrary problem. To a certain extent, Avi selects problems that fit his style.

To develop a style, you must be ready to apply it to problems, and you must also be prepared to walk away from problems where it really doesn’t apply. Otherwise, you will become the jack of all trades but the master of none. of course a great programmer can use shell scripts, C, and Python. But the programmer who has equal facility with all three problem domains is inferior to the programmer who “knows enough to be dangerous” in most domains but has obtained mastery in a very few of them.

So, Develop your style. Choose problems wisely. And learn your tools deeply, so that you may choose the right tools to express your style.

Labels: lispy

  ¶ 9:27 AM

  Quality Thanks so much to everyone who has submitted a solution to the 128-bit Programming Challenge.

I want to mention two things that are particularly touching. First, of course, is the quality of postings. When the last little challenge went round the blogs, there was nothing particularly odious about the responses, but there was little of value in the aggregate. That’s no smear on the respondents, it has a lot to do with the nature of the question.

With this challenge, I had an agenda. No, nothing to do with civil disobedience or freedom to watch movies you so-called purchased or rented on the operating system of your choice. My agenda was to demonstrate that the quality of responses is 100% driven by the quality of the challenge. When I saw the furor over the “illegal codes,” the challenge was a natural follow-up.

Thank you so much for reminding me how much tasty meat there is out there.

One more thing…

Now the second thing was very interesting. I checked my stats this morning, and at that time there were forty submissions, more or less. And there were thirty-five clicks on the link to The Essential Turing. So just as many submissions as click-throughs. And you know what? This is maybe even more interesting than the number of submissions.

Not because of the fifty cents or so in gift certificate money I will get from aggregate sales of the book. But because that particular book is one that appeals to people who are deeply, deeply interested in issues like Computability and Cryptography.

Issues touched on by the challenge, of course. And that’s also part of the point: the challenge is only peripherally “show off your coding chops.” At a deeper level, the challenge asks you to think about the relationship between programs and the data they produce.

I see the market for developer books following the same path as the market for developer tools: There is an increasing priority on “quick results”… I am as addicted to IntelliSense as the next guy. But I do believe this stuff come with a tradeoff. To some extent, the increasing emphasis on getting quick results comes at the expense of “deep understanding”. I define deep understanding as the knowledge of how stuff works “under the hood.”

Honestly, if you sit down to maximize sales on a page about a programming challenge, there are thousands of better choices. Most of them will be specific: books about a specific language outsell books about programming in general. But what can they tell us about programmers? Little of interest. Just that people skimming a site like reddit who are curious about a “programming challenge” are interested in books about specific programming tricks.

But when you put a book about Alan Turing’s work on the page, we find out that thirty-five people out there are interested in his work. And roughly the same number of people sat down to write code, debug, and submit code that is much more challenging than the code for the other programming problem.

I find that inspiring. Alan Kay lamented how few people know of Doug Englebart’s work. What I can say today is, there are almost as many people who are interested in the deep issues in computing as there are who have a bias for action and writing code. We just have to put that opportunity in front of our fellow programmers, and they will respond.

Thanks.  ¶ 5:58 PM

  The 128-bit programming challenge

Information wants to be free

Here’s a programming challenge:

Write a program that produces 128 specific bits of output. In the simplest case, output those 128 bits in a standard numeric form such as sixteen pairs of hexadecimal digits. Your program may not contain the 128 bits in literal form, it must manufacture them in some way.

Furthermore:

• Novelty counts. Try to think of a really unique way to generate the bits.
  
  Imagine you are determined to protect our freedom to program. Once the Monopolistic Overloards detect that a particular program produces illegal codes, they will send cease-and-desist letters to anyone distributing that program. Therefore, the greater good is served by collecting many different ways to generate the 128 bits of information.
  
  Or if you prefer a less weighty objective, imagine this is a job interview question. If the interviewers have seen your solution before, you won’t stand out from the crowd.


• You may express the 128 bits of output in any other form you like, however more complex or obscure formulations—such as steganographic images—should be accompanied by a method or program to recover the output into numeric form.


• Oh yes, the bits. Your program must produce 00001001 11111001 00010001 00000010 10011101 01110100 11100011 01011011 11011000 01000001 01010110 11000101 01100011 01010110 10001000 11000000, or any equivalent representation such as 09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0, as its output.

Come on folks, we’re programmers. We probably blew millions of person-hours writing programs that counted from one to one hundred. Let’s stand up and prove that when the chips are down, we can invest a few minutes and write something that’s both interesting to code and meaningful.

What’s the smallest program that produces the 128 bits? Can it be done in pure lambda calculus? Is there a self-modifying program that turns itself into the 128 bits? Can it be done with an Enterprisey Servive-Oriented Architecture? Haskell? C#? Let’s really do this right!!

Follow-up: Quality.  ¶ 9:40 AM

  Steve's road into the godawful swamp

You’ve all read about the Road to Lisp. I was on it for a little over a year. It’s a great road, very enlightening, blah blah blah, but what they fail to mention is that Lisp isn’t the at the end of it.

Lisp is just the last semi-civilized outpost you hit before it turns into a dirt road, one that leads into the godawful swamp most of us spend our programming careers slugging around in.

I guarantee you there isn’t one single Lisp programmer out there who uses exclusively Lisp. Instead we spend our time hacking around its inadequacies, often in other languages.

Labels: lispy

  ¶ 7:20 AM