Donner’s Daily Dose of Drama

So much has been written and preached about the difference between the words ‘principal’ and ‘principle’ that I don’t need to repeat it here. But in at least 50% of all job postings that are about a principal position you continue to use the term ‘principle’. You are not doing your clients a favor by publicly documenting your incompetence! The best candidates will skip right over such a job listing.

Fernand Point, by some considered the “father of modern French cuisine”, writes in his book “Ma Gastronomie”:

Perfection is a lot of little things done well.

This is certainly true in regard to cooking. It is rare, albeit possible, that on little thing that’s missing or was not done right ruins the dish. More often it is a few subtle things, not enough or too little of a spice, the temperature of the oil in the frying pan slightly too low, that will make the dish less than perfect.

I find that the perfect software application also requires a lot of little things done right. That’s what makes software engineering more an art than a trade.

The outsourcing and offshoring trends of the early 2000’s were an indication that executives increasingly thought of software development as a commodity, i.e. something that is purchased in large quantities at a certain price per unit (e.g. $/hour) and the less the company has to pay per unit, the greater the savings will be.

It turns out that something went wrong with that calculation, since these trends are seeing a reversal today. Now companies are ‘in-sourcing’ because the out-sourcing turned out to be not as economical as projected.

This development was predictable, but why?

Software development is a creative process. The software engineer has to make design decisions along the way that will impact the quality of the end product. Quality is an ambiguos term, so lets be more specific. The end product is characterized by the number of lines of code, how well the code adheres to the requirements, the structure of the code, its clarity, modularity, reusability, etc. All these factors contribute to how we generally defined good code. Good code can be understood and maintained more easily.
A good programmer can write good code and be very productive.
An moderately talented programmer can write good code and be less productive, or write not so good code and be less productive. There are also programmers who write poor code and are very productive in it.
A poor programmer writes poor quality code and is a lot less productive.

Each programmer’s code contributes directly to the quality and the success of the project. In a way, each programmer owns a small portion of the project. I’like to compare a software development project to a corporation, and each developer manages a department or a subsidiary. Nobody would in his right mind think of department heads or managers in a real company as a commodity that can be bought in China at the lowest possible hourly cost. The bottom line of such a company would suffer dramatically.
Similarly, the bottom line of a software development project suffers dramatically when developers are brought on as commodities.

Yesterday I got the basic Verizon Fiber Optic Service (FIOS) installed at my house. The installation took about 10 hours to complete, primarily because the installer hung the cable too low over my street and had to start over after I complained, and he initially did not have a backup battery for the power supply and had to come back in the evening to finish.

I have been using outgoing VPN connections through my PIX firewall for years - using all sorts of VPN clients (Cisco, Citrix, Nortel). When I started working for my most recent employer, I was unable to connect to their Microsoft VPN using the Windows XP VPN client.
After almost a year of on-and-off experimenting, I finally found the solution on Cisco.com: fixup protocol pptp 1723.

It worked without further ado, and this discovery was so exhilarating that I never want to forget it.

The presentation slides from the 30th Internationalization and Unicode Conference are available right here.
The presentation slides from the Gilbane Conference on Content Technologies are available right here.

The boom of the 90’s attracted many moderately talented gold diggers to becoming software engineers. Consequently, after the bust, the IT labor market today is flooded with people who have no concept of engineering techniques and quality of design. At the same time in corporate-land, the software-development-as-a-commodity mindset has created a similar situation at a global scale, and the practices of staffing firms have severely aggravated the situation. In short, good software engineers, and even people with the potential of becoming good software engineers, are difficult to spot among the masses. A hiring manager who wants to find a reasonably intelligent person with the ability to think and code at the same time has to go through a lot of effort in order to find suitable interview candidates. Working with recruiting firms is particularily straining, because the hit rate is low, interviewing is very time consuming, and communication with the staffing firm requires a lot of effort explaining to the recruiter that a good software engineer needs more than just a list of hot skills and a few projects on the résumé. In my experience, this communication most of the time yields no results, because …

When J2EE (Java 2 Enterprise Edition) became a mainstream standard about five or six years ago, EJB (Enterprise Java Bean) technology was sold by many evangelists as the silver bullet for every software problem. Organizations, so we were told, would now be able to minimize development time by utilizing container-provided features. Among those were object persistence and object/relational mapping, transaction handling, security, distributed objects and remote connectivity, etc. Simply put, all the problems that were supposed to be at the top of the priority list of most development organizations were now going to be solved single-handedly by using this new way of writing applications.
Consequently, most development organizations started to use EJBs heavily. Maybe not most of them, but certainly the ones that considered themselves at the forefront of innovation.

In 2001, I worked as a consultant on a trade-and-exchange web application for petroleum products. The client was a dot-com startup. We were using a product, MarketExchange by a company called Idapta. MarketExchange was a Java framework for auction and trade sites, and it was highly praised by the press - probably because it made heavy use of EJBs. As we eventually found out, every bidding transaction was represented internally as an Enterprise bean. Not surprising to us (the more surprising to Idapta’s support staff), the application did not perform, no matter how much hardware you threw at it. If there was more than a handful of users active on the site and bidding concurrently, response time went up into the minute timeframe.
We tried for a while to work with the vendor on the performance problems, but eventually gave up and wrote our own auction engine that did not use EJBs. It did not help the client much (Fuelspot.com), since their business model never produced any revenue, and they went out of business soon thereafter. If you Google fuelspot.com today, you will only get a few press releases and the résumés of former consulants and staff :-).

Needless to say, the vendor of the failing product, Idapta, did not have a bright future, either, and went under without the much predicted success stories. But I am grateful, since a got a trip to Atlanta for a 3-day training class at the Idapta offices out of this engagement, and I have not managed to get back to Atlanta since.
I am bringing these days back into the reader’s memory, because I believe that similar stories happened all over the country, probably all over the world.

And similar stories happen again today, except that today’s silver bullet is another 3-letter acronym: SOA. Yes, I mean Service-Oriented-Architectures. Oh, it sounds so sweet, doesn’t it? Standards-based application interopability, virtually for free. An IBM paper from 2003 predicts the “ubiquitous, open-standard, low cost network infrastructure”. And on they go with diagrams of heavily interconnected systems, such as this one (from this very IBM paper). The calculation of n(n-1) connections is mathematically correct, but nonsense nevertheless. There is no system of systems where every single node needs a connection to every other node. At least not in my and your worlds. And if there is in your’s, there is something seriously wrong with your world, much more than one can possibly fix with SOA (hint: you failed to manage the number of application dependencies properly - a large number of application interfaces is a symptom, not a problem).

I recently took over ownership of a portfolio of applications at an insurance company. Part of my portfolio is a small early-stage SOA infrastructure, and it appears that much of it was done for the sake of SOA, not because it made sense. There is a web service for logging errors to the database, for instance. Not a bad idea, unless you forget to implement any other means of logging in your application. There is another web service that does XML transformations. This one is used by a single application, which is itself heavily coupled with all sorts of XML transformation APIs. There is also a set of services that do useful things, but at what cost! The software engineers who are trying to fix bugs are complaining about three factors that make their work difficult:

• The application code lacks clarity because of its distributed nature
• Debugging is difficult
• The services introduce layers of application complexity that serve no other purpose but to enable services

And so I am learning the hard way what SOA does to my application infrastructure: it makes it awfully difficult to maintain. Unfortunately, it is already there, and it is working. How do I explain to the business that they could have saved hundreds of thousands of dollars by not going SOA? Down the drain go cost-effectiveness, speed to market, openness, and the love of my CIO.

Of course there are many more examples of technology hypes in the software industry. Client-Server, distributed object frameworks, multi-media, object-oriented databases, the list goes on. The pattern is always the same. Someone packages an idea that addresses a particular angst of many software development managers and IT executives who lack the ability to see through the fluff, wraps it into a compelling story, and the media picks it up and spreads it out like news of free beer. Unfortunately, many software development managers and IT executives have a hard time coming up with a sound and sane strategy for successful software applications on their own, and rely on bulls…, pardon, garbage dished to them by semi-knowledgeable journalists, book authors, and most persistently, sales people.

It seems that when used carelessly, EJBs made applications unusable because they did not perform, which was noticeable very soon, during prototyping or after the first components were tested. This made many development projects research alternative approaches before it was too late. With SOA, the problem is in maintainability and the long-term cost of ownership, both of which only hit you when it is too late to correct the course. Finding and retaining SOA-skilled workers who can maintain a complex network of services is only one additional difficulty that I am facing. None of this is apparent during development, though, and it may not become apparent until the first generation of engineers has jumped ship, which can be years after a SOA project is complete.

In my opinion, this makes SOA a hype that is a lot more dangerous than previous ones.

Nobody would design a web application for an auction site today with EJBs. I bet that 5 or 10 years from now, SOA will be a niche technology for very specific integration problems, but it will have disappeared from most middle-of-the-road enterprise applications.

The answer to the question what EJBs and SOA have in common therefore is: both are powerful concepts that were introduced with hyped expectations. Both were/are being used in ways that created poor overall results. The difference is that the performance issues caused by overuse of EJBs were apparent immediately, but that maintainance issues caused by overuse of SOA will not be apparent for some time after they are built.

Disclaimer: For the record, I am not denying the need for application integration, nor am I not recognizing the value of a services-based architecture. In fact, the most elegant application architectures that I have seen were very service-based, albeit written in the sixties using Cobol and CICS. I do criticize, however, the lemming-like adoption of web services and XML as a means of exchanging data, when the overhead can easily be avoided.

Only a few years ago, many commentators lamented about the flood of Indian software engineers that local colleges are spewing out in masses there. The fear was that cheap Indian tech workers would replace the engine that drives the US economy. Later, this diffuse and unqualified fear was directed against the Chinese. These reports painted bleak pictures of unemployed US engineers that could not compete against $20/hr off-shore laborers. Panicked voices could be heard in cubicles around the country. Some who lost their jobs after the dot-com burst were discouraged and found new jobs in education and other fields. Enrollment in software engineering college programs dropped substantially within a rather short period of time. The Washington Post reported in 2002:

At Virginia Tech, enrollment of undergraduates in the computer science department will drop 25 percent this year to 300. At George Washington University, the number of incoming freshmen who plan to study computer science fell by more than half this year. . . . In 1997, schools with Ph.D. programs in computer science and computer engineering granted 8,063 degrees. . . . [T]he numbers rose through 2001 when 17,048 [Ph.D.] degrees were awarded. . . . Nine hundred of the 2,000 plus undergraduates studying information technology and engineering at George Mason University were computer science majors last year. This year the enrollment in that major is down to 800, although a newly created and more general information technology major has attracted 200 students. . . . “Having it ease off for a while is a bit of a relief,” said a [George Mason] dean. “Particularly with the field as it has been, they don’t want to spend four years on something and then not get a job.”

On the other hand, Indian and Chinese engineering schools were producing graduates en masse, and continue to do so today:

India and China are producing the world’s largest number of science and engineering graduates — at least five times as many as in the United States, where the number has fallen since the early 1980s. (Reuters, 9/29/2006)

While the statistics have not changed much, these voices have largely disappeared. Maybe because the US economy has started to pick up and most software engineers who were without a job are employed again today, or maybe simply because nobody wants to here these stories anymore, the voices touting the end of the software engineering industry in the Unites States have somewhat quieted down.

But are we really out of the water? Where are we heading today?

I will be hosting a session at the 30th Internationalization and Unicode Conference this coming November in D.C. My topic will be “A Generic Content Localization Taxonomy”.
The program and more information are available at http://www.unicodeconference.org/.