The Demise of Software Engineering?

Only a few years ago, many commentators lamented about the flood of Indian software engineers that local colleges are spewing out in masses there. The fear was that cheap Indian tech workers would replace the engine that drives the US economy. Later, this diffuse and unqualified fear was directed against the Chinese. These reports painted bleak pictures of unemployed US engineers that could not compete against $20/hr off-shore laborers. Panicked voices could be heard in cubicles around the country. Some who lost their jobs after the dot-com burst were discouraged and found new jobs in education and other fields. Enrollment in software engineering college programs dropped substantially within a rather short period of time. The Washington Post reported in 2002:

At Virginia Tech, enrollment of undergraduates in the computer science department will drop 25 percent this year to 300. At George Washington University, the number of incoming freshmen who plan to study computer science fell by more than half this year. . . . In 1997, schools with Ph.D. programs in computer science and computer engineering granted 8,063 degrees. . . . [T]he numbers rose through 2001 when 17,048 [Ph.D.] degrees were awarded. . . . Nine hundred of the 2,000 plus undergraduates studying information technology and engineering at George Mason University were computer science majors last year. This year the enrollment in that major is down to 800, although a newly created and more general information technology major has attracted 200 students. . . . “Having it ease off for a while is a bit of a relief,” said a [George Mason] dean. “Particularly with the field as it has been, they don’t want to spend four years on something and then not get a job.”

On the other hand, Indian and Chinese engineering schools were producing graduates en masse, and continue to do so today:

India and China are producing the world’s largest number of science and engineering graduates — at least five times as many as in the United States, where the number has fallen since the early 1980s. (Reuters, 9/29/2006)

While the statistics have not changed much, these voices have largely disappeared. Maybe because the US economy has started to pick up and most software engineers who were without a job are employed again today, or maybe simply because nobody wants to here these stories anymore, the voices touting the end of the software engineering industry in the Unites States have somewhat quieted down.

But are we really out of the water? Where are we heading today?

In my role as Director of Distributed Systems at Aviva, a life insurer backed by a large international parent company, but with a comparably small footprint in the US insurance market, I have been agressively recruiting software engineers for a period of four months now. I have seen first-hand a trend that is indeed discouraging for the US software engineering industry, but it is only indirectly related to India and China. I am talking about the demise of the software engineering industry caused by recruiters and staffing firms.

Imagine you were shopping for a contractor to paint your house, and you asked questions to find out how qualified they are.

This may sound exaggerated, but it is very real. I wish I had recorded some of my interviews, so that I could provide you with accurate transcripts. When I ask a question that should be answered with Yes or No (yes, sometimes I ask such questions, too, you behavioral interviewers out there), I get a tirade of unrelated technical terms cobbled together to hard to understand sentences. When I ask open-ended questions and give lots of hints about the direction that I would like the candidate to take (e.g. When do you use stored procedures over embedded SQL, and why?), I get a response that contains the words “JSP”, “database”, “ODBC” (3 times), “JDBC”, and a few other not very relevant terms.

This kind of conversation I get with 9 candidates out of 10. The problem is not that there are so many clueless engineering school graduates (most of the ones that I interviewed came indeed from India, or were in India at the time of the interview), but that the recruiters think they are great, and keep sending more. Don’t get me wrong, I know, work with, and have worked with outstanding Indian software engineers. It just seems that today scores of students are graduating from Indian engineering schools who really don’t know much about practical software engineering, and are not able to acquire this even after several years on the job.

Now, if nature had its course, this would not be a problem, because they would not get hired and eventually disappear from the job market. Alas, it seems that nature does not have its course anymore. Misguided by an opportunity to save money, managers keep hiring and outsourcing to cheap labor, even if the resulting workforce is less than qualified. This creates a market that favors low-cost (i.e. poorly skilled) workers and qualified people are being forced into niche areas in order to survive. When someone like me comes along and wants to hire staff that can have a meaningful discussion about a technical problem, that can reason and explain design decisions that they make, then the market is blank. Zero. All gone.

Now this is a matter of communication style and interpersonal skill, one recruiter responded to my unfavorable feedback. In other words: the person is in reality really smart, she just does not communicate well. Huh? What’s the difference? If I can’t explain a problem and make myself understood about the intended solution, does it matter if the person doesn’t get it because of a lack of language skills, or a lack of intelligence? It does not. Most people I work with on daily basis are recent immigrants and many are not native English speakers. But they understand questions and can respond to them.
Why does the candidate not ask me to explain myself, when he or she did understand the question? Some may say that it is a cultural thing. In middle- and far-eastern cultures, asking questions is not encouraged. Indian software engineers like to go back and figure the answer out themselves, instead of asking more questions. This may be the case, but it does not justify throw-away work at the expense of the employer, because the purpose of a task was not understood.

But I don’t blame the candidates. I blame the recruiters for sending them. They are flooding the market with material that is not up to par with state-of-the-art practices and technologies, and that is abused and exploited. Because of this, many hiring managers have lowered the standards that they would normally apply to job candidates. They have resorted to asking for factual knowledge that can easily be learned out of books, instead of making sure that a person will put things in the right places when left alone. As a result, poorly designed applications and systems are produced that fail their companies more often.