Why Software Engineering Sucks
Updated: Additional clarification added to the bottom of the post.
I’m currently taking two Software Engineering Master’s courses. I’ll be graduating in May with a minor in Software Engineering, and I can say with absolute certainty that software engineering sucks.
I’m sitting in the back of one of my classes listening to a professor talk about software architectures. He’s explaining why and how architectures should be built. I can’t disagree with anything he’s saying…he’s talking about modular code and reusability, and explaining the levels of complexity of different types of architectures. That’s great, and I definitely see use for software engineering and how it can help make development simpler and easier. But software engineering sucks.
“But why does software engineering suck Brian?”
Software Engineering Does Not Promote Innovation
This can be summed up in a quote by a fellow student in one of my SE courses. I was explaining my reasons for not liking software engineering, which I’ll get to in a moment, and he responded:
“I think following Software Engineering processes is the most important thing”
What? Have we really gotten so abstracted from the end goal that the process becomes most important? What happened to actually developing a good product? Here’s my list of priorities when developing a product:
- Develop a product I’d be proud to have my name on
- Achieve the client’s goals
- Learn new technologies
Notice that there’s no mention of process on that list. I don’t care what process I use, I care about the result. Software Engineering makes people dumb…it encourages them to follow a series of steps without evaluating the appropriateness and applicability of each step along the way.
Too often I see people follow these processes to the letter, sometimes due to their own ignorance, and other times due to the process itself (and management’s enforcement of it) leaving little room for innovation. How can there be innovation when individual programmers aren’t encouraged to think about something as small as the process they follow. Companies should be encouraging programmers to question everything every step of the way. Some of the best business ideas come out of programmers questioning current business paradigms (see countless Silicon Valley startups if you need examples).
A-Teams Don’t Need Software Engineering
Consider the following two groups. Group 1 is full of brilliant, self-starting people who are good communicators. Group 2 has a spectrum of programmers with varying levels of programming and communication skills. They’re both working on an internal tool for a 100 person company that handles money-related tasks, let’s say payroll.
Group 1
Group 1 sits down with management and asks “what does this need to do?” They ask the right questions, and burn through the meeting quickly. Then the team sits down with a whiteboard, draws out what they need the system to do, and talks about security. They copy down their work to the intranet, split up tasks, and go write the system, checking in with each other every once in a while. After the system is done, they test it thoroughly, perhaps write an email to the QA department telling them to really pound the security aspects, and release it.
Group 2
Group 2 performs an initial requirements analysis. They pump out pages and pages of documents with complex diagrams. They review the documents. Then, when they have every item speced out, including class diagrams, functional requirements, and every little detail such that even a monkey can code it from there, the team meets. They split up the work and write it, documenting every tiny thing along the way, and meeting every morning to discuss how they’re doing. After the system is done, they collaborate with QA to produce another large document describing every little detail of what was written, and how to test it, again even a monkey could execute the test plan.
Both plans will produce perfectly good products. However, an extremely talented individual will be incredibly frustrated in Group 2, since he will be pumping out form documents and going over every little detail that won’t make him more productive. On the other side of things, a mediocre programmer will be incredibly frustrated in Group 1, because he’ll be forced to make a decision that he doesn’t have the experience or knowledge to make, and he won’t have the wherewithal to ask for help. He’ll inevitably fall behind and not think to mention it to anyone. The problem with the first situation is that it’s dependent on every programmer being brilliant.
So we have reached the core problem: not all programmers are brilliant (and I’m not saying that I am or am not). Microsoft needs to use SE processes since they need too many programmers to guarantee that each one is brilliant. I’ve seen 100 person companies that are careful enough with who they hire that they are all Group 1s. That’s right: every single one of their employees grasped the big picture, did their jobs well, and were smart enough to ask for help when needed. My personal principle is that if you’re not smart enough to be able to communicate well and produce good code without a strict formal process, I don’t want to be working with you.
Another way to think of this is that the amount you can rely on programmers to make decisions is capped at how much you can rely on the worst programmer (or, as Penelope Trunk, who’s blog I’ve recently become a fan of, puts it: “you should always hire A players because one B player brings everyone down”).
That’s why many software companies spend so much on employee perks - video game consoles, free food, big monitors - because attracting good programmers creates more ideas, better codes, and less managerial bullshit. If the best programmers may make you a million dollars practically overnight, don’t you only want the best?
Let me be clear in saying that I don’t think programmers should hack away all willy nilly. In my education I have found that one of the main caveats of SE is the idea of bringing a set group of processes to the table, and I resent that. Otherwise I have no gripe with the concept of a “process”, but I don’t think one should have any paradigm in mind when starting a project. As an example, I recently gave a presentation on a practicum I took to a large group of SE professors and researchers. I got into a discussion with one of them at the end of the talk, and I mentioned that my team, though technically following Scrum, only met once a week because there were just two of us contributing 12 hours a week each. He replied “then that’s not Scrum”. This is a perfect example of my point: SE in its current form places too much emphasis on process. That’s all I’m saying…perhaps my title is a bit misleading.
View Brian Krausz's profile
I absolutely know where you’re coming from, but I think you’re taking it too far. I don’t care how good your programmers are, there’s a point after which a project is too big or too complex for even good programmers to ship it without some kind of process.
The key thing to understand about good programmers and process is: Good programmers *use* a process to get their job done.
The problem arises when the process becomes the point in and of itself. Then you have the situation where the programmers must conform to the process, rather than the process conforming to the needs of the programmers.
Ideally, the programmers should *own* the process they are using. In the real world on large projects this is never entirely the case, but I think you’ll find that crack development teams will bend the organization’s standard development process to their will, so that they can get a process that helps them rather than one that hinders them.
I don’t imagine there are any software engineering classes that teach that last part though.
I think what you really hate is unnecessarily complex bureaucracy. Not all software engineering processes are frustrating and bloated. If anything, your classes should have at least mentioned agile development processes and the pros and cons compared with traditional waterfall methods (group 2). Many SE techniques are useful to both developers and companies. Without some conventions in place, how could developers coordinate with one another? Also, how would developers even know the intent and functional requirements of applications that are heavy on domain specific knowledge?
It’s hard to sit in a classroom environment and understand why SE is needed. I think that a lot of companies do have shitty processes in place. I’ve heard plenty of complaining from friends and coworkers at their past jobs. Yet it’s not always bad because I’ve an equal number of complaints for chaotic workplaces where people just do “whatever the hell they want”. I think Curtis hit it on the nail when he said that the process should be a *tool* for the programmer.
It also sucks because it get’s boring.
However, too many years of anything will drive you crazy.
way off the mark. Even the best programmers can’t handle a “big” project without proper processes.
I think many smart students feel the same way you feel is because most of their experiences are from toy assignments and small open source projects. They never experienced, especially in the center, of a multiple million lines of code with a huge legacy.
To clarify: I’m not saying everyone should run around all willy-nilly, I’m saying a process should naturally evolve from the team you’re working with. I’m working on a fairly large project now, and I felt that it needed proper documentation, so I am developing it. I’m sure my doc doesn’t conform to any particular “process” spec, but it’s exactly what I need in this situation. While I suppose that technically is a process in and of itself, it’s not a defined process, and those are the ones I take issue with.
I think mindless process is great. Large, tarpit-style mindless process created the towering behemoths of innovation, behemoths like HP and IBM. When it comes down to it, startups in the last several years have not and are not capable of creating revolutionary changes to the technology sector — our supposed agile startups will never create innovations like the disk drive, the memristor, or the ARPANet. If large, tarpit style process is what it takes, so be it.
You should read the “anniversary edition” of Fred Brooks’ book, THE MYTHICAL MAN MONTH. Ignore his sexist title–the first edition of this book was published in 1975. This should be required reading for every software engineering student.
I thin that Daniel Chen, one of your commentators, has read it since Brooks first chapter is called “The Tar Pit”.
Software engineering is really not about individual achievement but about achievements that require great numbers of individuals working together. Brooks draws on his experience managing efforts with hundreds and even thousands of technical contributors. Software engineering is about innovation and agility when an enormous amount of coordination is needed.
R Cohen: I have read the book, and appreciate a lot of its advice (such as the advice implied by the title about adding people to a project).
While I admit I am not the norm and I’m sure few will agree with me, I still feel that adequate process will naturally grow out of intelligent developers, and one of the main caveats of SE is the idea of bringing a set group of processes to the table. Updating the post now with some clarification.
Jerry is totally right here.
First, realize that Software Engineering is all about building complex software. It’s about the higher orders of complexity in software development, and how to build things that no 1 or 10 people can build by themselves. Of course in the classroom they have to scale it back to demonstrate the concepts, which makes it very very easy to miss the point.
Just like anything else, these processes and best practices that we have from 60 years of software development can be misapplied, but they weren’t the result of some ivory-tower academic ruminating about abstract organization. They were learned the hard way in the trenches of building operating systems and complex monolithic software. So when you’re in a classroom context working on contrived assignments and the teacher is speaking about these things, he is necessarily detached from the application of these principles. Your opinion of software engineering is now totally determined by the teaching ability of a single individual who may not even have any real world experience in software engineering.
Finally, I want to draw your attention to the importance of process. There is always a process in any organization, whether it is official or not. In your example of brilliant developers with good communication, you’re just talking about an ad-hoc process. Let me tell you, this works great in small groups but it won’t scale. In the three points you mention that your top priority is to “Develop a product I’d be proud to have my name on.” Well hopefully the product is everyone’s goal (and the vision is clear), but the process is how you actually get there. You may not care about the process, but you better hope somebody on your team does and that they have some talent at it. If not, then management of the project will fall to the collective, which means you all will end up being aware of everything else everyone is doing, and decisions will have to be made unanimously. Now instead of spending 10% of your time on process and 90% on your actual job (say, coding), now you (and everyone else) are forced to spend 50% of your time on dealing with each other. You may all have agreed that you hated managers, but now that’s what you’ve become, and you’re not very good at it.
I completely agree. Here’s another way of saying something similar, but from a completely different angle. http://compoundedthought.blogspot.com/2007/02/people-factor-why-startups-succeed.html
I’d be interested to see a follow-up on one of the themes here, which is how a B programmer can become an A one (and thus hireable per the maxim above).
Hope you don’t take your remuneration ideas from Spolsky. The linked article in that blog entry to inc magazine had him “pay” a conditional retention bonus in unpriced stock to a guy who made Spolsky a million bucks when not even full-time with him. “Surprisingly” million dollar guy takes Google’s offer instead and leaves Spolsky holding his bag of gold and his conditional warrants.
I think the guy took the Google offer because Google probably know the difference between innovation awards and golden handcuffs.
Mark: Personally I don’t envy Joel’s position (well, I do, he made a million bucks, but still…). He had an employee who went above and beyond, but the problem with recognizing one is that you then don’t recognize others. Often the existence of special statuses and rewards hurts those who don’t get it. Given the circumstances I think he did the best he could.
Also, I’m not so sure Google would do better (or even as well) given the same situation.
One thing i think you overlook, is that there have been plenty of projects that had teams more like team A then team B (brilliant, self-starting people who are good communicators) that never the less produced team B results. They start the project, and soon enough they come accross ambiguities and misunderstandings. Then there disagreements about how to handle those ambiguities, the customer doesnt know what they want, and the team manager picks the wrong way to handle it and boom your in a hole. Next thing you know you have to retrace your steps and redo many of the things you have already done. Your running behind schedule and the customer is wanting SOMETHING, ANYTHING. So you give them software that is buggy and incomplete. They begin using the product and then soon enough they find a bunch of bugs. You have to not only give them a new build but also fix the data they have already entered. Your project is already way over budget, and now you have to find a way to sell it to more customers just to keep the company afloat. So your sales guys sell a lot of things that the product doesnt have and then they promise to deliver it as if it is already in the product… and now you have just dug yourself further into a hole. In an effort to avoid these sorts of things is why we study the SDLC.
There are different methodologies for developing software, but in school they usually focus on the waterfall method because it includes all of the steps in the SDLC, but it is usually only needed on very large projects where there are many requirments and there is the most potential for misunderstandings and ambiguities. It is sad that they dont focus more on the agile methodology which is more applicable to smaller projects (which is what I usually work on).
I think you have mis-understood software processes (Possibly because most universities seem to have lecturers who work in professional Software Dev for 6 months before returning to universty).
Most modern processes; agile for example focus more on the people, in order to produce a quality product without the uneccesary bureaucracy. But even more traditional development processes if followed correctly should be effective (N.B most processes don’t discuss how much detail to use in documentation/design - just the phases to follow).
However good documentation is important. Imagine group 1 is working on a project that will make or break the company. 3 months before the end of the project two developers leave. Without good documentation i.e. functional and test specifications, it would be hard for the company to bring in new developers to pick up where the old ones left off. On the other hand group 2 who have complete documentation can easily transistion the work onto new developers.
This becomes ever more important in the real world if you have a customer running a product for 5-6 years and raise a bug. Potentially your development team who have to fix the problem, may never have installed the product before, so design documents are critical, as are test spec’s to ensure there are no regressions with the new fixes.
@Dave J
I’m not saying development processes are unnecessary, I’m saying that given sufficiently capable developers they should be organic and intuitive. I’m not saying “drop all software engineering practices”, I’m saying that competent people should not be forced to adhere to some obscure standard when the entire team may be more effective if they do things how they’re accustomed. I agree that this is a dangerous path, but if the developers are capable with regard to both coding and communication, this should not be a problem.
Its better to doubt it and challenge than march blindly and waste your time and money!
the same arguments could be made of algebra.
The team may be more effective, yes, but are they delivering what the customer needs and expects? Methodologies such as extreme and agile deliver constant, demonstratable updates to product in order to show progress to the customer. If each member of a team does what they like, can you still guarantee this? What if the customer wanted a demo TODAY? What if a recent change a few engineers made broke the entire demo?
This is just one of the tenets of SE methodologies, and there are other other customer-centric tenets which are being overlooked. SE methodologies are more than just organizing code and having a “process” to follow for less-experienced engineers — it reaches into the realm of the business model of the company, which us engineers often times do not fully understand.
You’ve obviously never worked on a large, complex system in the real world before…
Imagine microsoft trying to design windows with your method. Yeah……
@Rick: I addressed Microsoft specifically in my article:
If Microsoft had exclusively brilliant programmers, then a process would evolve naturally within and among teams and they wouldn’t have to all follow a certain process.
Get out of school, get a real job, work on a project with 100+ other developers for a few years, then get back to us…..
You’re alone on an island with this idea…..
@Rick: Thanks for the commands…I’ll keep that in mind.
Well, I’m out of school, have a job, and work with 70 developers. You’re reading a year old article.
http://www.codinghorror.com/blog/archives/001288.html
Yup…totally alone.
For the record I don’t bother conversing with closed-minded extremists who won’t consider other views constructively. Have fun commenting on my insignificant little blog.
lol, so you have what… 6 months or so experience in the real world?
Does your employer embrace your philosophy?
I’m just a student with some work experience and I think that you are right in a way. Software Engineering is a little over blown (and its boring), it doesn’t encourage innovation. However if you needed to design a software system for a nuclear power plant or the space shuttle then you would definitely need the software processes.
For the space shuttle software, they have no choice but to produce pages and pages of documents and specs so that even monkeys could code it. The processes helps Lockheed Martin produce good quality software, the current system has 420,000 lines of code and only 1 error. The last 11 versions had only 17 errors (this fact I got from an article, the link is below). If I were writing space shuttle software I would definitely use the software processes.
Check Out: http://www.fastcompany.com/magazine/06/writestuff.html (’They Write the Right Stuff’).