How do I prove to management that our programmers could work more productively?

Question

Backstory

I graduated less than a year ago with a degree in Computer Science (with extra courses in software engineering), and another degree in Software Engineering. I'd like to think that I'm familiar with modern software development methodologies (CI, Scrum, XP, etc).

The problem

I got my first (and current) job at a medium sized game development company. The company has been around for several years, has multiple successful titles under its belt, and is full of talented and dedicated people. However, software development happens in an entirely ad hoc way. We use SVN and a bugtracker, but beyond that there's very little planning, no automated testing, and you'd be hard pressed to find a UML diagram anywhere in the office.

I can think of two possible reasons for this:

1. Programmers are simply not given enough time to do things properly. Sadly, upper management has a tendency to set deadlines without consulting the programmers, which has led to many an all-nighter.
2. The programmers themselves lack the education / motivation to employ proper techniques.

The question

I recently spoke to management about this, and I was encouraged to suggest alternatives. I figure the first thing to do would be to rule out the second possibility listed above: that is, give management some evidence that if the programmers were given more breathing space, they could become much more productive. However, I'm not quite sure how to go about that. My best idea so far is to conduct a survey among programmers about what software development methodologies they know, as well as their attitudes towards code quality and such.

tl;dr

How can I assess whether our programmers are able/willing to employ modern software development methods, and prove to management that this would lead to increased productivity?

---

EDIT

I obviously failed to bring my point across properly, so let me elaborate on the situation a little.

First, I am fully aware that this looks like a bright-eyed fresh grad trying to save the world with his l33t education and changing the wicked ways of those grumpy old fossils who just don't know any better. I also understand that there is always a difference between theory and practice, and what they teach you in school is never what happens in real life. That is exactly why I'm being incredibly cautious here, trying to get a feel for the current situation instead of charging in going "Hai guise, let's all do teh unit tests from now on!"

Second, @psr raises a very good point. Let me name some of the reasons why I think that there is room for improvement:

• Our senior developer, who's a bit of a maverick but ultimately a skilled and reasonable guy, agrees that at this point in the company's history, we should have more sophisticated and repeatable development processes in place.
• One of the programmers who's much more experienced than me and has used modern methods agrees with me that we could be more productive if we used some of them.
• In some cases, throwing features together during an all-nighter and then hunting down all the bugs takes a ridiculous amount of time; I cannot imagine that doing things any other way could be any worse.
• The lack of planning, design, refactoring and documentation leads to poor code quality. Perhaps more importantly, it leads to programmers forgetting how their own code works in a matter of months (while they're still working on the same project!), and no way to find out apart from trial and error or wading through the spaghetti.
• Even management suffers, as the only quantifiable measure of progress in a project is the number of unresolved bugs. But since anything is liable to break at any time (due to the reasons above), this is a very unreliable figure.

Third, I don't actually believe that my colleagues lack the knowledge of modern methods or the willingness to use them, it's just that at this point, I cannot rule it out as a possibility. Which is kind of the reason why I came here asking what's the best way to find out what they know (besides lengthy personal interviews, obviously).

Fourth, don't take those comments about UML and automated testing too seriously, they were just examples. The fact is, code is produced on the immediate whims of designers/upper management without any kind of software design, and only ever tested by a couple guys pressing buttons to see if anything breaks. Perhaps I'm still wearing my university-issued rose tinted glasses, but that strikes me as garage-development at its finest.

tl;dr

I'm trying really hard not to be arrogant about this, but both my fellow programmers and management seem to be expressing a desire for more predictable and productive ways of developing software. How can I find a solution that could make for an easy transition?

Answer 1

Have you spoken to your development colleagues about this? How do you know they lack education? That's quite a sweeping statement and you'll probably find you're wrong.

I don't think it'd go down too well if a new grad started meddling with processes without understanding why they're like that in the first place. Managers love processes and love tracking and love being made look good, so the battle-hardened skeptic in me feels that your manager is interested in your views because it's something they can take credit for.

Also, be aware that software development methods in text-books don't always map out like that in the Real World. They're certainly not one-size-fits-all, and can look at the world through rose-tinted spectacles. UML is a load of crap, not having that around isn't that big of a deal.

Answer 2

First, be clear on the specific steps that management should take. "Give your programmers more breathing room" is too vague, not actionable, and not measurable.

Second, identify the actual problem. Why are your programmers pulling all-nighters? There's always a root cause or causes, and that root cause does extend beyond "we don't have enough time get things done."

Third, find authoritative sources and case studies that back up your claims. There are many resources available that discuss solutions to software development problems; use them to make your case.

Fourth, appeal to management on their own terms. What are they looking for? Chances are good that they're looking for ways to be more productive in terms that they can measure. In other words, "show them the money."

Fifth, consider the possibility that the problem that you are describing doesn't really exist. If the company is profitable, and they are adequately compensating the developers, all-nighters boil down to simple planning problems and inadequate management of expectations. In other words, it might not be a problem with your software development methods, but more a problem of managing scope and feature changes.

Finally, take into account your lack of practical experience. Education is great; I have one too, but that education didn't automatically make me an expert. What it did was ground me in the fundamentals. Getting into the "real world," and finding out how things really work is a whole 'nother education.

Answer 3

Your company uses SVN and a bugtracker? Consider yourself lucky! I just graduated and started a new job, my new company version controlled their >100 one-off, ~10 kLOC VB6 and VB.NET apps with ZIP files with dates in the file name. However, my situation is slightly different because they hired me with the expectation that I could help improve the development process, and it was just 3-5 previous developers, and all but one of these developers have moved on. Also, I have 1.5 years of internships at another, similar company with far better development practices to gauge my present employer against.

I readily acknowledge that as a junior developer I am not experienced enough to make major changes based on what I've read in my textbooks and heard from professors. Instead, draw on the wisdom of people with more experience: Professors with real-world experience, mentors from previous internships, and other developer's blogs. I found the following articles from Joel Spolsky insightful and applicable: The Joel Test: 12 Steps to Better Code, and Getting Things Done When You're Only a Grunt.

The Joel Test comprises the following questions:

1. Do you use source control?
2. Can you make a build in one step?
3. Do you make daily builds?
4. Do you have a bug database?
5. Do you fix bugs before writing new code?
6. Do you have an up-to-date schedule?
7. Do you have a spec?
8. Do programmers have quiet working conditions?
9. Do you use the best tools money can buy?
10. Do you have testers?
11. Do new candidates write code during their interview?
12. Do you do hallway usability testing?

It sounds like you score at least a 2, with your source control and bugtracker. My company scored a zero. Honest to goodness, a zero. Read the next line carefully: The truth is that most software organizations are running with a score of 2 or 3. My company has been profitable while operating at a Joel Test level of zero. A huge number of profitable, successful companies (like yours!) are making money and writing software with a process at or below the level of your company.

Next, "Getting Things Done When You're Only a Grunt" could be copied and pasted as an answer to your question. I've paraphrased it and made some adaptations for our situation here:

1. Just Do It: Use good software engineering practices personally, even if they're not official. I run a VCS, Jenkins CI and Redmine on my local machine. I work hard to eliminate bugs when I find them, to test my code well, and to do usability testing when possible. I write my own spec and schedule even though I can't influence them. I close my door, wear headphones, and apply a little focus. I brought in a spare second monitor, a couple sticks of RAM, and an SSD from home, and use a lot of open-source or free software to have better tools. You can give yourself a lot of the personal benefits of good software engineering even if your team doesn't want them,
2. Harness the Power of Viral Marketing: Allow change to spread organically rather than by executive fiat. I'm going to make a bold statement here: This is the only appropriate way to make changes as a <1 year junior developer. Don't email the entire department with a survey. Don't ask your managers to read research papers on software engineering. Let them ask you how you do your work so well. When you get some seniority, then you can consider making changes directly.
3. Create a Pocket of Excellence: Use good software engineering practices to write more and better code than other areas of the organization. As an individual developer, you pass the first ten tests on your own. I don't have the domain knowledge of my coworkers, but I can write good code. This is #1 applied to a small group.
4. Neutralize The Bozos: Minimize the damage that poor programmers can cause to your projects. This isn't really our place as new employees. You should probably ignore this one unless you're really good with people.
5. Get Away From Interruptions: Find a place and time that you can get good work done. Headphones can be helpful here, if they're OK with management.
6. Become Invaluable: Write lots of really good code so that you're respected (which helps #2 and #3 tremendously). This does not mean writing terrible code to give yourself job insurance. That is the opposite of invaluability and job insurance work. Being valuable does, and gives you career insurance, which is better than job insurance.

Read the article in full, it's gold. I'm about 6 months into the process, and the organization scores about a 1.5, which is infinitely better than the zero we had when I started. I consider that great progress, and will be ecstatic if we can get up to 2 or 3 by the time I've been here a year.

Answer 4

My experience has been that before my suggestions for process changes are even heard, I have had to develop a reputation with the group involved. The system the groups have had worked -- maybe not well, but good enough for products to get shipped for many years. Asking people to change to the unknown based on the word of someone with an unknown track record was a big ask.

"The status quo is working. Why change it? Why should we listen to you?"

I've been down this road before. So what did I do? Lots of things (20 years in SW development is a long time), but this seems to have worked best: I started by building my reputation. I showed myself accomplished in doing the tasks assigned. I also learned their systems and the reasons behind them. (Steven Covey presents the idea of a bank account. Asking for a change is equivalent to asking for a withdrawal from the account. Before you can make a big withdrawal, you have to put enough in to cover it.)

Once I had established my reputation, I would wait for opportunities to suggest small improvements:

"What would happen if we could automatically make a build of the code set every time it changed and have the results mailed to developers?"

"Now that we have the code building automatically, what if we had unit tests run with every build?"

"You know, instead of spending a month of everyone's time doing code reviews at the end of the development cycle, why don't we review every commit, before we have to re-work major sections?"

Did it work? Yes. A team that resisted the idea of checking in code more than once a week now commits multiple times a day. If the CI system is down for as little as an hour, I get calls. All of the source code is now under unit tests (95+% coverage!) for each build. The team now is doing "Continuous Code Reviews" as changes are committed.

And me, well, I'm starting to build my reputation again. They already like the idea of having integration builds with every commit... If only there was a way to make that happen... :-)

Answer 5

It should be fairly simple. Employ modern software development methods to be noticeably more productive than the others, and credit your productivity to these methods.

So... make a list of where your productivity will excel, and then over time, for each item on the list, keep a record of the methods you used to achieve excellence.

Answer 6

The copybook way of managing products/business can't fit for all. I have worked with medical imaging projects. Everyone wanted to work in agile way but the problem is we can't be completely agile because we've regulatory controlling our business and it's required to have hell lot of documentation to prove this product meets their terms and conditions to sell the product.

People who are core in to the work actually not much bothered about the process. They consider it as a shit job and slow up their activities (yeah most programmers do).

I guess they work with people make their plans and ship the product. That's how it's worked so far and they're not ready to change it.

For e.g companies like Valve doesn't have any managers at their office. It's an extreme case but most of the product companies have a vision, hack on certain things and ship. On introducing a process, ask these basic questions

• What exactly the problem you're going to solve?
• Are you finding without a proper process we are shipping low quality product? Or on introducing a new process, you're confident about improving the way of working within the team?
• If it's a problem with management, then which has to be fixed first? Are they lacking employee morale?
• How can you improve quality of the product. Some simple things like unit testing, coverage, coding standards and guidelines suits your project can improve the quality of the code
• How it's possible to streamline the process without hurting them much?

Agile way of working has to be tailored well to the needs of your business. For web development, it might work well. They ship early and often.

The UML diagrams are kind of nasty things and mostly people doesn't really need a too detailed level of diagram it's good to have but it's time consuming. Agile modelling is entirely different and fairly simple. May be you can ask people to follow this approach and they will be ready to do it as most of us discuss things in terms of simple hand-drawn diagrams.

People hate changes and I am pretty sure the survey is not really going to help you much because if they're frustrated about the way of working they would have fixed it much early as a self learning organization. But here you're asking their help to solve your problem than theirs also you can't clearly say what really works for them. I suggest you to sit back and think, and make the process streamlined into the development process without much overheads.

In the abstract level you will have to clearly identify the problems at every level and tackle it one by one (start from top, that's easy) and thrive the team towards success.

Answer 7

How can I assess whether our programmers are able/willing to employ modern software development methods?

It sounds like the first thing you need to do is talk to your coworkers. From your question I get the impression that you don't understand why the current processes or lack there of, are in place. So ask somebody! It may be that your coworkers are aware of the alternatives and are choosing not to use them for reasons you can then discuss, or it may be that they are unaware, and you can then introduce them and have a discussion about how they could benefit your team.

Trying to change team culture, procedure, or structure without understanding WHY it is currently the way it is makes a recipe for disaster. Work to understand the concerns that have led to this place and then discuss some solutions that you may see. A fresh pair of eyes is great, but only if you engage in dialog with people who know the territory well.

Answer 8

Most methodologies and best practices for software design never claim to be able to get an application built in the shortest time. Your company is proof that you can "cowboy code" your way through a few all-nighters and get something released. Management is happy because they got "it" done on time and usually that means on budget.

Bugs show-up and you'll spend a disproportionate amount of time on them, but management doesn't care. Somehow they're reinforced for hitting that release date. Someone will have a great idea to enhance your games, but your code is too brittle to make such a drastic change. Oh well, what might have been.

Suggest applying one of your ideas to a particular area of one of your games. This way you can track and measure any benefits compared to your other apps and present these findings to management. Example: Suggest automated testing. Your fellow programmers will display very quickly whether or not they can implement it. Very few devs who have taken the time to become fluent at unit testing protest their use.

IMHO - the arbitrary time-frame management determines development is your biggest problem. You need to get the devs closer to the users and/or decision makers who are coming up with these ideas. Discover the true requirements. Just because there is a need to make games available on different platforms, doesn't mean they have to run in a browser. There's always more than one way to skin a cat.

Answer 9

"predictable... ways of developing software."

Pure processes have predictable results. But software creation is literally the creation a process, not just the following of a process. An analogy to manufacturing: Software development is like designing a factory to create widgets; not the day-to-day running of the widget factory.

All software creation processes are impure. (outside of automatically-generated solutions)

Impure processes do not have the predictability of pure process. The more on-the-fly creative action that occurs in the process the less pure it is.

Purity does not imply good or bad.

Answer 10

I find your question quite daring! I understand completely what you mean, I also share some parts of what you experience, that's why I reply.

About the methods, I guess you'd have to adapt to the way they do because it's going to fire back to you. If you put yourself in their shoes they might think "a just graduated guy is going to tell me (>5 years development) how to work, if I got promotions and rises by doing what I'm doing the way I did it until now", as you can understand, pretty dangerous statement that can develop to not nice situations with your co-workers, something I always avoid because you need to learn first the ways they use and later try to improve without confrontation.

The reply from your management I understand it as "show me the money or keep it quite", that's also not good, that puts you in the spotlight without knowing 100% what are you dealing with (meaning how effective the previous methodologies are being profitable to them).

In my case, the way I deal is applying the methods I know only to myself, I use UML with the code I need to maintain (currently I do it out of working hours) and I tend to explain why I did it without implying that's the correct way. (As you know UML is for a code as a map for an explorer. You can explorer without map, but don't complain if you make 2 extra km in the way back home :) )

Currently I'm getting more questions about the code than the original creator, mostly because I'm fresh with the code, but also because I also add a nice map (UML) with some notes on how to change something, in a short time (the owner of the code also keep making changes).

My advice use what you know for yourself, and if you become more efficient, they will move the spotlight towards you, THEN you can start talking about effectiveness, before you may put your credibility in jeopardy.

Answer 11

I think there are some important aspects regarding productivity (I am not an expert in management, so this is just my 2 cents).

Here is a simplified scenario to illustrate my points.

Suppose you have 2 packages of features (X and Y) to implement, and that it takes 6 months to complete the first package X (first release). Suppose also that package X is used by later modules (e.g. Y).

I think often programmers say "I do not have time to do it properly" because when they write X they know X is going to be reused in Y and want to clean up the code and the design in order to make it more robust and reusable. Even if the code is not going to be reused, they know that a clean code will save them a lot of time if they have to fix bugs later. That's also what you learn in school and I think these are good principles.

But from a management point of view (i.e. in the industry, the so-called real world), release 1 is the one thing that is 100% sure, and any extra time spent on X after X is good enough for release 1 is considered a waste of time. The customer buys working software, not clean code.

Six months later, when working on package Y that makes heavy use of X, you wish you had produced proper diagrams and design, you wish you had made your design more extensible, and you wish you had done this six months earlier, when you had the whole problem still in your mind.

Anyway, now Y needs to be done and so X needs to be extended (extra work). While quickly adding extensions to X, X starts to get messy and new bugs appear. So, instead of

• 7 months for release 1 (doing X properly), and
• 7 months for release 2 (doing Y properly, with little work on X),

you have

• 6 months for release 1 (minimal X implementation that does the job), and
• 9 months for release 2 (implementing Y, implementing workarounds and bug fixing on X).

In the second scenario you have lower productivity on average but from a management point of view, you can only plan in the short / middle term. You basically do not optimize what you might not need: you cannot plan work on both release 1 and 2, if you are not 100% sure whether there is going to be release 2.

I think this is a bit of a paradox: managers prefer to save one month on release 1 and spend two extra months on release 2 because the extra cost on release 2 is less certain than the saving on release 1.

So, I think much of the frustration of many programmers that "there is no time to do things properly the first time, but there is always time to fix them later" (and the resulting loss of productivity) is inherent in the way projects are planned: since we do not know how the market will be in 2 years from now, we cannot make optimal plans and we must accepts a loss of productivity.