16

For many IT people, including myself a few years ago, the ideal software development process would involve the creation of detailed design documents with lots of UML diagrams before a line of code gets written. (This looks like a description of the waterfall model but it's the same with agile, except that the iterations are smaller.)

During the past two or three years, I have completely changed my mind. I still think that a detailed requirements specification with associated test cases is absolutely essential. For large projects, I would also require an outline of the overall architecture before starting to code. But all the rest should be done in code as much as possible. In the ideal case there should be no description of software design except the code itself.

How did I come to this conclusion? Here are some arguments:

Feedback

Tools for writing documents or creating diagrams provide little feedback. Yes, there are modeling tools that do some consistency checks on UML diagrams but they are limited and come with a lot of overhead.

Without feedback it is hard to recognize and fix errors.

As soon as you write code, you get lots of feedback, for example:

  • Errors and warnings from the compiler
  • Static code analysis results
  • Unit tests

Errors can be quickly recognized and fixed.

Consistency

To make sure that the code is consistent with your documents you have to check again and again. If there are frequent changes, it is hard to keep code and documents in sync.

Refactoring

There are powerful tools and techniques for refactoring code while refactoring textual descriptions or diagrams is usually hard and error prone.

There is one precondition to make this work: The code has to be easy enough to read and understand. This probably cannot be achieved with Assembler, Basic or Fortran but modern languages (and libraries) are much more expressive.

So if my arguments are valid, there should be a trend towards less or more lightweight software design specification and documentation. Is there any empirical evidence for this trend?

Martin Maat
  • 18,218
  • 3
  • 30
  • 57
Frank Puffer
  • 6,411
  • 5
  • 21
  • 38
  • 2
    Upfront design fell out of favor due to agile development gaining in popularity as the industry matured. Languages getting more expressive and tools getting lighter weight made it easier to rapidly prototype, enabling more agile development. I think there is some causation between the two. – Reinstate Monica May 08 '17 at 21:16
  • 14
    To my experience, "the creation of detailed design documents with lots of UML diagrams before a line of code gets written" was never a good idea - at least not in the time since I was working as a professional programmer, which is more than one decade longer than UML exists. Scetching a high level design before coding, however, is and was a good idea when systems are expected to have a certain size. But UML is IMHO not the right tool for this. – Doc Brown May 08 '17 at 21:23
  • 2
    Too lazy for a proper answer: more expressive programming languages and more powerful computers leads to needs for increasingly capable and complex programs, which leads back to more complicated requirements specifications. – whatsisname May 08 '17 at 21:32
  • @whatsisname: On the other hand, more expressive languages make protoyping and complex logic easier to develop in a shorter time frame, lessening the need for extensive specifications and planning. – Robert Harvey May 08 '17 at 22:21
  • 2
    Recommended reading: [Beating the Averages](http://www.paulgraham.com/avg.html). – Robert Harvey May 09 '17 at 04:02
  • @DocBrown: Same here: been programming since the early 90'-ies and never did a complete upfront design but only sketches of the overall architecture, way before I heard the word "agile". Other developers I knew were doing the same. I often wonder who were all these developers doing waterfall agile people speak about all the time. – Giorgio May 09 '17 at 05:35
  • 1
    I've worked on a project with a complete UML design. Which we code-generated from. I came to the conclusion that I never wanted to do it again. It was a *lot* harder to change the UML that it was to change the code; and a large UML model is at least as unwieldy as a lot of source code. The "generated" code was hard to read and the generator left "markers" in the code. – Nick Keighley May 09 '17 at 11:26
  • Coding is design. Not all of the design but it is design (it is not true to say, design is coding). If we use the manufacturing metaphor, then we see that all programming is design and the manufacturing phase become very small, it is the copying stage. – ctrl-alt-delor May 09 '17 at 16:35
  • This question seems more about development processes, but in my mind, a "more expressive language" would be one used to specify the behavior of a system, without using imperative code. The code would not be written at all, only the specification, and the "code" part would be automatically generated, or rather, reused. While specifications can have bugs, at least we don't deal with mutable state. – Frank Hileman May 11 '17 at 01:04
  • @FrankHileman: Code does not have to be imperative. It can be declarative as well. – Frank Puffer May 14 '17 at 13:49
  • @FrankPuffer which language are you speaking of? – Frank Hileman May 15 '17 at 22:18
  • 1
    Re;"who were all these developers..." - They were the ones getting the applications built that "had to work". If your product can be delivered in a state where it "sort of" works then you probably can truthfully say you didn't "need" a design. If you work on products that can't fail then good luck doing that without a decent design. There's only 1 reason that design went out of favor. Design is hard, most developers are really bad at it, which makes it a waste of time for them. People that are good at design find it an indispensable part of the process. – Dunk May 15 '17 at 22:32
  • @FrankHileman: For example all functional languages and languages with functional features like the latest versions of C++ and Java. – Frank Puffer May 16 '17 at 07:44
  • @Dunk: My question is absolutely not about having no design. It is about doing design in code. – Frank Puffer May 16 '17 at 07:48
  • @FrankPuffer OK I thought I had missed a new one. I was thinking of a non-imperative language whereby only rules (specifications) need to be declared, and the rest would be generated. – Frank Hileman May 16 '17 at 23:54
  • @FrankPuffer I guess you do realize that Java and C++, by adding functional features, are simply catching up to language designs that are very old. So the addition of such features can also be used as an argument that there is no continual progress, rather, steps forward and steps backward both. – Frank Hileman May 17 '17 at 00:21
  • I don't think agile development is necessarily related to the expressivity of the language. Sure modern languages might have become better but the same methodology can well be applied to languages such as Java as well. It's really more of a revolution in ways of thinking. – xji May 21 '17 at 09:03

5 Answers5

9

I question the premise that languages are more and more expressive. With today's ASP.NET code in c#, I write at about the same level as I did when I wrote ASP code in Visual Basic. We still use c++. Javascript has added features but overall the language has not changed. Same with SQL.

I think these other changes are more significant:

  1. Adoption of automated unit tests. Some would say that the tests are the specification. So we haven't removed the need for writing specs; rather, we write them in code rather than in Word documents.

  2. Changes in deployment methodology. In the past it was very expensive to make a mistake because you would need to ship an updated copy of your software. So you had to be careful. With web-driven applications, you can deploy fixes for immediate use, and you can afford to react to problems rather than anticipate them, restructuring code as you go.

  3. Adoption of design patterns. When everyone knows the patterns, you hardly have to design anything; you can just say "add a repository factory" and your team should be do it without needing to see UML.

  4. SOA's data contracts. Pretty much everything is SOA these days. All you need is a WSDL. The days of defining and documenting data transfer formats are gone. The current move toward more RESTful and micro-services continues this trend.

  5. Softwares are smaller. Partly as a result of SOA architectures, teams are writing smaller programs that are tied together. Each individual component is less complex and requires less up-front design; also, it is easier to change part of the architecture without breaking the overall solution because of the fire-breaks forced by the interface definitions between components.

  6. Much, much more use of established libraries. The .NET CLR offers a ton of functionality off the shelf, so no need to design, say, a scheme for caching session data. Third party libraries like jQuery UI or Bootstrap establish standards for writing code to work a certain way. You don't need to document these; teams should be able to just use them.

  7. Industry maturity. SWEs have learned that there is no such thing as a "Battlestar Galactica" project where you spend years and years trying to reach a specific goal; by the time those years have passed, the goal will have changed. Today we know that time to market is much more important than getting everything exactly the way we want it in the design.

  8. Better- and more consistently-educated engineers. These days you can hire engineers who understand best practices (hopefully) and will just implement them without a design document telling them what to do.

  9. Productivity tools like TFS allow you to write a simple task that references a use case and provides a couple bullet points for any ambiguous technical decisions. You don't need much more than that. Developers are able to review, estimate on, get code reviews on, check in, etc. everything through the tool. This is much more efficient than working with external documents since it ties everything together.

You still need design documents for some things... for example, if your application is split into different components that are developed by different teams, you at least need to tell them which component is responsible for which requirements. But for the most part, today's development methodologies allow much more freedom while giving you tools to manage and contain any risk that may arise from a developer making a poor decision.

John Wu
  • 26,032
  • 10
  • 63
  • 84
  • 3
    Our industry *has* matured... a little. Item 5 is by far the most important change; it positively impacts all of the others. But the fact that we change over all of our technologies every 5 years suggests that we still have a long way to go, and language expressiveness (the one thing that is relatively stable over time because meaningful improvement in that area is so painstakingly difficult) produces more gains than you give it credit for. – Robert Harvey May 09 '17 at 02:41
  • 1
    It is not all progress: the main hopp forward was, gracefully, with the invention of the compiler. But we still teach flow charts, an abstraction on assembler code (and many other obsolete practices). Probably because we have forgotten why? – ctrl-alt-delor May 09 '17 at 16:40
6

I would argue for No.

For simple reason that

For many IT people, including myself a few years ago, the ideal software development process would involve the creation of detailed design documents with lots of UML diagrams before a line of code gets written.

Was never considered "ideal", as Extreme Programming has existed since 1990s. And as you say:

In the ideal case there should be no description of software design except the code itself.

Was argued for ages ago. For example this legendary article from 1992 : What is Software Design.

The above shows that you can have "extreme" process with highly evolutionary architecture and iterative approach without the need for complex languages or IDEs.

Instead, I would say this "seeming" shift from design-up-front with lots of diagrams and use case documents to more evolutionary and iterative approach is simply "old-school" managers being replaced with new ones, who grew up in much more dynamic environment and for who it is much easier to accept and work in more "agile" environment.

Euphoric
  • 36,735
  • 6
  • 78
  • 110
6

I pretty much agree with this but I think it started much earlier than you imply. I also think that there is another big factor aside from expressiveness. When my father first started programming, he had to create punched cards, and schedule time on the computer. You might get one chance a day to run your program. There wasn't a lot time to waste building code, letting it fail, and then fixing it. You got maybe 2 or 3 shots at it and if it wasn't working, you were in trouble.

The risk of this meant it was crucial to spend a lot of extra time to plan out your program. People would write their code out in pencil, and then transfer it to the punched cards. As technology progressed, you could code right into the terminal but you were still using shared resources and CPU was expensive. A test-first methodology would be completely untenable in that world. If you didn't plan ahead, your peers would be at your desk with pitchforks.

Computing resources have become cheaper and better at a incredible pace. Many of the constraints under which all of these practices were developed are completely obliterated. As Euphoric points out, the move away from this really got started in the 90s. The continuation of much of the big up-front design has been pure inertia.

So, yes, the improved expressiveness of programming languages has had an impact on this from the simple fact that it's easier to use expressive code as it's own documentation. The cost of producing a document that tells you what the code says is very high and it's value (it's inevitably wrong at some level.) At the same time the cost of throwing shit at the wall and seeing what sticks is basically negligible.

JimmyJames
  • 24,682
  • 2
  • 50
  • 92
3

I think you forget the purpose of having design documents in the first place!

Design documents (requirements, use cases, mock-ups etc.) allows us to describe, understand and discuss the system at a high level. The amount of details left out of such documents is what makes them useful.

There is no need to have documents which describe the exact behavior of the system system in all details, since indeed the source code itself serves this purpose.

Software development could be considered the process of transforming high-level human readable specifications into low-level unambiguous specifications which are executable by a machine. But you need some input for this process.

JacquesB
  • 57,310
  • 21
  • 127
  • 176
  • Sure, high level documentation that doesn't deal with details is absolutely required. But that's exactly what I expect from a good programming language. It should allow to write code at different levels of detail. Code doesn't have to be an unstructured mess of low level instructions. – Frank Puffer May 09 '17 at 14:32
  • @FrankPuffer a diagram is worth 100,000 LoC. – RubberDuck May 09 '17 at 16:19
  • 1
    @RubberDuck It occurs to me that the ability (or lack thereof) to generate various useful diagrams from code could be a measure of the expressiveness of a language. There no diagram that you can draw that cannot be expressed in some sort of language. The question is whether that language can convey the same information in a way that can easily be written or read. – JimmyJames May 09 '17 at 16:53
  • 1
    @RubberDuck: Diagrams do make sense in some cases, for example to explain the overall architecture. But I don't think that they should be the default. There are definitely good and useful diagrams but unfortunately most UML I have seen is more confusing than helpful. And what's worse, it often differs from the actual implementation. – Frank Puffer May 09 '17 at 16:55
  • I like that you mention *generating* diagrams @JimmyJames. I prefer generation over manual creation. You've a very valid point, but I wonder if it's a function of expressiveness or tooling. – RubberDuck May 09 '17 at 16:56
  • @JimmyJames: I absolutely agree. All of the disadvantages I listed in my question do not apply to generated diagrams (and generated documentation in general). – Frank Puffer May 09 '17 at 16:58
  • @RubberDuck I had the pleasure of working with an IDE a long time ago for Java that basically treated UML and code as two sides of the same coin. Updating the code updated the diagrams and vice-versa in real time. It was really cool but in order to keep all the detail of the UML in the Java code, it had to use comments (this was before Java had annotations.) So I conclude that Java did/does not have the ability to express those elements of UML. I wonder if anyone has approached language design from this angle. – JimmyJames May 09 '17 at 17:00
  • The creator of Matlab Simulink did. – RubberDuck May 09 '17 at 17:04
1

In the ideal case there should be no description of software design except the code itself.

This is further off than you imply. Even things like dependent types (which, as I understand it, are quite promising theoretically) are several years out.

Formal verification is quite difficult, and for that reason the only place where formal verification is commonly used is cryptography libraries.

Unit tests

If property testing hasn't hit mainstream, I don't think this will be practical for a long time.

Moreover, writing good tests (that will not need to be edited with every refactor but will still catch enough errors) is quite difficult.

To make sure that the code is consistent with your documents you have to check again and again. If there are frequent changes, it is hard to keep code and documents in sync.

Probably easier to use a documentation testing tool at the moment.

There is one precondition to make this work: The code has to be easy enough to read and understand.

Unfortunately, designing languages that are not only extremely expressive but also eminently readable is difficult. Languages such as Go prioritize readability, and it hobbles higher-level thought.

Finally, in my experience, better languages and tooling do not lead to software with fewer bugs, but rather larger projects. There isn't really any plausible way pandoc would have been written in 1970.