8

I've been working as a programmer on a project designed for generic software for gas stations (to be redistributed for many customers) for 18 months. The project is big. Today we have around 150 tables. We haven't used a specifc approuch, it was not well managed.

The person table has today around 70 columns, but 15 months ago it had around 30 columns. These new fields emerged in order to integrate with other modules such as sales, financial and accounting. Also many fields were created then deleted.

As a result, we had many refactoring and rework. The project never gets ready because there are always new requirements emerging.

Here's my doubt: if we had used a usual approach of specification, we would had interviews, a requirements document, activity, sequence and class diagrams, so we would know since the begin that the "person" table would need 70 fields, then we had avoided a lot of refactoring.

Could scrum help in this project? I have a feeling that in this case scrum would end up in a lot of refactoring as well.

I'm only a programmer, not a project manager. I'm wondering how it should have been done: with scrum or with big design up front.

Edit

Just to complement the end of this story. Eight months later I asked this question, after put the project in production in some "test costumers" the project officialy failed. The product owner decided to abandon the project. It got hard to fix problems and a lot of perfomance problems occured.

Murilo
  • 219
  • 2
  • 7
  • 8
    In Scrum, it is assumed and accepted that you can't know everything up-front and that you will be doing lots of refactoring anyway. – Bart van Ingen Schenau Oct 29 '15 at 11:28
  • 2
    Refactoring is often considered a core part of any agile process, so it is likely that Scrum would have resulted in the same amount of refactoring or more. I suppose the question then is, why do you think this is a problem? What agile tries to address is that up front requirements capture and design is flawed, and you are better delivering value to a customer incrementally to discover the true requirements. If you could guarantee a perfect plan up front you could probably do it that way, and still deliver value incrementally. But would that initial requirements capture have been perfect? – Robin Oct 29 '15 at 11:32
  • 3
    I'm not sure the title reflects the question you're asking. – gbjbaanb Oct 29 '15 at 11:36
  • Thank you all for the comments. They cleared up some things on my mind about scrum. I can change the title of the question as well, just give a suggestion or edit it. Actually my main problem, as developer is that some changes affects things that were working. I had a doubt or even a fear that with scrum I couldn't have a "traceability matrix" of the requirements. I agree that initial requirements wouldn't be perfect. – Murilo Oct 29 '15 at 11:43
  • Is your system already running in production? For how long? If yes, why do you worry? I guess if you had tried your "big upfront design" approach with 70 fields for a single table, your system would still be under development for 18 months, with no benefit for any user. – Doc Brown Oct 29 '15 at 11:55
  • It's not in production yet. – Murilo Oct 29 '15 at 11:59
  • Tables? Fields? What are you talking about? – Lightness Races in Orbit Oct 29 '15 at 11:59
  • 6
    @LightnessRacesinOrbit: he is obviously talking about some kind of inhouse database application, with a DB schema with 150 tables. Some people seem to assume there is no other kind of software. I recommend them to read Joel Spolsky's [Five Worlds](http://www.joelonsoftware.com/articles/FiveWorlds.html). – Doc Brown Oct 29 '15 at 12:30
  • @DocBrown: Right, so he should add some explanatory text to the question to introduce the project, instead of just diving in with domain-specific terms. – Lightness Races in Orbit Oct 29 '15 at 12:40
  • 1
    Relevant Dilbert: http://i.stack.imgur.com/8zpS7.gif –  Oct 29 '15 at 13:16
  • 2
    @Murilo "Some changes affects things that were working." This is a common and expected problem with any changes in a software system, to help mitigate this problem Unit Tests can be developed (along side primary development) to give insight into what effect refactoring has on existing systems. – YoungJohn Oct 29 '15 at 14:28
  • @DocBrown: Thank you for the Five Worlds link. This should be required reading for every CS Bike Shed expert, *especially* for those on sites such as this. – JS. Oct 29 '15 at 18:08
  • Possible duplicate of [How do you deal with changing requirements?](http://programmers.stackexchange.com/questions/874/how-do-you-deal-with-changing-requirements) – gnat Oct 29 '15 at 18:23
  • 2
    70 columns in a user profile table - what in god's name are you guys storing, the user's underwear size? – JonH Oct 29 '15 at 18:58
  • Thank you all for the answers and discussion. Now I can recognize the problem was a combination of lack of requirements and bad architecture. You all gave me a lot of terms, concepts and links to study, so I can be a good project manager a day. I will be reading the post flagged as a possible duplicate to judge if I believe there is already an answer there. – Murilo Oct 29 '15 at 19:27
  • as a side problem to what your team is doing I'm almost certain that you are building a monolith. The Person table with 70 columns is a concrete proof of that. If Person table needs to hold data for several other projects then you are building a canonical model that you hope to fill all needs. Believe me it doesn't end well :) Have a look at [Bounded Contexts](http://martinfowler.com/bliki/BoundedContext.html) which is a core concept when building big applications using Domain-Driven Design – Songo Oct 29 '15 at 21:33
  • @Murilo: FWIW, if you still believe your problem is caused by a lack of requirements or bad architecture, please read the comments I gave to those two answers naming those things as a possible causes, and the reply of the authors of that answers. Both agreed that these two things are most probably **not** the cause. – Doc Brown Oct 29 '15 at 21:37
  • ... they are just symptoms of a completely different problem. – Doc Brown Oct 29 '15 at 21:45
  • Eight months later I just added the end of this story in the question @DocBrown and other guys – Murilo Jul 15 '16 at 20:43

4 Answers4

18

It looks like you have churned your way in an uncontrolled development process to create a never-ending development system. This occurs in agile systems too.

The root problem is a lack of requirements, and while your solution might seem to be to use an agile methodology to fix this (as agile is designed around changing requirements) it would not solve the problem.

Even agile methods require a fairly firm starting point. They respond well to changing requirements, but they are just as useless as any other methodology if you start with no requirements. You still have to have a plan that you're heading towards before you start. Agile helps when that goal drifts, it does not help you define that goal.

Now its true that a fixed up-front design is too rigid if you do not know exactly what you are building.

So, I think you're going to have to move from your current 'chaos' methodology to something more organised and will have to implement a fair amount of design and planning. You can try to do this in one go with a heavy methodology, or you can be more flexible with an agile one. What you cannot do is expect any methodology to fix your lack of initial planning.

Incidentally, Kanban sounds more appropriate for your needs. Scrum works best with small teams and projects. Kanban can work with larger projects but it also works with a throughput of work approach, so design changes are continual and not chunked into sprints. I think you'd have more success with that.

gbjbaanb
  • 48,354
  • 6
  • 102
  • 172
  • You did some very good statements. This is really a chaos. I wasn't clear enough: I don't want to fix it, because I'm just a developer of the project, so I have no ways to change it now, but I was wondering how it would be done correctly. Thanks you so much for your good answer. Ps.We use a Kanban board here in order to try help. – Murilo Oct 29 '15 at 12:05
  • 2
    Your answer is not bad, but "The root problem is a lack of requirements" ??? Honestly, for me it sounds quite the opposite. The OP has already far more requirements on the table than he process and manage seriously. – Doc Brown Oct 29 '15 at 13:12
  • @DocBrown Yes, fair enough - not a lack of requirements but one of properly managed and/or designed requirements - ie too much "just do this" and not enough "here's your thought-out spec" - which should have reduced the refactoring he talked about. Maybe I'm wrong though and interpreted the question badly. – gbjbaanb Oct 29 '15 at 13:59
  • @Murilo Based on my experience, if you're wondering "how it would be done correctly," the answer has remarkably little to do with the techniques applied, and everything to do with the people implementing the techniques and managing the project. – Cort Ammon Oct 29 '15 at 23:16
  • 2
    _"Agile helps when that goal drifts, it does not help you define that goal."_: This is an excellent summation of agile: – Bryan Oakley Oct 30 '15 at 00:34
12

18 months, 150 tables and still not in production? Sounds much like a death march for me. The only way you can fix this, if there is any chance to save this now, is to narrow the scope of your project dramatically - at least for your first production release. What you need is proper release planning, small, reachable goals and getting the system to the end user as soon as possible.

And when you have your first release in production, with only a tenth of the requirements implemented, you will have to extend it step-by-step by the next block of requirements, which will cause "refactoring". You will also get feedback, which means bugfixing and changed requirements, which will cause refactoring, too.

Now to you question - will Scrum help? Maybe, maybe not. Scrum is a tool to support iterative development and changing requirements, and focussing on the important things first. Other "Agile" methods do this too, and a not-so-formalized process might handle that also. But as long as you try to bring a monster like this in one "big bang" into production, it does not matter if you use "agile" or "upfront" development, both will fail.

So before thinking about Scrum, first rethink your goals and your release strategy, and then check if Scrum is the right tool for that, not vice versa.

Doc Brown
  • 199,015
  • 33
  • 367
  • 565
  • 1
    I would contend that "bring a monster like this all-at-once into production" isn't agile at all. Since the OP claims to be using [kanban](http://programmers.stackexchange.com/questions/301189/should-i-use-scrum-for-big-projects/301199#comment627354_301192), the focus on a minimal viable product appears to have been completely lost from the very start of the project. –  Oct 29 '15 at 13:14
  • @MichaelT: I agree, it is debatable if a so-called "agile development" without delivering something usable can be really called "agile": – Doc Brown Oct 29 '15 at 13:21
  • I agree with this answer. You have to start somewhere, start small publish your code, get feedback, build on the feedback, its just a never ending cycle. At least at that point you have someone using this tool. – JonH Oct 29 '15 at 19:01
  • 1
    @MichaelT where I work, project teams are agile, but the support infrastructure definitely is not. So we as a team deliver something that can be used every few weeks, but we only have one production deploy every 3-4 months at best which is the state of what's ready for production at that time. Organisation in flux, hopefully. Took them 3 months to get us a test server for example. So we just made do with a laptop as a project internal test server until that time... – jwenting Oct 30 '15 at 06:34
8

If you can't manage requirements and don't have people capable of implementing requirements properly, SCRUM isn't going to help you (much), and that seems to be the real problem you're facing.
SCRUM can help you better deal with changing requirements than more static project management systems, but it's not the holy grail that will magically make everything work. In fact, unless your people are on board, willing and capable of working with SCRUM, and so is the rest of the organisation, it may end up making things worse.

If you've a table that's grown so much to fit in stuff to link with other systems, I postulate your database design is seriously flawed for example. No amount of SCRUM is going to improve your database design without you including people who are good at database design in your team and not being afraid of those design changes and the changes they will cause to the rest of your system.

jwenting
  • 9,783
  • 3
  • 28
  • 45
2

Please note that when I wrote this answer, I didn't realise that the system was not in production yet.

They way you describe your product, I don't think that your immediate problem is that of managing requirements, nor your development process. It is that of your system architecture.

You have managed to create a monolith - and a rather large one at that. 150 tables is a lot for one system*. In particular, you mention that you have 40 new fields during the last 15 months just for integrating to external systems. I would seriously consider splitting your system into several autonomous services, probably starting with services for integrating to external systems - but later identifying separate business areas implemented in you monolith, and then perhaps split those concerns into separate services.

If you manage to split this monolith into separate maintainable code bases, you can also split your developers into smaller teams with well-defined responsibilities in specific areas of your business, and you can have a lot of smaller agile teams maintaining their own code base, instead of all hacking in the same code base.

As to why you have arrived at such an architecture, the root cause for your problem, there can be many answers. Maybe it's rooted in you development process, maybe all your developers are only experienced with transactional consistent software, or maybe it's a consequence of how your organization is structured (you are the victim of Conway's Law). I think there's a good chance it is a combination of the latter two.

I don't think that implementing scrum, or being better at managing requirements will help solve your immediate problem, nor the root cause. Adjusting the architecture for the complexity of you system will, and addressing the root cause for why you have build such a system will.

* Some will probably argue that they can maintain a system with 150 tables - or they have maintained much larger systems, but I do believe that most developers will consider this a large number of tables for one system.

Pete
  • 8,916
  • 3
  • 41
  • 53
  • 1
    This sounds for me like a technical suggestion to an organizational problem - which to my experience seldom works. A system architecture with 150 tables for a single system can be fine - problems start is when you try to develop and deploy such a system in one "big bang". – Doc Brown Oct 29 '15 at 13:17
  • @DocBrown - I agree with you. At the time I wrote this answer, it wasn't clear that the project was not in production yet. I then incorrectly guessed that it was. – Pete Oct 29 '15 at 17:17