PM opting for an overly-complex setup which nobody has experience with

Question

Recently I started a project which didn't seem too hard to make, the concept was a fairly simple application that had to accept input every now and then (maybe 10x a day), and try to perform some operations on them and collect all results at the end. This application would then get a front-end web portal that customers could use to view the results, not exactly rocket science.

For this I initially made smart use of Python's built-in concurrency libraries (ThreadPoolExecutor) and use an easy-to-use library for the front-end (I chose Flask as it's easy for beginners and is relatively easy to maintain and test).

Once we were halfway the project, the PM stated we had to use third party message queue capabilities instead of threads and had to implement load balancing, what eventually ended up happening was that we eventually started working with Celery, Redis, RabbitMQ, Nginx, uWSGI and a bunch of other large third party services which nobody had any real experience with.

In the end this lead to a bunch of spaghetti code, untestable tasks (because of the complexity of third party libraries, patching the code didn't even work) and a bunch of headaches because nobody even knew what the added value of these services were.

Before you say "Yes you should use those services", keep in mind nobody knows how to use these or even knows what they do besides introduce race-condition plagued code.

What should I do about this? At this point it would simply be too costly to revert back to what we had and the PM is dead-set on using these services, even though the end-product is now worse off than it was in the beginning. Is there even any use in discussing this with him? Do I ask for more time? Or the harsh answer, am I just too stupid for my job?

What problem does concurrency solve for you? Who will use this system? What business value does it accomplish? Are there significant scalability issues that will need to be addressed? As a developer you should the PM _why_ these tools and libs are needed. Then you might begin to understand how these tools will help if at all. — RibaldEddie, Jul 11 '17 at 22:50
You are working with an unproductive PM. You can either stay or go. Probably the same kind of silliness will happen with other projects under the same PM. — Frank Hileman, Jul 11 '17 at 23:18
Why is a PM making technical decisions?! That's a real project smell if I ever did smell one. — RubberDuck, Jul 11 '17 at 23:19
This is like buying a child a chainsaw and pressuring them to go outside and find a tree to cut down so it wasn't a waste of money. — JeffO, Jul 12 '17 at 00:00
Sounds like this project needs a strong technical lead who isn't afraid to laugh hysterically at a project manager pretending to be a solutions architect. You really should just nod your head in agreement, and then build the sensible solution anyway. Yeah. This wouldn't fly with me. — Greg Burghardt, Jul 12 '17 at 00:26
Isn't this more of a workplace-type question? This is really about how to handle the PM situation, no? Where's the _specific_ software engineering problem? Should this be migrated? — code_dredd, Jul 12 '17 at 10:29
Did you ask the PM **why** he insists on using third party message queue capabilities instead of threads and load balancing? Did you ask this at least [5 times](https://en.wikipedia.org/wiki/5_Whys)? And are you really sure about "it is too costly to revert back"? It is not unlikely this might be a misconception, at least in the long run. — Doc Brown, Jul 12 '17 at 12:56
@RubberDuck this times a million. Non-technical managers and product owners who solutionise rather than just give requirements are one of the biggest sources of wasted effort and friction I can think of. — GoatInTheMachine, Jul 12 '17 at 15:00
Oddly enough I would have expected a concurrency library to lead to more pitfalls like race conditions than message queueing — but I do not know the environments in question, and five+ products for a simple job does sound a lot! — PJTraill, Jul 12 '17 at 21:55
@ray: worth thinking about migration, but perhaps there is something about the PM specifying technology that is especially common in software engineering practice. — PJTraill, Jul 12 '17 at 21:57
@PJTraill The problem seems to be that a PM is overstepping their boundaries and demanding some tech is used and so on, not that OP is stuck on how the tech should be used, etc. My question is: where's the software engineering issue, exactly? I don't see it; OP is not about the tech, but the PM's demands. That's why I thought migration seemed like a good idea. — code_dredd, Jul 12 '17 at 22:01

score 89 · Accepted Answer · edited Jul 12 '17 at 08:55

89

Once we were halfway the project, the PM stated we had to use third party message queue capabilities instead of threads and had to implement load balancing

This isn't an appropriate thing for a PM to "state" unilaterally. Two reasons:

Design decisions should be made by a technical resource and only in response to NFRs. So politely ask your PM if there is a new NFR and if you could please have details.
If an NFR is being introduced halfway through the project, it should probably be done via a change control. The change control is very important from a governance perspective; it would not only be an input to your requirements, but also is an important input to QA's test cases, operations' deployment and support handbook, and (here is the really important part) the PM's schedule. If the new requirement introduces more work, the development team ought to have an opportunity to communicate new development estimates, and the PM will have to decide whether they can live with the new date, add more resources, or push back on the stakeholder who introduced the NFR.

Now if there really is a bona fide NFR, and there is no getting around it, it may also be appropriate to request new or different resources who have familiarity with the technologies that are being introduced, or request a training budget for some of your existing resources. So there is a cost aspect as well.

If you speak the PM's language – schedule and cost – I think you will get more traction than speaking about how developers feel about the resulting design. Those things have real impact.

A PM ought to know better than to introduce stuff like this on the fly with no governance, no controls, and no consensus. If they just don't get it, you may need to escalate to product management or program management, as (s)he is putting quality and schedule at risk unnecessarily.

edited Jul 12 '17 at 08:55

Søren D. Ptæus

399
1
7

answered Jul 12 '17 at 00:09

John Wu

26,032
10
63
84

21

Ok, this is the answer. A project manager should never be making these kinds of decisions. Money? Time? Project management handles this. RabbitMQ? Not a chance. – Greg Burghardt Jul 12 '17 at 00:31
I like this answer a lot. There are controls in place to ensure you don't just have hell just dropped on you. Sit down with him and talk about it. – Rhys Johns Jul 12 '17 at 02:32
3

However, one thing is that sometimes while it sucks, you might just have to learn a new technology or library. Will it take time, yes, but it might just be worth it. – Rhys Johns Jul 12 '17 at 02:32
5

As a project manager, I could not agree with this answer more. – James McLeod Jul 12 '17 at 03:03
13

In smaller organisations the "Project Manager" is often the boss. They may have the owner's\CEOs ear, and may effectively be the Technical Lead Developer or Architect or some ungodly combination. In this cases the scope of their remit is not clear. – Sled Jul 12 '17 at 14:42
An NFR, at least according to [the Wikipedia article](https://www.wikiwand.com/en/Non-functional_requirement), seems to be more typically something like 'the system should be stable', not 'the system should use '. – Kenny Evitt Jul 12 '17 at 19:48
@Kenny, are you saying that the requirement to use RabbitMQ is a *functional* requirement? If not, then it is automatically an NFR, by definition. – John Wu Jul 12 '17 at 20:03
@JohnWu No, I was just pointing out that the requirement is not typical – actually *abnormal* even – tho perhaps that's a deficit of the Wikipedia article given how relatively common requirements like this are. – Kenny Evitt Jul 12 '17 at 20:54
@RhysJohns: If you think learning new technologies sucks, software development may not be for you. – TMN Jul 13 '17 at 12:52
@TMN: I don't think learning new technologies sucks, i wouldn't be here if i did. I'm referring to the process of being forced to learn it. It doesn't matter if you love something or not, if you are forced to learn it suddenly mid project, it can be a terrible experience. – Rhys Johns Jul 14 '17 at 00:19

candied_orange · Answer 2 · 2017-07-14T02:58:06.547

What would be stupid is to let yourself get death marched.

What you are describing is that you've lost critical feel. There is no sense of control and no clear way back to it.

The last thing you should do is work hard, keep your head down, and quietly suffer until they finally admit that the project is doomed.

What you should do is think very hard about what you have every right to expect.

If they want you to use technologies that you don't understand you should expect time to learn them. Don't be ashamed of what you don't know. Use your ignorance as a cudgel. When they demand you use something ask why. Don't accept 'because'. Don't accept 'modern best practices'. Don't accept 'scale-ability' without getting real, testable, expectations.

By testable I mean they HAVE to tell you how many requests per day/hour/minute they want it to be able to do. Make clear you intend to build something to exercise this system according to those specs.

This way you can use a 30 day free trial to prove that the latest wiz bang thing they want is worth it or if your better off sticking to what you already know.

Now keep in mind. It isn't the tools that introduced race-condition plagued code. You guys did that. You need to learn HOW you did that so you can undo that.

And no. It isn't too costly to revert back to what you had. The PM can't have what they want just by demanding it. You have to push back until you can effectively use what the PM wants or prove it's not what the project needs.

Seriously, just giving in to this is unprofessional and deadly to the project.

I've been here man. More then once. It does make you feel stupid. That's really not it. You're just lost.

Talk to the PM. Honestly. Lay it all out. Show you're willing to learn but don't want to be taken for a ride. Never ever ever design or code based on faith. Make the PM show you how to do what they want. Don't pretend you understand when you don't. Don't say it'll be done when it wont. If you're going to believe in something believe in yourself. You have to be willing to say NO.

If that doesn't work polish the resume because you're going to need it soon. One way or another.

`Now keep in mind. It isn't the tools that introduced race-condition plagued code. You guys did that. You need to learn HOW you did that so you can undo that.` Yeah, this part especially sticks out at me. Whether it be celery or threads, any type of concurrency can introduce race conditions. The same issues may well have existed in the thread-based code. — Izkata, Jul 12 '17 at 17:02

score 10 · Answer 3 · answered Jul 12 '17 at 08:13

This should really be on workplace.stackexchange.com, because the problem is not really a software development question, but about workplace relationships.

If you are sure that your simple approach would have worked and produced a working result rather quickly, then your PM is a destructive force in your company that should be removed. Figure out how to get the news to the level above him: That your team had a simple, working solution that had made good progress, and for reasons that nobody can explain your PM forced you to attempt a much more complex solution, using a multitude of tools that nobody knows, nobody understands, nobody knows if they are useful at all, and that unfathomable decision of your PM caused you all the trouble and causes the project to be late and not working.

score 1 · Answer 4 · answered Jul 13 '17 at 15:06

There are two approaches to third-party libraries (and other components):

Use as many of them as possible
Use as few of them as possible

My approach is (2). It sounds like your approach too is (2), but the project manager likes the approach (1).

There are three ways you can handle this situation. Either you let the PM do whatever the PM wants, you try to convince the PM to change the approach to third-party libraries, or you vote with your feet and select another job.

If you want to convince the PM to change the approach, consider these arguments:

Time to learn. Each external library requires time to learn, in which time a competent programmer may write the desired functionality especially if a huge library was chosen just to do a very simple thing that could be done in few hundreds of lines of code.
Replaceability. If you have an external library, how do you ensure that if its development is stopped, you can replace it with another similar library? My solution is to avoid external libraries whenever I can, and whenever it isn't feasible, I write a simple wrapper to abstract the part of the programming interface I want. Usually the interface I want is much simpler than the interface the library offers. Then my code accesses the external library only through this wrapper, making replacements easy. Building your entire application on some framework is a big no-no. Servlets? Yes, they have been here for a long amount of time and continue to be here for the foreseeable future. Template engines? Yes, although they aren't exactly replaceable (you usually pick one and stay with that), the value they bring is huge, so select carefully -- and keep in mind that when switching template engines, you can have two template engines in the same application but you can't have two frameworks in the same application usually. Apache Struts? No, frameworks come to fashion and go out of fashion in a quick manner, and you usually can't have two frameworks in the same application.
Version hell. By selecting an external library, you have to update it to avoid security vulnerabilities, and updating it may break things. Well-designed components (such as Java JRE) are compatible with different versions, but my experience is that most libraries are crap due to imposing huge version hell. Also, component X may require Z version 1 and component Y may require Z version 2, and you may not necessarily be able to link Z version 1 and Z version 2 in the same application.
Security vulnerabilities. By selecting an external library, the number of easily exploitable security vulnerabilities against your application increases. Some could claim that in-house developed code resembles security through obscurity, but then again I would say it is still a form of security.
License issues. Each external library imposes its own license on parts of your program. For example, GPL libraries cannot be used in non-GPL programs, and LGPL libraries also require distribution of the source code along with binaries, which may take considerable amounts of bandwidth.
Application startup time. Each big external library slows down the startup time of your application. By writing a simple, lean library in-house, you can make the startup time of your application much quicker.
Memory footprint. By having X requiring Y requiring Z requiring A requiring B, you need X+Y+Z+A+B in the memory at the same time. By implementing only the equivalent of X, let's call it X', in-house, you need just X' in the memory. And usually the memory footprint of X' is less than the memory footprint of X.
Bug risk. The more lines there are in the external component, the higher is the risk that you will run into a bug that will be hard to fix due to the huge amount of code you need to understand. If you do the thing in-house, you usually do it with less lines of code (to do just what you need, nothing else), and thus, smaller risk of bugs.
Customizability. If I write SQL queries myself, I know what the query looks like and how well it performs on a given database engine and given set of indices. If, on the other hand, the SQL query is written by an external component, I don't know anything about its performance. I used to work in a company where each web page took several seconds to fetch. I suspected the cause was the Hibernate library they used fetched too much data automatically from the database when all you needed was one item and not all the items related to this particular item. I left the company before discovering the true cause of the slowness, because I didn't like the approach of using huge numbers of existing libraries.

Beware especially if a library calls itself a framework. This means the library requires you to build your entire application around itself. You generally cannot have two frameworks in the same application; they will fight with each other without peacefully coexisting. Web development utility library? Yes please, there are too few of these. If I ever find a better library than what I use now, I can use the newly found library in new code while continuing to use the old library in old code. Web development framework? A big honking NO!

Christophe · Answer 5 · 2017-07-11T23:19:09.180

Not knowing the context and the product strategy pursued by your management, it's difficult to objectively answer your question.

Here some objective arguments. It's however possible that it's not what you expected:

"Keep in mind that nobody know how to use these products yet".
Using only tools and techniques known perfectly will ensure high productivity. But it will considerably limit the ability to innovate. On some markets, it might be fatal to your product. For example, almost 30 years ago, I proposed to use windows 3.0 to develop a new version of a CAD product that was successful under MS-DOS. The product manager objected that this was not a proven environment, that it would be too complex and too difficult to learn for the team, and anyway, that "Windows will never be a mainstream environment"... I let you guess the success of his product 2 years later.
It's all a matter of costs and benefits. The cost of experimenting vs. the benefit of a scalability and deployability ensured by a third party that is experienced with huge installations and heavy workload.
The drawbacks of adding a new tech can be smoothened, with appropriate training, or initial support/coaching by an experienced consultant.

Ultimately, the economic choice is the responsibility of your product manager. Discuss with him the pros and cons, to ensure that he makes a well informed decision and does not underestimate the added complexity. And if he stays on his track, try to achieve your best: you've nothing to loose and in the worst case, you'll have a new tech on your CV.

score 0 · Answer 6 · answered Jul 13 '17 at 14:20

I think your PM is aiming for a hard to manage system that will produce a lot of maintenance work while it is live, so it will ensure your income.

Personally, you seem to be stuck with python, just forget python for a while, don't code in python for a year, learn new stuff, you will see there are other languages that can do the same, and probably better.

As others have stated, learn the tools before you start coding with them. Maybe suggest that it would be good to evaluate the necessary stack together, based on research of different tools that seem suitable to the task. Or maybe ask how he came up with that list, he could have had help from someone who is up to date.

score -2 · Answer 7 · answered Jul 12 '17 at 04:33

Developers should not be scared of learning to use new libraries, frameworks, technologies, etc. That is a core part of a developer's job description, and it is perfectly reasonable for someone to suggest that the team work with third party things that nobody has any experience with, or even to demand that the team do so if they're in a position to be making authoritative technical decisions for the team.

However, it isn't reasonable to expect that you can just pull a new technology (let alone several new technologies at once) into your stack and keep making progress. Significant time should have been scheduled to learn the ins and outs of the new approach and figure out a good design to incorporate the new pieces, during which no real progress on the actual product would be expected (from the people doing this learning/design work, which may or may not be the entire team, though if it's not there probably needs to be more scheduled time down the road for the people who learned to transfer knowledge to the rest of the team). That is the cost of making this sort of major change. Learning new technologies is part of the developer's job, but it's not something that just happens with zero time cost.

It sounds like that didn't happen from the question. People dove right in trying to somehow make build good implementations on top of technologies they didn't themselves understand. Of course the resulting code is terrible.

Try to convince your PM that the company will need to spend more time on this. It will either come in the form of stopping now, learning and assessing the new technologies, figuring out a good design and cleaning up the current implementation mess. Or it will come in the form of more time wasted on bugs, maintenance, costlier development, etc.

It's impossible to say whether the technical choices described in the question (load balancing, message queues, etc) are actually appropriate. I don't think that "no-one on the team has experience working with this before" is a good reason to absolutely rule out a decision, but it does increase the short-run cost of making that decision (which can alter the "best" decision for the context), and if your PM isn't considering that and is expecting the team to immediately become as productive as experienced people would be then you should push back on those grounds; they will be setting highly unrealistic project schedules, which isn't in anyone's best interest.

PM opting for an overly-complex setup which nobody has experience with

7 Answers7