35

The system I'm working on started as a small training project within the company I work for but quickly caught interest from management as a means to help to standardize the existing excel-heavy workflows. I'm one of the lead developers on the project, despite having no lead experience prior and only being in my third year as a software engineer (NOTE: this is my first professional project).

I recently used the strategy pattern to write a validator to handle different possible validation tasks for a given object (working in Java with Spring Boot currently). My goal was to make it easy to add new validations without risking affecting other parts of the code and make it easy for other members to extend it. I thought this was a good approach, until I told my manager. He said he doesn't want new logic to be written each time there are new validation paths because he doesn't want to do an official release each time we have logic changes (we have no DevOps pipeline, Kubernetes, etc. due to IT department restrictions, so everything deployment-wise is manual at this point). He wants everything to be configurable to the point that no additional logic is needed or housed in the RDBMS so that we can just insert new records into some table or modify a stored procedure/function and call it a day. He also claims this will reduce complexity for new members.

Personally, this seems like a poor design decision. Moving everything to SQL is going to add more complexity over time, even if in the short-term things work out. Also, SQL - while powerful when it comes to data querying/organizing - doesn't seem all that great when you have to write tons of cases to account for different validation rules, etc.

Are my concerns with my manager's approach justified, or is he correct?

EDIT 2023-04-05:

I wanted to provide additional context to the validation process mentioned in the above post. The part of the application that handles the validation is new. Right now, most of the company's core data housing and management is in countless Excel sheets. Upper management has expressed a desire to move away from Excel, particularly to streamline some of the more tedious internal processes. As a part of this, one of the departments is attempting to standardize the available options in their existing excel-heavy process (previously, all of the options were free-form text fields). At the moment, not all options have been standardized, but each needs a series of validation checks if chosen. So far, they've been straightforward and amounted to various equivalence checks. However, I don't know if that precedent will hold over time because neither I nor my manager are familiar with this department and all of its processes.

ElderFuthark
  • 467
  • 2
  • 5
  • 35
    Whatever you do, just make sure to avoid recreating the [Enterprise Rules Engine](https://thedailywtf.com/articles/The_Enterprise_Rules_Engine) – ThatCoolCoder Apr 05 '23 at 06:51
  • 7
    "I'm in my third year working as a software engineer" uh-huh "and I cannot help but feel that maybe my designs are awful" yeah that's to be expected, the odd thing would be if they *weren't*. Don't beat yourself over it. You may actually still be right in this particular case! – Jared Smith Apr 05 '23 at 17:10
  • 1
    One way this could be handled is to embed a Python (or Lua or whatever) interpreter into the program, and then you can distribute Python/Lua/whatever scripts by whatever mechanism, and these scripts can embody the rules and call your program's APIs to enact them. That can avoid the need to rebuild/reinstall software, although it won't avoid the need to test and verify the new code (now you'll need to test/verify Python code instead of Java) – Jeremy Friesner Apr 05 '23 at 20:38
  • 1
    Worth saying that if you move your strategies to Groovy, you can persist those as Groovy files or text in any RDBMS row. Groovy code can be executed in runtime by Java code. The problem is testing. You want any groovy snippet you code, to be testable and tested as any other source code. – Laiv Apr 06 '23 at 11:13
  • I would write up a document explaining the advantages and disadvantages of each approach (and be fair and thorough about it), and send it to your boss, and cc anyone else who might be useful. At worst, your rear will be covered if the manager's preferred process goes badly, and at best they might read it and reconsider. – nasch Apr 06 '23 at 16:35
  • *Personally, this seems like a poor design decision* -> NO. It IS a bad decision. Argue your position, do not equivocate. They want confidence that you are confident. Be prepared with short, focused, reasoned answers to specific counterpoints. Connect these to desired goals. Focus on what is right; the roots of stupid ideas run deep - explaining why something is wrong is surprisingly very hard. – radarbob Apr 07 '23 at 01:53
  • *"just modifying a stored procedure or function"* absolutely requires a formal release process, with traceability, test sign-off etc. It scares me that anyone would dismiss it as a trivial change you could do on the fly. – William Robertson Apr 13 '23 at 10:00

6 Answers6

62

Problem scenarios like these tend to suffer from a very confused train of thought that mixes ideas. In an attempt to answer this, I am going to have to systematically unravel all of the discrete elements here.

I'm one of the lead developers on the project, despite having no lead experience prior.

That's not necessarily a problem. You can be a lead because you are a generally experienced developer, but you can also be a lead because you know the business requirements and are able to articulate those needs well. I suspect you are in the latter category here.

I recently used the strategy pattern to write a validator to handle different possible validation tasks for a given object (working in Java with Spring Boot currently). My goal was to make it easy to add new validations without risking affecting other parts of the code and make it easy for other members to extend it.

There's not enough info for me to judge your implementation but it seems you have the right idea and goal.

He said he doesn't want new logic to be written each time there are new validation paths

Is your manager aware he is managing a software development team, and not a software development avoidance team? Joking aside, there could be some merit to his comment depending on what a "new validation path" entails.

For example, if you have validation that checks if a field contains any swear words, I do agree with your manager that you should be able to update your list of swear words without needing redevelopment and redeployment.

However, if the new validation path is algorithmically and logically different in a way that is more than just a value change for an already existing validation; then redevelopment and redeployment is very much warranted.

The rest of this answer assumes the previous paragraph is the case. If this is a matter of configurable values, your manager is correct that this should be configurable without needing development and deployment.

He wants everything to be configurable to the point that no additional logic is needed or housed in the RDBMS so that we can just insert new records into some table or modify a stored procedure/function and call it a day.

Your manager would effectively be arguing that you should be writing a Turing-complete validation rule engine so that new rules can be designed without making changes to the engine, which is a massively different remit compared to what you have described the current project to be like.
Would it be cool if you had such an engine available? Sure thing. Is your manager willing to commit the time and money needed to design such an engine? Doubt it.

because he doesn't want to do an official release each time we have logic changes (we have no DevOps pipeline, Kubernetes, etc. due to IT department restrictions, so everything deployment-wise is manual at this point)

This is the core issue. It's not a development issue, it's a management issue. The deployment process is too cumbersome, and rather than either sucking it up or improving the process, your manager is telling you to halt your development.

This is the equivalent of an inefficient hospital refusing to improve their ability to treat sick people, and instead tell the ambulance drivers that they should not be bringing in new people so frequently.

Not only is it an idiotic solution to the presented problem, it is the antithesis of what your manager should be achieving. They should be empowering their department to develop software bigger, better and faster, and instead they're undercutting the development of software so they don't have to be bothered with having to manage all this development.

He also claims this will reduce complexity for new members.

I refer back to my Turing-complete validation engine comment as a counter to their claim.

I suspect, based on my own personal history, that your manager might be more familiar with SQL (and Excel) than they are with programming languages and general software development methodologies.

In that light, SQL might be easier for them to understand. I've seen countless managers make these kinds of blanket statement subconsciously based on their own knowledge. It's usually not willful but it does still miss the mark.

Personally, this seems like a poor design decision. Moving everything to SQL is going to add more complexity over time, even if in the short-term things work out. Also, SQL, while powerful when it comes to data querying/organizing, doesn't seem all that great when you have to write tons of cases to account for different validation rules, etc. Am I crazy?

You are the only sane man (irrespective of your gender, I'm just referring to a specific idiom). If you haven't already, I feel like you're going to enjoy reading Dilbert cartoons because of its similarities to your workplace as you've described it.

Flater
  • 44,596
  • 8
  • 88
  • 122
  • 51
    Of note: if you implemented a VM that interprets rules in the DB, you still have software development and releases => they just happen on an underspecified language with no tooling and the release process is updating the DB. It's going to fail, and hard. – Matthieu M. Apr 05 '23 at 15:16
  • 4
    @MatthieuM.: Either that or you end up with a [customer-friendly system](https://thedailywtf.com/articles/the_customer-friendly_system) that is managed exactly as if it *was* a "real" development platform, but was not designed to function as one. – Kevin Apr 05 '23 at 17:53
  • 1
    It seems a bit more like a hospital asking paramedics to treat patients more thoroughly on the ambulances so the hospital doesn't have to do it, than asking them not to bring in patients at all – user253751 Apr 06 '23 at 01:27
  • 10
    @Kevin this anti-pattern has a generic name: [the inner-platform effect](https://en.wikipedia.org/wiki/Inner-platform_effect) and it [has been featured](https://thedailywtf.com/articles/the_inner-platform_effect) on thedailywtf – user253751 Apr 06 '23 at 01:28
  • 1
    @user253751 Your analogy focuses on the complex Turing-complete rule engine that's being asked for. My analogy focuses on wanting to avoid the deployment (and therefore wanting to avoid the development that leads to it - I doubt the manager considered the extra effort required to do what they ask). Both make sense in a way. – Flater Apr 06 '23 at 01:33
  • 1
    One counterpoint to the rule engine: if you are using a Prolog-like engine inside your application, you can (theoretically) store a relatively compact declarative representation of your desired validation logic in the database. I'm sure there are substantial problems with this model in real life, but it has a lot of theoretical appeal. – shadowtalker Apr 06 '23 at 14:48
  • 9
    There is usually a reason, why the deployment process is cumbersome (often quality-gates stemming from regulatory requirements) - as soon as someone from quality control understands you are actually deploying code into SQL tables, they will (rightly so) require the same cumbersome rules for any update of these tables and you have won nothing, but actually made it worse. – Falco Apr 06 '23 at 17:27
  • 1
    Spending so much effort developing a custom Turing-complete rules engine for this specific task when the OP is already using the rather powerful general-purpose Turing-complete rules engine called Java illustrates the problem here. It's all programming: the only question is whether you'd rather program this logic in Java or do a lot of work to be able to do it in some language you're making up. Occasionally, treating code as data in a domain-specific language really can be the best solution, but that's both rare and requires a lot of work to build the tooling to overcome its problems. – Zach Lipton Apr 07 '23 at 06:22
  • 1
    @user253751 I'd say they want OP to replace the regular ambulances with a big truck that has a operation table, dentist chair, x-ray machine, CT scanner etc. all built in such that you can pull them out while driving around the hospital - you just bring the patient to the reception to process their card and then back into the ambulance. That way, there is minimal chance the health inspector sees that you regularly forget to clean your operation table and you don't even have enough power for the xray scan on the truck. ^^ But: it's way faster to add new machines! – Frank Hopkins Apr 08 '23 at 04:49
24

Monkey Patching

It seems that the requirement from your boss is to have the ability to Monkey Patch the system while bypassing all other software development tools, processes and methodologies.

Monkey patching can be a workable strategy in any system when its scope is limited to things with a low business risk and business reach/impact.

There are clear advantages in some circumstances when the boss can just walk over to a developer's desk, let them know what needs doing then watch the magic happen right in front of their eyes over a cup of coffee; such an environment tends to be optimal for way-out-there crackpot innovation where any failure is easily contained and the risk to the business of the entire thing imploding on itself is something that everybody in the business understands and accepts.

Risks

If monkey patching is something which bypasses all development tooling and process, then it's important to understand everything which may potentially be sacrificed and the risks:

  • Changes are not version controlled; no the history of earlier revisions, no 'revert' mechanism, no traceability of who changed something, why, when or how.
  • Potential for vast differences and divergence between what developers see and what users see
  • No way to prevent non-peer-reviewed changes from making their way into production.
  • No static code analysis or vulnerability scanning prior to deployment.
  • No automated testing prior to deployment
  • No manual acceptance testing prior to deployment
  • No sanity checks against potential impact to the stability of the system or integrity of the data
  • Debugging and fault diagnosis happens in production.
  • Faults will likely require developer tools and configuration to be deployed into production (Need to assess the potential impact of this - particularly on security and performance)
  • Real, live, end-users are the QA testers - realistically they will often be the very first people to notice a new bug or when something breaks. (Are they OK with this?)
  • No protection against accidental (or deliberate) misuse or damage of live data
  • More humans given access to the live database only increases the potential surface for security threats to the database

Cowboy Coding

What your boss is proposing here sounds dangerously close to Cowboy Coding. It's entirely possible that your boss hasn't realised this yet, in which case I would pose the following questions about how the solution will work in practice:

  • How will QA/testing work for changes going into production?
  • How will developers' own local environments keep in-sync with production changes?
  • Will there be test environments and test data?
  • What provisions will exist for diagnosing and debugging faults?
  • Will changes be source controlled?
  • Will changes be properly tracked and versioned?
  • Will there be a process to promote changes between environments?
  • How would changes be covered by static analysis and automated tests?
  • How will peer code reviews work?
  • What level of risk and potential for bugs is acceptable for end-users?
  • How will the data be protected against the risk of getting broken?
  • Has information security been considered?
Ben Cottrell
  • 11,656
  • 4
  • 30
  • 41
8

That code is in stored procedures inside a SQL-based database does not excuse it from version control, version management, testing, and all the usual dance steps of software development.

For validation, as for many other workflow processes, there is range of possible solutions: on one end, some values in a table (as Flater's answer pointed out); on the other end, code. Somewhere along the line there are complex embodiments of code in data.

Perhaps you need an ERP system? Those are plagued with code-as-data. And implementing one in a company is no trivial matter. But they already have solutions for many many company problems.

Pablo H
  • 598
  • 2
  • 7
6

we have no DevOps pipeline, Kubernetes, etc. due to IT department restrictions, so everything deployment-wise is manual at this point

While you added it as a passing comment, this is the most important part of your question. Continuous integration is a fundamental part of software development today - so fundamental that in my opinion, a company that lacks it cannot reasonably be called a software development company. And if they don't have CI, they almost certainly have other fundamental problems - likely they don't use a reliable systems development lifecycle, which implies poor code quality and insufficient oversight. Essentially, a "cowboy coder" environment.

As its name suggests, such an environment breeds cowboys, so-called because they're "quick on the draw" to fix problems that arise in their projects. This makes them extremely popular with management, who perceive this reactive nature as a positive and reward the cowboys. The actual issue is that a lack of an SDLC is what is causing the problems!

Thus, any attempt to introduce an SDLC into that environment will receive massive pushback from the cowboys, because they understand that making the development process more predictable and transparent threatens their good reputation with management. Especially since most cowboys are also middling to poor developers, an SDLC represents a massive threat to them - and they'll use their political clout with management to resist it.

As for management, you need to remember that their job first and foremost is to avoid risk. But the fact that SDLC-less development is so fraught with bugs, SNAFUs, and broken releases means it has a lot of risk, so their inherent risk aversity means they come to see any possible new risk in development - like, say, using the strategy pattern vs SQL - as unacceptable.

In short, if you don't have CI, you almost certainly don't have other fundamentals like an SDLC - and if you don't have these things, you as an ordinary developer are probably not going to be able to implement them, due to the aforementioned politics. You're also unlikely to get your "new-fangled" ideas approved by management.

If the above sounds like your company, you need to get out. Seriously. The chance that you as a developer, are going to be able to change things for the better, are as close to zero as they can be without actually being zero. Wasting your time and effort in an attempt to fix the company's problems simply isn't worth it - take that advice from someone who has tried, and failed.

Your alternative is to simply shut up and do what you're told by management, but your post indicates to me that you're not developing software because it pays the bills, but rather because you are indeed a software engineer in the true sense of the term. Your "only" three years of experience is irrelevant, what matters is your desire to build software well - and that desire shines through strongly in your writing. If you are a software engineer, then forcing yourself to do things the wrong way will eventually break you. Again, I speak from experience.

Looking for a good new job is difficult and scary. Staying in a bad one is ultimately much, much worse - both professionally and personally.

Ian Kemp
  • 379
  • 1
  • 11
  • 4
    +1 I'm willing to bet that the reason the company is looking to migrate its infrastructure away from a huge pile of Excel sheets is not because Excel doesn't work, but rather because maintaining a business of a sufficient size as a collection of Excel sheets is hell due to the fact that Excel is extremely antagonistic to any sensible software development life cycle when it comes to version control, testing, managing deployments and multiple environments, etc). It makes no sense to spend so much effort moving away from Excel, only to recreate this exact same problem in a new environment. – Zach Lipton Apr 07 '23 at 06:47
  • 1
    @ZachLipton this is a good argument: OP's manager needs to understand that to get the benefits of non-excel, they need to stop turning the other system into excel and take the drawbacks / effort the other system comes with into account. – Frank Hopkins Apr 08 '23 at 04:52
3

You are both right, on different levels.

Your boss is right that given the constraints that full releases are slow and time-consuming, it makes sense to move business rules into the database, which can be updated on the fly. (Assuming you do need to be able to quickly add or change business rules.)

But you are correct that it complicates the system to move business rules into stored procedures if they are better expressed in regular code.

So the question is if the current deployment procedure is set in stone or if you can push for more CI/CD, making code deployments less expensive. This is probably a question of organizational culture and may not be something you or even your boss can change.

You probably don't need to go full Kubernetes. It is easier to introduce changes gradually, e.g. by replacing some of the manual deployment steps with batch scripts.

Alternatively, if you can't introduce CI/CD, you might consider using some kind of scripting language for some of the business logic, where the scripts can be updated on the fly.

JacquesB
  • 57,310
  • 21
  • 127
  • 176
2

because he doesn't want to do an official release each time we have logic changes (we have no DevOps pipeline, Kubernetes, etc. due to IT department restrictions, so everything deployment-wise is manual at this point)

This sounds like you are being enlisted in an interdepartmental politics war to help fight via technical means.

Probably, The intent is to lever out, sabotage, or undermine a change management process which is broken or perceived as broken or simply counter to your department's interest.

Don't consider technical pros and cons as first priority. Understand the underlying politics first, and who on whose side has which kind of clout. You might be being used, or you might have been enlisted to fight on the winner's side.

In case the change management process is due to regulatory requirements, stop and really find out what is going on before you shoot yourself or the whole team or company in the foot with a cannon.

rackandboneman
  • 155
  • 2