3

I'm working on a portion of an ERP system where I need to process data in a way that's similar to a series of batch jobs, and I'm struggling with deciding the best program architecture to use. I'm asking here because I know I'm ignorant of a lot of state-of-the-art programming methods, and I'm worried about designing my software in an archaic way without realizing it.

My system needs to process batches of data and have the processing jobs be able to be scheduled at regular intervals, and also to be run on demand. Some of the processing also relies on outside web services.

Right now my architecture is that I have one single Visual Studio C# solution (with several projects inside of it). The solution produces a program which has an interface for running jobs on demand also for configuring a schedule for the jobs to run automatically. The single program also has all of the batch processing code in it.

Within the program, each "batch job" is called by calling a method on my main controller class, such as public async Task BillOrders(), which then calls the appropriate method from a high-level service class, which calls lower-level services, and so on. The classes themselves are all pretty well separated in terms of the single responsibility principle. The main controller calls each batch job method asynchronously and handles monitoring them, reporting errors, throttling, and so on.

This all works, but I'm worried that this isn't the way I should be doing it. Specifically, my concerns are:

  1. Since I have several separate batch-process-type functions all in the same program, I'm worried that a crash or bug in the program could cause all the batch processes to stop working. It also seems bad to have one single program that does a bunch of different things.

  2. Even though each batch process is related to one area of my ERP, I'm worried that having a single codebase with different functions in it will end up creating something monolithic and difficult for several developers to work on different aspects of simultaneously.

One of the main reasons why I've done this as a single program (and Visual Studio solution) is my Entity Framework context and service classes are shared between each batch job. If I split the program into several smaller programs, each program would have a lot of classes and the EF context that are the same code. And any bugfix or extension to a service in one program would need to be copied to the others. Also, my EF context has several dozen tables that are done with Fluent API.

I considered making a series of microservices, but there are so many cross cutting concerns that each microservice would need a lot of the same EF context and service classes that I described above, thereby defeating the goal of using microservices.

Is there a commonly accepted architecture for this sort of thing? Is what I'm doing OK, or am I programming as if it's 15 years ago?

Ben Rubin
  • 151
  • 1
  • 5
  • Are you designing an ERP made of pieces with batch jobs as a glue ? If it's the case, and sorry to be direct, you'd be designing as if it's 25 years ago. First question: why not design the flows between the pieces real time ? And if you're considering microservice (which would really be state of the art), why not organize your information flows on an event-stream, to which all your services could subscribe ? – Christophe Sep 21 '18 at 21:38
  • I'm not sure what you mean by "design the flows between the pieces real time". The main reason that I'm hesitant to use microservices is that each batch job, which is incremented as a single service, would use a lot of the same service/EF code. If I split out the common services (for instance services to retrieve various bits of information about orders), I would wind up with so many microservices that communication and speed between them would become an issue. Is there a good way to share the code for an Entity Framework `DbContext` between all microservices? – Ben Rubin Sep 21 '18 at 21:44
  • That's not to say that I'm totally against microservices, I'm just struggling to see how the gains would outweigh the costs both in terms of programming time and service communication time. – Ben Rubin Sep 21 '18 at 21:47
  • 2
    Let's take an example with SAP: there's a purchase module handling purchase orders, a logistic order handling inventory, and an accounting module handling invoices. There is no batch job that tells the inventory module how much new inventories are expected to be received. And when the pieces arrive, there's no batch job to tell the accounting module how much the inventory is worth. It's all done "real time", i.e. every single transaction in any module will automatically and immediately update all the other modules. The alternative is batch jobs and delays to get up-to-date infos. – Christophe Sep 21 '18 at 21:54
  • So my question to you is: what is the purpose of the batch jobs: make some processing that make only sense on a consistent group of data (that's ok) ? Or organize the flows between your modules in an asynchronous manner ? – Christophe Sep 21 '18 at 21:56
  • The purpose of the batch jobs is because I usually want to accumulate a bunch of data first, and then process it. For instance, it'll be faster for me to process 1,000 invoices once, than to process one invoice 1,000 times, mainly due to database latency. – Ben Rubin Sep 21 '18 at 21:59
  • Here's the main thing I'm struggling with regarding your example (which I agree is a better way of doing things than how I am): the purchasing module and logistics module both share a lot of the same tables, which are in my Entity Framework context. I don't want to have 10 different Visual Studio solutions that all have basically the same database-related code. Is it possible to have a bunch of microservices within the same VS solution? – Ben Rubin Sep 21 '18 at 22:01
  • 1
    IMO, this is not necessarily suited for microservices arch. style, which is more about supporting extreme levels of scalability required by services like Amazon and Facebook, and arguably requires infrastructure supporting [CI and CD](https://www.atlassian.com/continuous-delivery/ci-vs-ci-vs-cd), and things like [Event Sourcing / event streams](https://blog.couchbase.com/event-sourcing-event-logging-an-essential-microservice-pattern/), design based on bounded contexts, separated databases, eventual consistency, etc. (See [this](https://martinfowler.com/articles/microservice-trade-offs.html).) – Filip Milovanović Sep 21 '18 at 23:50
  • 1
    My first instinct, based on your description, would be to initially go for some form of [pipes-and-filters architecture](https://docs.microsoft.com/en-us/azure/architecture/patterns/pipes-and-filters) (which yes, is ages old, but that's not an issue), and move to full-blown microservices style later if necessary. This will introduce some extra complexity, so make sure to weigh the pros and cons. You'll also have to think about what can be processed independently and in parallel, and what cannot, what must be atomic, how to communicate between filters/processors, etc. – Filip Milovanović Sep 21 '18 at 23:51
  • @FilipMilovanović Thanks for the suggestion. I'd never heard of pipes-and-filters before, so that's very helpful. I'm going to look into it. – Ben Rubin Sep 22 '18 at 01:12

2 Answers2

1

Since I have several separate batch-process-type functions all in the same program, I'm worried that a crash or bug in the program could cause all the batch processes to stop working.

Are you running multiple jobs at the same time in the same process? In that case there is a bit of a risk that one error might terminate other jobs. This will only happen for errors which damage the whole process. This is fairly rare. Examples are out of memory conditions, stack overflows or people terminating the process. Otherwise, you can wrap your processing code in try-catch to isolate them from one another. This is what web applications do and normally it works very well.

You could also run each job as a separate OS process. Then, the jobs are totally isolated. You can reuse the same EXE file for this. Make an integer or string command line argument that tells the EXE which of the many possible jobs it is supposed to launch.

Reusing the same EXE file saves you from the need to create many C# projects and have the code management overhead that would come with those projects.

It also seems bad to have one single program that does a bunch of different things.

What is your concern? Maybe you are worried that this might go against the single responsibility principle. This principle is about the same code doing multiple things. As I understand it your jobs are fairly isolated from one another. Also consider that typical web application are doing 100s or 1000s of different things and it's not an issue. The SRP just a principle, not a hard rule. Do what works for you.

Even though each batch process is related to one area of my ERP, I'm worried that having a single codebase with different functions in it will end up creating something monolithic and difficult for several developers to work on different aspects of simultaneously.

If you are concerned the codebase might become unwieldy then I'd argue that splitting the code into multiple projects would not gain you anything. It's still the same code, the same dependencies and architecture in a different place.

Solving entanglement is never possible just by putting code into a different place (additional projects or services). You must make architectural changes.

So whether you build one EXE/microservice per job or if you make each job merely it's own C# class is not much of an architectural difference. It just adds a lot of development complexity.

Since your jobs seam fairly isolated from one another I'm not concerned about putting all of them in the same project.

It also sounds like you want to share some infrastructure such as Entity Framework models. That's also a good reason to keep things together.

usr
  • 2,734
  • 18
  • 15
  • 1
    Thank you for the answer. You did a great job of addressing my concerns. Regarding your first point about running multiple instances of the same .exe, that's an interesting solution. I hadn't though of that. – Ben Rubin Sep 22 '18 at 13:26
1

Your domain is so different from mine and the comments below your question from experts in this domain are going way over my head, but with "batch processing" and your two specific questions related to your very immediate design concerns, immediately I think of the command pattern and IPC (interprocess communication) as my goto techniques for solving these issues.

Even though each batch process is related to one area of my ERP, I'm worried that having a single codebase with different functions in it will end up creating something monolithic and difficult for several developers to work on different aspects of simultaneously.

A solution for that is to replace your central "main controller" interface with decentralized concrete command objects which all implement the same abstract Command interface. You can execute them asynchronously from a central place if that's a need. You can also combine that with a factory to instantiate commands by a string indicating the name which could be useful for the second point below. This way you can keep adding new commands to your system non-intrusively without some central interface growing to become a monolith.

Since I have several separate batch-process-type functions all in the same program, I'm worried that a crash or bug in the program could cause all the batch processes to stop working. It also seems bad to have one single program that does a bunch of different things.

The solution for that if you want something really robust is spawn a separate process to execute these commands (or spawn it in advance and communicate with an existing one to tell it new things to do with IPC). You don't have to program separate binaries for every single possible task/job (command) that could be performed as usr pointed out. You can have one generalized command executor binary or even reuse the same one you already have now with command arguments supplied to tell it what command to execute, and some form of IPC (there are many options here from named pipes to file mappings to mailslots and others on Windows -- I've generalized the file mapping and shared memory method as the one most generalized for my purposes, though it requires semaphores) to share relevant state between two or more processes. If you do it that way, then if some sub-process crashes, it won't bring the host application performing these batch jobs down with it in flames.

I believe that's how many of our browsers work to avoid crashing when Flash or some JavaScript or WebGL crashes or something like that. They likely spawn these things in separate processes, and you can see with Chrome at least that it has its own task manager for that (shift+esc) and that it tends to launch many instances of Chrome.exe independently of how many Chrome instances you opened yourself.