1

Background

My task is to create new system (X) for realtime work tracking for internal use. You should think it like an app that you have opened on a separate monitor/ on phone all day and you are logging your work progress there. In fact, there is already a system (Y) that solves this and internally its working well, but it's really annoying to use because of slowness, unresponsiveness, and terrible UX design. The legacy system cannot be replaced at this time (a lot of reasons) and data from app new must be synchronized with it. The legacy system Y is also integrated with a lot of other systems and plan is to take over these integrations and rebuild them to system X. There is intent to completely replace the legacy system with a new one in the future - not only because of unresponsiveness but also because of monster technical debt and obsolete tech stack.

I proposed to create new system X as a standalone software with its own database and integrate it with Y using its API. The problem is - API calls (http requests) to Y are slow (up to 10 seconds). System X should have to be very responsive, all requests have to be processed in < 1s. So my plan is to push events from X to some kind of queue and sent them to Y asynchronously. Additionally, it's important to process responses from system Y in system X (example below) but it can be delayed. Also, other systems can eventually consume the same events from the queue.

Example of communication

User ->    X    : Create job request 
X    ->    User : Job(id_X=1,id_Y=null) created and Event1 pushed to queue
Y               : Starting processing event1
User ->    X    : Add comment to Job(id_X=1,id_Y=null)
X    ->    User : Comment(id_X=2,id_Y=null) created Event2 and pushed to queue
Y               : Starting processing event2
Y    ->    X    : Event1 processed, returning id_Y=8876 for job with id_x=1
X               : Update job (id_X=1,id_Y=8876) internally from resnponse
X    ->    User : Display updated job info

.. and so on.

Questions

  • Is my proposed communication pattern meaningful or are there any design flaws?
  • Should be queue functionality part of the system X or is it better to make it standalone?
  • Can you suggest any existing software for the queue?

Thank you in advance for any advice

edit: To be little more specific

  • System X based on Java/Spring framework powered backend service and SPA (Angular) frontend web app
  • Websocket API / REST API on backend service
  • System Y is Python/Zope web app with web API
  • X and Y both using relation database
  • Deployment of X and Y in a private datacenter, also any other software should be optimally self-hosted
  • Low workload for all services, about ~50 users online at the same time max
Pettan
  • 11
  • 2
  • 1
    You're probably on the right track. The queue should definitely be _separate_ from your service - there's an entire class of software for that - see the tag message-queue or simply search the web for message queue/messaging queuing software. I don't know if the rules here allow for recommending software, and in any case, the question at this point is to vague to recommend anything, e.g., we don't even know private datacenter vs public cloud. – davidbak Jul 09 '20 at 17:10
  • Since you mentioned "Also, other systems can eventually consume the same events from the queue." I would suggest to check out some streaming softwares like kafka, in case of a queue you might end up creating multiple of them if you intend to pass the same data to multiple systems. BTW can the legacy system support new tech? – Lalit Mehra Jul 10 '20 at 07:14

0 Answers0