3

I have this Java project called Server. Server is essentially a single-threaded application that listens to events from external applications. These events are crucial and missing one is a huge deal.

What Server does is that it listens to events, does some calculations based from the events received, and stores the calculated value in the database. Also, some events are dependent on each other and I have a single static map that keeps track of all the events coming in.

I'm handling call events. There are three events that I'm currently handling:

  1. Join events: Where two people are in a conversation

  2. Talking events: Where either person A or B are talking

  3. Leave events: Where either person leaves the conversation

My goal is to calculate how much time each person talked for the duration of the conversation.

What I do right now is, when I get a Join event, I add that conversation to the map. While I get talking events, I get the conversation from my map and do some calculation. When someone leaves the conversation, I remove the conversation from the map and save whatever I have in the database.

Since I can't afford to miss a single event, how do I implement redundancy? I thought about clustering but if ever I do cluster, then both clusters will have the same events coming in, leading to data duplication.

mpmp
  • 781
  • 3
  • 8
  • 12
  • Are these events uniquely identified _in the events themselves?_ –  Feb 26 '16 at 19:22
  • Yes they are @Snowman – mpmp Feb 26 '16 at 19:41
  • You might want to look at [Apache Kafka](http://kafka.apache.org/) or, if you're on AWS, [Amazon Kinesis](https://aws.amazon.com/kinesis/). – Derek Elkins left SE Feb 26 '16 at 19:43
  • 1
    Having unique events makes this easier, and a messaging system like a JMS provider would make this trivial. This is a solved problem: why reinvent the wheel? –  Feb 26 '16 at 19:47
  • What happens if that provider has to be restarted though? The catch is I can't afford to miss events. @Snowman – mpmp Feb 26 '16 at 20:21
  • @MiguelPortugal no different from any homebrew solution. If nothing is listening, then the message is either lost or the client needs to retry. –  Feb 26 '16 at 20:22
  • Yes, what you are looping for is a message queue. – flup Feb 26 '16 at 21:39
  • Please also note that you will likely lost some of the events: anything between your clients and the server could fail. In order to keep as much information as possible, I would simply (redundantly and) persistently store each event as fast as possible after reception, and perform some computations "from time to time". In such a way, if your computing process/server/handler fails, you can always compute all derivated values again from your "history". Redundant events will then be handled by simply ignore them. – mgoeminne Feb 27 '16 at 12:37
  • 1
    Another consideration / reason to look at messaging solutions is that you say this is single-threaded but it's crucial you miss no events. Are you sure your application can handle multiple concurrent events? I simple fact of reality is that it's much harder to guarantee anything when you have a single point of failure. If it's truly critical to catch 'every' event you need to scale horizontally. These messaging solutions can do this without duplicating the events. – JimmyJames Apr 07 '16 at 00:20

1 Answers1

2

The phrase "crucial and missing one is a huge deal" really commands a different design. Right now, a single instance of the server app is getting the job done, and someone is rightfully thinking about redundancy should that server app go down or be unavailable, or cannot finish serving the last event quick enough.

Solving these problems would require recreating a lot of wheels that have been around for years to specifically handling the heavy lifting. A message queuing (MQ) system would fit well. You would insert brokers between your clients and servers, and they handle connections with events, and queuing all events and marshaling to the server to the server without missing any. If load increases where two brokers can't handle all events as required, you simply add more brokers and scale out. It may seem overly complex, but it really isn't and I can't think of a better way to guarantee no events are missed. That is what these technologies were designed for. These solutions include the ability to monitor brokers/queues, which is important if missing events is a big deal.

Other option would be to run the server process on two boxes, and a basic load balancer fronting the connections, but wouldn't be as solid as an MQ broker solution.

Because the title of this question has to do with start/shutdown of a java server app, I expected the question to be about how to increase the reliability of a java app by wrapping something around it that handles JVM crashes do to crashes in native code that are outside your control, etc. I use YAJSW which stands for "Yet Another Java Service Wrapper" which is an open source variation of Tanuki's product. Does a lot of things, but the main thing it does is make sure your process at least gets restarted if it crashes or hangs.

Thomas Carlisle
  • 1,293
  • 9
  • 11
  • I really think a message queue system is the right tool for this as well. That was the first thing that came to mind for me. With that said, a highly critical need would require the MQ to have a failover instance as well. – jleach Apr 06 '16 at 22:36