-1

I have one design problem for one of my projects based on location tracking. The scenario is I have multiple devices(for some business) all over the city and have GPS installed inside, the device is calling the Ping API for updating its most recent location every 10 seconds. I am managing a queue-based solution(RabbitMQ) for accepting the request asynchronously (have a dedicated queue for each device) and also have a dedicated consumer for each of those queues performing some business logic on the request or persisting it in some store.

Right now I am having around 10000 devices in the field. this might grow in the future as per business requirements, but I don't have any number or scaling factor yet. So the store is ingesting 10000 messages every 10 seconds or on an average 1000 messages per second. with this number in a month I will have this many messages = 1000 * 30 * 24 * 60 * 60 = 2,59,20,00,000

For searching, I have 2 requirements:

  1. Fetch the current location of a device
  2. Fetch the location of a device at any instant

both the operation will be polled frequently(like every 10s or 20s). So if I will use an RDMS solution for persisting these messages the query time will scale down once the no of messages will grow. How and what should I pick as my store for solving this problem?

Praveen kumar
  • 11
  • 1

1 Answers1

0

My first thought is that you could use a time-series database, this could speed the operation of your second requirement (= Fetch the location of a device at any instant). PostgreSQL for example has hypertables, which you could use per device. This would split your database structure down.

But one problem with your requirement as you wrote is that you said, that you will fetch the location of a device at any instant frequently. You need to define if you do this for every instant since the beginning of the recordings or just for the last 24 hours or so. With this you could introduce a hot/warm/cold tier. So that your database will not keep and query 5 year old data although you don't need it regulary.

hideous
  • 109
  • 1