2

I have a question regarding the design of an application where the database system has a rollover. To clarify the context :

  • the database system (a cloudant instance) is fed with sensor data by an iot platform
  • the iot platform performs a database rollover each day, week or month (configurable but fixed once it's configured)
  • at the beginning of month x, the database x+1 is created with some indexes.

So the sensor data from month x of year y will go in the cloudant database mydb-<y>-<x>.

Disregarding the fact that this is a painful way to do things, I have to create an application that will have to query data from database x, x-1 and eventually x-2.

My question is is there any 'correct' way to do this ? Are there any known patterns or gotchas to be aware of that would allow me to proceed in this direction or should I look into other solutions, like replicating the data from the databases x to x-n into one aggregating db to simplify things ? It will make me lose the benefit of having more lightweight databases in the end but it would be more simple.

joel
  • 23
  • 2

1 Answers1

0

the iot platform performs a database rollover each day, week or month

Why? I would only consider this necessary if the data entry volume were massive.

More likely, the database was created by someone with a "spreadsheet" mentality so that the only way to keep the database performing well is to regularly chop the data off and store it in another sheet.

An alternative suggestion:

Do not read anything from databases x-1, x-2, ....

Just after the "rollover", extract all of the data from database x-1 and load it into a properly organised/ indexed database that can hold all of the data that you need, and which has a sensible archiving mechanism, should the data really start to get too large to handle.

That way, your application only has to read database x and your new, cumulative database.

Phill W.
  • 11,891
  • 4
  • 21
  • 36
  • It has the potential to be massive but yes, that's not a sound argument. It is like this and we have to deal with that at the moment. In the end, we only need to get the latest data so it is acceptable to query only the first two databases (two months), which is equivalent to your solution. I don't think I will get more answers so I'm marking your answer as the solution, since it is equivalent to what we ended up doing. – joel Nov 17 '16 at 08:43