4

I'm planning to extend an application with logging of user events. (Client application via REST-API to Symfony). What is the preferred type of databases?

Currently the log data is stored in a relational database along with the rest of the data. Click events are to be stored as well. I suspect that a relational database would no longer be suitable for storing data sets in the range of a million.

Would it make more sense to use a non-relational database, or should I continue to work with the relational database and use the possibilities for the optimization of the requests(e.g. elasticsearch)? Since the relational database seems to work slowly for querying large amounts of data, would it be possible to have the data prepared in advance via cronjob stored in auxiliary tables?

The goal is to evaluate the log data for the user interface and to display different data in several charts.

cngzz1
  • 45
  • 9
yellowduck
  • 57
  • 1
  • 3
    Based on an experience I had at a large company with a very large application - the database will become a bottleneck for your logging, and thus, for your app itself. Plus if the database connection for logging is lost, for whatever reason, does your app stop?? Sad! Don't do it that way. Just write to files, the normal way, and use any available log/metrics collector to feed those files to a log/metrics database. (There are several choices available for any platform. I'm speaking of things like Prometheus...) – davidbak Dec 24 '20 at 16:43
  • Just as an aside... "data sets in the range of a million" are not even remotely a problem for (most) relational databases. You'd have to go a few orders of magnitude higher before there's any sense in worrying about performance issues with a properly designed relational database. – Eric King Dec 24 '20 at 18:52
  • 1
    @EricKing - its not the number of rows in the table, its the frequency with which you _add_ rows to the table - especially in a situation with many (dozens, or hundreds) of servers adding rows to a centralized logging table. Plus, the fact, that in many (most?) database client libraries there are no _asynchronous_ table inserts! That alone - i.e., synchronous logging - will cause throughput problems in your app. – davidbak Dec 24 '20 at 20:24
  • 1
    If you do insist on using a database stick a reliable (i.e., guaranteed delivery) message queuing system between your log writer and your database - you can get _those_ to handle high throughput writes and buffer messages while waiting for the database to catch up. – davidbak Dec 24 '20 at 20:26
  • Welcome to Software Engineering! – CryptoJones Dec 26 '20 at 18:49

0 Answers0