10

Is it wise to use kafka as the 'source of truth' for mission critical data?

The setup is:

  • kafka is the underlying source-of-truth for the data. -querying is done on caches (I.e. Redis, ktables) hydrated from kafka
  • Kafka configured for durability (infinite topic retention, 3+ replication factor etc)
  • architecture follows CQRS pattern (writes to kafka, reads from the caches)
  • architecture allows for eventual consistency between reads & writes

We are not allowed to lose data in any circumstances

In theory the replication should guarantee durability & resiliency. Confluent themselves encourage the above pattern.

The only flaws I can think of, are:

  • cache blows up and needs to be rehydrated from scratch -> query
  • broker disk gets wiped/corrupted -> kafka rebalance, resulting in prolonged downtime if topics contain mountains of data

Has anyone run and battle tested this kind of setup in production? I.e. encountered disk corruption, brokers going down, but still retain data?

My gut feel is, this is a bad setup, because kafka wasn't originally designed for RMDBS levels of durability, but can't point to a concrete reason why this would be the case.

major_clanger
  • 111
  • 4
  • https://kafka.apache.org/documentation/#design_filesystem implies that data is not fsync()'d to disk (let's pretend that's a strong synchronous guarantee that data is written to the hard disk). This implies to me that it doesn't offer RDBMS levels of durability. But I'm not sure... In: https://kafka.apache.org/intro under "Kafka as a storage system", it says: "Kafka allows producers to wait on acknowledgement so that a write isn't considered complete until it is fully replicated and guaranteed to persist even if the server written to fails." So it's "unlikely" to lose data :) – Jake_Howard Mar 28 '19 at 15:06
  • @Jake_Howard Kafka uses [zero-copy](https://en.wikipedia.org/wiki/Zero-copy#cite_note-j-zerocopy-5) for reads (at least). It's not clear to me what that might mean for writes in this context. – JimmyJames Dec 17 '19 at 17:10
  • @Jake_Howard Kafka can be configured to force an fsync of data written to the log after every message: **flush.messages**: This setting allows specifying an interval at which we will force an fsync of data written to the log. For example if this was set to 1 we would fsync after every message; if it were 5 we would fsync after every five messages. In general we recommend you not set this and use replication for durability and allow the operating system's background flush capabilities as it is more efficient. https://kafka.apache.org/documentation/#topicconfigs – Denis Stafichuk Jan 12 '20 at 05:41
  • _broker disk gets wiped/corrupted -> kafka rebalance, resulting in prolonged downtime if topics contain mountains of data._ What rebalance are you talking about here, and why would it result in prolonged downtime if topics contain mountains of data? – Denis Stafichuk Jan 12 '20 at 05:43

1 Answers1

2

We are not allowed to lose data in any circumstances

Another possible flaw could be in Kafka itself. I mean in some rare circumstances (a potential future version of) Kafka can corrupt its own data.

It happened to Postgres several times. I remember a recent issue which led to corrupted data with different glibc versions on master and replica. Also, until Postgres started to handle errors during fsync with panic (shutdown), it could also corrupt data. That's for a product that is dedicated to persisting your data, albeit with more features, therefore larger bug surface.

In my opinion, the lesson is to have an archive of your data stored separately in a colder storage where there's lower chance that some process will mess them up. Also, having two storage systems, e. g. Kafka and S3 means that if one of them messes up your data, it's less likely the other one does it at the same time.

I guess it all depends on your definition of any circumstances.

Krab
  • 186
  • 5