Software Engineering Stack Exchange
Software Engineering Stack Exchange

Questions tagged [apache-kafka]

Apache Kafka is publish-subscribe messaging rethought as a distributed commit log.

Apache Kafka is publish-subscribe messaging rethought as a distributed commit log.

87 questions
17
votes
2 answers

Traditional Message Brokers and Streaming Data

According to the Kafka site: "Kakfa is used for building real-time data pipelines and streaming apps." Searching the internet far and wide, I've found the following generally-accepted definition of what "stream data" is: Stream data is data that…
smeeb
  • 4,820
  • 10
  • 30
  • 49
11
votes
2 answers

Event sourcing, replaying and versioning

I am designing a system that uses Event Sourcing, CQRS and microservices. I am lead to understand this isn't an uncommon pattern. A key feature of the service needs to be the ability to rehydrate/restore from a system of record. Microservices will…
Antony Woods
  • 353
  • 1
  • 3
  • 10
10
votes
1 answer

Data durability guarantees in Kafka

Is it wise to use kafka as the 'source of truth' for mission critical data? The setup is: kafka is the underlying source-of-truth for the data. -querying is done on caches (I.e. Redis, ktables) hydrated from kafka Kafka configured for durability…
major_clanger
  • 111
  • 4
8
votes
3 answers

Decoupling microservices with gRPC

I'm setting up a microservices architecture, and am confused about how gRPC can loosely-couple services (compared to a pub-sub message service like Kafka). Doesn't the request go directly to the server, and not through a pub/sub system? While gRPC…
skunkwerk
  • 189
  • 1
  • 3
7
votes
2 answers

Is Kafka needed in a realtime chat application?

I'm developing a realtime chat application with an Angular frontend and Java backend. I've found a couple of examples that resemble what I am trying to achieve, such…
Babyburger
  • 211
  • 1
  • 3
  • 6
7
votes
1 answer

Is it appropriate to say that RabbitMQ and Apache Kafka solve similar problems?

I have used RabbitMQ but I haven't used Apache Kafka. Is it a similar problem that these products solve, or is there no connection?
Niklas Rosencrantz
  • 8,008
  • 17
  • 56
  • 95
6
votes
1 answer

Can Event Sourcing and an API Management platform leverage each other?

I'm evaluating the use of Kafka to implement Event Sourcing in a microservices environment that already makes use of an API Management platform. I was wondering if it is common and/or considered a good practice to use the API Management as a gateway…
ivarec
  • 315
  • 1
  • 5
5
votes
2 answers

kafka consumer properties session.timeout.ms vs heartbeat.interval.ms

I am reading Kafka, the definitive guide and I came across the below point for consumers. heartbeat.interval.ms must be lower than session.timeout.ms, and is usually set to one-third of the timeout value. May I know why is it, I thought both should…
Ramya B
  • 59
  • 2
4
votes
3 answers

Alternating between Java streams and parallel streams at runtime

This is a question I constantly ask myself when designing a data intensive application: When is it appropriate to use stream() over parallelStream()? Would it make sense to use both? How do I quantify the metrics and conditions to intelligently…
user0000001
  • 263
  • 3
  • 7
4
votes
2 answers

Event Driven Microservice Race Condition

Say we have 4 services, A through D, which communicate (for the most part) through some sort of asynchronous event-driven system. When a new entity is created in A, B & C receive that event. B creates an entity of its own based on that event, and C…
Asmodean
  • 149
  • 1
  • 4
4
votes
2 answers

Two processes in a single docker container or two services connecting to the same db?

I recently started moving a monolithic application to microservices architecture using docker containers. The general idea of the app is: scraping data -> format the data -> save the data to MySQL -> serve data via REST API. I want to split each…
rogamba
  • 151
  • 6
3
votes
1 answer

Data processing pipeline design for processing data

I have a use case for which I need to build a data processing pipeline Customer contact leads data coming from different data sources like csv, data base, api has to be first mapped to a universal schema fields. There could be ~100k rows coming…
Sam Rohn
  • 139
  • 1
3
votes
1 answer

How to organize exchanges of non critical messages between servers in multiple data centers by using Kafka (or other solution)?

We need to organize a way to exchange messages between servers in multiple data centers. Messages are not critical. We just need to be able to send messages from any servers in any data centers to any other servers in any data centers. We are…
Oleksandr
  • 141
  • 7
3
votes
0 answers

Designing clickstream analysis?

I have a application where user purchases/click the certain products. I need to design the click stream analysis here which product got clicked how many number of time, user/geographical detail click those product Here is the design i am…
user3198603
  • 1,896
  • 2
  • 16
  • 21
3
votes
3 answers

Kafka, is it possible to get the message offset after producing?

While using Apache Kafka, is it possible to get the offset of the message that has been produced? From the Kafka documentation page it says: each message in the partition are assigned a unique sequential id/number called Offset Now suppose when I…
user2720864
  • 255
  • 1
  • 3
  • 9
1
2 3 4 5 6