Stream processing is consuming an input stream or sequence of bytes with some format, processing the data as it arrives, and translating it into another format in a related output stream.
Questions tagged [stream-processing]
85 questions
57
votes
3 answers
Is it an antipattern to use peek() to modify a stream element?
Suppose I have a stream of Things and I want to "enrich" them mid stream, I can use peek() to do this, eg:
streamOfThings.peek(this::thingMutator).forEach(this::someConsumer);
Assume that mutating the Things at this point in the code is correct…
Bohemian
- 1,956
- 2
- 17
- 24
50
votes
4 answers
Why should I use "functional operations" instead of a for loop?
for (Canvas canvas : list) {
}
NetBeans suggests me to use "functional operations":
list.stream().forEach((canvas) -> {
});
But why is this preferred? If anything, it is harder to read and understand. You are calling stream(), then forEach() using…
Saturn
- 3,887
- 9
- 30
- 40
42
votes
4 answers
What is a byte stream actually?
Can anyone explain me what byte stream actually contains? Does it contain bytes (hex data) or binary data or english letters only? I am also confused about the term "raw data". If someone asked me to "reverse the 4 byte data", then what should I…
user2720323
- 991
- 3
- 12
- 11
20
votes
3 answers
Is it a sane thing to return Streams wherever we would normally return Collections?
While developing my API that is not tied to any legacy code, I often find myself writing methods that are purely Streams pipeline terminated by collecting the results. Like this one:
ImmutableSet deriveSomethingMeaningfulFromPrivateState() {
…
jojman
- 303
- 1
- 7
17
votes
2 answers
Traditional Message Brokers and Streaming Data
According to the Kafka site:
"Kakfa is used for building real-time data pipelines and streaming apps."
Searching the internet far and wide, I've found the following generally-accepted definition of what "stream data" is:
Stream data is data that…
smeeb
- 4,820
- 10
- 30
- 49
14
votes
2 answers
Know file size with a base64 string
I do not have very clear what a base64 string is or how it is related to a file.
Right now I have a website that uploads an image as a base64 string to my server. This string is converted back into a bitmap and resized.
However, I would like the…
luisgepeto
- 257
- 1
- 2
- 4
13
votes
1 answer
How does sorting with java 8 stream work under the hood?
When I call Stream.sort(..) is there a new array of elements created and the stream iterates over the newly created sorted array?
In other words, how Java 8 Stream does sort under the hood?
InformedA
- 2,991
- 2
- 19
- 28
10
votes
3 answers
Why do you want to avoid flushing stdout?
I stumbled upon a question in Codereview, and in one answer the feedback was to avoid std::endl because it flushes the stream. The full quote is:
I'd advise avoiding std::endl in general. Along with writing a new-line to the stream, it flushes the…
klutt
- 1,428
- 1
- 10
- 25
8
votes
1 answer
What type of buffer should I implement for a one-way streaming audio device?
I'm working on a project where audio data is streamed to a device. The audio data is encoded via opus and streamed at 20 ms payloads at a time. The streaming is done via TCP to avoid packet loss completely. The goal of the streaming is to have as…
Mikey A. Leonetti
- 297
- 1
- 8
6
votes
2 answers
Are there any techniques for detecting redundancies a stream of changes to a filesystem?
I'm working on a file-synchronization client that currently produces a stream of changes to the underlying filesystem. That is a stream of create/update/move/delete events is produced for each synchronization "target".
Each event includes a…
Louis Thibault
- 2,178
- 3
- 16
- 17
6
votes
1 answer
Why does the C stdio 'ungetc' function exist?
In the C programming language (and many subsequent languages that either directly interfaced with or built a facsimile of the C's Standard IO functions), there exists a function called ungetc: int ungetc(int char, FILE *stream);. It 'puts back' a…
Qqwy
- 4,709
- 4
- 31
- 45
6
votes
1 answer
How is one supposed to deal with the intermediate buffer of DataReader class?
Summary
I am developing a WAV file format reader under WinRT, and for this I need to read random amounts of structs consisting of fundamental types such as int, uint, float and so on.
Back in desktop development one would rely on BinaryReader, now…
aybe
- 727
- 6
- 16
5
votes
2 answers
What are the distinction and relation between batch processing and stream processing systems?
Design Data Intensive Applications says
Batch processing systems (offline systems) Chapter 10
A batch processing system takes a large amount of input data, runs a
job to pro‐ cess it, and produces some output data. Jobs often
take a …
Tim
- 5,405
- 7
- 48
- 84
4
votes
1 answer
Download file stream as it's being generated
Scenario
We are building a UI that allows users to query our data in bulk. The return format is CSV and there is a decent amount of processing that needs to happen, so this processing processing is done in parallel and then sets of rows are streamed…
Jared Goguen
- 200
- 1
- 1
- 8
4
votes
2 answers
Are .NET Streams like 'real' streams?
To me a stream implies that I should be able to:
Put things in one end and receive it from the other end in the same order.
Do these things at the same time. i.e. continuously be adding to the top of the stream and reading from the bottom.
The…
BanksySan
- 714
- 2
- 6
- 15