Software Engineering Stack Exchange
Software Engineering Stack Exchange

Questions tagged [compression]

65 questions
36
votes
5 answers

Does storing plain text data take up less space than storing the equivalent message in binary?

As a web developer I have very little understanding of binary data. If I take the sentence "Hello world.", convert it to binary, and store it as binary in an SQL database, it seems like the 1s and 0s would take up more space than the letters. It…
john doe
  • 1,007
  • 1
  • 9
  • 9
30
votes
5 answers

What is the most efficient way to store a numeric range?

This question is about how many bits are required to store a range. Or put another way, for a given number of bits, what is the maximum range that can be stored and how? Imagine we want to store a sub-range within the range 0-255. So for example,…
rghome
  • 668
  • 5
  • 12
29
votes
7 answers

How can lossless compression ever exist?

If all data is essentially just a bit string, then all data can be represented as a number. Because a compression algorithm, c(x), must reduce or keep the same length of the input, then the compressed file must be smaller or equal to the input (and…
Mercury
  • 465
  • 1
  • 2
  • 5
14
votes
2 answers

Fast, lossless compression of a video stream

I have a video coming from a stationary camera. Both the resolution and the FPS are quite high. The data I get is in Bayer format and uses 10 bit per pixel. As there's no 10 bit data type on my platform, the original data is stored in memory using…
Headcrab
  • 243
  • 2
  • 6
11
votes
4 answers

How to review sql changes more effectively?

From my experience, sql code changes almost always tend to be NOT incremental: someone creates a new stored procedure, or modifies an entire embedded sql query for optimization purposes, or creates a brand new table. When I receive one of these code…
CEGRD
  • 235
  • 1
  • 7
9
votes
2 answers

Best compression algorithm for timelapse photos

I have a folder containing about 9,000 JPEG photos (about 30Gb), which I want to archive with some sort of compression. I understand that compressing JPEGs is not normally very effective, but these photos are frames of a timelapse, so there is a…
Stephen
  • 201
  • 1
  • 5
7
votes
2 answers

some misunderstanding in concept of Huffman algorithm

What is difference between Average length of codes and Average length of codewords in Huffman Algorithm? is both the same meaning? I get stuck in some facts: I see a fact that marked as False: for a text that it's characters get from set of n…
Emma Nic.
  • 183
  • 5
7
votes
2 answers

How does conditional compilation impact product quality, security and code complexity?

Software libraries targetting resource constrained environments like embedded systems use conditional compilation to allow consumers to shave space by removing unused features from the final binaries distributed in production. What are the tradeoffs…
TZubiri
  • 435
  • 1
  • 4
  • 9
7
votes
4 answers

compression algorithm for non-repeating integers

I have an array of unique integers, for example: {1,3,,7,9,31,46,...}, which I want to compress. I have found compression techniques and algorithms for the list of integers, where some integers are repeated but I couldn't find any compression…
Angela
  • 99
  • 1
  • 6
6
votes
1 answer

How is it possible to GZIP a stream before the entire contents are known?

Two things you can do in Java: Send a gzipped JSON body in response to an HTTP request Send a StreamingOutput response to an HTTP request, where you begin sending a response before you know the entirety of the response (e.g. if reading entries from…
Malcolm Crum
  • 261
  • 2
  • 7
6
votes
1 answer

How can I reduce the amount of storage needed for a gravitational n-body simulation?

I am currently attempting to create a gravitational n-body simulation using a modified Barnes-Hut algorithm, to be more amicable to GPU computation. This is primarily as a learning project. My goal is to simulate a number of stars comparable to that…
john01dav
  • 879
  • 1
  • 7
  • 14
5
votes
2 answers

How to deal with large data in Websocket message?

I wrote a websocket server in Spring Boot and a client in Javascript. These work fine. I also wrote a second client in Java. When this one attempts to handle a frame after connecting to the host, I get this…
Jeff
  • 1,854
  • 3
  • 16
  • 19
5
votes
2 answers

finding optimal token definitions for compression

I have a collection of strings which have a lot of common substrings, and I'm trying to find a good way to define tokens to compress them. For instance, if my strings are: s1 = "String" s2 = "Bool" s3 = "String -> Bool" s4 = "String -> String" s5 =…
ErikR
  • 296
  • 1
  • 4
5
votes
4 answers

Fast compression in C++ and decompression in C#

Overview I am working on a client-server application. The client is written in C++ (working on Windows, planning to support Linux) and the server is a .NET RESTful service. I need to HTTP POST some data to the server. The data is a unicode string…
oleksii
  • 1,196
  • 1
  • 9
  • 20
5
votes
4 answers

Randomly Accessing Data Packets in a Compressed File

In my line of work I deal with very large files, hundreds of Gigabytes in size. The nature of the data in these files is such that compression would greatly reduce their size. The problem is, the records/data packets within the file must be…
Robert Harvey
  • 198,589
  • 55
  • 464
  • 673
1
2 3 4 5