Software Engineering Stack Exchange
Software Engineering Stack Exchange

Questions tagged [protobuf]

Protocol Buffers are a serialization format with an interface description language developed by Google. Source: en.wikipedia.org/wiki/Protobuf

Protocol Buffers are a serialization format with an interface description language developed by Google. Source: en.wikipedia.org/wiki/Protobuf

23 questions
29
votes
1 answer

Why Protobuf 3 made all fields on the messages optional?

The syntax 3 of protobuf made all the fields optional dropping the keywords required and optional from previous proto2 syntax. Reading some comments from developers it seems that it was done for enhancing forward/backward binary compatibility. But…
tonicebrian
  • 413
  • 1
  • 4
  • 6
24
votes
2 answers

Protobuf design patterns

I am evaluating Google Protocol Buffers for a Java based service (but am expecting language agnostic patterns). I have two questions: The first is a broad general question: What patterns are we seeing people use? Said patterns being related to…
Apoorv
  • 1,128
  • 1
  • 6
  • 15
5
votes
1 answer

Is it rational to convert protobuf into json to send it to a web server?

Protobuf is a good thing, it allows c++ devs to not bother with class serialization/deserialization, is fast and .proto format is very nice. It also allows automatic data validation. But the messages are not human readable, and most importantly not…
Zeks
  • 221
  • 2
  • 5
3
votes
2 answers

Reverse engineering an ETL pipeline

I have a one-way large ETL pipeline in Scala where I start with protobuf schema A and end up with protobuf schema B. I want to create a many-to-many mapping between the fields of schema A and schema B, where schema B uses a subset of the fields from…
Conor
  • 109
  • 7
2
votes
1 answer

Protobuf: compile specification once, use generated code everywhere?

Once written, a protobuf specification can be compiled using protoc to a variety of implementations in different languages (e.g. python, c++, go). That's great. But to compile that generated code, you need a protobuf development library and, if I am…
AkiRoss
  • 129
  • 5
1
vote
1 answer

Recommended generic approach to checking scalar field presence in protobuf 3

There are several questions related to presence field tracking of scalar fields in protobuf 3, but I didn't find any with generic default approach recommendation. It may be useful to be able to track field presence in protobuf message. I see at…
Alex Che
  • 113
  • 4
1
vote
0 answers

Is non-contiguous protobuf field numbering an anti-pattern?

Is it an anti-pattern to have gaps in field numbers to aid readability? The need being address by this idea is that metadata fields, which are "aspect" fields, can be placed in the proto file away from the main fields, and by numbering them using…
Bohemian
  • 1,956
  • 2
  • 17
  • 24
1
vote
1 answer

Using protobuf to store large array of doubles

For my job, I'm working on a project that involves working with robot sensor data. We've been using protobuf to serialize and transfer data. I'm trying to set up a new protobuf type but this data involves a large array (e.g. a million elements) of…
user81147
  • 119
  • 3
1
vote
0 answers

Separating generated flatbuffer/protobuf files from domain model - best practices

I'm using flatbuffers for the first time. I've generated my java classes and have tested out serializing / deserializaing a message. Now I'm thinking about how to integrate these in to my JavaFx and Android applications. Is it valid to pass a…
Stealth Rabbi
  • 137
  • 2
1
vote
0 answers

What's the most intuitive way to organize a set of endpoints to fetch users and their roles in groups?

What's the most intuitive way to organize a set of endpoints to fetch users and their roles in groups? i.e. Users and their roles in a specified group. Groups a user is in, and their role in each group. Attempt 1 We began with the…
Andrew Cheong
  • 425
  • 2
  • 9
1
vote
2 answers

What are the "scalar fields" and "composite fields" in JAVA?

Need to understand the below definition, from the protobuf tutorial: mergeFrom(Message other): (builder only) merges the contents of other into this message, overwriting singular scalar fields, merging composite fields, and concatenating repeated…
RafiAlhamd
  • 113
  • 4
1
vote
1 answer

How to handle adding new field / partial data update for a gRPC API?

Assuming we have following message that will be used to update data and it just got updated to version 2. message HelloRequest { string name = 1; // version 1 bool is_valid = 2; // version 2 } Now assuming we updated our apps like so: server :…
Jan Paolo Go
  • 119
  • 7
1
vote
1 answer

Why protobuf is said to be in binary format although we write it in text format similar to json

JSON { "hello": "world" } Protobuf message HelloWorld { required string name = 1; } Why do we say that protobuf uses binary format during network exchange and json don't, even though network always will transfer using binary data format?
pzk
  • 29
  • 4
1
vote
2 answers

Options for having model, parsers and serializers for a given data-format generated in multiple languages?

I am member of the Apache PLC4X (incubating) project. Here we are currently implementing multiple industry PLC protocols. While we initially focussed on creating Java versions of these, we are currently starting to work on also providing C++ and…
Christofer Dutz
  • 127
  • 1
1
vote
0 answers

Can I add new function for protobuf generated struct

I am using protobuf definitions to define message types. Structure looks like message1 { } message2 { bytes msg1 = 1 } So message2 has a field which is a byte array representation of message1. I am: Creating a msg1 struct instance Serializing…
Korba
  • 111
  • 1
1
2