5

Protobuf is a good thing, it allows c++ devs to not bother with class serialization/deserialization, is fast and .proto format is very nice. It also allows automatic data validation. But the messages are not human readable, and most importantly not human craftable. Which is a problem when you need to quickly test the response to a certain stimuli from a remote server.

The solution, as it seems to me, is to craft protobuf message, convert it to json via some library and send this, using the same library to deserialize it to protobuf on the server and back to c++.

It seems like we get best of both words: message validation, automatic class generation from .proto files and readability of json. Sure it is slower, but for my task speed is not essential. Still, is it sane? Admittedly, I have little knowledge of networking and problems that can arise from described approach. Can someone tell me if there are future problems?

Zeks
  • 221
  • 2
  • 5
  • 5
    Protocol Buffers already has a "text format" that is similar (if not identical) to JSON. See https://developers.google.com/protocol-buffers/docs/overview#whynotxml – Robert Harvey Sep 18 '15 at 01:40
  • @RobertHarvey: That text format is not used when transmitting a Protocol Buffers message over the wire, but it is a language-independent way to specify the messages. – Bart van Ingen Schenau Sep 18 '15 at 07:40
  • No, @RobertHarvey is correct, there is a text format for passing messages around.not sure if it is better to use that or json though as they are not the same – Zeks Sep 18 '15 at 12:34
  • @BartvanIngenSchenau: That is the `.proto` file, not the text format. Protocol Buffers allows you to specify a binary or text wire format, and the text format is modeled on JSON. – Robert Harvey Sep 18 '15 at 15:07
  • @Zeks: The exact format of the message format is not important so long as Protocol Buffers is on both sides of the wire. – Robert Harvey Sep 18 '15 at 15:18
  • I have since asking my question stumbled upon gRPC which does look like an even better solution to my real problem as it eliminates custom transport protocol entirely. And generates RPC calls directly from protobuf files – Zeks Sep 18 '15 at 15:18
  • *[shrug]* Apparently I misunderstood your actual problem. – Robert Harvey Sep 18 '15 at 15:34
  • Nah, I just had a realization that my problem is slightly different. You answer is still valid given the above problem definition and will be used. Just that the thing I am working on right now is a bit different from how I thought it was this night – Zeks Sep 18 '15 at 15:41
  • IMO, using protobuf when you don't have specific performance requirement is a bad practice. The "advantages" you're quoting is due to the protobuf library you're using also providing built in schema validation. It's implementation rather than inherent advantage of protobuf format. Many xml and json libraries also provides similar schema definition and validation, and they're much more widespread, more mature, and more choices available than protobuf's. – Lie Ryan Apr 21 '17 at 11:22
  • Where protobuf's binary format really shines is for chatty internal protocols (within a single data center/rack) where each messages gets passed around to be processed by multiple machines and require very low parsing overhead. For traffic going across the larger networks/internet, protobuf is usually a premature optimization. – Lie Ryan Apr 21 '17 at 11:30
  • @LieRyan Yep, pretty fair points. The application referred to in my answer started life as a little interface layer onto a financial data database, intensively used by one client calculation system, over protobuf; we even cached the PB messages to reduce encoding overhead. Gradually, usage spread, and a diagnostic web-interface mutated into a full, independent web API. So we kept the PB interface for the one premium client and advertised the web interface for everyone else - handy because we don't have to tell anyone how to use it. – SusanW Apr 24 '17 at 09:57

1 Answers1

4

Converting Protobuf to JSON and back is perfectly workable, as long as you can handle the performance overhead. I worked on a project where we did something like that, using JSON as the preferred format for ReST calls for lower-performance, high-friendliness. The web-layer converted to Protobuf, which the underlying services used directly. We included a direct Protobuf content-type so you could bypass the JSON encoding/decoding and use Protobuf over HTTP(S).

Certain privileged high-performance clients could access a raw socket interface that implemented a Protobuf/RPC interface onto the same services.

All worked nicely. I don't think any clients converted their protobuf to JSON - but they could have done; the schema was available, and that could have been used for migration or debugging.

EDIT:

As @RobertHarvey observes in his comment, there is a "text formatter/parser" library in protobuf, described here: https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.text_format. This provides:

Utilities for printing and parsing protocol messages in a human-readable, text-based format

For clarity, this is not a protobuf wire protocol (though I guess it could be crowbarred into being one!), but is a toolkit for helping you to work with protobuf messages as text. It probably means you wouldn't need to use JSON specifically. You could write a tool that would display a protobuf msg as text, edit it, and re-encode.

SusanW
  • 1,035
  • 10
  • 14