Good examples, articles and books on marshalling [closed]

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance. Closed 11 years ago.

While working on a software protection libr开发者_运维知识库ary for smart card based dongle I realized I need to transfer some tree-like data structures back and forth between client application and code inside the dongle.

Well, when working with web services the technologies like XML-RPC or JSON-RPC are reasonable way to consider. However, that is not the case with embedded devices like smart cards. You need to use some binary formats to optimize memory usage and to achieve good performance.

I guess what I need is to implement some binary data marshaling algorithm. I don't like the idea of reinventing the whole wheel and I pretty sure there are great books, articles and examples on marshalling issues like these.

What would you recommend?

UPD. I am using C and C++ on Linux, but the question is about info on marshalling algorithms in general.

---

You can look at ASN.1 as a general serialization solution. There are free books and and free implementations: asn1c, a2c, snacc.

But ASN.1 is a huge. Simpler protobuf or xdr can be good enough for your task.

---

Google's Protobuf

To start I'm differentiating between marshaling and RPC (which uses marshaling). Google Protobuf is hands down the best solution for marshaling over a network. It has a minimal footprint and encodes/decodes at lightning speed.

If you're still interested in how to implement efficient marshaling checkout the documentation for protobuf encoding.
One example from the encoding page is varint. Varint is how protobuf encodes unsigned integers in binary format. Varint is optimized to for small numbers. eg 1 only uses one byte on the wire while 300 uses two bytes.
Of course the protobuf designers realized that often we use integers as bit masks, etc. So they also provide an integer data type that is always four bytes (that way masks with the msb bits set don't consume extra space).

There is also ample documentation on how to implement RPC with protobufs.

---

You might also want to have a look at Messagepack (http://msgpack.org). It claims to be up to 4 times faster than Protobuf. Also in contrast to Protobuf, this library supports inheritance.

---

There's not a lot of context in your question about what platform / language you are targetting.... however!

The most popular ones were (are?) DCOM and CORBA. There is embedded CORBA..... you can use something like TAO from the (ACE TAO libraries)

but if this is reasonably small scale, you can just serialize it yourself, main thing to remember is to version the serialization format so you can change it and support legacy versions (if thats a concern in your project)

---

serialisation/marshalling and tree like data structures together sounds like quite big problem. Here's some aspects of how I would start solving it:

1. consider encoding of the data. 4 bytes for int transfer or transfer as ascii?
2. consider if you need to transfer whole data or just sync between two tree data structures
3. every bit of data needs to know where it is in the tree structure. Identifying data bits will help quite much in this problem.
4. consider what needs to be transferred?
   • just data inside the tree nodes?
   • changes to the tree structure?
   • the whole tree?
   • location/id of the data?
5. identifying data
   • use paths from root to the tree node?
   • use subtrees?
   • use simple data nodes
   • identifying elements inside tree node?
6. consider how to publish the data in the tree
   • maybe write interface that gives out all the data?
   • can you recursively go through all the data with an algorithm
   • is there pointers inside the data which cannot be transferred easily?

Obviously some libs like the google protobuf can help...

---

You might consider going with Thrift.