Hi Hubert,
Thanks for forwarding that document.
I guess I made some wrong assumptions in what direction compact JSON is going.
This idea about removing the repeating property names to reduce the size is really interesting.
I would like to make the suggestion to actually calculate the effect of different approaches. We could decide on some representative responses to run through: traditional JSON, compact JSON, and binary serialization. All with and without
compression (gzip). Beside comparing size we should also look into the effort to write client code dealing with this.
Christof
From: Hubert Heijkers <
Hubert.Heijkers@nl.ibm.com>
Sent: Tuesday, August 4, 2020 00:59
To: Christof Sprenger <
chrispre@microsoft.com>
Cc:
odata@lists.oasis-open.org Subject: [EXTERNAL] RE: action item 3913 - Concept for binary data format
Hi Christof,
Yeah, that doc is what is supposed to be the start of the formal normative specification for Compact JSON, if you haven't seen my contribution from back then you can find it here:
https://www.oasis-open.org/apps/org/workgroup/odata/download.php/62898/odata-compact-json-format-v4.0.pdf This was enthusiastically received by and I had a follow up meeting with the SAP guys, at their offices, a week or two later, which lead to a number of great ideas/improvements. I'm still eager to take that next
step and then move to a binary 'equivalent'.
Let me know what you think, happy to discuss further.
Cheers,
-H
Hubert Heijkers STSM, Chief Architect TM1 Server and OData Evangelist IBM Business Analytics +31-621-394123
"The crisis of today is the joke of tomorrow" - H.G. Wells
Christof
Sprenger ---03-08-2020 05:23:33 PM---Thanks Hubert for the reply. Would you please share the current stage of the discussion on compact J
From: Christof Sprenger <
chrispre@microsoft.com >
To: Hubert Heijkers <
Hubert.Heijkers@nl.ibm.com >
Cc: "
odata@lists.oasis-open.org " <
odata@lists.oasis-open.org >
Date: 03-08-2020 05:23 PM
Subject: RE: [EXTERNAL] Re: action item 3913 - Concept for binary data format
Thanks Hubert for the reply.
Would you please share the current stage of the discussion on compact JSON ? I was only able to find this
document which is relatively empty.
There is a fundamental difference between MessagePack and ProtoBuf. In ProtoBuf message format is defined upfront and needs to be known and carefully versioned for compatibility. MessagePack, has a fixed but very flexible and simple model (see
type system , and yes, array s are length prefixed similar to all variable length structures like maps, strings, ). In that sense this is similar to JSON and XML where typically there is a second step in Memory that transforms message into Programming
Language Types.
In summary, my impression is that MessagePack is so much closer to JSON that I would expect that it should be easier for any OData implementation to add such a format.
As mentioned above, I do not have any insights into the discussion on compact JSON but it might also be worth investigating compression on top of a binary format.
Happy to help with any investigations.
Christof
From: Hubert Heijkers <
Hubert.Heijkers@nl.ibm.com >
Sent: Sunday, August 2, 2020 09:19
To: Christof Sprenger <
chrispre@microsoft.com >
Cc:
odata@lists.oasis-open.org Subject: [EXTERNAL] Re: action item 3913 - Concept for binary data format
I still have a note here to reach back out to Mike as well because he mentioned ProtoBuf a couple of months ago by now as well in this same realm.
For me this has always been an action item that was closely related to the Compact JSON work, hoping to iron out how we'd be able to specify very concisely what's to be expected in the payload in Compact JSON already and then come up with a representation in
which we could use a more binary (read: less human readable) format to optimize the communication further. Back then I'd looked at a bunch of binary formats that would potentially make sense, MessagePack and ProtoBuf (the later one even more so since we use
build on GRPC), but the time couldn't find any that I liked because either 1) they didn't have a separation between metadata data and a data file, 2) didn't support collections without needing to know the size up front easily (a definite no go for us) or 3)
didn't feel we could easily be embedded in our serialization abstraction logic.
So to answer your question, other then early investigations, I've not gotten to this again, wasn't planning to until we'd have taken the next step on Compact JSON (which is effectively taking my proposal - which we have in production - and merge in all the
ideas/thoughts we had since and come up with a normative proposal for it).
Cheers,
-H
Hubert Heijkers STSM, Chief Architect TM1 Server and OData Evangelist IBM Business Analytics +31-621-394123
"The crisis of today is the joke of tomorrow" - H.G. Wells
Christof Sprenger ---31-07-2020 08:29:54 PM---Hubert, We recently stumbled
upon MessagePack< INVALID URI REMOVED
From: Christof Sprenger <
chrispre@microsoft.com >
To: Hubert Heijkers <
hubert.heijkers@nl.ibm.com >
Cc: "
odata@lists.oasis-open.org " <
odata@lists.oasis-open.org >
Date: 31-07-2020 08:29 PM
Subject: [EXTERNAL] action item 3913 - Concept for binary data format
Hubert,
We recently stumbled upon
MessagePack and I remembered the action item
3913 which seems to be assigned to you.
May I ask in which stage this action item is? I would love to explore if/how Message Pack fits into this since it appears to be at first sight so similar to JSON that this should give a nice balance between efficiency and compatibility.
Christof
Tenzij hierboven anders aangegeven: / Unless stated otherwise above:
IBM Nederland B.V.
Gevestigd te Amsterdam
Inschrijving Handelsregister Amsterdam Nr. 33054214
Tenzij hierboven anders aangegeven: / Unless stated otherwise above:
IBM Nederland B.V.
Gevestigd te Amsterdam
Inschrijving Handelsregister Amsterdam Nr. 33054214