Advanced Message Queuing Protocol (AMQP) Bindings and Mappings (AMQP-BINDMAP) TC

 View Only
  • 1.  Re: [amqp-bindmap] Raw notes from AMQP Redmond workshop

    Posted 04-18-2013 01:40
    Dave, Thankyou. Regarding websockets, I'm very worried that you want to map AMQP frames to websocket frames. This makes it impossible for anyone to then use one of the many tcp to websocket gateways, so reducing interoperability, and requiring deep broker integrations, etc. In some ways this point mirrors the discussion we had about TLS in AMQP. It does slightly complicate writing a js client, but very very slightly (one has to hold onto the last few bytes in a buffer and append them to the next buffet). Frankly, given that it's very hard to write a js client in any event, this is exceedingly minor. Raphael Cohn Chief Architect raphael.cohn@StormMQ.com StormMQ Limited UK Office: Gateshead int'l Business Centre, Mulgrave Terrace, Gateshead, NE8 1AN, United Kingdom Telephone: +44 845 3712 567 Registered office: 16 Anchor Street, Chelmsford, Essex, CM2 0JY, United Kingdom StormMQ Limited is Registered in England and Wales under Company Number 07175657 StormMQ.com On 18 Apr 2013, at 00:09, David Ingham <David.Ingham@microsoft.com> wrote: > A few folks have asked me for my notes from the Redmond workshop that was held back in February so here they are attached. They are a little raw I'm afraid. There is reference to the AMQP management spec in these notes. Rob Godfrey, Rob Dolin and I have been iterating on that quite a bit since the workshop and I suspect that we'll have something to distribute for review within the next couple of weeks. > > Thanks, > Dave. > <mime-attachment> > <2013.02-AMQP-TechnicalWorkshop-Redmond-Notes-Public.pdf>


  • 2.  RE: [amqp] Re: [amqp-bindmap] Raw notes from AMQP Redmond workshop

    Posted 04-18-2013 05:01




    Moving member section and core TC to BCC as this is a bindmap topic.
     
    Hi Raph,
     
    We considered mapping AMQP frames to Websocket frames but ruled it out primarily because different hops in a multi-hop Websocket chain could use different frame sizes. For example, an intermediary may read a Websocket message that is
    split over two frames and send it on re-encoded in to 1 frame or 3 frames. Because of this there’s no way to know what AMQP frame size should be negotiated to ensure that it could safely be transmitted in a single Websocket frame. We took advice on this from
    several folks who worked on the spec and they all made the same recommendation that we should map AMQP frames to Websocket messages.
     
    All - I’ve sent out a meeting request for a Websockets discussion next Wednesday at 9am Pacific time. I’ve only included those folks that I know are interested in contributing to the spec. If you don’t see the invite but would like to
    attend then ping me. For those of you interested in reviewing the resultant spec, hang on a while, we’ll have a dedicated review when we have agreement on direction between the authors.
     
    Thanks,
    Dave.
     



  • 3.  RE: [amqp] Re: [amqp-bindmap] Raw notes from AMQP Redmond workshop

    Posted 04-18-2013 05:01




    Moving member section and core TC to BCC as this is a bindmap topic.
     
    Hi Raph,
     
    We considered mapping AMQP frames to Websocket frames but ruled it out primarily because different hops in a multi-hop Websocket chain could use different frame sizes. For example, an intermediary may read a Websocket message that is
    split over two frames and send it on re-encoded in to 1 frame or 3 frames. Because of this there’s no way to know what AMQP frame size should be negotiated to ensure that it could safely be transmitted in a single Websocket frame. We took advice on this from
    several folks who worked on the spec and they all made the same recommendation that we should map AMQP frames to Websocket messages.
     
    All - I’ve sent out a meeting request for a Websockets discussion next Wednesday at 9am Pacific time. I’ve only included those folks that I know are interested in contributing to the spec. If you don’t see the invite but would like to
    attend then ping me. For those of you interested in reviewing the resultant spec, hang on a while, we’ll have a dedicated review when we have agreement on direction between the authors.
     
    Thanks,
    Dave.
     



  • 4.  RE: [amqp] Re: [amqp-bindmap] Raw notes from AMQP Redmond workshop

    Posted 04-18-2013 05:01




    Moving member section and core TC to BCC as this is a bindmap topic.
     
    Hi Raph,
     
    We considered mapping AMQP frames to Websocket frames but ruled it out primarily because different hops in a multi-hop Websocket chain could use different frame sizes. For example, an intermediary may read a Websocket message that is
    split over two frames and send it on re-encoded in to 1 frame or 3 frames. Because of this there’s no way to know what AMQP frame size should be negotiated to ensure that it could safely be transmitted in a single Websocket frame. We took advice on this from
    several folks who worked on the spec and they all made the same recommendation that we should map AMQP frames to Websocket messages.
     
    All - I’ve sent out a meeting request for a Websockets discussion next Wednesday at 9am Pacific time. I’ve only included those folks that I know are interested in contributing to the spec. If you don’t see the invite but would like to
    attend then ping me. For those of you interested in reviewing the resultant spec, hang on a while, we’ll have a dedicated review when we have agreement on direction between the authors.
     
    Thanks,
    Dave.
     



  • 5.  Re: [amqp-ms] RE: [amqp] Re: [amqp-bindmap] Raw notes from AMQP Redmond workshop

    Posted 04-18-2013 06:03
    Dave, Thank for the reply. I'm a bit confused as to what you're actually proposing now, but I'm happy to wait until the call to discuss things in more detail. For me, a websocket message IS a set of websocket frames (binary+continuation). The websockets API in the browsers raises one onMessage callback per complete websocket message received. At my end, I've built a complete websockets binding and a trivial AMQP js client (ie just enough to understand the problem space). I see absolutely no reason to do anything at all to AMQP to work with WebSockets. We simply use it as a transport wrapper. Frame size negotiation does not need to change at all. This means onMessage will contain one, several, part of one, or several and part of one, AMQP frame (or handshake). Not a problem, Just concatenate the array buffer in the client code. Far bigger fish to fry - like state management or decoding... Indeed, it is essential to not change the main spec, because otherwise, we exclude EVERYONE who wants to just put a simple Websockets TCP gateway in front of an AMQP connection - no matter from whom they are getting that AMQP connection. Instead, we bias the market to favour only:- - bespoke WebSockets vendors - those broker vendors who support this variation In other words, it could be seen as anti-competitive and not in our customer audience's best interests. Raph PS Having implemented WebSockets, I'm very unimpressed by the WebSockets RFC. It can't make its mind up if its a transport protocol or an application protocol. Raphael Cohn Chief Architect, stormmq Co-Chair, OASIS MQTT Standard Secretary, OASIS AMQP Standard raphael.cohn@stormmq.com +44 7590 675 756 UK Office: Hamblethorpe Farm, Crag Lane, Bradley BD20 9DB, North Yorkshire, United Kingdom Telephone: +44 845 3712 567 Registered office: 16 Anchor Street, Chelmsford, Essex, CM2 0JY, United Kingdom StormMQ Limited is Registered in England and Wales under Company Number 07175657 StormMQ.com On 18 April 2013 06:00, David Ingham < David.Ingham@microsoft.com > wrote: Moving member section and core TC to BCC as this is a bindmap topic.   Hi Raph,   We considered mapping AMQP frames to Websocket frames but ruled it out primarily because different hops in a multi-hop Websocket chain could use different frame sizes. For example, an intermediary may read a Websocket message that is split over two frames and send it on re-encoded in to 1 frame or 3 frames. Because of this there’s no way to know what AMQP frame size should be negotiated to ensure that it could safely be transmitted in a single Websocket frame. We took advice on this from several folks who worked on the spec and they all made the same recommendation that we should map AMQP frames to Websocket messages.   All - I’ve sent out a meeting request for a Websockets discussion next Wednesday at 9am Pacific time. I’ve only included those folks that I know are interested in contributing to the spec. If you don’t see the invite but would like to attend then ping me. For those of you interested in reviewing the resultant spec, hang on a while, we’ll have a dedicated review when we have agreement on direction between the authors.   Thanks, Dave.  


  • 6.  RE: [amqp-bindmap] Re: [amqp-ms] RE: [amqp] Re: [amqp-bindmap] Raw notes from AMQP Redmond workshop

    Posted 04-18-2013 08:18




    To be clear, there are no changes to the core spec implied by the mapping to WebSockets. 

     
    The proposal to map an AMQP Frame to a WebSocket message means that there is no need to alter frame size negotiation – the AMQP Frame size and WebSocket frame
    size are independent (if the AMQP frame is bigger than a WebSocket frame, continuation frames are used).  The only implication is that at an AMQP Frame boundary one must start a new WebSocket message (and thus a new WebSocket frame).  This is obviously trivial
    to implement.  In the browser this means that one onMessage will contain exactly one AMQP frame which removes the need to replicate the buffer concatenation and frame parsing which WebSockets can do for you. It took me about half an hour to implement a TCP
    <-> WebSocket gateway based on this scheme.
     
    -- Rob
     

    From: amqp-bindmap@lists.oasis-open.org [mailto:amqp-bindmap@lists.oasis-open.org]
    On Behalf Of Raphael Cohn
    Sent: 18 April 2013 07:03
    To: David Ingham
    Cc: amqp-bindmap@lists.oasis-open.org
    Subject: [amqp-bindmap] Re: [amqp-ms] RE: [amqp] Re: [amqp-bindmap] Raw notes from AMQP Redmond workshop

     

    Dave,

     


    Thank for the reply. I'm a bit confused as to what you're actually proposing now, but I'm happy to wait until the call to discuss things in more detail. For me, a websocket message IS a set of websocket frames (binary+continuation). The
    websockets API in the browsers raises one onMessage callback per complete websocket message received.


     


    At my end, I've built a complete websockets binding and a trivial AMQP js client (ie just enough to understand the problem space). I see absolutely no reason to do anything at all to AMQP to work with WebSockets. We simply use it as a transport
    wrapper. Frame size negotiation does not need to change at all. This means onMessage will contain one, several, part of one, or several and part of one, AMQP frame (or handshake). Not a problem, Just concatenate the array buffer in the client code. Far bigger
    fish to fry - like state management or decoding...


     


    Indeed, it is essential to not change the main spec, because otherwise, we exclude EVERYONE who wants to just put a simple Websockets TCP gateway in front of an AMQP connection - no matter from whom they are getting that AMQP connection.
    Instead, we bias the market to favour only:-


     


    - bespoke WebSockets vendors


    - those broker vendors who support this variation


     


    In other words, it could be seen as anti-competitive and not in our customer audience's best interests.


     


    Raph


     


    PS Having implemented WebSockets, I'm very unimpressed by the WebSockets RFC. It can't make its mind up if its a transport protocol or an application protocol.






    Raphael Cohn
    Chief Architect, stormmq

    Co-Chair, OASIS MQTT Standard

    Secretary, OASIS AMQP Standard
    raphael.cohn@stormmq.com


    +44 7590 675 756



    UK Office:
    Hamblethorpe Farm, Crag Lane, Bradley BD20 9DB, North Yorkshire, United Kingdom
    Telephone: +44 845 3712 567

    Registered office:
    16 Anchor Street, Chelmsford, Essex, CM2 0JY, United Kingdom
    StormMQ Limited is Registered in England and Wales under Company Number 07175657
    StormMQ.com



     

    On 18 April 2013 06:00, David Ingham < David.Ingham@microsoft.com > wrote:


    Moving member section and core TC to BCC as this is a bindmap topic.
     
    Hi Raph,
     
    We considered mapping AMQP frames to Websocket frames but ruled it out primarily because different hops in a multi-hop Websocket chain could use different frame sizes. For example, an intermediary may read a Websocket message that is split
    over two frames and send it on re-encoded in to 1 frame or 3 frames. Because of this there’s no way to know what AMQP frame size should be negotiated to ensure that it could safely be transmitted in a single Websocket frame. We took advice on this from several
    folks who worked on the spec and they all made the same recommendation that we should map AMQP frames to Websocket messages.
     
    All - I’ve sent out a meeting request for a Websockets discussion next Wednesday at 9am Pacific time. I’ve only included those folks that I know are interested in contributing to the spec. If you don’t see the invite but would like to
    attend then ping me. For those of you interested in reviewing the resultant spec, hang on a while, we’ll have a dedicated review when we have agreement on direction between the authors.
     
    Thanks,
    Dave.


     



  • 7.  Re: [amqp-bindmap] Re: [amqp-ms] RE: [amqp] Re: [amqp-bindmap] Raw notes from AMQP Redmond workshop

    Posted 04-18-2013 09:10
    Thanks for the clarification Rob. I think it should just be the raw stream. That way AMQP can be transmitted over WebSockets by third parties using generic libraries and products - with no need to understand AMQP at all. Hence we can drive adoption across the very widest possible community. This does make life slightly harder for the writer of a JS client, but, in all likelihood, there's only ever going to be one JS WebSockets AMQP client... The flip side, is it makes it much *Easier* for a non-browser based WebSockets client to simply pass data to a regular AMQP client library - it becomes nothing more than another implementation of Socket in the Java world, say. A corollary to this point is the way we use TLS - we deliberately decided NOT to align AMQP frames to TLS records, even though there would be some TLS optimisation advantages. As a result, we can use TLS with libraries, balancers, routers, etc that know nothing about AMQP. My point above is not that it may or may not be simple to implement in the originator - but that WebSockets should be a tunnel that is independent of AMQP, so users can users things that wrap it in WebSockets, etc. I presume we're also keeping the handshakes (although a TLS handshake would seem to be redundant, except in some weird edge case involving tunnelling through to a broker only expecting a TLS handshake - perhaps not as mad as it sounds - especially if the broker is remote on the internet, and the WebSocket is terminated at a local gateway)? Raph Raphael Cohn Chief Architect, stormmq Co-Chair, OASIS MQTT Standard Secretary, OASIS AMQP Standard raphael.cohn@stormmq.com +44 7590 675 756 UK Office: Hamblethorpe Farm, Crag Lane, Bradley BD20 9DB, North Yorkshire, United Kingdom Telephone: +44 845 3712 567 Registered office: 16 Anchor Street, Chelmsford, Essex, CM2 0JY, United Kingdom StormMQ Limited is Registered in England and Wales under Company Number 07175657 StormMQ.com On 18 April 2013 08:59, Godfrey, Robert X < robert.godfrey@jpmorgan.com > wrote: To be clear, there are no changes to the core spec implied by the mapping to WebSockets.    The proposal to map an AMQP Frame to a WebSocket message means that there is no need to alter frame size negotiation – the AMQP Frame size and WebSocket frame size are independent (if the AMQP frame is bigger than a WebSocket frame, continuation frames are used).  The only implication is that at an AMQP Frame boundary one must start a new WebSocket message (and thus a new WebSocket frame).  This is obviously trivial to implement.  In the browser this means that one onMessage will contain exactly one AMQP frame which removes the need to replicate the buffer concatenation and frame parsing which WebSockets can do for you. It took me about half an hour to implement a TCP <-> WebSocket gateway based on this scheme.   -- Rob   From: amqp-bindmap@lists.oasis-open.org [mailto: amqp-bindmap@lists.oasis-open.org ] On Behalf Of Raphael Cohn Sent: 18 April 2013 07:03 To: David Ingham Cc: amqp-bindmap@lists.oasis-open.org Subject: [amqp-bindmap] Re: [amqp-ms] RE: [amqp] Re: [amqp-bindmap] Raw notes from AMQP Redmond workshop   Dave,   Thank for the reply. I'm a bit confused as to what you're actually proposing now, but I'm happy to wait until the call to discuss things in more detail. For me, a websocket message IS a set of websocket frames (binary+continuation). The websockets API in the browsers raises one onMessage callback per complete websocket message received.   At my end, I've built a complete websockets binding and a trivial AMQP js client (ie just enough to understand the problem space). I see absolutely no reason to do anything at all to AMQP to work with WebSockets. We simply use it as a transport wrapper. Frame size negotiation does not need to change at all. This means onMessage will contain one, several, part of one, or several and part of one, AMQP frame (or handshake). Not a problem, Just concatenate the array buffer in the client code. Far bigger fish to fry - like state management or decoding...   Indeed, it is essential to not change the main spec, because otherwise, we exclude EVERYONE who wants to just put a simple Websockets TCP gateway in front of an AMQP connection - no matter from whom they are getting that AMQP connection. Instead, we bias the market to favour only:-   - bespoke WebSockets vendors - those broker vendors who support this variation   In other words, it could be seen as anti-competitive and not in our customer audience's best interests.   Raph   PS Having implemented WebSockets, I'm very unimpressed by the WebSockets RFC. It can't make its mind up if its a transport protocol or an application protocol. Raphael Cohn Chief Architect, stormmq Co-Chair, OASIS MQTT Standard Secretary, OASIS AMQP Standard raphael.cohn@stormmq.com +44 7590 675 756 UK Office: Hamblethorpe Farm, Crag Lane, Bradley BD20 9DB, North Yorkshire, United Kingdom Telephone: +44 845 3712 567 Registered office: 16 Anchor Street, Chelmsford, Essex, CM2 0JY, United Kingdom StormMQ Limited is Registered in England and Wales under Company Number 07175657 StormMQ.com   On 18 April 2013 06:00, David Ingham < David.Ingham@microsoft.com > wrote: Moving member section and core TC to BCC as this is a bindmap topic.   Hi Raph,   We considered mapping AMQP frames to Websocket frames but ruled it out primarily because different hops in a multi-hop Websocket chain could use different frame sizes. For example, an intermediary may read a Websocket message that is split over two frames and send it on re-encoded in to 1 frame or 3 frames. Because of this there’s no way to know what AMQP frame size should be negotiated to ensure that it could safely be transmitted in a single Websocket frame. We took advice on this from several folks who worked on the spec and they all made the same recommendation that we should map AMQP frames to Websocket messages.   All - I’ve sent out a meeting request for a Websockets discussion next Wednesday at 9am Pacific time. I’ve only included those folks that I know are interested in contributing to the spec. If you don’t see the invite but would like to attend then ping me. For those of you interested in reviewing the resultant spec, hang on a while, we’ll have a dedicated review when we have agreement on direction between the authors.   Thanks, Dave.  


  • 8.  RE: [amqp-bindmap] Re: [amqp-ms] RE: [amqp] Re: [amqp-bindmap] Raw notes from AMQP Redmond workshop

    Posted 04-18-2013 09:58




    I think there are two distinct use cases here, mapping AMQP to WebSockets and tunneling AMQP (or any other TCP stream) over WebSockets. 

     
    For tunneling there is no specific knowledge of AMQP at a proxy, it simply wraps what it receives over TCP into WebSocket frames, and unwraps incoming WebSocket
    frames as TCP. This would include tunneling a TLS encrypted stream.  This case seems primarily useful where WebSockets is being used to circumvent firewalls, and there is no expectation that either AMQP endpoint speaks WebSockets natively.  Any TLS encryption/authentication
    will be between the ultimate TCP endpoints and they are unaware of the use of WebSockets as part of the transport.
     
    A mapping of AMQP to WebSockets is useful where endpoints to are speaking WebSockets natively.  Where we wish to bridge such an endpoint to a TCP network it
    will be necessary to introduce a TCP/WebSocket bridge with specific understanding of the mapping of AMQP to WebSockets (in practice it should be easy to make any AMQP endpoint speak WebSockets in addition to AMQP if decent WebSocket libraries exist – in Java
    this is again a matter of minutes).  In a mapping, where there is bridging the TLS encryption/authentication would be between the endpoint and the bridge.  The secure WebSocket would then be established between the bridge and the other endpoint.  If a raw
    (encrypted) stream were to be passed through I believe it would require a browser client to have a pure _javascript_ TLS implementation (I’ve looked – one does exist – but this seems a sub-optimal solution).
     
    For tunneling there is no need to define any AMQP-specific subprotocols… the WebSocket connection is purely carrying tunneled TCP.  It is up to the endpoints
    to discover the nature of the tunneled stream (whether it be AMQP or stream video or whatever).  In the mapping case the endpoints are natively speaking WebSockets and should utilize the features that this provides, and TLS should be provided by the WebSocket
    layer (between the WebSocket endpoints).
     
    -- Rob
     

    From: Raphael Cohn [mailto:raphael.cohn@stormmq.com]

    Sent: 18 April 2013 10:10
    To: Godfrey, Robert X
    Cc: David Ingham; amqp-bindmap@lists.oasis-open.org
    Subject: Re: [amqp-bindmap] Re: [amqp-ms] RE: [amqp] Re: [amqp-bindmap] Raw notes from AMQP Redmond workshop

     

    Thanks for the clarification Rob. I think it should just be the raw stream. That way AMQP can be transmitted over WebSockets by third parties using generic libraries and products - with no need to understand AMQP at all. Hence we can drive
    adoption across the very widest possible community. This does make life slightly harder for the writer of a JS client, but, in all likelihood, there's only ever going to be one JS WebSockets AMQP client... The flip side, is it makes it much *Easier* for a
    non-browser based WebSockets client to simply pass data to a regular AMQP client library - it becomes nothing more than another implementation of Socket in the Java world, say.

     


    A corollary to this point is the way we use TLS - we deliberately decided NOT to align AMQP frames to TLS records, even though there would be some TLS optimisation advantages. As a result, we can use TLS with libraries, balancers, routers,
    etc that know nothing about AMQP. My point above is not that it may or may not be simple to implement in the originator - but that WebSockets should be a tunnel that is independent of AMQP, so users can users things that wrap it in WebSockets, etc.


     


    I presume we're also keeping the handshakes (although a TLS handshake would seem to be redundant, except in some weird edge case involving tunnelling through to a broker only expecting a TLS handshake - perhaps not as mad as it sounds -
    especially if the broker is remote on the internet, and the WebSocket is terminated at a local gateway)?


     


    Raph


     






    Raphael Cohn
    Chief Architect, stormmq

    Co-Chair, OASIS MQTT Standard

    Secretary, OASIS AMQP Standard
    raphael.cohn@stormmq.com


    +44 7590 675 756



    UK Office:
    Hamblethorpe Farm, Crag Lane, Bradley BD20 9DB, North Yorkshire, United Kingdom
    Telephone: +44 845 3712 567

    Registered office:
    16 Anchor Street, Chelmsford, Essex, CM2 0JY, United Kingdom
    StormMQ Limited is Registered in England and Wales under Company Number 07175657
    StormMQ.com



     

    On 18 April 2013 08:59, Godfrey, Robert X < robert.godfrey@jpmorgan.com > wrote:


    To be clear, there are no changes to the core spec implied by the mapping to WebSockets. 

     
    The proposal to map an AMQP Frame to a WebSocket message means that there is no need to alter frame
    size negotiation – the AMQP Frame size and WebSocket frame size are independent (if the AMQP frame is bigger than a WebSocket frame, continuation frames are used).  The only implication is that at an AMQP Frame boundary one must start a new WebSocket message
    (and thus a new WebSocket frame).  This is obviously trivial to implement.  In the browser this means that one onMessage will contain exactly one AMQP frame which removes the need to replicate the buffer concatenation and frame parsing which WebSockets can
    do for you. It took me about half an hour to implement a TCP <-> WebSocket gateway based on this scheme.
     
    -- Rob
     

    From:
    amqp-bindmap@lists.oasis-open.org [mailto: amqp-bindmap@lists.oasis-open.org ]
    On Behalf Of Raphael Cohn
    Sent: 18 April 2013 07:03
    To: David Ingham
    Cc: amqp-bindmap@lists.oasis-open.org
    Subject: [amqp-bindmap] Re: [amqp-ms] RE: [amqp] Re: [amqp-bindmap] Raw notes from AMQP Redmond workshop



     

    Dave,

     


    Thank for the reply. I'm a bit confused as to what you're actually proposing now, but I'm happy to wait until the call to discuss things in more detail. For me, a websocket message
    IS a set of websocket frames (binary+continuation). The websockets API in the browsers raises one onMessage callback per complete websocket message received.


     


    At my end, I've built a complete websockets binding and a trivial AMQP js client (ie just enough to understand the problem space). I see absolutely no reason to do anything at all
    to AMQP to work with WebSockets. We simply use it as a transport wrapper. Frame size negotiation does not need to change at all. This means onMessage will contain one, several, part of one, or several and part of one, AMQP frame (or handshake). Not a problem,
    Just concatenate the array buffer in the client code. Far bigger fish to fry - like state management or decoding...


     


    Indeed, it is essential to not change the main spec, because otherwise, we exclude EVERYONE who wants to just put a simple Websockets TCP gateway in front of an AMQP connection
    - no matter from whom they are getting that AMQP connection. Instead, we bias the market to favour only:-


     


    - bespoke WebSockets vendors


    - those broker vendors who support this variation


     


    In other words, it could be seen as anti-competitive and not in our customer audience's best interests.


     


    Raph


     


    PS Having implemented WebSockets, I'm very unimpressed by the WebSockets RFC. It can't make its mind up if its a transport protocol or an application protocol.






    Raphael Cohn
    Chief Architect, stormmq

    Co-Chair, OASIS MQTT Standard

    Secretary, OASIS AMQP Standard
    raphael.cohn@stormmq.com


    +44 7590 675 756



    UK Office:
    Hamblethorpe Farm, Crag Lane, Bradley BD20 9DB, North Yorkshire, United Kingdom
    Telephone: +44 845 3712 567

    Registered office:
    16 Anchor Street, Chelmsford, Essex, CM2 0JY, United Kingdom
    StormMQ Limited is Registered in England and Wales under Company Number 07175657
    StormMQ.com



     

    On 18 April 2013 06:00, David Ingham < David.Ingham@microsoft.com > wrote:


    Moving member section and core TC to BCC as this is a bindmap topic.
     
    Hi Raph,
     
    We considered mapping AMQP frames to Websocket frames but ruled it out primarily because different hops in a multi-hop Websocket chain could use different frame sizes. For example, an intermediary may read a Websocket message that is split
    over two frames and send it on re-encoded in to 1 frame or 3 frames. Because of this there’s no way to know what AMQP frame size should be negotiated to ensure that it could safely be transmitted in a single Websocket frame. We took advice on this from several
    folks who worked on the spec and they all made the same recommendation that we should map AMQP frames to Websocket messages.
     
    All - I’ve sent out a meeting request for a Websockets discussion next Wednesday at 9am Pacific time. I’ve only included those folks that I know are interested in contributing to the spec. If you don’t see the invite but would like to
    attend then ping me. For those of you interested in reviewing the resultant spec, hang on a while, we’ll have a dedicated review when we have agreement on direction between the authors.
     
    Thanks,
    Dave.


     



  • 9.  Re: [amqp-bindmap] Re: [amqp-ms] RE: [amqp] Re: [amqp-bindmap] Raw notes from AMQP Redmond workshop

    Posted 04-18-2013 11:16
    Maybe. With use case 2, there is quite a performance hit for the bridge. It took years for things like HAProxy to get to the level of performance that they currently do (indeed, the HAProxy code base would be a good place to start, as its core is a Janus-faced edge polling generic socket to socket connector with transforms, and Linux kernel pipe buffers - vsplice() and friends). I agree raw TLS to be decrypted by a JS client would be sup-optimal - but its not the only use case (although I emphasise I think it's very unlikely to be a common case). There's an increasing use of WebSockets for non-browser clients - mostly because of its firewall tunnelling, use of existing infrastructure and auth, etc. We should consider their needs too. If we really care about performance, then forcing a sender to sync his network writes to WS frames *is* sub-optimal, as they can no longer use writev() to do scattered writes as efficiently  - a lot more checks need to be made, and far fewer scenarios will fit. We should be careful. WebSockets use cases will involve 10,000s to 100,000s of clients - any small performance or bandwidth gains are substantial in such an use case. I don't quite agree with the para 3 - I still think there's a WS subprotocol, and its still amqp. I do strongly disagree that we should use the features WebSockets provide. They're half-baked and poorly thought through. Are we thus also planning to use its ping-pong for heartbeats? Gzip extensions? My desire here is to keep things really simple to drive maximum interop, and to allow the widest range of uses with off-the-shelf tech. So, for me, there's jsut one use case. The simplest thing over WebSockets, so everyone can work with the AMQP protocol. Raph Raphael Cohn Chief Architect, stormmq Co-Chair, OASIS MQTT Standard Secretary, OASIS AMQP Standard raphael.cohn@stormmq.com +44 7590 675 756 UK Office: Hamblethorpe Farm, Crag Lane, Bradley BD20 9DB, North Yorkshire, United Kingdom Telephone: +44 845 3712 567 Registered office: 16 Anchor Street, Chelmsford, Essex, CM2 0JY, United Kingdom StormMQ Limited is Registered in England and Wales under Company Number 07175657 StormMQ.com On 18 April 2013 10:57, Godfrey, Robert X < robert.godfrey@jpmorgan.com > wrote: I think there are two distinct use cases here, mapping AMQP to WebSockets and tunneling AMQP (or any other TCP stream) over WebSockets.    For tunneling there is no specific knowledge of AMQP at a proxy, it simply wraps what it receives over TCP into WebSocket frames, and unwraps incoming WebSocket frames as TCP. This would include tunneling a TLS encrypted stream.  This case seems primarily useful where WebSockets is being used to circumvent firewalls, and there is no expectation that either AMQP endpoint speaks WebSockets natively.  Any TLS encryption/authentication will be between the ultimate TCP endpoints and they are unaware of the use of WebSockets as part of the transport.   A mapping of AMQP to WebSockets is useful where endpoints to are speaking WebSockets natively.  Where we wish to bridge such an endpoint to a TCP network it will be necessary to introduce a TCP/WebSocket bridge with specific understanding of the mapping of AMQP to WebSockets (in practice it should be easy to make any AMQP endpoint speak WebSockets in addition to AMQP if decent WebSocket libraries exist – in Java this is again a matter of minutes).  In a mapping, where there is bridging the TLS encryption/authentication would be between the endpoint and the bridge.  The secure WebSocket would then be established between the bridge and the other endpoint.  If a raw (encrypted) stream were to be passed through I believe it would require a browser client to have a pure _javascript_ TLS implementation (I’ve looked – one does exist – but this seems a sub-optimal solution).   For tunneling there is no need to define any AMQP-specific subprotocols… the WebSocket connection is purely carrying tunneled TCP.  It is up to the endpoints to discover the nature of the tunneled stream (whether it be AMQP or stream video or whatever).  In the mapping case the endpoints are natively speaking WebSockets and should utilize the features that this provides, and TLS should be provided by the WebSocket layer (between the WebSocket endpoints).   -- Rob   From: Raphael Cohn [mailto: raphael.cohn@stormmq.com ] Sent: 18 April 2013 10:10 To: Godfrey, Robert X Cc: David Ingham; amqp-bindmap@lists.oasis-open.org Subject: Re: [amqp-bindmap] Re: [amqp-ms] RE: [amqp] Re: [amqp-bindmap] Raw notes from AMQP Redmond workshop   Thanks for the clarification Rob. I think it should just be the raw stream. That way AMQP can be transmitted over WebSockets by third parties using generic libraries and products - with no need to understand AMQP at all. Hence we can drive adoption across the very widest possible community. This does make life slightly harder for the writer of a JS client, but, in all likelihood, there's only ever going to be one JS WebSockets AMQP client... The flip side, is it makes it much *Easier* for a non-browser based WebSockets client to simply pass data to a regular AMQP client library - it becomes nothing more than another implementation of Socket in the Java world, say.   A corollary to this point is the way we use TLS - we deliberately decided NOT to align AMQP frames to TLS records, even though there would be some TLS optimisation advantages. As a result, we can use TLS with libraries, balancers, routers, etc that know nothing about AMQP. My point above is not that it may or may not be simple to implement in the originator - but that WebSockets should be a tunnel that is independent of AMQP, so users can users things that wrap it in WebSockets, etc.   I presume we're also keeping the handshakes (although a TLS handshake would seem to be redundant, except in some weird edge case involving tunnelling through to a broker only expecting a TLS handshake - perhaps not as mad as it sounds - especially if the broker is remote on the internet, and the WebSocket is terminated at a local gateway)?   Raph   Raphael Cohn Chief Architect, stormmq Co-Chair, OASIS MQTT Standard Secretary, OASIS AMQP Standard raphael.cohn@stormmq.com +44 7590 675 756 UK Office: Hamblethorpe Farm, Crag Lane, Bradley BD20 9DB, North Yorkshire, United Kingdom Telephone: +44 845 3712 567 Registered office: 16 Anchor Street, Chelmsford, Essex, CM2 0JY, United Kingdom StormMQ Limited is Registered in England and Wales under Company Number 07175657 StormMQ.com   On 18 April 2013 08:59, Godfrey, Robert X < robert.godfrey@jpmorgan.com > wrote: To be clear, there are no changes to the core spec implied by the mapping to WebSockets.    The proposal to map an AMQP Frame to a WebSocket message means that there is no need to alter frame size negotiation – the AMQP Frame size and WebSocket frame size are independent (if the AMQP frame is bigger than a WebSocket frame, continuation frames are used).  The only implication is that at an AMQP Frame boundary one must start a new WebSocket message (and thus a new WebSocket frame).  This is obviously trivial to implement.  In the browser this means that one onMessage will contain exactly one AMQP frame which removes the need to replicate the buffer concatenation and frame parsing which WebSockets can do for you. It took me about half an hour to implement a TCP <-> WebSocket gateway based on this scheme.   -- Rob   From: amqp-bindmap@lists.oasis-open.org [mailto: amqp-bindmap@lists.oasis-open.org ] On Behalf Of Raphael Cohn Sent: 18 April 2013 07:03 To: David Ingham Cc: amqp-bindmap@lists.oasis-open.org Subject: [amqp-bindmap] Re: [amqp-ms] RE: [amqp] Re: [amqp-bindmap] Raw notes from AMQP Redmond workshop   Dave,   Thank for the reply. I'm a bit confused as to what you're actually proposing now, but I'm happy to wait until the call to discuss things in more detail. For me, a websocket message IS a set of websocket frames (binary+continuation). The websockets API in the browsers raises one onMessage callback per complete websocket message received.   At my end, I've built a complete websockets binding and a trivial AMQP js client (ie just enough to understand the problem space). I see absolutely no reason to do anything at all to AMQP to work with WebSockets. We simply use it as a transport wrapper. Frame size negotiation does not need to change at all. This means onMessage will contain one, several, part of one, or several and part of one, AMQP frame (or handshake). Not a problem, Just concatenate the array buffer in the client code. Far bigger fish to fry - like state management or decoding...   Indeed, it is essential to not change the main spec, because otherwise, we exclude EVERYONE who wants to just put a simple Websockets TCP gateway in front of an AMQP connection - no matter from whom they are getting that AMQP connection. Instead, we bias the market to favour only:-   - bespoke WebSockets vendors - those broker vendors who support this variation   In other words, it could be seen as anti-competitive and not in our customer audience's best interests.   Raph   PS Having implemented WebSockets, I'm very unimpressed by the WebSockets RFC. It can't make its mind up if its a transport protocol or an application protocol. Raphael Cohn Chief Architect, stormmq Co-Chair, OASIS MQTT Standard Secretary, OASIS AMQP Standard raphael.cohn@stormmq.com +44 7590 675 756 UK Office: Hamblethorpe Farm, Crag Lane, Bradley BD20 9DB, North Yorkshire, United Kingdom Telephone: +44 845 3712 567 Registered office: 16 Anchor Street, Chelmsford, Essex, CM2 0JY, United Kingdom StormMQ Limited is Registered in England and Wales under Company Number 07175657 StormMQ.com   On 18 April 2013 06:00, David Ingham < David.Ingham@microsoft.com > wrote: Moving member section and core TC to BCC as this is a bindmap topic.   Hi Raph,   We considered mapping AMQP frames to Websocket frames but ruled it out primarily because different hops in a multi-hop Websocket chain could use different frame sizes. For example, an intermediary may read a Websocket message that is split over two frames and send it on re-encoded in to 1 frame or 3 frames. Because of this there’s no way to know what AMQP frame size should be negotiated to ensure that it could safely be transmitted in a single Websocket frame. We took advice on this from several folks who worked on the spec and they all made the same recommendation that we should map AMQP frames to Websocket messages.   All - I’ve sent out a meeting request for a Websockets discussion next Wednesday at 9am Pacific time. I’ve only included those folks that I know are interested in contributing to the spec. If you don’t see the invite but would like to attend then ping me. For those of you interested in reviewing the resultant spec, hang on a while, we’ll have a dedicated review when we have agreement on direction between the authors.   Thanks, Dave.