Advanced Message Queuing Protocol (AMQP) Bindings and Mappings (AMQP-BINDMAP) TC

 View Only
  • 1.  JMS Destination handling

    Posted 12-17-2014 12:32
    Hi everyone, I would like to have a discussion around JMS destination handling in the JMS Mapping for AMQP 1.0, in particular around how to handle JMS Destination names via the AMQP "address" field of a link (producer/consumer) source/target and the "to", and "reply-to" field of messages. Apologies for the length of the mail, there is a fair bit to outline. I moved some information for full context to the end to help a tiny bit. I didn't use JIRA for this as mentioned on the call yesterday, as I noticed a problem with my account. JMS defines multiple Destination types that each have their own inherent name space, so it is possible for example to have a Queue and a Topic with the same name (e.g "foo"). AMQP defines an "address" field on the source/target of links (producers/consumers), and a "to" and "reply-to" field are available on messages, to indicate the destination node (e.g queue/topic) address. These are typically string values, and they form a single space since as there is no additional node type information only the address name itself. This is is mostly an issue for non-temporary Queues and Topics since TemporaryQueue and TemporaryTopic destinations will be given generated addresses by the 'broker' peer through use of dynamic nodes, and so can naturally be prevented from having the same addresses as each other, and be made unlikely or unable to clash with non-temporary nodes. To handle this mapping between JMS and AMQP it would seem we must either: 1. Not support JMS Queues and Topics with the same name existing at all, OR 2. Allow multiple nodes to have the same address string but use type metadata (via capabilities + annotations, see additional context) to discriminate between them, OR 3. Utilise address string naming conventions (e.g prefixes) for them to separate the types into subspaces. The first option is an issue for implementations that already do, and wish to continue to, allow Queues and Topics with the same name via other protocols while also supporting AMQP, and would be a limitation in terms of full JMS support. The second option would break reply-to usage for any clients or intermediaries that don't understand the message annotations and/or source+target capabilities carrying type metadata (see additional context). The third option either requires clients to always utilise the full address strings in session.createQueue("<queue-prefix>foo") etc calls, or providing a means to configure the prefixes within the client so that they are added/removed behind the scenes and the application just uses session.createQueue("foo"), but the resulting AMQP address string would be "<queue-prefix>foo". The main issue with requiring clients always use the full address as the session.createQueue(..) value would be for bridging between different systems using different conventions, though the values for those methods are noted as being provider-specific. Both the old Qpid AMQP 1.0 JMS client, and the new JMS client we are creating that implements the JMS Mapping for AMQP being worked on, currently do some form of the third option, providing a way to configure a 'queue prefix' and 'topic prefix' that are used to prefix the application provided strings in session.createQueue(..) etc for outgoing addresses used for links and messages and be stripped from incoming addresses on messages to give the names used for the JMSDestination and JMSReplyTo objects. Temporary destinations are named by the 'broker' peer and their addresses are used as provided. The main issue with this approach is that such configuration makes it more difficult to use the client against a number of different brokers, which is a goal, since this configuration is likely to differ between them meaning even the simplest HelloWorld type example may be unable to work against them without additional configuration. An idea to handle this was to have the brokers use connection properties to inform the client of the prefixes (if any) they require it to use, allowing different brokers to supply their own specific value (if any) to meet their requirements, and allowing clients/simple applications to work against many of them without further configuration change. An alternative suggestion was to have the JMS Mapping define a set of standard name prefixes the client would use by default, such that the issue of Topics and Queues with the same name is addressed by the mapping, while also allowing brokers to specify their own values via connection properties so that their specific needs can still be met if different (e.g they have existing naming conventions they wish/need to retain). There was also a suggestion that something beyond a simple prefix may be needed, I will let the person behind those thoughts expand further to stop this getting any longer for now. Thoughts? Robbie Additional Context: We also need to transmit the destination type information during link (e.g producer/consumer) attachment and on sent messages to ensure we can support the required JMS behaviours (e.g. to indicate we are attaching to a particular type of node, say a queue, and for carrying JMSDestination and JMSReplyTo type on messages to indicate/discover where a message was sent). For handling these points we are defining the following behaviour: # During link attachment for producers/consumers: - Node name carried in source/target "address" field string. - JMS Destination type represented by capabilities on the source/target (e.g "queue", "topic", "temporary-queue", "temporary-topic"). - Clients can optionally assert on the attach response that the required capabilities exist in the source/target to ensure they have attached to a node that meets their requirements. - 'Broker' peers can use the capabilities to determine the type of node to create if it is a dynamic node being requested (or if they support auto-creation of non-temporary nodes). # When sending messages: - Node names carried in "to" and "reply-to" fields as appropriate. - JMS Destination type carried in "x-opt-jms-dest" and "x-opt-jms-reply-to" message annotations as a byte.


  • 2.  RE: JMS Destination handling

    Posted 12-17-2014 13:19
    As per the TC call yesterday, I think option 3 is the only truly viable solution here. > There was also a suggestion that something beyond a simple prefix may be needed, > I will let the person behind those thoughts expand further to stop this getting > any longer for now. That'd be me, I guess :-) My comment on prefixes is mainly that the work on the "global" addressing would potentially impact on "valid" address names, and that a simple prefix such as "queue:" might prevent the use of such addresses in a global context. As such I would probably favour a more flexible (e.g. regex) mechanism for name "mangling". A quick thought on Destination equivalence - I'm guessing that in JMS terms two Destinations are equal iff they are of the same type (Topic/Queue/etc - derived from meta-data) and use the same *mangled* AMQP address? -- Rob Legally required information for business correspondence/Gesetzliche Pflichtangaben für Geschäftskorrespondenz: http://www.jpmorgan.com/pages/international/germany/email


  • 3.  RE: JMS Destination handling

    Posted 12-17-2014 13:47




    +1 on option 3 for sure.
     
    >> My comment on prefixes is mainly that the work on the "global" addressing would potentially impact on "valid" address names, and that a simple prefix such as "queue:" might prevent the use of such addresses in a global context. 
    As such I would probably favour a more flexible (e.g. regex) mechanism for name "mangling".   
     
    Fair enough on the wire, but since JMS supports only topics/queues for addressing today, why not define such fixed prefixes for use by JMS and all other topic/queue based MoMs and then avoid the complex broker/client interaction of making
    these configurable/vendor-specific?   Ie. why would we want them to be different?
     
    Allowing them to be different by implementation, imo, adds yet more complexity to the client/broker interaction and will only further complicate any kind of broker-to-broker addressing of topic/queues required for federation.
     
    Shawn
     



  • 4.  Re: [amqp-bindmap] RE: JMS Destination handling

    Posted 12-17-2014 15:03



  • 5.  RE: JMS Destination handling

    Posted 12-17-2014 18:48




     
     
    >> Fair enough on the wire, but since JMS supports only topics/queues for

    >> addressing today, why not define such fixed prefixes for use by JMS

    >> and all other topic/queue based MoMs and then avoid the complex broker/client
    >> interaction of making these configurable/vendor-specific?   Ie. why would we
    >> want them to be different?
    >
     
    >We might not want them to be different.
     
    >The main issue is likely to be that they already do differ in existing systems because the address string doesn't have a defined syntax, >and changing them to a particular fixed value might not be feasible in all cases.
     
    right - and that can be a problem for systems developed before a spec is completed – and it is a hassle. 

     
    However, there are presumably going to be
    other incompatibilities in addition to this issue between any existing systems (where the incompatibilities are different for each existing system) and what the final mappings specification says and I would hate to set a precedence for needing a negotiation/dynamic
    mechanism of handling each of those that we find – and then maybe still not enough since we don’t know the ones we don’t know.
     
    I don’t think we should let pre-standards work impose complexity that we will need to live with forever.  Ie. more configuration or negotiation, needing to do mappings both for published messages and subscriptions
    and then imposing these issues on administrators – it will not help encourage adoption.
     
    If existing systems/products did something different, I would prefer that they use the typical solution of them using system-specific ways to go from their own on pre-standard version to the standard – and do this as they see fit vs
    burdening the standard with (dynamic) mechanisms to adapt to pre-standard versions whose characteristics we likely don’t fully understand.
     
    That’s why I’d prefer to just define a specific prefix for the identifier of a topic or queue to differentiate them from each other (and others in the Addressing spec) and leave it at that.
     
    Shawn
     



  • 6.  Re: JMS Destination handling

    Posted 12-18-2014 14:46



  • 7.  RE: [amqp-bindmap] RE: JMS Destination handling

    Posted 12-17-2014 20:26
    Having differing prefixing schemes across vendors will add complexity to the client, and particularly in cases where an application might connect to two or more different AMQP implementations having different prefix strings or regex expression to manage. More dirty laundry users won't want to have to deal with and are likely to struggle with to get right. Obtaining custom prefixes wrapped in a centrally administered connection factory (or possibly provided by the broker implementation during the Open performative exchange?) would at least have the benefit of shielding the app from this complexity. If we are leaning towards option #3 route I'm with Shawn though about biting the bullet and trying to standardize on a single prefix convention without further complicating the specification and in turn client and broker implementations having to accommodate this variability. Looking at option #2, annotations could provide a standard, consistent way of indicating destination type ("to" and "reply-to"), irregardless of the address syntax per vendor, and also not have to expose any of this complexity either in app code or in configuration. Robbie indicated this won't work because existing client and intermediaries wouldn't likely understand these attributes. Wouldn't we assume since the JMS mapping is not yet finalized that any vendors that went out on the limb and built implementations ahead of the game might have to change to conform to the final specification, and therefore any clients/intermediaries would have to change to support the defined annotations/capabilities ? . . . Or the above if we standardized on a single prefix scheme? Another thing to think about is how this will impact the interaction between JMS and non-JMS clients and what would be reflected to non-JMS clients from the interface point of view. Neither is ideal and fully transparent. A non-JMS client would be exposed directly to the mangled name and have to deal with parsing that (and would need to be aware the other side is a JMS client to know it needs to do the parse, and, "how do I find the proper regex expression to use?"). Using the option #2 annotations approach would require the client to gain access to the particular field and decipher the dest type. Again, option #2 might be less ugly by having a consistent attribute to look for that indicates dest type irrespective of what shows up in the address field, and possibly it would make sense having a more generic field name for this (i.e. "x-opt-dest" instead of "x-opt-jms-dest" - that way a non-JMS client could set dest type when it wants to communicate dest-type scope without necessarily assuming a JMS broker). Mark


  • 8.  Re: [amqp-bindmap] RE: JMS Destination handling

    Posted 12-19-2014 10:48



  • 9.  Re: [amqp-bindmap] RE: JMS Destination handling

    Posted 12-17-2014 14:52
    In case my expansions below dont make it clear enough, I also think option 3 is the way to go. I was thinking of you with that comment :) Yes, certain prefixes, some of which may already be in use as they do form 'valid' addresses, might not play well in a future global addressing context, certainly something to consider. Any more concrete thoughts/examples on a regex 'mangling' mechanism? In terms of Destination equivalence, if the name mangling is conducted on the way out of the client to form an AMQP address from the initial 'JMS name', and reversed on the way in to the client to give the 'JMS name' used for getJMSDestination/getJMSReplyTo, then I guess two Destination objects would be considered equal if they had the same type and same 'JMS Name'. Robbie


  • 10.  Re: [amqp-bindmap] JMS Destination handling

    Posted 12-18-2014 10:40
    Hi Robbie, Clearly, you seem to be in favor of the option 3. But it is not clear to me what impact will the prefixes have in other clients. From my brief experience with ActiveMQ/A-MQ, I didn't really liked the way it introduced the queue/topic concept into clients written in other languages (e.g. the addresses with different prefixes for queues versus topics which have to be used in the qpid.messaging C++ API). I'm sure that it can help in the JMS clients, but it isn't nice to other clients which are not based on the queue/topic concept and it basically breaks the cross-implementation compatibility there. It is also not clear how this concept would work with a larger network of AMQP components. E.g. when JMS client connects to AMQP server X which tells him to use prefix "queue://" and "topic://" when connecting, but the message with your reply-to address would be forwarded from server X somewhere to server Y where the software works with prefixes "queue::" and "topic::"? ----- To me it seems that there are several working streams - JMS, addressing and management - which very closely relate to each other but are being designed separately. - Addressing tries to define addresses which allow routing across global networks and thus are also compatible across implementations - JMS is trying to create addresses which would define the node type / node behavior - Management already talks about types ... e.g. org.amqp.queue Shouldn't we use the Management stream to define the basic type hierarchy ... e.g. org.amqp.topic and org.amqp.queue (+ some other specific or generic nodes) ... afterwards incorporate these into the addressing ... and at the end use the address with org.amqp.queue or org.amqp.topic in the JMS? The fact that we define the type hierarchy doesn't necessarily mean everyone has to implement everything. But it would be clear that if you implement topic it should inherit from the org.amqp.topic type and be able to handle org.amqp.topic somewhere in the address. Thanks & Regards Jakub ---------------------------------------------------------------------------- Deutsche Börse Services s.r.o. Managing Directors/Geschäftsführung: Michael Gassmann, Mats Andersson. Limited liability company with registered office at Sokolovská 662/136B, CZ-186 00 Prague 8 recorded in the Commercial Register IC: 275 77 015. Maintained by the city court in Prague, Sec. C, File No. 116874. From: Robbie Gemmell <rgemmell@redhat.com> To: amqp-bindmap@lists.oasis-open.org, Date: 17/12/2014 13:31 Subject: [amqp-bindmap] JMS Destination handling Sent by: <amqp-bindmap@lists.oasis-open.org> Hi everyone, I would like to have a discussion around JMS destination handling in the JMS Mapping for AMQP 1.0, in particular around how to handle JMS Destination names via the AMQP "address" field of a link (producer/consumer) source/target and the "to", and "reply-to" field of messages. Apologies for the length of the mail, there is a fair bit to outline. I moved some information for full context to the end to help a tiny bit. I didn't use JIRA for this as mentioned on the call yesterday, as I noticed a problem with my account. JMS defines multiple Destination types that each have their own inherent name space, so it is possible for example to have a Queue and a Topic with the same name (e.g "foo"). AMQP defines an "address" field on the source/target of links (producers/consumers), and a "to" and "reply-to" field are available on messages, to indicate the destination node (e.g queue/topic) address. These are typically string values, and they form a single space since as there is no additional node type information only the address name itself. This is is mostly an issue for non-temporary Queues and Topics since TemporaryQueue and TemporaryTopic destinations will be given generated addresses by the 'broker' peer through use of dynamic nodes, and so can naturally be prevented from having the same addresses as each other, and be made unlikely or unable to clash with non-temporary nodes. To handle this mapping between JMS and AMQP it would seem we must either: 1. Not support JMS Queues and Topics with the same name existing at all, OR 2. Allow multiple nodes to have the same address string but use type metadata (via capabilities + annotations, see additional context) to discriminate between them, OR 3. Utilise address string naming conventions (e.g prefixes) for them to separate the types into subspaces. The first option is an issue for implementations that already do, and wish to continue to, allow Queues and Topics with the same name via other protocols while also supporting AMQP, and would be a limitation in terms of full JMS support. The second option would break reply-to usage for any clients or intermediaries that don't understand the message annotations and/or source+target capabilities carrying type metadata (see additional context). The third option either requires clients to always utilise the full address strings in session.createQueue("<queue-prefix>foo") etc calls, or providing a means to configure the prefixes within the client so that they are added/removed behind the scenes and the application just uses session.createQueue("foo"), but the resulting AMQP address string would be "<queue-prefix>foo". The main issue with requiring clients always use the full address as the session.createQueue(..) value would be for bridging between different systems using different conventions, though the values for those methods are noted as being provider-specific. Both the old Qpid AMQP 1.0 JMS client, and the new JMS client we are creating that implements the JMS Mapping for AMQP being worked on, currently do some form of the third option, providing a way to configure a 'queue prefix' and 'topic prefix' that are used to prefix the application provided strings in session.createQueue(..) etc for outgoing addresses used for links and messages and be stripped from incoming addresses on messages to give the names used for the JMSDestination and JMSReplyTo objects. Temporary destinations are named by the 'broker' peer and their addresses are used as provided. The main issue with this approach is that such configuration makes it more difficult to use the client against a number of different brokers, which is a goal, since this configuration is likely to differ between them meaning even the simplest HelloWorld type example may be unable to work against them without additional configuration. An idea to handle this was to have the brokers use connection properties to inform the client of the prefixes (if any) they require it to use, allowing different brokers to supply their own specific value (if any) to meet their requirements, and allowing clients/simple applications to work against many of them without further configuration change. An alternative suggestion was to have the JMS Mapping define a set of standard name prefixes the client would use by default, such that the issue of Topics and Queues with the same name is addressed by the mapping, while also allowing brokers to specify their own values via connection properties so that their specific needs can still be met if different (e.g they have existing naming conventions they wish/need to retain). There was also a suggestion that something beyond a simple prefix may be needed, I will let the person behind those thoughts expand further to stop this getting any longer for now. Thoughts? Robbie Additional Context: We also need to transmit the destination type information during link (e.g producer/consumer) attachment and on sent messages to ensure we can support the required JMS behaviours (e.g. to indicate we are attaching to a particular type of node, say a queue, and for carrying JMSDestination and JMSReplyTo type on messages to indicate/discover where a message was sent). For handling these points we are defining the following behaviour: # During link attachment for producers/consumers: - Node name carried in source/target "address" field string. - JMS Destination type represented by capabilities on the source/target (e.g "queue", "topic", "temporary-queue", "temporary-topic"). - Clients can optionally assert on the attach response that the required capabilities exist in the source/target to ensure they have attached to a node that meets their requirements. - 'Broker' peers can use the capabilities to determine the type of node to create if it is a dynamic node being requested (or if they support auto-creation of non-temporary nodes). # When sending messages: - Node names carried in "to" and "reply-to" fields as appropriate. - JMS Destination type carried in "x-opt-jms-dest" and "x-opt-jms-reply-to" message annotations as a byte. --------------------------------------------------------------------- To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at: https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php ----------------------------------------- Diese E-Mail enthaelt vertrauliche oder rechtlich geschuetzte Informationen. Wenn Sie nicht der beabsichtigte Empfaenger sind, informieren Sie bitte sofort den Absender und loeschen Sie diese E-Mail. Das unbefugte Kopieren dieser E-Mail oder die unbefugte Weitergabe der enthaltenen Informationen ist nicht gestattet. The information contained in this message is confidential or protected by law. If you are not the intended recipient, please contact the sender and delete this message. Any unauthorised copying of this message or unauthorised distribution of the information contained herein is prohibited. Legally required information for business correspondence/ Gesetzliche Pflichtangaben fuer Geschaeftskorrespondenz: http://deutsche-boerse.com/letterhead


  • 11.  Re: [amqp-bindmap] JMS Destination handling

    Posted 12-19-2014 11:47
    Hi Jakub, I favoured option 3 as a means for implementations that wish to avoid option 1 to do so, essentially thinking of improving the optional functionality some JMS clients already offered to do so by defining a further optional mechanism to convey any inherent default prefix of the server (retaining client side ability to control/override it). Following this discussion and others, I am now of the opinion we should treat this only as an implementation detail, like JMS Destination names themselves are considered, and AMQP 1.0 addresses also. In terms of using other clients, they would continue to utilise whatever address strings they are already required to by the server they are connecting to, as the actual AMQP address of the nodes would still be the same from any client. In the case of networks of different [server] components, I think you would need to ensure the resulting node addresses used the same format everywhere if you wanted that level of cross-server compatibility right now. Ultimately, I would be looking to the Addressing spec to really offer a way to solve that issue through defined global addresses for nodes. As Rob pointed out in previous calls/mails, any client doing that type of prefixing would want to avoid changing such addresses. The idea of incorporating management types into addressing interesting. One possible issue could be that people don't necessarily implement management but could use the global addressing. Really a subject for those groups at the core TC I think (though it may come up as we consider use of management for any JMS functionality, e.g shared subscriptions). Robbie