I don't think the words that Brian quotes
from the charter are an issue here. They say
· The TC will not identify MQTT topics nor prescribe
any mechanism or convention for classification of MQTT topics or
topic spaces.
The field in question is a Topic Filter,
not a Topic, so we're free to define syntactic devices in that field to
express things that aren't part of the topic name. The + and # characters
are examples of this, and it would in principle be ok to define a new syntax
that lets us insert additional flags or subfields into the Topic Filter
should we want to do that.
There is a bit of a problem doing that,
as at the moment any leading character other than + / or # is assumed to
be the first character of the topic name. Moreover Server implementations
are permitted to use strings that start with a $ as real topic name. The
compatibility sentences that Ken quotes mean that - if we want to go this
way - we'd have to use a syntax that doesn't take away anything that people
might already be using. I would have thought that starting with $$
or ++ might be safe.
Anyway, before we get too wrapped up
in the syntax, it's important to agree on the semantics of the thing, and
what restrictions might be imposed on it. Andrew is suggesting quite a
drastic restriction, but I suspect it will meet the requirements of a large
number of users.
1.The simplest case to consider is one
where the consuming application has a collection of threads, all subscribed
to the a single topic. We need a way to place them all in the the same
"consumer group" so that each incoming message is delivered to
exactly one thread. The application should be able to add or remove
threads from this group dynamically, e.g. by subscribing or unsubscribing
2. The next case is the same, but with
a wildcarded topic _expression_. Again, we want the messages to be delivered
to exactly one consuming thread.
Note that in both these cases, I am
assuming that the application doesn't require exact message ordering (or
those messages that do need to be ordered come sufficiently far apart from
each other that you are prepared to take a risk). If you need more guarantees
you would do it by partitioning the topic into subtopics and not used shared
subscriptions as you want to make sure that there's only one consuming
thread per subtopic.
3. Now consider the case where the application
submits two or more subscriptions. There seem to be two cases:
i) They still want to have a single
pool of threads each processing exactly one of the messages. In effect
this is the same as 2; the only reason you would need multiple subscriptions
is if you couldn't express the filter with a single wildcard _expression_.
In this case each thread would submit the same set of subscriptions. It's
unlikely that the subscriptions would overlap, but I guess it could happen.
ii) They want to control the number
of threads dedicated to the two subscriptions separately. In this case
the threads in each different subpool would use the same topic filter,
but the filters would be different for the different subpools. Again there's
the unlikely possibility of overlap. The case where one of the subpools
only has a single thread is in effect a non-shared subscription.
I think Andrew's suggestion covers 1,
and 3 i). To do 3 ii) I think you would have to say that you get a different
subscription group for each topic filter/share name combination, as otherwise
we would force people to use different sessions for each subpool
I don't really understand what would
happen with his suggestion if there is overlap between the topic filters
in case 3.
I think there might be other semantics
we have to think about, with regards to in-flight QoS 1 or Qos 2 messages
and shared subscriptions.
Peter Niblett
IBM Senior Technical Staff Member
Member of the IBM Academy of Technology
+44 1962 815055
+44 7825 657662 (mobile)
From:
Andrew Schofield/UK/IBM@IBMGB
To:
mqtt@lists.oasis-open.org Date:
09/04/2016 09:52
Subject:
Re: [mqtt] Recharter:
Shared subscriptions and $share
Sent by:
<
mqtt@lists.oasis-open.org>
One final thought. I think a sticking
point here is that it's hard to handle a mixture of shared and nonshared
subscriptions from the same client. I think supporting that mixture is
unnecessary and, if it makes it too complex, let's prevent it.
How about making the share name a property of a session? When a client
connects it can provide an optional client ID and it could also supply
an optional share name. For a client with a share name, any subscriptions
it makes are shared. Sharing is still a property of the subscription, but
the way in which we choose which subscriptions are shared would be different.
This would work with clean-session and so on, just making the identity
of the session a pair <client ID, share name> instead of just the
client ID.
From: Nicholas
O'Leary/UK/IBM@IBMGB
To:
mqtt@lists.oasis-open.org Date: 09/04/2016
01:02
Subject: Re:
[mqtt] Recharter: Shared subscriptions and $share
Sent by: <
mqtt@lists.oasis-open.org>
Okay, let's say it is a property of the subscription and can be used with
any topic. It would be helpful to me if we could flesh out the expected/desired
behaviour in the following scenarios.
Both Andrews have implied the existence of a share name that goes alongside
the topic. I presume that is so you can have more than one group of clients
sharing the messages amongst themselves.
Eg client A, B and C subscribe to foo with a share name of alpha. Clients
D, E and F subscribe to foo with a share name of beta. So there are effectively
two groups of clients for which the messages are shared between the members
of each group.
1. Is the share name scoped to the topic? If client G then subscribes to
topic NotFoo with a share name of alpha, is that considered a different
share?
2. If the share name is scoped to the topic, I presume more accurately
it is scoped to the topic filter used in the subscription request? That
is to say a sub to foo/# with share name gamma would not be related to
a sub to foo/+ with share name gamma even when messages are published to
foo/bar (which matches both filters)
3. I realise after I typed this one out its what Ken addresses below -
what to do when a client has overlapping subs that are a mix of shared
and non shared. I'm inclined to agree the cleanest answer would be to prohibit
overlapping subs of mixed type.
I don't agree with the argument that using topic prefix is akin to adding
point to point to mqtt and that using a share name on the subscription
isn't. The whole shared sub concept is, as far as the participating
clients are concerned, queuing. But that's just semantics and not worth
arguing over. (Because I expect one of you to say that in the share name
case, you can still have normal subscribers on the topic receiving everything...
To which I say, great, we now have a hybrid of queuing and pub/sub....)
Andrew S - I take your point that using a topic prefix on its own loses
flexibility - that you cannot have a mix of subscribers. But you could
combine it with the share name concept to allow different groups of subscribers.
The key difference being if a client didn't provide a share name it would
be put in a default share group - allowing 3.1.1 clients to partake without
needing to know about share names. Just a thought.
I think my preference for a topic prefix approach came from the sense it
had little, if any, impact on client implementation and on-the-wire formats.
We could get shared sub behaviour quite easily. As has been identified,
that approach has its limitations - the question is whether those limitations
make it not fit for purpose.
The share name approach is more flexible at the cost of implementation
complexity and client churn.
I think I find myself back on the fence at this point.
Nick O'Leary
IBM Emerging Technology Services
Twitter: @knolleary
IBM United Kingdom Ltd registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants, PO6 3AU
Ken Borgendale --- Re: [mqtt] Recharter: Shared subscriptions and $share
---
From:
"Ken
Borgendale" <
kwb@us.ibm.com>
To:
mqtt@lists.oasis-open.org Date:
Fri,
8 Apr 2016 22:45
Subject:
Re:
[mqtt] Recharter: Shared subscriptions and $share
As a charter issue, whether or not $share is an attribute
of a subscription or of a topic, we would be defining semantics of the
topic space which is what the existing and proposed charter do not allow.
I agree with Andrew that making shared an attribute of the topic makes
it a queue rather than a pub/sub topic. While queues do have some useful
semantics, this would be a radical departure for MQTT and we would certainly
need to make this an in-scope item if that is what we want to do.
It is perfectly reasonable for there to be multiple subscriptions to a
topic, some shared and some not. There is not a significant semantic issue
with this. The significant semantic issue comes only with overlapping shared
and non-shared subscriptions by the same client. This is messy today (see
MQTT-217) and shared subs is only likely to make it messier. We could solve
this for shared subs by saying that a new shared subscription MUST be rejected
if it overlaps an existing non-shared subscription and vice versa. We could
make all overlapping subscriptions invalid, but that again would strain
the compatibility clauses of the charter :
Changes to the input document should be compatible with implementations
of previous versions of the standard such that it is possible for a client
or server to implement multiple versions of the standard, allowing a client
coded to an older version of the protocol to connect to and use a server
that implements both the previous and current versions of the standard.
Specifying that either single or all subscriptions in a client must be
honored in the case of overlapping subscriptions would most certainly break
this.
Ken Borgendale --
kwb@us.ibm.com 1-207-805-6708 1-207-371-8082
Senior Programmer -- IBM MessageSight and Watson Internet of Things Connect
- Architect
Andrew
Schofield ---04/08/2016 04:30:19 PM---Hi Nick, If I've understood your
comment correctly, you are suggesting that a topic
From: Andrew Schofield <
andrew_schofield@uk.ibm.com>
To:
mqtt@lists.oasis-open.org Date: 04/08/2016 04:30 PM
Subject: Re: [mqtt] Recharter: Shared subscriptions
and $share
Sent by: <
mqtt@lists.oasis-open.org>
Hi Nick,
If I've understood your comment correctly, you are suggesting that a topic
starting with the prefix "$shared" would actually treat all subscriptions
as shared subscriptions. Unfortunately, I think that's a really bad idea.
It's gone from N publishers->M subscribers to N publishers->1 subscription->M
consumers. That's not pub/sub - it's just a queue with multiple consumers.
In a publish/subscribe system, ideally I want the publishers to be totally
unaware of the number and availability of the subscribers. By having this
rule for "$shared", if I want to use workload balancing among
a set of consumers, I have to publish on a topic starting "$shared"
and each message published can be consumed by at most one consumer. I can
no longer dream up another purpose for the messages and add another subscription
- it would just end up getting a small share of the existing shared subscription.
What I would seek to enable is exactly what you didn't like: I want subscribers
that can take all of the messages to be able to have them all, and subscribers
who cannot keep up with the message rate to be able to share the stream
of messages with like-minded consumers.
So, I think sharing is a property of the subscription, not the topic. I
prefer having a separate share name which is not part of the topic name.
Thanks,
Andrew
Andrew Schofield
Chief Architect, Hybrid Cloud Messaging
Senior Technical Staff Member
IBM Cloud Application Services
IBM United Kingdom Limited
Mail Point 211
Hursley Park
Winchester
Hampshire
SO21 2JN
Phone ext. 37248357 (External:+44-1962-818357), DE2J22
Internet mail:
andrew_schofield@uk.ibm.com From: Nicholas O'Leary/UK/IBM@IBMGB
To:
mqtt@lists.oasis-open.org Date: 08/04/2016 19:46
Subject: [mqtt] Recharter: Shared subscriptions and
$share
Sent by: <
mqtt@lists.oasis-open.org>
There are two fundamental approaches here.
1. The client chooses whether a subscription is shared or not
2. The ‘shared’ nature is a property of the topic itself
If the client gets to choose, then you need to define what happens if clients
A and B subscribe to topic foo with the shared flag set, but client C subscribes
without the shared flag set. I think trying to support that type of combination
would overly complicate implementations. We could say that in that scenario
client C’s sub is rejected, but it doesn’t feel like the right approach
to me.
I think it is much cleaner for the shared nature to be a pre-existing property
of the topic - in that way no changes are needed in client implementations
at all, as demonstrated by the fact some broker implementations already
support shared subs within the confines of 3.1.1.
The question is then whether it should be done by virtue of a well-known
topic prefix - $shared, or have it as a property on the topic that can
be administratively set on the broker.
The advantage of the latter is that any topic could be set as a shared
topic - and would be within the constraints of our charter to not define
topic spaces. The downside is it requires administrative action to use
the feature - something that isn’t required anywhere else in the protocol.
To deploy a new application that requires a shared sub would now require
co-ordinated administrative action on the broker - that doesn’t feel lightweight
to me and would fail the Just Works test that so much of MQTT benefits
from.
That brings me to the conclusion that the cleanest solution is to define
any topic that is prefixed with $shared as being a shared topic. We shouldn’t
let the current text of the charter, which we’re currently redrafting
anyway, stop us making the right technical choices for the protocol.
Nick O'Leary
IBM Emerging Technology Services
Twitter: @knolleary
IBM United Kingdom Ltd registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants, PO6 3AU
Brian Raymor --- [mqtt] Recharter: Shared subscriptions and $share ---
From:
"Brian
Raymor" <
Brian.Raymor@microsoft.com>
To:
mqtt@lists.oasis-open.org Date:
Mon,
4 Apr 2016 16:25
Subject:
[mqtt]
Recharter: Shared subscriptions and $share
Forwarding to the mailing list for broader awareness and discussion.
The conversation is related to this out-of-scope item in the draft charter:
· The TC will not identify MQTT topics nor prescribe
any mechanism or convention for classification of MQTT topics or
topic spaces.
…Brian
From: Andrew Banks [ mailto:
andrew_banks@uk.ibm.com ]
Sent: Friday, April 1, 2016 12:35 PM
To: Ken Borgendale <
kwb@us.ibm.com>
Cc: Brian Raymor <
Brian.Raymor@microsoft.com>
Subject: Re: MQTT shared subs and charter
Ken, using the $share topic prefix seems the more natural way to go to
me.
To avoid using the $ prefix we would still have to encode the sharename
into the subscribe packet, probably using the same metadata encoding method
as for the Publish Metadata. The existence of the sharename should signal
the shared subscription and so remove the need to use one of the Qos bits.
However, the Subscribe packet can carry multiple subscriptions and SubAck
can carry multiple outcomes so that is where it starts to look unnatural.
The $ prefix was reserved in the V3.1.1 specification for uses such as
this, and it is being used in practice, so I think we should alter the
charter to make its use allowed.
Andrew Banks
Telephone (44) 1962 816123
From: Ken Borgendale/Austin/IBM
To: Andrew Banks/UK/IBM@IBMGB
Date: 31/03/2016 22:18
Subject: MQTT shared subs and charter
The issue I have with charter and shared subs in the prohibition in the
charter of defining topics. I would prefer to keep this charter limitation
but it means that using $shared or any other $ topic to indicate shared
subs would not be allowed. I think it makes much more sense to use one
of the bits in the QoS byte of SUBSCRIBE for this purpose. This is especially
true if we also implement nolocal in this way, as there needs to be an
explanation of the intersection of shared subs and nolocal. The problem
is that there is no corresponding bit on UNSUBSCRIBE but perhaps we should
have an options byte per UNSUBSCRIBE and on UNSUBACK. This makes them symetical
with SUBSCRIBE and allows for a return status on individual UNSUBSCRIBES.
On the issue of shared subs, how you specify shared subs is the simple
issue. Some of the semantics of shared subs are very messy especially any
text around overlapping subs and shared subs.
Ken Borgendale --
kwb@us.ibm.com 1-207-805-6708 1-207-371-8082
Senior Programmer -- IBM MessageSight - Architect
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU