Comments inline. Pim From: Moberg Dale [mailto:
dmoberg@axway.com] Sent: 31 January 2012 16:51 To: Pim van der Eijk;
ebxml-msg@lists.oasis-open.org Subject: RE: [ebxml-msg] Semantics of AS4 Receipt Hi Pim I have been responding to what I thought had been some questions about the semantics of NRR or of simple receipt. That does not involve product feature details. Also there are differences in what can be done when an ack of any sort is returned on a separate connection than on the same connection. If you need to say something about delivery, then the original sender’s message has been delivered when the recipient has received it. If you want to say something about what subsequent operations of the delivered message have completed, then that is more a question of disposition, which for AS4 purposes need to be semantically aligned with the concept of “processed” The ebMS 3.0 Core specification makes a difference between RM-Deliver and MSH-Deliver and between receiving a message and delivering it. I'm saying that I think AS4 Receipts should be MSH-Deliver rather than RM-Deliver, and that MSH-Deliver is closer to (without perhaps going as far as) EDIINT concepts. Please have a look at the definitions of RM-Deliver and MSH-Deliver and explain if you think an AS4 receipt should signal RM-Deliver or MSH-Deliver. In essence, MSH-Deliver seems to require that only the first of any subsequent steps (the step of getting the message and its metadata out of the MSH) succeeds. It does not say anything about further processing steps. But unlike RM-Deliver it seems to rule out a situation where an MSH sends an acknowledgment, but then fails in some other SOAP module above the reliability layer and/or subsequently fails to transfer responsibility to the first component outside the MSH. As I mentioned, it is difficult to sum up what to say about that next step after processed. There may not actually be a next step (because the data had been placed on a file system during processing, and the next step would have been to place the data on the filesystem. Since that has been done, it is not necessary to do it again.) Exactly, writing the message to a file system is a way of transferring a message out of the MSH, if that's the selected configuration. Whether any other application or user actually does anything with that message is not its responsibility and perhaps even invisible to the MSH. But if the disk was full, or the partition not mounted, or some other error happened, MSH-Delivery has failed though RM-Deliver has succeeded. The MSH can detect this failure. Or queues. Or another transfer ove r the same or different protocols, and so on. For a messaging system, the deliver() operation sh ould probably be defined as submit() and not as send(). But w e dont't need to, and shouldn't define what delivery means for all sorts of integration backends. AS4 does not specify what integration backends to use. Now just how many of these kinds of next steps’ statuses do you want to check? I'd say we're only interested in the first step from inside to outside the MSH. Your little pseudo code snippet abstracts over a lot of very different cases. Many involve delays not accompanied by a timeout value. Those would pose problems. We can impose or recommend that the deliver() operation has a timeout value if you want. The most commonly used delivery mechanisms used by B2B gateway products (write to disk, JMS or other message queues, HTTP posts, FTP) support timeouts. The MSH can itself wrap an arbitrary delivery operation in a timeout and decide how a timeout is to be interpreted. But that would be out of scope for the AS4 protocol. Retry, resend, resubmit options are available. Resend means that while the message had been sent (as indicated by a 2xx status for HTTP if that is involved), no expected ack (NRR or simple MDN) has been received. If the ack was lost, “resending” (which viewed from the sending perspective is a re-deliver) will result in generating another ack, as I think it should. The point I was making was that i n some products, the receiving MSH (not the sending MSH as in your description) has a (re)deliver option. This does not require the message to be retried/resent/resubmitted by/to the sending MSH. It is an internal function in the MSH that allows previously received messages to be delivered again (e.g. writing once more to a queue, or to file system, or ..). An AS4 product that associates a Receipt with receive() rather than deliver() would not generate a new receipt. I mentioned one product that does generate a new Acknowledgment for redelivered ebMS 2.0 messages. Backend failure is possible. NRR does not intend to magically solve difficulties by removing the possibility of backend failures. It does demarcate a kind of “responsibility” to some degree. The distinction between RM-Deliver and MSH-Delivery is a subtle one and indeed relates to where the boundary of responsibility is placed. But neither RM-Deliver not MSH-Deliver depend on what happens within backends or enterprise business processes. Customizations or implementation choices of product are quite another matter. I had thought the question you were raising was what the standard means or specifies. You are right that we are specifying a standard protocol, not implementations. The questions are about how the Receive(), Acknowledge() and Deliver() operations that are defined in the ebMS and Web Services specifications relate to each other. From: Pim van der Eijk [mailto:
pvde@sonnenglanz.net] Sent: Tuesday, January 31, 2012 12:49 AM To: Moberg Dale;
ebxml-msg@lists.oasis-open.org Subject: RE: [ebxml-msg] Semantics of AS4 Receipt Dale, For example, the payload might be placed on a queue, but whether the payload was taken off the queue is an open question as far as the signal is concerned.) As I am writing for the third time now, I am just asking for the success of the enqueue operation to be checked. This does not require any new coordination protocols, it just imposes some ordering in the structure of your application code. Instead of something like .... acknowledge(message) .... deliver(message) You would write something like .... if deliver(message) == Success: .... .... acknowledge(message) .... else: .... .... errormessage(message,'Delivery Failure') In at least one Axway product, ebMS 2.0 acknowledgments seem to be triggered by delivery. I know this because if a system administrator manually re-delivers a message from this product (which is an option that is useful in some exceptional situations), an acknowledgment is sent to the trading partner). So assuming the AS4 implementation will follow this pattern, your company's product already does what I'm asking for. Pim From: Moberg Dale [mailto:
dmoberg@axway.com] Sent: 30 January 2012 22:49 To: Pim van der Eijk;
ebxml-msg@lists.oasis-open.org Subject: RE: [ebxml-msg] Semantics of AS4 Receipt I think that AS4 NRR is now roughly the same as AS1-3 NRR. My concern was that we not proceed beyond secured, acknowledgement of receipt (and, when security features are OK, an indication that the next processing step was engaged). I am uncertain about “successful transfer” – the MSH did its step in “processing,” but whether the next processing step was successful is not assured. ([Many requests from “end user communities” have emerged over time to add on various assurances about further processing. I don’t think we should go there with data exchange protocols. That does not prevent embedding data exchange protocols in richer coordination protocols that might add on various notification services. But I would not want to get into these additions within AS4. I think the additional protocols should be developed as applications of existing IETF protocols, maybe as an EDIINT-NG or B2B-NG interest group. I don’t think it should be an OASIS effort. Just my opinion (or using the new catch phrase), just sayin’] From: Pim van der Eijk [mailto:
pvde@sonnenglanz.net] Sent: Monday, January 30, 2012 1:36 PM To: Moberg Dale;
ebxml-msg@lists.oasis-open.org Subject: RE: [ebxml-msg] Semantics of AS4 Receipt Thanks for the response and background information. The proposal I made was quite modest: to link the sending of an AS4 receipt to the ebMS 3.0 concept of MSH-Delivery . If ebMS 3.0 and WS-Reliability optionally support this concept, it would be inconsistent for AS4 to commit to less. AS2 (
http://www.ietf.org/rfc/rfc4130.txt ), section 7.1, relates an MDN to delivery (my emphasis): The signed MDN, when received by the sender of the EDI Interchange, can be used by the sender [... as] an acknowledgement that the EDI Interchange sent was delivered and acknowledged by the receiving trading partner [..] . MSH-Delivery seems in fact to correspond exactly to what you call successful transfer to the first receiving agent that is indeed often just the first step of a pipeline . The defining requirement is that the message has been transferred successfully out of the AS4 MSH. This is the case for all the examples in my message. The increase in latency is minimal, the impact on implementations limited, and fewer false receipts are produced. The functionality better matches the expectations of end-users and is supported in commonly used B2B gateway products for other protocols. Pim From:
ebxml-msg@lists.oasis-open.org [mailto:
ebxml-msg@lists.oasis-open.org] On Behalf Of Moberg Dale Sent: 30 January 2012 17:53 To: Pim van der Eijk;
ebxml-msg@lists.oasis-open.org Subject: RE: [ebxml-msg] Semantics of AS4 Receipt The first observation is that the requirements internet-draft was never published as an informational RFC. If it had been, some of the tradeoffs involved in being an “applicability statement” and in seeking to make AS1, AS2 and AS3 as much alike as possible would have necessitated some revisions to the original draft. The original draft also included several ideas that were not implemented. Going back over all the tradeoffs would be tedious. I think the main one relevant here is that the MDN was reused as the message type for receipts. In an email context, a MDN is more associated with what the a UA does upon seeing what the final MTA has come up with. So POP3 and IMAP based UA access can also generate MDNs. EDIINT, however, only makes use of a success code from the MDNs of “Processed”. Here is where the semantics that got implemented got defined. RFC 2298 was I think in scope when the EDIINT work started, but I don’t think all the subsequent “obsoleting or updating” RFCs in this area change the situation. processed The message has been processed in some manner (i.e., by some sort of rules or server) without being displayed to the user. The user may or may not see the message later, or there may not even be a human user associated with the mailbox. The “processed” code then indicates the semantics of any kind of receipt/delivery/disposition on the receiver side. So the overly ambitious requirement of the draft were aligned with the constraints of reuse, and what already existed. The reasons for the tradeoff were many and varied. Mainly however they all reflected the unpredictable latencies between MTA/MSH/WebServer/FTPserver reading of data and the acquisition by the next processing step(s). Would it be a POP3 UA or was there an automated pipeline pushing to a UA feeding a translator,JMS, MFT, ESB, or any of the integration technologies in use? Or for HTTP, would a servlet, appserver, CGI push to the next app, and would a success from that app be relevant to count as “disposition”. The ultimate “backend application” is only in simple middleware cases just one hop from reception; the receiving agent is just the first step of a pipeline (where there can be branching or forking process flows). How is all that to be summed up in one code? In general the A2A visibility problem remains unsolved for general deployment situations by any current standard. It will take some significant effort and cost to make A2A visibility commonplace. AS4 will not be doing it IMO. The backflow of status notifications down the integration pipeline is just too varied to smooth over, without considerable thought given to lightweight notification technologies, with built in subscription, expiration, digest, and other capabilities. Hello, I was working on an AS4-related topic over the weekend and have some questions on AS4 non repudiation receipts. The AS4 profile is presented as designed to meet EDIINT requirements. According to section 3.7 of
http://tools.ietf.org/id/draft-ietf-ediint-req-09.txt EDIINT defines the requirements for non repudiation as: Receipt by the sender of a signed receipt, is an implicit acknowledgment of successful mailbox delivery. It is also an explicit acknowledgment that the Interchange was retrieved from the mailbox - pick-up notification. By having the receiver sign the receipt, it authenticates that the intended receiver picked up the EDI Interchange -- mailbox authentication -- and that the intended receiver verified the integrity of the EDI Interchange, and the identity of the sender. By returning the original message id and the one-way hash value of the received contents back in the signed receipt, the sender can reconcile the acknowledged EDI Interchange with what was sent. According to section 8 of
http://docs.oasis-open.org/ebxml-msg/ebms/v3.0/core/ebms_core-3.0-spec.odt the ebMS 3.0 Core Specification distinguishes between RM-Delivery and MSH-Delivery. Case 2 : The acknowledgment is on MSH-delivery (supported in WS-Reliability). In that case, notification option 1 can be supported as well as option 2. In order for option 1 to be supported, an RMP must implement RM-Deliver operation so that it is only considered successful (worthy of sending an acknowledgment) if the Deliver operation from MSH to Consumer also succeeds. It is RECOMMENDED that an implementation support this acknowledgment semantics. In
http://www.oasis-open.org/apps/org/workgroup/ebxml-msg/download.php/44888/AS4-profile-v1.0-wd22.odt section 3.4, the following is stated about AS4: The Receiving MSH is taking responsibility for processing this user message. However, no guarantee can be made that this user message will be ultimately delivered to its Consumer application (this responsibility lays however now on the Receiver side). Now this seems to be more like the RM-Delivery (only option supported in WS-ReliableMessaging) than MSH-Delivery ? The EDIINT definition of non repudiation of receipt seems to require MSH-Delivery rather than RM-Delivery. If we're claiming that AS4 meets EDIINT requirements, is this enough ? I'm not saying that AS4 products should support a VAN mailbox interface based on pickup of a message, and should delay sending a receipt until such a pick-up happened (possibly hours or days later). Drawbacks of this have been discussed in the past when we looked at multi-hop reliability models. But in the AS4 context, it could be defined as: the MSH successfully passed responsibility for delivering the message to some third entity (system, program, user). Examples of succesful or unsuccessful delivery could depend on implementation: - Delivery to a mailbox: Succes: message was added to mailbox. Failure: quota exceeded, mailbox full. (No guarantee that any user or system actually processes the message) - Delivery to a file system: Success: the MSH successfully created, wrote data to, and closed a file representation of the message metadata and contents. Failure: Permission denied, I/O errors (There is no guarantee that any user or program actually processes the created file) - Delivery to a message queing system Success: successful enqueue() Failure: enqueuing failed (There is no guarentee that any other application dequeue the item) - Delivery to another B2B messaging system: Success: the Submit() operation was invoked successfully Failure: Submit() operation failed (No dependency on any downstream receipts) In the case of failure, which the AS4 MSH can detect immediately after receiving a message, I would prefer the AS4 to not send a Receipt but to send an Error instead. An alternative idea is discussed in section 8.2.4 of the ebMS 3.0 Core Specification, which has a discussion on false delivery failures: False DF - which can never be completely eliminated - can always be detected outside the reliable messaging (RM) layer, in a tractable and less urgent way - e.g. the sending party may synchronize on a daily basis by communicating its list of assumed delivery failures, for confirmation by receiver. The Status Request feature (which may be described in a forthcoming Part 2 of the ebMS specification) could facilitate this function. Pim