OASIS ebXML Messaging Services TC

  • 1.  Groups - Initial thought draft Large Paylod handling (Largepayloadsupport.htm) uploaded

    Posted 10-31-2007 20:59
    The document named Initial thought draft Large Paylod handling
    (Largepayloadsupport.htm) has been submitted by Mr Ian Jones to the OASIS
    ebXML Messaging Services TC document repository.
    
    Document Description:
    This is an intial thought draft of the large payload handling module for
    part 2 based on the work at the September 2007 Face to Face
    
    View Document Details:
    http://www.oasis-open.org/apps/org/workgroup/ebxml-msg/document.php?document_id=25928
    
    Download Document:  
    http://www.oasis-open.org/apps/org/workgroup/ebxml-msg/download.php/25928/Largepayloadsupport.htm
    
    
    PLEASE NOTE:  If the above links do not work for you, your email application
    may be breaking the link into two pieces.  You may be able to copy and paste
    the entire link address into the address field of your web browser.
    
    -OASIS Open Administration
    


  • 2.  RE: [ebxml-msg] Groups - Initial thought draft Large Paylod handling (Largepayloadsupport.htm) uploaded

    Posted 03-26-2008 21:26
    Hello Ian,
    
    Amid all the discussions on multihop, and with a F2F meeting coming up, it
    may be good to take up this item again too as this is an important issue in
    many user deployments.  Here are some comments/ideas/questions, mainly on
    the fragmented payload concept.  
    
    1) One interpretation of fragmenting at MSH level is that in some "message
    service interface", the Submit() and Deliver() operations would refer to one
    "logical" message.  After Submit(), a suitably configured MSH would split
    the message in fragments, and send them as separate lower-level messages.
    Similarly, prior to invoking Deliver(), it would reassemble the message
    fragments into a single "logical" unit.  An MSH that does not support this
    natively but still conforms to v3.0  Core would not support the
    splitting/merging in the MSH, but could allow applications to set the
    relevant metadata programmatically? 
     
    2) Would we expect each fragment to contain a copy of the (or a) full ebMS
    header?  If yes what would be the ebXML MessageId of a fragment message?  Do
    we assume multiple parts messages with the same ebMS message Id, or
    different ones?  If different, there has to be some reference to the larger
    structure they should be reassembled into.  Fragments would have different
    message Ids at WSRM / WSR level and (if used) at WS-A level. To support
    routing, each fragment message should contain all relevant ebMS metadata,
    like From, To, Service, Action, ConversationId, too?  
    
    3) Similarly, in the response to a message sent as fragments, the
    RefToMessageId would refer to the whole message, never to a single part. If
    that response message is a large message itself, multiple fragment messages
    would all refer to the same RefToMessageId.  This is to express that the
    ebMS concept of a MEPs is more a logical concept from a business process
    point of view concept?
    
    4) Most natural use cases for large messages sent in fragments assume
    reliable messaging, and reliable messaging for all fragments. So either the
    entire logical message (all message fragments) is delivered or it is not
    delivered at all.  If the message is split in fragments, any resending of
    last parts would not trigger a retransmission of the full "logical" ebMS
    message but just of the missing lower-level message containing the message
    fragment.   So a requirement could be that all parts are sent over the same
    WSRM SequenceId, with the ordering of parts (1 < i < n) reflected in the
    message number.  The fragments could be sent over some existing sequences,
    so fragments from different logical ebMS messages could be sent on single
    WSRM sequence, as long as the partial order is preserved.
    
    5) We need to think about the mapping of message fragments and the logical
    structure of message parts. An ebMS message transporting a single 2GB MRI
    image could still have just one eb:PayloadInfo element.  An ebMS message
    transporting 35 small JPEG images as 35 eb:PayloadInfo elements may not need
    to split them in as many separate messages. 
    
    6) A common messaging pattern is Claim Check pattern, see
    http://www.enterpriseintegrationpatterns.com/StoreInLibrary.html. 
    I remember discussions in the past about a need to "push" part of the
    information and allow other information to be "pulled" on demand.  Perhaps
    different fragments could have their own P-Modes. A common use case is that
    structured metadata (typically a small XML document) is pushed immediately
    to the destination, but that large supporting data (XML or binary) could be
    pulled with lower priority. Gateway intermediaries could add value here too.
    
    Pim
    
    
    
    


  • 3.  RE: [ebxml-msg] Groups - Initial thought draft Large Paylod handling (Largepayloadsupport.htm) uploaded

    Posted 01-19-2010 20:49
     
    
    A separate question to discuss:
    
    The AS4 draft defines a compressed payload option.  This
    does not provide all the functionality of the "payload
    fragmentation" protocol described in Ian's message
    (referenced below).  Compressing individual payloads does
    not help with messages that have large payloads that are
    already compressed (e.g. PDF documents or ZIP archives), are
    very large even after compression (e.g. medical images, data
    cubes, a day's worth of billing data from a telco etc.), and
    messages with many payloads (none of which are too large,
    but the result is a very large message).  
    
    In a multi-hop environment, splitting large messages and
    sending them separately is also preferable to storing and
    forwarding large messages as atomic units.  If they are
    split, an intermediary can forward each part immediately
    after it is received, without having to wait for the other
    units.  With very large messages, storing and forwarding can
    introduce an unacceptable delay.  With end-to-end
    reliability, only message fragments that fail need to be
    resent.
    
    Many enterprise messaging products have limits on message
    size and automatically split and send individually messages
    that exceed message sizes.  Since we already have bundling
    in part 2, I think we should consider a fragmenting protocol
    too.
    
    Messages bundles may be large. Fragmenting and bundling
    could be composable.  Bundle two user messages, one with a
    99 MB payload, one with a 1 M payload. Then fragment into 20
    5 MB parts for efficient transmission through an icloud.
    
    Pim
    
    
    


  • 4.  RE: [ebxml-msg] Groups - Initial thought draft Large Paylod handling (Largepayloadsupport.htm) uploaded

    Posted 04-18-2010 11:24
     
    
    Hello Ian,
    
    I've borrowed some of the ideas in your initial proposal in
    the draft I just uploaded.
    Looking forward to your comments.
    
    Pim