To OASIS Members:
A draft TC charter has been submitted to establish the Testing and Monitoring
Internet Exchanges (TaMIE) Technical Committee. In accordance with the OASIS TC
Process Policy section 2.2:
(http://www.oasis-open.org/committees/process.php#2.2) the proposed charter is
hereby submitted for comment. The comment period shall remain open until 11:45
pm ET on 30 November 2007.
OASIS maintains a mailing list for the purpose of submitting comments on
proposed charters. Any OASIS member may post to this list by sending email to:
mailto:oasis-charter-discuss@lists.oasis-open.org. All messages will be publicly
archived at:
http://lists.oasis-open.org/archives/oasis-charter-discuss/. Members who wish to
receive emails must join the group by selecting "join group" on the group home
page:
http://www.oasis-open.org/apps/org/workgroup/oasis-charter-discuss/. Employees
of organizational members do not require primary representative approval to
subscribe to the oasis-charter-discuss e-mail.
A telephone conference will be held among the Convener, the OASIS TC
Administrator, and those proposers who wish to attend within four days of the
close of the comment period. The announcement and call-in information will be
noted on the OASIS Charter Discuss Group Calendar.
We encourage member comment and ask that you note the name of the proposed TC
(TaMIE) in the subject line of your email message.
Regards,
Mary
---------------------------------------------------
Mary P McRae
Manager of TC Administration, OASIS
email: mary.mcrae@oasis-open.org
web: www.oasis-open.org
phone: 603.232.9090
===========
PROPOSED CHARTER FOR REVIEW AND COMMENT
Charter for the "Testing and Monitoring Internet Exchanges" Technical Committee
1a. Name and abbreviation
Testing and Monitoring Internet Exchanges (TaMIE) TC
1b. Purpose
Electronic collaborations over Internet between business partners (e-Business /
e-Government) appear to be converging toward well-established types of message
exchange patterns that involve both user-defined standards and infrastructure
standards. At the same time, the notion of event is increasingly promoted for
asynchronous communication and coordination in Event-Driven Architectures (EDA)
that are considered as either complementary to or part of SOA systems. In both
cases collaboration between partners or between components is achieved by the
means of choreographed exchanges of discrete units of data - messages or events
- over an Internet-based protocol. The TC will define an event-centric test case
scripting mark-up and execution model for such systems.
In e-Business transactions as in EDAs, partners or components must agree on the
use of a combination of standards in order to interoperate with each other.
Typically, these standards can be classified into three layers.
* Messaging infrastructure standards, ranging from transport level to
higher-level messaging protocols and quality of service (QoS) including
reliability and security, such as those defined as SOAP extensions, or REST
(Representational State Transfer).
* Multi-message exchange standards as manifested in business processes and
choreographies. Included standards may conform to UMM business transaction
patterns, ebXML Business Process Specification Schema (BPSS or ebBP),
WS-Choreography or WS-BPEL.
* Business document standards. These may be business content structure and
semantics, taxonomies in use, code lists, semantic rules, or the XML schema
modeling style. They are industry-specific (e.g. RosettaNet PIP schemas, AIAG
Inventory Visibility and Interoperability schemas,), horizontal document
standards (e.g. based on UN/CEFACT Core Components, OAGIS Business Object
Documents schemas), or regional guidelines (e.g., KIEC's e-document modeling
guideline).
There have been conformance and interoperability test suites (e.g, ebXML
messaging V2 test suites, WS-I test assertions) and testing tools (e.g., ebXML
IIC, NIST QOD, NIST Business Document Content testbed, KorBIT ebMS testbed, WS-I
test tools) for each above layer individually. But the testing of integrations
of standards has been ad-hoc (e.g. RosettaNet Self-Test Kit is tied to
RosettaNet protocol), or limited mostly to standards in the messaging
infrastructure (WS-I).
Although the need for testing some form of integration of standards has been
well recognized for infrastructure standards (e.g. WS-I profiles), there has
been little support for testing integrations that extend to the use of standards
specific to a business - e.g. for documents or choreographies. Such integrations
can be construed as user-defined profiles. For example, the level of QoS
required for a business transaction may depend on the nature of business data
being exchanged, or on some property defined by the related business process.
Testing and monitoring these infrastructure layers and their integration also
requires that test cases access a combination of contracts - agreements or
policies - represented by meta-level documents (WS-Policy, WSDL, ebXML CPPA,
ebBP definitions, XML schemas).
This compliance objective goes beyond quality assurance for the messaging
function: it requires the monitoring of live transactions in production
environments, as well as verifying conformance of business endpoints in
operation conditions. This calls for a flexible test execution model that can
accommodate performance constraints as well as different timeliness constraints
- e.g. where tests are either deferred over log data, or executed on live
exchanges in a monitoring mode.
The output of a monitoring script also must provide more information than a
report of the type pass / fail. Different ways of "passing" or "failing" must be
reported on, as well as identifying the types of business transactions. This
reporting must be easy to consolidate, for Service Level Agreements (SLA) and
Business Activity Monitoring (BAM) purposes.
The TC will define a testing and monitoring model, as well as a test script
mark-up, so that test cases or monitoring cases can be fully automated, and
portable across test environments.
1c. Scope
The scope of activity for this TC must be within the following topics:
- Use cases and Requirements: Investigation of use cases that may combine
standards - or specifications submitted to an SDO - in all areas concerned by
message exchanges: protocols, QoS, choreography, documents and business content.
Selection and prioritization of related requirements.
- Test execution model: A model for executing test cases or monitoring cases
addressing the previously described requirements will be defined. In particular,
an event-centric approach such as the one described in the event-driven Test
Scripting Language (see Input material) will be considered. This execution model
also includes a representation and access model for log data.
- Test case scripting: An XML mark-up that complies with the execution model
will be designed. This may reuse or profile subsets of existing dialects that
are already in use for domains that have similar functional requirements (if
their IP status is compatible with the TC IPR mode), in which case the mark-up
will act as a coordination or integration language, treating these dialects
either as language extensions or as external resources used via adapters. In
particular, openness to the reuse of XML processing dialects and tools based on
recognized standards - such as XPath, XSLT, XQuery - is a requirement. The
design approach will favor (a) simplicity of use - a small number of constructs
that are easy to learn and to implement as opposed to feature-rich dialects; and
(b) modular units of scripts that allows for functional reuse and extensibility.
Test scripts will process run time materials (i.e., messages, events, signals),
as well as configurative artifacts such as policies and agreements, schemas and
interface definitions. Test cases or monitoring cases will also provide for more
detailed outputs than pass/fail, and will strive to support SLA monitoring, BAM
(Business Activity Monitoring) and Event-Driven Architectures (EDA). However,
the objective will not be to specify a BAM system or an SLA enforcement system,
but only the monitoring technology that can support these.
- Evaluation and Investigation: of third-party mark-ups and dialects, log
formats and existing test practices, for possible reuse or interfacing in the
test case scripting language. However no third-party specification or tool can
be made essential or necessary to an implementation of the specification to be
delivered, unless its licensing is free and unrestricted.
- Methodology and Guidelines: In scope of this activity, are all forms of
support for users, such as example scripts, best practices, primers and
guideline documents, concerning all topics listed above.
The following documents are input material to this TC, which will deserve prime
attention from the TC assuming their IP status is compatible with the IPR mode
of the TC:
- The event-driven Test Scripting Language (eTSL) V0.85,
- http://www.oasis-open.org/committees/document.php?document_id=26036
- Conformance testing and Certification Framework (OASIS, Conformance TC, June
2001)
- QA Framework: Specification Guidelines (W3C, November 2004)
- Test Assertion Documents for WS-I profiles (WS-I, 2003-2005).
- OASIS IIC Test Framework 1.1
Other documents may be considered by the TC, subject to a TC decision.
Explicitly Out-of-scope:
- Defining or specifying detailed test case metadata or artifacts that would
complement the above scripting - e.g. such as supported in ATML, for
representing test environments, complete test suites and/or their result.
- Definition of particular test harnesses.
- Use cases and requirements that refer to infrastructure specifications that
are not submitted to an SDO, or that refer to business documents or content that
are not approved by an organization or consortium representative of this
business domain.
1d. Deliverables
The TC will produce the following deliverables:
(a) A requirements document, which may include use cases for Internet exchanges,
test assertions for related standards, references to existing test case dialects
or existing logging formats or systems.
(b) A specification defining a test case execution model and scripting, that
supports both the testing and monitoring of message and business data exchanges.
(c) A set of examples and use cases illustrating the use of above specification
for testing or monitoring business exchanges based on (a) some business content
standards, (b) some choreography standards, and (c) some specifications in the
domain of SOAP messaging or others.
(d) An implementation of the above specification used for proof of the proposed
concept and principle.
Timeline:
The TC will aim at a Public Review draft of the Test Case Execution Model and
Scripting by the end of 2008.
1e. IPR Mode
Royalty-Free on Limited Terms.
1f. Audience
The TC is welcoming any OASIS member with an interest and/or experience in
testing and monitoring.
1g. Language
English
Non-Normative Information
2a. Identification of similar or applicable work that is being done in other
OASIS TCs or by other organizations, why there is a need for another effort in
this area and how this proposed TC will be different, and what level of liaison
will be pursued with these other organizations
In general the following works fall short of addressing TaMIE charter, in that
they:
* either target a specific protocol stack,
* or a particular domain of business.
Their support for testing and monitoring the implementation of a combination of
standards is either inexistent or limited, and they do not provide a versatile,
extensible scripting of test cases. The requirements that are specific to B2B
environments are not well addressed either: none is addressing both run-time
monitoring of systems in production, and conformance testing prior to
deployment. Only the latter is generally supported by existing work, e.g.
neglecting real-time monitoring cases that may lead to dynamic notification or
remedial actions, or ignoring real-time constraints about events throughput and
recovery after shutdown time. These test environments below - at the exception
of WS-I tools - do not support the combined processing of electronic documents
representing agreements or policies (metadata) and of run-time logs. Finally,
none supports a general-enough notion of event, along with adequate time
primitives and event correlation and search capability.
(a). The OASIS ebXML IIC test framework (TF v1.1)
http://www.oasis-open.org/apps/org/workgroup/ebxml-iic/document.php?document_id=
10896
This framework was designed specifically to test the conformance and
interoperability of the software implementing the ebXML Messaging Specification.
The framework defines components and harnesses (configurations) necessary to
simulate and control the environment for conformance and interoperability tests.
A shortcoming of the IIC TF is that it has been designed for ebXML messaging
2.0, and provides weak extensibility options, both for external events and for
specialized evaluations. The IIC TF does not support other suites of B2B
integration standards such as other SOAP profiles integrating several Web
Services standards. It also does not support ebMS 3.0 and electronic documents
testing.
(b). RosettaNet Self-Test Kit (RNSTK)
RNSTK
(http://serg.telecom.lth.se/education/master_theses/docs/72_Kjellin_slutrapport.
pdf )
is a test system provided by the RosettaNet consortium. The system allows
software solutions to perform self-tests for compliance to business
collaboration scenarios and the RosettaNet Integration Framework (RNIF)
specification. The RNSTK tests an integrated system implementing particular B2B
collaboration scenarios, yet is tightly dependent on RNIF messaging. RNSTK test
suites are encoded using the RNIF specification itself. Due to these reasons,
the RNSTK cannot be used to test other suites of B2B standards including the
Document Type Definition semantics [20]. Another weakness is that its
architecture only supports conformance test configurations.
(c). Web Services Interoperability (WS-I) Test Tools (for Basic Profile 1.1)
These tools do not allow for the scripting of a test suite; instead they apply a
battery of predefined tests to any Web service material provided as input. The
objective is to verify the conformance of this material to one of the WS-I
profiles. The tools only passively analyze message traffic and have no ability
to control the systems under test. Nevertheless, this capture and analysis
architecture is un-intrusive, simple, and supports deferred testing. The
analyzer tool of WS-I cannot be used as a general test engine, being tied to
profile definitions; but, it can be leveraged as a specialized module by a more
general monitoring capability, when the conformance of message material to WS-I
profiles is required.
(d). TTCN-3 (Testing and Test Control Notation Version 3, ETSI, June 2001)
TTCN-3 (http://www.ttcn-3.org/StandardSuite.htm) is a powerful, programmatically
complete procedural language with specialized constructs for defining test
procedures, test verdicts, matching mechanisms for evaluation, timer handling
and distributed test components. Due to its original focus on telecommunication
systems, it also has the ability to specify encoding information, to communicate
both synchronously and asynchronously, and to perform passive monitoring. These
capabilities are all essential for testing in message-based B2B integration.
Relative to previous test frameworks, TTCN-3 is more powerful and flexible.
However, TTCN-3 notation can be difficult for a business or domain expert to
use. TTCN-3 has not been designed to delegate some functions to external modules
and it is weak when it comes to validating business transaction events and XML
electronic documents, which are essential for B2B integration testing.
(e). ATML (Automatic Test Mark-up Language, IEEE, December 2004)
ATML
(http://grouper.ieee.org/groups/scc20/tii/ATML/Working%20Groups/Management/ATML%
20Overview.doc) has been designed for exchange of diagnostic information,
configuration data, and test instrument definitions between conforming software
applications. It was not designed to support test scripting and execution.
However, ATML representations could be complementary to executable test case
scripts.
(f). General Software Test Environments and Scripting JXUnit.
JXUnit is an XML-based test-scripting system built on top of the JUnit Java
classes. It is a data-driven testing system in which input to the system under
test and expected output are specified and then the actual output is evaluated.
There is no built-in support for B2B testing, in particular the communication
and the event-driven tests. However, as a general, test-scripting platform that
relies on a common programming language, JXUnit and JUnit could be used as an
implementation platform for essentially any test framework including the one
proposed here.
2b. First Meeting:
Tentative date is January 30, 2008. The first meeting will be a 2h conference
call. Fujitsu or KIEC will sponsor the call.
2c. Projected Meeting Schedule:
A 90mn meeting every two weeks will be schedule, unless the TC decides of a
different frequency. Assuming the first meeting is a conference call, a
face-to-face meeting will be set within 2 months of the first meeting once the
work has actually started, to maximize the value of face time.
2d. Co-proposers
AIAG (Automotive Industry Action Group): Mohammad Abidi,
Mabidi @ aiag.org
Axway: Dale Moberg,
dmoberg @ axway.com
Fujitsu: Jacques Durand,
jdurand @ us.fujitsu.com
KIEC (Korea Institute for Electronic Commerce): Hyunbo Cho,
hcho @ postech.ac.kr
NIST (National Institute of Standards and Technology): Nenad Ivezic,
nivezic @ nist.gov
OAGI (Open Applications Group, Inc.): Dave Connelly,
Dconnelly @ openapplications.org
2e. Convener
Jacques Durand, Fujitsu
2f. Member Section
N/A
2g. List of Contributions
The following work, already submitted in October 2006 to OASIS ebXML
Implementation, Interoperability and Conformance TC , is also contributed to
this TC:
- eTSL draft V0.85, (originally derived from the TestFramework 1.1 produced
OASIS IIC, 2004-2005)
2h. Draft of F.A.Q.
Will be provided later.
2i. Proposed Working Title for deliverables
Deliverable #1: eTSM - event-driven Test Scripting and Model.
A specification defining a test case execution model that supports both the
testing and monitoring of message and business data exchanges, and the scripting
language for defining and executing these test cases.
Deliverable #2: eTSM Use Cases Document - A set of examples and use cases
illustrating the use of above specification for testing or monitoring business
exchanges based on (a) some business content standards, (b) some choreography
standards, and (c) some specifications in the domain of SOAP messaging or
others. This document may serve to illustrate requirements that have been
addressed.
Deliverable #3: An implementation of the above specification used for proof of
the proposed concept and principle.