To OASIS Members:
A draft TC charter has been submitted to establish the OASIS Test Assertions Guidelines (TAG) TC Technical Committee.
In accordance with the OASIS TC Process Policy section 2.2 (http://www.oasis-open.org/committees/process.php#2.2) the
proposed charter is hereby submitted for comment. The comment period shall remain open until 11:45pm ET on 28 March
2007.
OASIS maintains a mailing list for the purpose of submitting comments on proposed charters. Any OASIS member may post
to this list by sending email to:
mailto:oasis-charter-discuss@lists.oasis-open.org. All messages will be publicly archived at:
http://lists.oasis-open.org/archives/oasis-charter-discuss/. Members
who wish to receive emails must join the group by selecting "join group" on the
group home page:
http://www.oasis-open.org/apps/org/workgroup/oasis-charter-discuss/. Employees of organizational members do not require
primary representative approval to subscribe to the oasis-charter-discuss e-mail.
A telephone conference will be held among the Convener, the OASIS TC Administrator, and those proposers who wish to
attend within four days of the close of the comment period. The announcement and call-in information will be noted on
the OASIS Charter Discuss Group Calendar.
We encourage member comment and ask that you note the name of the proposed TC (TAG) in the subject line of your email
message.
Regards,
Mary
---------------------------------------------------
Mary P McRae
Manager of TC Administration, OASIS
email: mary.mcrae@oasis-open.org
web: www.oasis-open.org
phone: 603.232.9090
OASIS Symposium:
"eBusiness and Open Standards:
Understanding the Facts, Fiction, and Future"
15-18 April 2007 San Diego, CA USA
http://www.oasis-open.org/events/symposium/
===========
PROPOSED CHARTER FOR REVIEW AND COMMENT
OASIS TEST ASSERTIONS GUIDELINES TECHNICAL COMMITTEE
Name and abbreviation:
OASIS Test Assertions Guidelines (TAG) TC
Purpose:
The design of Test Assertions (TAs) associated with a specification or standard - referred to in this charter as target
specification - has the following recognized benefits: (i) it improves the quality of this specification during its
design, and (ii) it reduces the lead time necessary to create a test suite for this specification.
A test assertion (TA), also sometimes defined as test specification, is understood in this charter with the following
general meaning:
A TA is an independent, complete, testable statement for requirements in the specification.
A TA always refers to an item under test (IUT), which is assumed to implement all or parts of the target specification,
so that this IUT is concerned with the requirements addressed by the TA. This reference is either implicit or explicit
if it is necessary that the TA identifies the item under test in some unambiguous manner. A TA describes the expected
output or behavior for the item under test within specific operation conditions, in a way that can be measured or
tested. A TA may refer to a test harness architecture, of which a description limited to the interactions between its
components and the IUT may be sufficient.
Test assertions are generally different from test cases, which are more detailed and contingent to a concrete test
framework: TAs are the basis to write test cases, and relate the latter to the narrative of the target specification.
The general objective served by this TC is to facilitate the creation and usage of test assertions by any group involved
in designing a specification or standard of which software implementations are expected to be developed, with a primary
focus on OASIS technical committees.
The first step in achieving this is to establish a common and reusable model, metadata, methodology and representation
for TAs.
This is aligned with the intent of the former OASIS Conformance TC, although the focus in the current initiative is not
on the various aspects of quality and conformance, but on a specific one, namely test assertions.
This TC will submit its deliverables to the OASIS Technical Advisory Board (TAB) with the ultimate intent of having
these deliverables recommended to the OASIS Board for contribution to a future revision of the OASIS general TC process,
aiming at improving the quality and adoption of OASIS output. The TC would accept feedback and recommendations from the
TAB and OASIS Board.
The TC will also facilitate the promotion of its deliverables and give them the visibility necessary to reach potential
users in other standard organizations.
Scope:
The scope of activity for this TC must be within the following topics:
- TA model: A model for designing Test Assertions (TA model), that
is independent from any particular target specification or standard, but that may recognize different types of test
assertions, and may accommodate these in a specific way. Test assertions may be for verifying conformance of an
implementation to a specification, or interoperability between implementations of the same specification. The TA model
may address any useful relationship identified between TAs, such as pre-requisites or pre-conditions. It may include
support for grouping several TAs - or grouping entities -, but will not pretend to fully model such entities as
conformance profiles, specification modules or implementation roles.
- Test Environment modeling: Guidelines for characterizing the
test environment or test harness assumed by the test assertions, as well as the item under test or IUT (an
implementation of all or part of the target specification). Such characterization may remain abstract by just focusing
on the interaction between test environment and IUT. This characterization may state the expected properties and mode of
operation required from the test environment in order to verify the TAs. It may be seen as some of the requirements for
a real test harness intended to process test cases based on these TAs .
- Related Notions: Within scope is the selection and/or refinement
of definitions of concepts expected to be related to TAs, even if not directly targeted by the modeling and methodology
work of this TC. Such concepts may include: test case, conformance profile or level, test environment / harness, test
execution.
- Methodology: A methodology to make use of this model. This may
include examples derived from applying the above TA model to particular specifications or standards.
- XML Mark-up: An XML representation for TAs and - if appropriate - for their grouping entities. Additional
notations supporting the modeling of TAs (e.g. UML) are also within scope. The intent of the XML representation is left
at the discretion of the TC. It could be intended as an exchange format for editing tools, or as a source for
rendering/publishing, or as an input for a test tool, or a combination of these.
- Case Studies: Investigation off current practices in other OASIS
TCs or other organizations which have already written test assertions for their specifications.
- Liaison and Promotion: Within scope are efforts to liaise with
other organizations than OASIS, and cooperation with external contributors and groups that can provide input to the TC
as well as become users of its deliverables. Joint deliverables are within scope.
The following documents are input material to this TC, that will deserve prime attention from the TC assuming their IP
status is compatible with the IPR mode of the TC:
- the Test Assertion Guideline draft, (originally initiated within
OASIS TAB, 2004-2005)
http://www.oasis-open.org/committees/document.php?document_id=20661&wg_abbrev=ebxml-iic
- Conformance requirements for Specifications (OASIS, March 2002)
http://www.oasis-open.org/committees/download.php/305/conformance_requirements-v1.pdf
- Conformance testing and Certification Framework (OASIS,
Conformance TC, June 2001)
http://www.oasis-open.org/committees/download.php/309/testing_and_certification_framework.pdf
- QA Framework: Specification Guidelines (W3C, November 2004)
http://www.w3.org/TR/qaframe-spec/
- Variability in Specifications, WG note (W3C, 2005)
http://www.w3.org/TR/2005/NOTE-spec-variability-20050831/
- Test Assertion Documents for WS-I profiles (2003-2005).
http://www.ws-i.org/Testing/Tools/2005/01/BP11_TAD_1-1.htm
- Test Metadata, QA Interest Group note, (W3C, September 2005)
http://www.w3.org/TR/2005/NOTE-test-metadata-20050914/
Other documents may be considered by the TC, subject to a TC decision.
Explicitly Out-of-scope:
- Defining test case metadata or representation. (Several developments are occurring in this area.)
- Modeling or methodology of dimensions of variability such as conformance profiles / levels / modules / discretionary
items.
Substantial prior work exists in these areas. However the TC may use this prior work as a foundation and may also
accommodate these constructs when designing an XML mark-up for TAs.
Deliverables:
The TC will produce the following deliverables, in this order:
(a) Test Assertions Guidelines document, including definitions, abstract model and structure not excluding the use of a
modeling notation (e.g. UML), methodology and examples showing how to extract TAs from a target specification.
(b) A representation of TAs in XML, that refers to and supports the TA model of (a).
Timeline:
The TC will aim at a Public Review draft of the Test Assertions Guidelines (a) before end of 2007. The guidelines will
be processed at least to Committee Specification level, and up to OASIS standard if considered appropriate by the TC.
The TC will aim at a Public Review draft of the TA XML Mark-up (b) within first half of 2008. The TA Mark-up will be
processed at least to Committee Specification level, and up to OASIS standard if considered appropriate by the TC
IPR Mode:
Royalty-Free on Limited Terms.
Audience:
The TC is welcoming any OASIS member with an interest and/or experience in conformance, testing and quality assurance.
Users of the TC deliverables are expected to be members of any OASIS technical committee.
Language:
English
=============================== complement to charter =============================
Complement to the Test Assertions Guidelines Technical Committee
1. Why Other Works do not Address TAG Charter
A common point in the following works, is that they treat Test Assertions at best very lightly. The main focus is on
other aspects of QA. In contrast, the TAG objectives is to exclusively emphasize the test Assertions, as the single most
critical factor of the testability of specifications, and the foundation to any conformance statement. The TAG
deliberately ignores other QA aspects to force users' attention on TAs.
1. Conformance requirements for Specifications (OASIS, March 2002)
http://www.oasis-open.org/committees/download.php/305/conformance_requirements-v1.pdf
The focus is on Conformance statements, a small section is dedicated to test Assertions. This is not sufficient for TAG
charter, rather complementary to it.
2. Conformance testing and Certification Framework (OASIS,
Conformance TC, June 2001)
http://www.oasis-open.org/committees/download.php/309/testing_and_certification_framework.pdf
is about testing guidelines and certification process. This concerns steps that are subsequent to the TA definitions. In
fact, proper TA definitions are a prerequisite to this testing / certification phase.
3. QA Framework: Specification Guidelines (W3C, November 2004)
http://www.w3.org/TR/qaframe-spec/
This work is an exhaustive QA guideline, yet only makes a small coverage of test Assertions, for which the reader does
not get much guidance. It does not convey the importance that TAs should have according to TAG charter. We see TAG
charter as a concrete, detailed guidance to achieve a piece of what the QA Framework recommends.
4. Variability in Specifications, WG note (W3C, 2005)
http://www.w3.org/TR/2005/NOTE-spec-variability-20050831/
Is a detailed guide on conformance technology, focused on a technical aspect of conformance. A complement to [3], and an
in-depth study of some aspects defined in [1]. Same comments apply as for [1].
5. Test Assertion Documents for WS-I profiles (2003-2005).
http://www.ws-i.org/Testing/Tools/2005/01/BP11_TAD_1-1.htm
Is an instance of test assertions for a particular standardization work (here, WS-I profiles). While this work
represents valuable input, it cannot pretend - as is - at being a guideline for a larger number of specifications.
6. Test Metadata, QA Interest Group note, (W3C, September 2005)
http://www.w3.org/TR/2005/NOTE-test-metadata-20050914/
This work provides some ontology and terminology, most of it appears to be relevant to TAs as well as to test
development and execution (the prime focus of the document). However it is no substitute for modeling and methodology
of TAs. Rather, a terminology resource for a TA guideline.
2. First Meeting:
Day: Thursday May 3rd (US date)
Time: 2pm GMT (5pm StPetersb, 11pm Beijing, midnight Seoul, 7am California, 10am EastCoast, 2pm UK)
Style: Conference call (info will be sent later)
Duration: 90 min maximum
Sponsor: Fujitsu
3. Projected Meeting Schedule:
Either 1h every week, or 2h every two weeks, depending on TC member preferences. Assuming the first meeting is a
conference call, a face-to-face meeting will be set within 3 months of the first meeting - June or July - once the work
has actually started, to maximize the value of face time.
4. Co-proposers
Lynne Rosenthal, NIST (National Institute of Standards and Technology) lsr@nist.gov
Patrick Curran, Sun
Patrick.Curran@Sun.COM
Hyunbo Cho, KIEC (Korea Institute for Electronic Commerce) hcho@postech.ac.kr
Jacques Durand, Fujitsu
jdurand@us.fujitsu.com
5. Convener
Jacques Durand, Fujitsu
6. Member Section
N/A
7. List of Contributions
This work, already submitted to OASIS, is contributed to this TC:
- Test Assertion Guideline draft, (originally initiated within
OASIS TAB, 2004-2005)
http://www.oasis-open.org/committees/document.php?document_id=20661&wg_abbrev=ebxml-iic
8. Draft of F.A.Q.
Will be provided later.
9. Proposed Working Title for deliverables
Deliverable #1: Test Assertion Guidelines - name: "TA Guidelines".
Deliverable #2: Test Assertion XML representation - name: "TA Mark-up".