Motivation

ITU-T is producing a large number of Recommendations. To achieve interoperability, it is essential that implementations of these Recommendations conform to the Recommendations. Techniques currently exist for doing this but are not being widely used due to a lack of familiarity with the methodology. These techniques also impact the way Recommendations are written. It is essential to maintain and promote the testing methodology and ensure that new Recommendations lead to implementations that are testable.

It is essential that conformance and interoperability testing methodologies used by all study groups engaged in testing are aligned and consistent with each other. To achieve interoperability on a global scale, all Recommendations must be developed and maintained with conformance and interoperability in mind according to the methodology Recommendations.

In conformance testing, the objective is to determine how completely and how correctly the normative requirements stated in the Recommendation have been met by the implementation. In interoperability testing, the objective is to determine if two or more implementations of the same Recommendation communicate and correctly exchange information with each other. It is generally assumed that the implementations have been conformance tested prior to interoperability testing.

Three essential parts that need to be developed and maintained are: Testing methodology, Test specification language and Generic test suites for conformance and interoperability testing. Testing methodology and test specification language must not be done by separate groups. Otherwise, separate methodologies will be developed for different test specifications and the developers of testing tools will have to develop different tools for different systems at a great expense. This is likely to result in a lack of commercial testing tools.

Methodology experts from this Question must strongly interact with experts from other study groups engaged in developing test specifications in their respective areas of responsibility. Two such areas are NGN and requirements and framework for commercial off-the-shelf (COTS) software. This is to ensure that the testing methodology is widely applicable, correctly interpreted and its test specification languages are used correctly.

In particular it is required to address Testing and Test Control Notation, version 3 (TTCN-3) test specification language and its application to the generic as well as specific test suites.

These Recommendations need to be maintained and updated when appropriate. New Recommendations or other documentation may be identified, to meet the needs of users in ITU, the industry, and other organizations such as OMG.

Recommendations and Supplements under responsibility of this Question as of 1 December 2008: X.290, X.291, X.292, X.293, X.294, X.295, X.296, Z.161, Z.162, Z.163, Z.164, Z.165, Z.166, Z.167, Z.168, Z.169, Z.170, Z.500 and Supplements 4 and 5 to X-series.

Question

Study items to be considered include, but are not limited to:

1. What extensions or enhancements to existing Recommendations on protocol testing, testing languages and generic test suites based on formal models are required to meet evolving needs of users?
2. What new Recommendations, Supplements or other provisions are required (if any) to define or revise the definitions of new or existing testing languages, methodologies and frameworks?
3. What changes are needed to allow the use of ISO/IEC 10646 text to aid human understanding in annotations, names and identifiers and in text handled in TTCN-3?
4. What is needed for the testing methodology to measure impact on climate change?

Tasks

Tasks include, but are not limited to:

1. Progress work in the area of TTCN-3.
2. Maintenance of X.290, X.291, X.292, X.293, X.294, X.295, X.296, Z.161, Z.162, Z.163, Z.164, Z.165, Z.166, Z.167, Z.168, Z.169, Z.170 and Z.500.
3. Develop generic test suites expressed in TTCN-3, which can be specialized by test specifiers in other study groups for their specific testing needs.
4. Extend existing conformance testing methodology and framework for use with NGN and commercial off-the-shelf (COTS) software components, including conformance of components selected for “carrier grade” open environment (CGOE).
5. Develop interoperability testing methodology and framework for general use giving consideration to next generation network (NGN) testing and off-the-shelf software components testing.
6. Coordinate conformance and interoperability testing activity with all other study groups, and in particular with Study Groups 2, 9, 11, 13, 15 and 16. These study groups develop test specifications for a large number of Recommendations. Other study groups are also likely to develop test specifications based on the methodology produced under this Question.
7. Assist Recommendation developers in all study groups in applying conformance and interoperability testing methodology to their specific needs.
8. Develop and present tutorial material on conformance and interoperability testing methodology and framework as it evolves.
9. Progress work on testing languages and conformance testing based on formal models.
10. Assist the relevant Questions in the development of a TTCN Unified Modelling Language (UML) profile. Progress work in the area of UML Testing profile.
11. Consider extending the TTCN-3 notation to allow expanded use of ISO/IEC 10646 characters, with the possible exception of keywords.
12. Consider how testing methodology can be extended to help in measuring the contribution to climate change caused by implementations of ITU-T Recommendations.

Relationships

Recommendations: Z.10x-, Z.12x-, Z.13x-, Z.15x-, X.68x-, X.69x- and X.900-series

Questions: ITU-T Qs 12, 13 and 15/17

Study groups: ITU-T SGs 2, 9, 11, 13, 15 and 16

Standardization bodies: ISO/IEC JTC 1; ETSI; OMG

Other bodies: SDL Forum Society