Test specifications for protocols, networks and services for emerging technologies, including benchmark testing
(Continuation of Questions 9/11, 10/11 and 11/11)
MotivationWTSA Resolution 76 – Studies related to conformance and interoperability testing, assistance to developing countries, and a possible future ITU Mark programme – resolves that ITU-T Study Group 11 coordinates the sector’s activities related to the ITU Conformance and Interoperability (C&I) programme across all study groups and reviews the Recommendations in the C&I business plan for the long term implementation of the C&I programme.
ITU-T is producing a large number of Recommendations. To achieve interoperability and conformity, one of the important aspects of the ITU C&I Programme concerns the development and maintenance of testing frameworks and methodologies.
It is essential that conformance and interoperability testing methodologies used by all study groups engaged in testing are aligned and consistent with each other. To achieve interoperability on a global scale, the ITU-T Recommendations must be developed and maintained with conformance and interoperability in mind according to the relevant methodology.
The objective of conformance testing is to determine how completely and correctly the requirements stated in the Recommendation have been met by the implementation. On the contrary, in interoperability testing, the objective is to determine if two or more implementations of the same Recommendation communicate and correctly exchange information with each other. It is generally assumed that the conformity of an implementation has been tested prior to perform an interoperability testing assessment.
The recent trends (such as IMT-2020, IoT technologies) may lead to many changes with regards to the extant network architecture that will require higher network performance. This in turn would impact on the specifications of the terminal unit as in CPE, Mobile Units, and Phones etc.
Most of the telecom operators are implementing various emerging technologies and migrate from circuit-switched to the packet-switched networks, trying to provide their services using “all over IP” concept. As a result, operators face some issues which are in general related to the compatibility and interoperability of the used ICT equipment and the interconnection of IP-based networks (e.g. 4G, IMT-2020), which, among others, will be used for roaming/nomadic services. For example, providing voice and video over LTE (VoLTE/ViLTE) roaming services pose some challenges to the operators due to the lack of agreed roaming procedures among stakeholders, various available VoLTE/ViLTE implementation scenarios and other not standardized issues (e.g. ENUM, emergency call, etc.). The conformance and interoperability testing of network-network interfaces (NNI) against ITU-T Recommendations may help operators to be sure that their VoLTE/ViLTE solutions are ready for interconnection. This interconnection approach may be also used for future packet-based networks, e.g. IMT-2020 network and beyond.
In general, benchmarking is a common approach used for the measurement and testing of signalling parameters against the performance design objectives, which should help to provide the end to end service delivery and ensure network reliability.
Benchmarking in the case of IMT-2020 is not limited only to the transport layer and includes performance, quality and reliability of virtual platforms.
The issues of finding the values of network performance and productivity for services with required quality of service (QoS) is an important question for operators and their users.
For example, the Internet-related performance measurement systems, which are publicly available on the Internet, do not provide reliable and comparable measurements. Beyond the lack of standardized metrics for Internet speed in ITU-T Recommendations, the obtained test results, which can be achieved by an existing testing method, may vary from the results achieved by other methods. Obviously, the testing results depend on the network segments which are used during e2e connection. Particularly, it is not possible to guarantee that the e2e connection is based on the telecom operator’s network only and does not include other network segments which may belong to other operators.
The development of a unified approach to Internet-related performance measurement is important for all ICT stakeholders.
Also, this standardized approach will support the development of the “Framework for ICT service delivery with the guaranteed QoS and requested bitrate on fixed & mobile data networks, for development of efficient economic mechanisms and models of interaction in the "operator-provider-user" chain”.
NOTE − The network performance QoS/QoE, including the networks and services KPIs, parameters and requirements, are defined by ITU-T SG12 and related SDOs. This Question is responsible for Q.3900-Q.4099-series (testing for next generation networks), Q.1912.x-series, X.290-series (except X.292), X.Suppl.4, X.Suppl.5 and Z.500-series.
QuestionStudy items to be considered include, but are not limited to:
- What is the testing methodology for testing emerging technologies?
- What are the existing ITU-T Recommendations which include test suites?
- What is the architecture of the testbed or testing facilities to be used for testing emerging technologies?
- Which technologies that are being developed for the ICT market require conformity and interoperability testing (taking into account market needs)?
- What kind of test suites are needed for testing interconnection of IP-based networks (e.g. 4G, IMT-2020 network and beyond)?
- What kind of equipment could be benchmark tested?
- What kind of test procedures could be used for benchmark testing?
- What type of traffic could be simulated for benchmark testing?
- What design objectives need to be benchmark tested?
- What test procedures for standardized network parameters, including those defined in the framework of Internet related performance measurements (ITU-T Q.3960), such as e2e bit rate within and outside the fixed and mobile operator’s network (that is between users of the operator’s network and a particular Internet resource), need to be developed?
- How parameters/technologies/services may be tested remotely?
- What procedures need to be developed to implement remote testing?
- What is the network architecture to be used for remote testing?
- What is the methodology for testing IMT-2020 mobile devices?
- What is the methodology for testing services which require ultra-low latency?
TasksTasks include, but are not limited to:
- study the testing methodology for testing emerging technologies;
- identify the existing ITU-T Recommendations which include test suites;
- identify the architecture of the testbed or testing facilities to be used for testing emerging technologies;
- identify market driven ICT technologies, which require conformity and interoperability testing;
- develop test suites to be used for testing interconnection of IP-based networks (e.g. 4G, IMT-2020 network and beyond);
- determine the types of equipment, which could be benchmark tested;
- develop the test procedures for benchmark testing;
- identify the type of traffic to be simulated for benchmark testing;
- define the design objectives that need to be benchmark tested;
- develop test procedures for standardized network parameters, including those defined in the framework of Internet-related performance measurements (ITU-T Q.3960), such as e2e bit rate within and outside the fixed and mobile operator’s network (that is between users of the operator’s network and a particular Internet resource);
- determine the parameters/technologies/services which may be tested remotely;
- develop the procedures needed to implement remote testing;
- specify the network architecture to be used for remote testing;
- develop a methodology (guide) that would extend current experiences and testing approaches to IMT-2020;
- determine the methodology for testing IMT-2020 devices;
- determine the methodology for testing services which require ultra-low latency.
An up-to-date status of work under this Question is contained in the SG11 work programme (
https://itu.int/ITU-T/workprog/wp_search.aspx?Q=16/11).
Relationships
Recommendations
- Q, Y, H, G, E, I, M, P, X, Z and F-series
Questions
- All ITU-T SG11 Questions
Study groups
- ITU-T SG13 on future networks (e.g. SDN, NFV), cloud computing, IMT-2020, machine learning
- ITU-T SG15 on core and access technologies
- ITU-T SG16 on multimedia services, applications and e-health
- ITU-T SG17 on testing languages, including TTCN-3
- All other ITU-T SGs that are involved with C&I, IMT-2020 network and beyond, machine learning activities
- ITU-D SG1 and ITU-D SG2 on remote test centres
- ITU-T SG3 on policy issues
- ITU-T SG12 on parameters and requirements of QoS
Other bodies
- ETSI (especially ETSI TC INT and ETSI TC NTECH)
- IETF
- IEEE
WSIS action linesSustainable Development Goals