Geneva, Switzerland, 11 March 2019
Contact: tsbevents@itu.intIntroductionThe question of whether customers may or may not test the performance of the specific technology, equipment/system and application announced by vendors/operators becomes very important in the era of digital development. According to the term defined in Recommendation ITU-T E.800, benchmark is “evaluation of performance value/s of a parameter or set of parameters for the purpose of establishing value/s as the norm against which future performance achievements may be compared or assessed”. Moreover, describing the process, Recommendation ITU-T Q.3930 defines benchmarking as “performance tests of a system based on a suite of standardized performance tests. The main purpose of a performance benchmark is to produce a metric that can be rated and compared with the metric values produced by other systems using the same benchmark”. In principal, the benchmarking approach can be used for performance assessment of parameters of different systems, networks, services and applications. For instance, among these parameters could be call drops, call set-up delay, CPU load, latency, jitter, download transmission speed, upload transmission speed to name but a few. In general, there is a need for standards, which define performance test suites, for rating each of the above mentioned parameters for every particular system, network, service and/or application. The existence of such standards can allow the availability in the market of applications that can be used by the customers to produce comparable results. Following the results of WTSA-16, some of the current ITU-T SG11 activities focus on service and networks benchmark testing and remote testing including Internet related performance measurements (Q9/11). As of now, ITU-T SG11 developed a number of Recommendations on benchmarking of IMS and IMS/NGN platforms, including:
- ITU-T Q.3930: Performance testing of distributed systems - Concepts and terminology;
- ITU-T Q.3931.1-Q.3931.4: Performance benchmark for the PSTN/ISDN emulation subsystem of an IP multimedia system;
- ITU-T Q.3932.1-Q.3932.4: IMS/NGN performance benchmark;
- ITU-T Q.3933: Reference benchmarking, background traffic profiles and KPIs for VoIP and FoIP in fixed networks.
In addition, SG11 approved Recommendation ITU-T Q.3960 “Framework of Internet related performance measurements” and, in close collaboration with ETSI TC INT, SG11 is working on testing methodologies of Internet related performance measurements. The common idea of such methodology is based on the assessment of network performance parameters, which directly influence the experience of customers about the quality of the Internet connection provided by operators. This workshop was dedicated to share experiences and highlighting the approaches on benchmarking and Internet related performance measurements of different stakeholders, including telecom operators, regulators, SDOs, customers and any active player. The Workshop focused on:
- Benchmarking concept and key terminology;
- ITU and other SDOs activities on benchmarking and Internet related performance measurements;
- Benchmarking of equipment/systems/applications;
- Performance evaluating systems for signaling networks and applications;
- Key challenges faced by operators and regulators on using existing approaches on Internet related performance measurements;
- Technical approaches/solutions to be used for Internet related performance measurements.
The Workshop took place during the ITU-T Study Group 11 meeting from 6-15 March 2019 at the same venue.
Rohde & Schwarz Demo
A successful demo was showcased during the break from Rohde & Schwarz, Germany, following their presentation on "Why Operations Support Systems (OSS) Are Not the Answer to All? Enhanced Network Performance Evaluation with Mobile Probes".
Target Audience
ITU members and non-members were invited. Participation was free of charge.
