​​​​Executive summary: Meeting of ITU-T Study Group 12 (Performance, Quality of Service, Quality of Experience), Geneva, 1-10 May 2018

Note: This is not an official record and is subject to correction or modification

Overview

ITU-T Study Group 12 held its third meeting of the study period in Geneva, 1-10 May 2018. The meeting was attended by 128 participants from 43 countries. 13 delegates used the remote participation facilities.

The meeting received 74 Contributions​ and handled 183 TDs​. Delegates reviewed 43 incoming Liaison statements and adopted 18 outgoing Liaison statements.

Nine Recommendations were consented (4 new, 2 revised, 3 Amendments), and three non-normative texts were agreed (1 Amendment, 1 Implementers’ Guide, 1 Technical Report) by the meeting (see below for details).

A webinar (announcement | recording | slides) was held on 10 May 2018 following the closing plenary, summarizing key meeting results. Photos of the SG12 opening plenary can be found here​.

Major accomplishments

Completed work

The meeting consented new ITU-T Recommendations on
(1) ​ Artificial noise-fields in laboratory conditions (will be known as ITU-T P.570);
(2) ​ Subjective evaluation of speech quality with a crowdsourcing approach (ITU-T P.808);
(3) ​ Subjective evaluation methods for gaming quality (ITU-T P.809);
(4) ​ Statistical framework for end to end network-performance benchmark scoring and ranking (ITU-T E.840).

The meeting consented revisions and amendments to existing ITU-T Recommendations on
(5) ​ Test signals for use in telephonometry (ITU-T P.501), Amendment 1 on AM-FM test signal for super-wideband and fullband applications;
(6) ​ Framework and methodologies for the determination and application of QoS parameters (ITU-T E.802), Amendment 2 on updates and additional information on the degree of variability function in support of E.802;
(7) ​ Opinion model for video-telephony applications (ITU-T G.1070);
(8) ​ Measurements in IP networks for inter-domain performance assessment (ITU-T Y.1543); and
(9) ​ Hand-over performance among multiple access networks (ITU-T Y.1546), Amendment 1 on the IP-based service availability function.

SG12 agreed on three non-normative texts:​
(1) ​ Hybrid perceptual bitstream models for objective video quality measurements (ITU-T J.343), Amendment 1, new Appendix containing information about test vectors for the J.343 family of Recommendations;
(2) ​ Implementers’ Guide for Recommendation ITU-T P.863 (03/18) on Perceptual objective listening quality assessment;
(3) ​ Technical Report on subjective evaluation of quality of media with a crowdsourcing approach.

The meeting also endorsed the way forward developed by the ad-hoc group on E.CEMI (Customer experience management index for popular services in operators’ network to score service quality that customers experience in terms of key network performance parameters), a text consented at the previous meeting. SG12 agreed to convert existing E.CEMI into a new work item, E.CEMIF (Framework), and to consequently not to approve the existing text of E.CEMI as an ITU-T Recommendation.

New work items

SG12 established new work items on
(1) ​ Application guide for the concept of QoE (G.CQoE);
(2) ​ Super-wideband E-model (G.107.2);
(3) ​ Computational model used as a QoE/QoS monitor to assess one-to-one videotelephony services (G.CMVTQS);
(4) ​ Crowdsourcing embedded solutions for measuring end-to-end QoS in fixed broadband networks (E.CrowdESFB);
(5) ​ QoE factors for new services in 5G network (G.QoE-5G);
(6) ​ QoE factors of augmented reality (AR) (G.QoE-AR);
and initiated work on two new technical reports on (1) the considerations on the use of algorithms based on machine learning and artificial intelligence for QoS and QoE purposes, and (2) on the concept of broadband access.

WTSA matters

The update of SG12’s operational plan for implementation of WTSA Resolution 95 (ITU-T initiatives to raise awareness on best practices and policies related to service quality) was approved. The meeting reviewed the preliminary results of the questionnaire on service quality regulatory frameworks circulated after the last meeting via TSB Circular 62 and agreed on follow-up activities.

A comprehensive overview of the results is scheduled for presentation at an ITU Workshop on Telecommunications Service Quality Regulatory Frameworks and Experience Driven Networking, to take place in Geneva, 26 November 2018, a day before opening the next SG12 meeting.

In addition to advancing its work related to WTSA Resolution 95, SG12 liaised with other ITU-T Study Groups on topics including alternative calling procedures (WTSA Resolution 29), SG12 work related to software-defined networking (WTSA Resolution 77), and the impacts of counterfeit and non-compliant devices on the QoS/QoE of mobile networks (WTSA Resolution 96).

Other activities

Work is advancing on the ITU-T Software Tool Library (Recommendation ITU-T G.191) on GitHub​, with inputs from 3GPP and the wider community. A revision of ITU-T G.191 is likely to be considered at the next SG12 meeting.

A lab was set up in the basement of the ITU Montbrillant building to advance the practical work on P.NATS phase 2: Parametric bitstream-based and pixel-based quality assessment of progressive download and adaptive audiovisual streaming services over reliable transport for HEVC and UHD.

Technical tutorials were given by on the topics of (1) extended KPI for train-related end-to-end quality metrics for train-related use cases, and (2) assessing the impact of environmental factors on perceived speech quality using smartphone sensors. ITU/TSB introduced delegates to the new Rapporteur Group Meeting (RGM) Document Management System.

Approximately 20 people from 13 countries attended the SG12 newcomers’ session.

SG12 in the news

• ITU News: New ITU standards for video gaming experience (WEBINAR)
• Flickr: ITU-T SG12 Opening​

Results by group directly reporting to Plenary

Q1/12

• Coordination on WTSA Resolution 95 (Hammamet, 2016);
• Approved proposal for workshop on Telecommunications Service Quality Regulatory Frameworks and Experience Driven Networking;
• Initiated work on a technical report on the considerations on the use of algorithms based on machine learning and artificial intelligence for quality of service and quality of experience purposes;
• Adopted two outgoing liaison statements.

Q2/12

• Established new work item G.CQoE;
• Initiated work on a technical report on the concept of broadband access;
• Adopted two outgoing liaison statements.

SG12 Regional Group on QoS for the Africa Region (SG12RG-AFR)

• Reviewed its action plan
• Coordinated African contributions to SG12

Quality of Service Development Group (QSDG)

• Provided briefings on recent workshops in Rio de Janeiro and Dakar;
• Reviewed and updated its work programme for 2018-19.

Results by working party

Working Party 1/12

• Produced the following Recommendations for consent in plenary:
• New Recommendation ITU-T P.570: Artificial noise-fields in laboratory conditions;
• New Recommendation ITU-T P.808: Subjective evaluation of speech quality with a crowdsourcing approach;
• New Recommendation ITU-T P.809: Subjective evaluation methods for gaming quality;
• Amendment 1 to Recommendation ITU-T P.501: Test signals for use in telephonometry.
• Produced the following non-normative text for agreement in plenary:
• Technical Report on subjective evaluation of quality of media with a crowdsourcing approach.
• Adopted two outgoing liaison statements.

Working Party 2/12

• Produced the following non-normative text for agreement in plenary:
• Implementers’ Guide for Recommendation ITU-T P.863 (03/18) on Perceptual objective listening quality assessment;
• Amendment 1 to Recommendation ITU-T J.343: Hybrid perceptual bitstream models for objective video quality measurements.
• Established new work items G.107.2 and G.CMVTQS.
• Adopted one outgoing liaison statement.

Working Party 3/12

• Produced the following Recommendations for consent in plenary:
• New Recommendation ITU-T E.840: Statistical framework for end to end network-performance benchmark scoring and ranking;
• Amendment 2 to Recommendation ITU-T E.802: Framework and methodologies for the determination and application of QoS parameters;
• Revised Recommendation ITU-T G.1070: Opinion model for video-telephony applications;
• Revised Recommendation ITU-T Y.1543: Measurements in IP networks for inter-domain performance assessment;
• Amendment 1 to Recommendation ITU-T Y.1546: Hand-over performance among multiple access networks.
• Established new work items E.CrowdESFB, G.QoE-5G, G.QoE-AR.
• Adopted eleven outgoing liaison statements.

Future meetings and workshops

Meetings of SG12

• 27 November - 6 December 2018, Geneva, Switzerland
• TBC: 7-16 May 2019, Geneva, Switzerland
• TBC: 26 November – 5 December 2019, Geneva, Switzerland

Meetings of Rapporteur groups

• Q14/12: 19-21 June 2018, virtual meeting
• Q12/12: 5-6 September 2018, Istanbul, Turkey (Türk Telekom) (jointly with workshop and QSDG)
• Q4/12: 27-28 September 2018, Herzogenrath, Germany (Head acoustics)
• Qs 13 and 17/12: 2-3 days in September or October 2018, venue TBC

Meeting of Quality of Service Development Group (QSDG)

• 7 September 2018, Istanbul, Turkey (Türk Telekom)

SG12 Workshops

• ​3-4 September 2018, Istanbul, Turkey (Türk Telekom) (with Q12/12 and QSDG)
• 26 November 2018, Geneva, Switzerland: ITU workshop on Telecommunications Service Quality Regulatory Frameworks and Experience Driven Networking (followed by SG12)​