Recommendation ITU-T P.913 (06/2021) Methods for the subjective assessment of video quality, audio quality and audiovisual quality of Internet video and distribution quality television in any environment
Summary
History
Keywords
FOREWORD
Table of Contents
Introduction
1 Scope
     1.1 Limitations
2 References
3 Definitions
     3.1 Terms defined elsewhere
     3.2 Terms defined in this Recommendation
4 Abbreviations and acronyms
5 Conventions
6 Source stimuli
     6.1 Source signal recordings
     6.2 Video considerations
          6.2.1 Deviating from these criteria
          6.2.2 Coding complexity
          6.2.3 Subject matter
          6.2.4 Production quality and aesthetics
          6.2.5 Post-production effects and scene cuts
          6.2.6 Unusual properties
          6.2.7 Novelty and convenience sampling
     6.3 Audio considerations
     6.4 Audiovisual considerations
     6.5 Duration of stimuli
     6.6 Number of source stimuli
7 Test methods, rating scales and allowed changes
     7.1 List of methods
          7.1.1 Absolute category rating method
               7.1.1.1 Comments
          7.1.2 Degradation category rating or double stimulus impairment scale method
               7.1.2.1 Comments
          7.1.3 Comparison category rating, double stimulus comparison scale or pair comparison method
               7.1.3.1 Comments
          7.1.4 Subjective assessment of multimedia video quality description for video and audiovisual tests
               7.1.4.1 SAMVIQ overview
               7.1.4.2 SAMVIQ scale
          7.1.5 Multi-stimuli with hidden reference and anchor points description for audio tests
               7.1.5.1 MUSHRA overview
               7.1.5.2 The MUSHRA listening panel
               7.1.5.3 The MUSHRA scale
               7.1.5.4 Test instructions
               7.1.5.5 Comments
     7.2 Acceptable changes to the methods
          7.2.1 Changes to level labels
          7.2.2 ACR with hidden reference (ACR-HR)
               7.2.2.1 Comments
     7.3 Discouraged but acceptable changes to the methods
          7.3.1 Increasing the number of levels is discouraged
               7.3.1.1 Comments
8 Environment
     8.1 Controlled environment
     8.2 Public environment
     8.3 Viewing distance
9 Subjects
     9.1 Number of subjects
     9.2 Subject population
     9.3 Sampling subjects
     9.4 Sampling techniques
10 Experimental design
     10.1 Size of the experiment and subject fatigue
     10.2 Special considerations for transmission error, rebuffering and audiovisual synchronization impairments
     10.3 Special considerations for longer stalling events
     10.4 Pre-pilot testing and pilot testing
     10.5 Study design
          10.5.1 Within subject
          10.5.2 Between subject
11 Experiment implementation
     11.1 Informed consent
     11.2 Overview of subject screening
     11.3 Optional pre-screening of subjects
     11.4 Post-screening of subjects
     11.5 Instructions and training
     11.6 Study duration, sessions and breaks
          11.6.1 Short stimuli designs
          11.6.2 Long stimuli designs
     11.7 Stimuli play mechanism
          11.7.1 Computer playback and compressed playback
          11.7.2 Self-paced sessions
          11.7.3 Fixed paced sessions
          11.7.4 Stimuli randomization
          11.7.5 Types of stimuli in each session
     11.8 Voting
     11.9 Questionnaire or interview
12 Data analysis
     12.1 Documenting the experiment
     12.2 Calculate MOS or DMOS
     12.3 Evaluating objective metrics
     12.4 Significance testing, subject bias and standard deviation of scores
     12.5 Ratings from multiple laboratories
     12.6 Improve MOS or DMOS data quality under challenging test conditions
13 Elements of subjective test reporting
     13.1 Documenting the test design
     13.2 Documenting the subjective testing
     13.3 Data analysis
     13.4 Additional information
Annex A  Method for post-experimental screening of subjects using Pearson linear correlation
     A.1 Screen by PVS
     A.2 Screen by PVS and HRC
Appendix I  Sample informed consent form
Appendix II  Sample instructions
Appendix III  Reference code for clause 12.6
Bibliography