Recommendation ITU-T P.910 (10/2023) Subjective video quality assessment methods for multimedia applications
Summary
History
FOREWORD
Table of Contents
Introduction
1 Scope
     1.1 Limitations
2 References
3 Definitions
     3.1 Terms defined elsewhere
     3.2 Terms defined in this Recommendation
4 Abbreviations and acronyms
5 Conventions
6 How to use this Recommendation
7 Source stimuli
     7.1 Source signal recordings
     7.2 Video considerations
          7.2.1 Deviating from these criteria
          7.2.2 Coding complexity
          7.2.3 Subject matter
          7.2.4 Production quality and aesthetics
          7.2.5 Post-production effects and scene cuts
          7.2.6 Unusual properties
          7.2.7 Novelty and convenience sampling
     7.3 Image considerations
     7.4 Audio considerations
     7.5 Audiovisual considerations
     7.6 Duration of stimuli
     7.7 Number of source stimuli
     7.8 Spatial information (SI) and temporal information (TI) metrics for scene selection
          7.8.1 Pre-processing of luma values and post-processing steps for SI / TI
               7.8.1.1 Normalization
               7.8.1.2 Range scaling
               7.8.1.3 Electro-optical transfer function
               7.8.1.4 Opto-electronic transfer function
               7.8.1.5 Post-processing
          7.8.2 Spatial perceptual information measurement
          7.8.3 Temporal information measurement
          7.8.4 Aggregation of SI / TI scores
          7.8.5 Usage of SI / TI with scene cuts
          7.8.6 Usage of SI / TI with different frame resolutions
          7.8.7 Usage of TI with different frame rates
8 Test methods, rating scales and allowed changes
     8.1 Absolute category rating (ACR) method
          8.1.1 Comments
     8.2 Degradation category rating (DCR) or double stimulus impairment scale (DSIS) method
          8.2.1 Comments
     8.3 Comparison category rating (CCR) or double stimulus comparison scale (DSCS) method
          8.3.1 Comments
     8.4 Pair comparison (PC) method
          8.4.1 Comments
     8.5 Subjective assessment of multimedia video quality (SAMVIQ) description for video and audiovisual tests
          8.5.1 SAMVIQ overview
          8.5.2 SAMVIQ scale
     8.6 Acceptable changes to the methods
          8.6.1 Changes to labels
          8.6.2 ACR with hidden reference (ACR-HR)
               8.6.2.1 Comments
          8.6.3 Skip option
               8.6.3.1 Comments
          8.6.4 DCR played more than once
          8.6.5 Side-by-side presentation for DCR or CCR
          8.6.6 Split video presentation for DCR or CCR
     8.7 Controversial changes to the methods
          8.7.1 Increasing the number of levels
               8.7.1.1 Comments
          8.7.2 Watch again option
               8.7.2.1 Comments
9 Environment
     9.1 Controlled vs uncontrolled environment
     9.2 Homogenous vs heterogenous environment
     9.3 Viewing distance
     9.4 Viewing conditions
10 Subjects
     10.1 Number of subjects
     10.2 Subject population
     10.3 Sampling subjects
     10.4 Sampling techniques
     10.5 Few observers with repetitions (FOWR) subject protocol
11 Experiment design
     11.1 Size of the experiment and subject fatigue
     11.2 Conventional vs unrepeated scene experiment designs
     11.3 Framework for evaluating specific tasks
     11.4 Special considerations for transmission error, rebuffering and audiovisual synchronization impairments
     11.5 Special considerations for longer stalling events
     11.6 Repetitions
     11.7 Pre-tests
     11.8 Pilot study
     11.9 Within subject and between subject experiment designs
          11.9.1 Within subject
          11.9.2 Between subjects
12 Experiment implementation
     12.1 Informed consent
     12.2 Overview of subject screening
     12.3 Optional pre-screening of subjects
     12.4 Post-screening of subjects
     12.5 Instructions and training
     12.6 Experiment duration, sessions and breaks
          12.6.1 Short stimuli designs
          12.6.2 Long stimuli designs
     12.7 Stimuli play mechanism
          12.7.1 Computer playback and compressed playback
          12.7.2 Self-paced sessions
          12.7.3 Fixed paced sessions
          12.7.4 Stimuli randomization
          12.7.5 Types of stimuli in each session
     12.8 Voting
     12.9 Questionnaire or interview
13 Data analysis
     13.1 Documenting the experiment
     13.2 Calculate MOS or DMOS
     13.3 Evaluating objective metrics
     13.4 Significance testing, subject bias removal and standard deviation of scores
     13.5 Ratings from multiple laboratories
     13.6 Bias-subtracted consistency-weighted MOS method for subject screening
     13.7 Disagreement rate for lab-to-lab and method-to-method comparisons
14 Mandatory information to report on a subjective test
     14.1 Documenting the test design
     14.2 Documenting the subjective testing
     14.3 Data analysis
Annex A  Method for post-experimental screening of subjects  using Pearson linear correlation
     A.1 Screen by PVS
     A.2 Screen by PVS and HRC
Annex B   Details related to the characterization of the test sequences
     B.1 Sobel filter
     B.2 Definitions of EOTF and OETF functions
     B.3 How to use spatial information and temporal information for test sequence selection
     B.4 Examples
Appendix I  Sample informed consent form
Appendix II  Sample instructions
Appendix III  Reference code for bias-subtracted consistency-weighted MOS method for subject screening
Appendix IV  Obsolete CRT display technologies
Bibliography