(modified former Question 14/9) Motivation
In digital systems the quality of each communication medium is influenced by
a number of interacting factors, such as source coding and compression, bit rate
(fixed or variable), delay, bandwidth, synchronization between the media,
transmission impairments, and many others. New services that use IP, wireless,
mobile, NGN, ISDN, B-ISDN, ATM, etc. are providing ubiquitous access for
multimedia services. Audiovisual multimedia cover multichannel audio, television, 3D video
and LSDI applications including interactive ones, in addition to other
applications such as videoconferencing, personal computer desktop conferencing,
interactive educational and training services, groupware, interactive gaming,
and videotelephony. In order to develop the two-way measurement techniques
required for conversational applications a basis in one-way audio and video
quality evaluation must first be defined and validated. Considering the spread
of broadband connections to business and the home, the bandwidths will support
both low resolution (e.g., QCIF) and standard and high definition imagery. As an
example, audio multimedia applications currently range from audio for narrowband
applications (e.g. video telephony) to the enhanced audio contained in 7.1
surround sound systems for interactive gaming. In the future, 3D programs and 3D
games are expected to become more widely available. Methods for assessing the quality
of these media services are needed.
Objective methods
Current objective quality measuring techniques for audiovisual applications
do not correlate to the user opinion on the perceived audiovisual quality with
the desirable accuracy. It is therefore necessary to identify objective
techniques for measuring the various individual and combined effects of factors
such as digital compression, transmission, storage, and others on the perceived
quality of audiovisual systems. It is also important to verify that these
techniques are meaningful by correlating proposed objective tests with
corresponding subjective test data.
Subjective methods
There is a need to continue to develop new subjective methods to address new
audiovisual services. The perceived quality depends on the kind of application
and on the tasks the applications are used for. For example, in a free
conversation through a videophone, the perceived quality may primarily depend on
delay, lip-synchronization and audio quality, while in a mainly one-way
application like remote-teaching the perceived quality could be primarily
related to the quality of graph and low motion picture sequences.
These studies include the maintenance of and enhancements to Recommendations
P.910, P.911, P.920, P.930, P.931, J.146, J.148, J.149, and the development of
new Recommendations as needed.
Much of the work on this Question (and its predecessors) was and will be done
in cooperation with the Video Quality Experts Group (VQEG).
Question
- Interaction of media: What subjective and objective measurement methods
should be used to evaluate end-to-end quality of each medium (e.g. video,
audio, television, 3D video) and the interactions between the media, with particular
attention to the audiovisual quality assessment of systems used for
videoconferencing/videotelephony and other interactive multimedia services?
What are the quality levels that can be defined by objective or subjective
methods in different applications (or tasks) taking into account the
interactions between media?
- 2) Transmission errors: What objective methods could be used for
in-service measurement and monitoring of transmission systems for such
multimedia services in the presence of transmission errors? What new
subjective measurement methods should be used for the evaluation of
transmission quality of real time audiovisual services by expert observers
resulting in the identification of specific flaws in the transmission
equipment or environment? What procedures should be used, and which
dimensions, transforms, and partial or differential signals should be viewed
by experts to evaluate specific impairments of real time audiovisual
services?
What objective and subjective methods can be used to evaluate audiovisual
signals with time-varying quality?
- Impairment characterizations: Among the most significant factors (e.g.
spatial resolution, temporal resolution, colour fidelity, audio and visual
artefacts, media synchronization, delay, cross-talk etc.) affecting the overall quality
of multimedia services, what objective and subjective methods assess the
extent of or can differentiate between these factors? How can the mutual
interaction between these factors be objectively and subjectively measured
with respect to their influence on overall audiovisual quality? For what
applications can the assessment methods be shown to be useful and robust
over a range of conditions?
- Evaluation of specific services: What assessment methods (objective and
subjective) can be used to characterize the quality effects of multipoint
distribution for interactive communication and other new audiovisual
services such as remote monitoring, interactive gaming, and mobile
audiovisual communication?
- Test methodologies: What objective or subjective methods and assessment
tools are required to fully describe perceived audiovisual impairments in
terms of measurable system parameters? What kind of references should be
used in subjective tests? What methods can be used to measure the video
quality of 3D display monitors? What methods can be used to measure the
eye fatigue in 3D video applications?
- Combination of test results: In some cases it may be useful to combine
objective measures (e.g. video measures, audio measures, media
synchronization) to provide a single figure of merit. In this regard, which
objective measures and/or techniques should be combined, and in what manner,
so that the figure of merit correlates satisfactorily with subjective test
results?
- Test sequences: While the library of test sequences has increased
greatly during the last study period (e.g. VQEG Phase I), there is still a
need for more test sequences, especially those with audio included. Which
audiovisual test material (e.g. audiovisual test sequences, 3D video) can be used for
subjective and objective evaluations? In addition to the definitions of SI
and TI in P.910, which criteria (objective and/or subjective) should be used
to characterize and classify multimedia test material?
- Validation and applicability of objective methods: There are three basic
methodologies of objective picture quality measurement. Full-reference (FR)
uses the full bandwidth video input. Reduced-reference (RR) uses lower
bandwidth features extracted from the video input. No reference (NR) has no
information about the video input. What objective methodology should be used
for different multimedia applications? What subjective methods should be
used to validate each of the three basic objective methodologies?
- What enhancements to existing Recommendations are required to provide
energy savings directly or indirectly in Information and Communication
Technologies (ICTs) or in other industries? What enhancements to developing
or new Recommendations are required to provide such energy savings?
Tasks
Tasks include, but are not limited to the ones detailed below.
- Quality assessment in multimedia services requires on the one hand the
continuous updating of Recommendations under the responsibility of Study
Group 9 and also the definition of new task oriented/application-dependent
evaluation methods for the combined evaluation of audio and video signals.
- A new Recommendation utilizing expert viewers is expected in this Study
Period. Three Recommendations defining objective methods for assessing
audiovisual quality in multimedia services are expected to be approved in
this Study Period.
- Initial work on quality assessment of interactive gaming applications
will result in a new Recommendation in this Study Period.
An up-to-date status of work under this Question is contained in the
SG 9 Work Programme.
Relationships
Recommendations:
Questions:
Study Groups:
- ITU-T Study Groups 12, 13, 15 and 16 and ITU-R Study Group 6
Standardization bodies:
- IETF and regional standardization bodies, e.g. ATIS
|