​​​​​​​​​​​​Title​

A study of the extended perceptually weighted peak signal-to-noise ratio (XPSNR) for video compression with different resolutions and bit depths

Abstract

Fast and accurate estimation of the visual quality of compressed video content, particularly for quality-of-experience (QoE) monitoring in video broadcasting and streaming, has become important. Given the relatively poor performance of the well-known peak signal-to-noise ratio (PSNR) for such tasks, several video quality assessment (VQA) methods have been developed. In this study, the authors’ own recent work on an extension of the perceptually weighted PSNR, termed XPSNR, is analyzed in terms of its suitability for objectively predicting the subjective quality of videos with different resolutions (up to UHD) and bit depths (up to 10 bits/sample). Performance evaluations on various subjective-MOS annotated video databases and investigations of the computational complexity in comparison with state-of-the-art VQA solutions like VMAF and (MS-)SSIM confirm the merit of the XPSNR approach. The use of XPSNR as a reference model for visually motivated control of the bit allocation in modern video encoders for, e. g., HEVC and VVC is outlined as well.

Keywords

Data compression, HD, HEVC, PSNR, QoE, SSIM, UHD, video coding, VMAF, VQA, VVC, WPSNR

Author​s

​​​Christian R. Helmrich 
Fraunhofer HHI, Germany 

Christian R. Helmrich received the B. Sc. degree in Computer Science from Capitol Technology University (formerly Capitol College), Laurel, MD, USA, in 2005 and the M. Sc. degree in Information and Media Technologies from the Technical University of Hamburg-Harburg, Germany, in 2008. Between 2008 and 2013 he worked on numerous speech and audio coding solutions at the Fraunhofer Institute for Integrated Circuits (IIS), Erlangen, Germany, partly as a Senior Engineer. From 2013 until 2016 Mr. Helmrich continued his work as a research assistant at the International Audio Laboratories Erlangen, a joint institution of Fraunhofer IIS and the University of Erlangen-Nuremberg, Germany, where he completed his Dr.-Ing. degree with a dissertation on audio signal analysis and coding. In 2016 Dr. Helmrich joined the Video Coding & Analytics Department of the Fraunhofer Heinrich Hertz Institute, Berlin, Germany, as a next-generation video coding researcher and developer. His main interests include audio and video acquisition, coding, storage, and preservation as well as restoration from analog sources. Dr. Helmrich participated in and contributed to several standardization activities in the fields of audio, speech, and video coding, most recently the 3GPP Enhanced Voice Services standard (3GPP TS 26.441), the MPEG-H 3D Audio standard (ISO/IEC 23008-3), and the MPEG-I/ITU-T Versatile Video Coding standard (to be specified in ISO/IEC 23090-3 and ITU-T H.266). He is co-author of more than 40 scientific publications and a Senior Member of the IEEE.


​​​​Sebastian Bosse
Fraunhofer HHI, Germany 

Sebastian Bosse is a senior researcher with the Machine Learning Group and the Image & Video Coding Group in the Video Coding & Analytics Department of Fraunhofer Heinrich Hertz Institute, Berlin, Germany. He studied Electrical Engineering and Information Technology at RWTH Aachen University, Aachen, Germany and Polytechnic University of Catalonia, Barcelona, Spain. He received the Dipl.-Ing. degree in Electrical Engineering from RWTH Aachen University, Aachen, Germany and the Dr.-Ing. degree in Electrical Engineering and Computer Science from the Technical University of Berlin, Germany.As a student, Sebastian Bosse was a visiting researcher at Siemens Corporate Research, Princeton, USA. In 2014, he was a guest researcher at Stanford University, USA.His major research interests are image and video compression, human visual perception for image communications, and neural correlates of perceived image quality and quality of experience, as well as signal processing.

​​​Heiko Schwarz
Fraunhofer HHI and Free University Berlin, Germany 

Heiko Schwarz received the Dipl.-Ing. degree in electrical engineering and the Dr.-Ing. degree, both from the University of Rostock, Rostock, Germany, in 1996 and 2000, respectively. In 1999, Heiko Schwarz joined the Fraunhofer Heinrich Hertz Institute, Berlin, Germany. Since 2010, he is heading the research group “Image and Video Coding” at the Fraunhofer Heinrich Hertz Institute. In October 2017, he became Professor at the Free University (FU) Berlin. Heiko Schwarz has actively participated in the standardization activities of the ITU-T Video Coding Experts Group (ITU-T SG16/Q.6-VCEG) and the ISO/IEC Moving Pictures Experts Group (ISO/IEC JTC 1/SC 29/WG 11-MPEG). He successfully contributed to the video coding standards ITU-T Rec. H.264 | ISO/IEC 14496-10 (MPEG-4 AVC) and ITU-T Rec. H.265 | ISO/IEC 23008-2 (MPEG-H HEVC) and their extensions. Heiko Schwarz co-chaired various ad hoc groups of the standardization bodies. He was appointed co-editor of ITU-T Rec. H.264 and ISO/IEC 14496-10 and software coordinator for the SVC reference software.

​​​​​​Detlev Marpe 
Fraunhofer HHI, Germany

Detlev Marpe received the Dipl.-Math. degree (with highest honors) from the Technical University of Berlin (TUB), Germany, and the Dr.-Ing. degree in Computer Science from University of Rostock, Germany. He is Head of the Video Coding & Analytics Department and Head of the Image & Video Coding Group at the Fraunhofer Heinrich Hertz Institute, Berlin. He is also active as a part-time lecturer at TUB.
For nearly 20 years, he has successfully contributed to the standardization activities of ITU-T VCEG, ISO/IEC JPEG, and ISO/IEC MPEG in the area of still image and video coding. During the development of the H.264 | MPEG-4 Advanced Video Coding (AVC) standard, he was chief architect of the CABAC entropy coding scheme as well as one of the main technical and editorial contributors to its Fidelity Range Extensions (FRExt) including the now ubiquitous High Profile. Marpe was also one of the key people in designing the basic architecture of Scalable Video Coding (SVC) and Multiview Video Coding (MVC) as algorithmic and syntactical extensions of H.264 | AVC. He also made successful contributions to the recent development of the H.265 | MPEG-H High Efficiency Video Coding (HEVC) standard, including its Range Extensions and 3D extensions. 
For his substantial contributions to the field of video coding, he received numerous awards, including a Nomination for the 2012 German Future Prize, the Karl Heinz Beckurts Award 2011, the 2009 Best Paper Award of the IEEE Circuits and Systems Society, the Joseph von Fraunhofer Prize 2004, and the Best Paper Award of the German Information Technology Society in 2004. As a co-editor and key contributor of the High Profile of H.264 | AVC as well as a member and key contributor of the Joint Collaborative Team on Video Coding for the development of H.265 | HEVC, he was co-recipient of the 2008 and 2017 Primetime Emmy Engineering Award, respectively. Detlev Marpe is author or co-author of more than 200 publications in the area of video coding and signal processing, and he holds numerous internationally issued patents and patent applications in this field. His current citations are available at Google Scholar. He has been named a Fellow of IEEE, and he is a member of the DIN (German Institute for Standardization) and the Information Technology Society in the VDE (ITG). He also serves as an Associate Editor of the IEEE Transactions on Circuits and Systems for Video Technology. His current research interests include image and video coding, signal processing for communications as well as computer vision and information theory.

​​​​​Thomas Wiegand 
Fraunhofer HHI and Technische Universität ​Berlin, Germany​

Thomas Wiegand is a professor in the department of Electrical Engineering and Computer Science at the Technical University of Berlin and is jointly heading the Fraunhofer Heinrich Hertz Institute, Berlin, Germany. He received the Dipl.-Ing. degree in Electrical Engineering from the Technical University of Hamburg-Harburg, Germany, in 1995 and the Dr.-Ing. degree from the University of Erlangen-Nuremberg, Germany, in 2000. As a student, he was a Visiting Researcher at Kobe University, Japan, the University of California at Santa Barbara and Stanford University, USA, where he also returned as a visiting professor. He was a consultant to Skyfire, Inc., Mountain View, CA, and is currently a consultant to Vidyo, Inc., Hackensack, NJ, USA.
Since 1995, he has been an active participant in standardization for multimedia with many successful submissions to ITU-T and ISO/IEC. In 2000, he was appointed the Associated Rapporteur of ITU-T VCEG and from 2005-2009, he was Co-Chair of ISO/IEC MPEG Video. The projects that he co-chaired for the development of the H.264/MPEG-AVC standard have been recognized by an ATAS Primetime Emmy Engineering Award. He was also a recipient of an ATAS Primetime Emmy Engineering Award for the development of H.265/MPEG-HEVC and a pair of NATAS Technology & Engineering Emmy Awards. For his research in video coding and transmission, he received numerous awards including the Vodafone Innovations Award, the EURASIP Group Technical Achievement Award, the Eduard Rhein Technology Award, the Karl Heinz Beckurts Award, the IEEE Masaru Ibuka Technical Field Award, and the IMTC Leadership Award. He received multiple best paper awards for his publications. Since 2014, Thomson Reuters named him in their list of “The World’s Most Influential Scientific Minds” as one of the most cited researchers in his field. He is a recipient of the ITU150 Award. He has been elected to the German National Acdemy of Engineering (Acatech) and the National Academy of Science (Leopoldina). Since 2018, he has been appointed the chair of the ITU/WHO Focus Group on Artificial Intelligence for Health.