Title
Data-driven intra-prediction modes in the development of the versatile video coding standard
Abstract
In this paper, intra-prediction modes for video coding that were designed using data-driven methods are presented.
These predictors were incorporated into a test model of the emerging versatile video coding (VVC) standard and yield compression benefit over state-of-the-art intra-prediction tools. However, most of the use cases for video coding require severe complexity and memory restrictions, in particular at the decoder side. As data-driven methods typically result in predictors that are described by a large set of parameters and operations, satisfying these constraints turned out to be a difficult task. The purpose of this paper is to outline key steps in the complexity reduction of the trained intra-prediction modes that were discussed in the VVC standardization activity. These simplifications finally led to matrix-based intra-prediction (MIP) which is part of the current VVC draft international standard.
Keywords
Video Coding, intra-prediction.
Authors
Jonathan Pfaff
Fraunhofer HHI Berlin,Germany
Jonathan Pfaff received his Diploma and his Dr. rer. nat. degree in Mathematics from Bonn University, Bonn, Germany, in 2010 and 2012. After a postdoctoral research stay at Stanford University, he joined the Video Coding & Analytics Department at the Heinrich Hertz Institute in Berlin, Germany in 2015, where, since 2020, he is heading the research group on video coding technologies. He has contributed to the efforts of the ITU-T Video Coding Experts Group in developing the Versatile Video Coding standard since 2017. His current research interests include image and video coding and machine learning.
Philipp Helle
Fraunhofer HHI Berlin, Germany
Philipp Helle received the Dipl.-Ing. degree in computer engineering from the Technical University of Berlin, Berlin, Germany, in 2009. From 2004 to 2008, he was with MikroM - Mikroelektronik f¨ ur Multimedia GmbH, Berlin, where he designed software and hardware components for digital cinema video decoders. Since 2008, he has been with the Video Coding and Analytics Group, Fraunhofer Institute for Telecommunications - Heinrich Hertz Institute, Berlin. He has successfully contributed to the HEVC video coding standard of JCT-VC of ITUT VCEG and ISO/IEC MPEG and participates in the development of the evolving VVC video coding standard. His current research interests include video coding and machine learning.
Philipp Merkle
Fraunhofer HHI Berlin, Germany
Philipp Merkle received the Dipl.-Ing. degree in electrical engineering and the Dr.-Ing. degree from Technical University of Berlin, Berlin, Germany, in 2006 and 2016, respectively. He joined Fraunhofer Institute for Telecommunications – Heinrich Hertz Institute, Berlin, in 2003 and has been a Research Associate since 2006. Since 2005, he has been an active participant in standardization activities of ISO/IEC Moving Picture Experts Group (MPEG) and ITU-T Video Coding Experts Group (VCEG) with successful contributions to the Multiview Video Coding (MVC) amendment of the H.264/Advanced Video Coding (AVC) standard, the multiview and 3D video extensions of the H.265/High Efficiency Video Coding (HEVC) standard, and the H.266/Versatile Video Coding (VVC) standard. His research interests include video coding, 3D video and television, representation and compression of multiview video plus depth scenes, 2D and 3D video-based rendering, and machine learning based video compression.
He is senior member of the IEEE and received the Rudolf-Urtel Prize of the German Society for Technology in Television and Cinema for his journal paper on MVC in IEEE Transactions on Circuits and Systems for Video Technology in 2006, as well as the EURASIP Best Paper Award for his journal paper on multiview depth compression in Signal Processing: Image Communication (Elsevier) in 2014.
Michael Schäfer
Fraunhofer HHI Berlin, Germany
Michael Schäfer received the M.Sc.degree in Mathematics from the Freie Universität Berlin, Germany, in 2017. During his studies in 2015, he joined the Fraunhofer Heinrich Hertz Institute, Berlin, Germany as an associate of its Video Coding research group. He has contributed to the efforts of the ITU-T Video Coding Experts Group in developing the Versatile Video Coding standard since 2017.
Björn Stallenberger
Fraunhofer HHI Berlin, Germany
Björn Stallenberger received the M.Sc. degree in electrical Engineering from the Technical University of Berlin, Berlin, Germany in 2014. Since 2016, he is working as a Research Associate on the topic of video compression at the Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute, Berlin, Germany.
Tobias Hinz
Fraunhofer HHI Berlin, Germany
Tobias Hinz received the Dipl.-Ing. degree in electrical engineering from the Technical University of Berlin, Berlin, Germany, in 1999. He is currently a Research Engineer with the Department of Video Coding and Analytics, Fraunhofer Institute for Telecommunications-Heinrich Hertz Institute, Berlin. His current research interests include processing, coding and transmission of video and audio content, as well as software design and optimization.
Heiko Schwarz
Fraunhofer HHI Berlin; FU Berlin, Germany
Heiko Schwarz received the Dipl.-Ing. degree in electrical engineering and the Dr.-Ing. degree, both from the University of Rostock, Rostock, Germany, in 1996 and 2000, respectively. In 1999, Heiko Schwarz joined the Fraunhofer Heinrich Hertz Institute, Berlin, Germany. Since 2010, he is heading the research group "Image and Video Coding" at the Fraunhofer Heinrich Hertz Institute. In October 2017, he became Professor at the FU Berlin. Heiko Schwarz has actively participated in the standardization activities of the ITU-T Video Coding Experts Group (ITU-T SG16/Q.6-VCEG) and the ISO/IEC Moving Pictures Experts Group (ISO/IEC JTC 1/SC 29/WG 11-MPEG). He successfully contributed to the video coding standards H.264/AVC, H.265/HEVC, and the current standardization project Versatile Video Coding (VVC). Heiko Schwarz co-chaired various ad hoc groups of the standardization bodies and coordinated core experiments. He was appointed as co-editor of H.264/AVC and as software coordinator for the SVC reference software. Heiko Schwarz served as reviewer for various international journals and international conferences. He was associate editor of the IEEE Transactions on Circuits and Systems for Video Technology.
Detlev Marpe
Fraunhofer HHI Berlin, Germany
Detlev Marpe (M’00–SM’08–F’15) received the Dipl.- Math. degree (Highest Hons.) from the Technical University of Berlin, Berlin, Germany, in 1990 and the Dr.-Ing. degree from the University of Rostock, Rostock, Germany, in 2004. In 1999, he joined the Fraunhofer Institute for Telecommunications Heinrich Hertz Institute (HHI), Berlin, where he is currently the Head of the Video Coding & Analytics Department and the Image and Video Coding Group. He was a major technical contributor to the entire process of development of the H.264/MPEG4 Advanced Video Coding (AVC) Standard and the H.265/MPEG High Efficiency Video Coding (HEVC) Standard, including several generations of major enhancement extensions. In addition to the CABAC contributions for both standards, he particularly contributed to the Fidelity Range Extensions (which include the High Profile that received the Emmy Award in 2008) and the Scalable Video Coding Extensions of H.264/MPEG-4 AVC. During the development of its successor H.265/MPEG-HEVC, he also successfully contributed to the first model of the corresponding standardization project and further refinements. He also made successful proposals to the standardization of its Range Extensions and 3-D Extensions. He has authored numerous publications in the research area of image and video coding, and holds several hundreds of internationally issued patents and patent applications in this area. His current research interests include still image and video coding, signal processing for communications and computer vision, machine learning, and information theory. Dr. Marpe is Fellow of the IEEE and member of the Informationstechnische Gesellschaft of the Verband der Elektrotechnik Elektronik Informationstechnik e.V. He was a co-recipient of three Technical Emmy Awards as a Key Contributor and a Co-Editor of the H.264/MPEG-4 AVC Standard in 2008 and 2009, and as Key Contributor of H.265/MPEG-HEVC in 2017, respectively. He received several best paper awards for his publications and he was recipient of the Karl Heinz Beckurts Award in 2011 and the Joseph von Fraunhofer Prize in 2004. From 2014 to 2018, he served as an Associate Editor of IEEE Transactions on Circuits and Systems for video Technology. In recognition of his dedicated contributions and excellent management of the review process, he was awarded the 2016 Best Associate Editor Award of the IEEE Circuits and Systems Society.
Thomas Wiegand
Fraunhofer HHI Berlin; TU Berlin, Germany
Thomas Wiegand (M’05–SM’08–F’11) is a professor in the department of Electrical Engineering and Computer Science at the Technical University of Berlin and is jointly heading the Fraunhofer Heinrich Hertz Institute, Berlin, Germany. He received the Dipl.-Ing. degree in Electrical Engineering from the Technical University of Hamburg-Harburg, Germany, in 1995 and the Dr.-Ing. degree from the University of Erlangen-Nuremberg, Germany, in 2000. He was a Visiting Researcher with Kobe University, Kobe, Japan; University of California at Santa Barbara, Santa Barbara, CA, USA; and Stanford University, Stanford, CA, where he also returned as a Visiting Professor. He served as a consultant to several start-up ventures. He has been an active participant in standardization for video coding multimedia with many successful submissions to ITU-T and ISO/IEC. He is an Associated Rapporteur of ITU-T VCEG. The projects that he co-chaired for the development of the H.264/MPEGAVC standard have been recognized by the ATAS Primetime Emmy Engineering Award and a pair of NATAS Technology & Engineering Emmy Awards. For his research, he received numerous awards and multiple Best Paper Awards for his publications. Since 2014, Thomson Reuters named him in their list of The World’s Most Influential Scientific Minds as one of the most cited researchers in his field. He has been elected to the German National Academy of Engineering (Acatech) and the National Academy of Science (Leopoldina). Since 2018, he has been appointed the chair of the ITU/WHO Focus Group on Artificial Intelligence for Health.
