Performance Evaluation of Objective Quality Assessment Methods for Omnidirectional Images Under Emerging Compressions

Abstract

This paper presents a unique study of objective and subjective image quality assessment (IQA) of 360 degrees images, distorted by emerging and legacy compressions - AV1 Image File Format (AVIF), Joint Photographic Experts Group (JPEG), High Efficiency Image File Format (HEIC), JPEG XL from the cross-lab experiments carried out by two universities. The performance evaluation of objective conventional IQA methods relied on statistical data analysis employing not only traditional approaches with correlation coefficients but also advanced techniques such as Receiver Operating Characteristic (ROC) analysis, which offers several statistical advantages. The comprehensive statistical examination presented, utilizing an outcome from the subjective experiment with 66 observers, confirmed that the performance of specific IQA metrics depends not only on the image features but also on the applied image compression algorithm. Overall, the study examined nine IQA objective metrics. For instance, when evaluating distortion caused by the AVIF and HEIC codecs, the Multi-Scale Structural Similarity (MS-SSIM) metric exhibited the best performance, while the Feature Similarity chrominance (FSIMc) metric excelled in assessing JPEG-based compressions. Additionally, detailed outcomes of the subjective experiments, including eye-tracking data, are available in a public dataset. This study provides previously unexplored insights into the performance of conventional objective metrics for 360 degrees images degraded by emerging image codecs. The presented results suggest suitable metrics for specific application scenarios (codecs) and, along with the dataset, lay the groundwork for further research on new objective quality assessment methods for omnidirectional images under emerging compressions.This paper presents a unique study of objective and subjective image quality assessment (IQA) of 360 degrees images, distorted by emerging and legacy compressions - AV1 Image File Format (AVIF), Joint Photographic Experts Group (JPEG), High Efficiency Image File Format (HEIC), JPEG XL from the cross-lab experiments carried out by two universities. The performance evaluation of objective conventional IQA methods relied on statistical data analysis employing not only traditional approaches with correlation coefficients but also advanced techniques such as Receiver Operating Characteristic (ROC) analysis, which offers several statistical advantages. The comprehensive statistical examination presented, utilizing an outcome from the subjective experiment with 66 observers, confirmed that the performance of specific IQA metrics depends not only on the image features but also on the applied image compression algorithm. Overall, the study examined nine IQA objective metrics. For instance, when evaluating distortion caused by the AVIF and HEIC codecs, the Multi-Scale Structural Similarity (MS-SSIM) metric exhibited the best performance, while the Feature Similarity chrominance (FSIMc) metric excelled in assessing JPEG-based compressions. Additionally, detailed outcomes of the subjective experiments, including eye-tracking data, are available in a public dataset. This study provides previously unexplored insights into the performance of conventional objective metrics for 360 degrees images degraded by emerging image codecs. The presented results suggest suitable metrics for specific application scenarios (codecs) and, along with the dataset, lay the groundwork for further research on new objective quality assessment methods for omnidirectional images under emerging compressions.