Improved Image Caption Rating – Datasets, Game, and Model

Conference Paper


How well a caption fits an image can be difficult to assess due to the subjective nature of caption quality. What is a good caption? We investigate this problem by focusing on image-caption ratings and by generating high quality datasets from human feedback with gamification. We validate the datasets by showing a higher level of inter-rater agreement, and by using them to train custom machine learning models to predict new ratings. Our approach outperforms previous metrics – the resulting datasets are more easily learned and are of higher quality than other currently available datasets for

Conference Name

2023 CHI Conference on Human Factors in Computing Systems

Year of Publication


Date Published



Association for Computing Machinery

Conference Location

Hamburg, Germany