Abstract
Scene graph generation has emerged as a prominent research field in computer vision, witnessing significant advancements in the recent years. However, despite these strides, precise and thorough definitions for the metrics used to evaluate scene graph generation models are lacking. In this paper, we address this gap in the literature by providing a review and precise definition of commonly used metrics in scene graph generation. Our comprehensive examination clarifies the underlying principles of these metrics and can serve as a reference or introduction to scene graph metrics. Furthermore, to facilitate the usage of these metrics, we introduce a standalone Python package called SGBench that efficiently implements all defined metrics, ensuring their accessibility to the research community. Additionally, we present a scene graph benchmarking web service, that enables researchers to compare scene graph generation methods and increase visibility of new methods in a central place.
Links
Contributions
Precise Definitions of Scene Graph Metrics
We rigorously define the following scene graph metrics using pseudo code and equations:
- Recall@k
- Mean Recall@k
- Pair Recall@k
- No Graph Constraint Recall@k
- Mean No Graph Constraint Recall@k
- Instance Recall
- Predicate Rank
Python Package
The code to calculate the described metrics is bundled as an easy to install pip package. It has only four dependencies in total and should work with any big framework you are currently using.
You can install the python package using
pip install sgbench
The source code will be made available here: https://github.com/lorjul/sgbench
Benchmarking Server
To compare recent scene graph methods, we developed a benchmarking server where users can upload scene graph evaluation files.
License
The structure of this page is taken and modified from nvlabs.github.io/eg3d which was published under the Creative Commons CC BY-NC 4.0 license .