Source Benchmarks

The challenge reuses two existing benchmarks, referenced here rather than re-published.

Bench4KE (CQ Generation)

An extensible, API-based benchmarking system for knowledge engineering automation, focused on evaluating CQ generation. It provides a gold standard of 843 manually authored CQs from 17 real-world ontology engineering projects and a suite of lexical and semantic similarity metrics.

Citation: Ciancarini, P., Lippolis, A. S., Nuzzolese, A. G., Presutti, V., & Ragagni, M. D. (2026). Bench4KE: Benchmarking Automated Competency Question Generation. In European Semantic Web Conference (pp. 212–231). Cham: Springer Nature Switzerland.

• DOI: 10.5281/zenodo.17817277
• Code: github.com/fossr-project/ontogenia-cini
• License: Apache 2.0
• Maintained by the University of Bologna and ISTC-CNR.

CQ4OE (Ontology Generation)

A benchmark for the systematic and reproducible evaluation of LLM-based ontology generation from competency questions, with CQ-aligned gold standards and fine-grained CQ-to-term and CQ-to-axiom provenance. It defines the CQ2Term (99 CQs) and CQ2Onto (118 CQs) tasks over six source ontologies.

Citation: Li, J., Wang, Z., Garijo, D., & Poveda-Villalón, M. (2026). CQ4OE: A benchmark for assessing LLM-assisted ontology generation from competency questions [Computer software]. https://doi.org/10.5281/zenodo.20080309

• Code: github.com/oeg-upm/cq4oe-benchmark
• Zenodo: 10.5281/zenodo.20080309   Hugging Face: 10.57967/hf/8712
• Metric definitions: w3id.org/cq4oe/metrics
• License: Apache 2.0
• Maintained by the Polytechnic University of Madrid and the OEG group.