Related Works

Source Benchmarks

The challenge reuses two existing benchmarks, referenced here rather than re-published.

Bench4KE (CQ Generation)

An extensible, API-based benchmarking system for knowledge engineering automation, focused on evaluating CQ generation. It provides 843 manually authored reference CQs from 17 real-world ontology engineering projects and a suite of lexical and semantic similarity metrics. For the challenge, the evaluation metrics have been finalized. The selected metrics have been extended with additional metrics from AskCQ. We refer the reader to the references below for a detailed discussion of the metrics and the benchmark.

Citations

Alharbi, R., Tamma, V., Payne, T. R., & de Berardinis, J. (2026). A Comparative Study of Competency Question Elicitation Methods from Ontology Requirements. In Acosta, M., et al., The Semantic Web. ESWC 2026. Lecture Notes in Computer Science, vol 16549. Springer, Cham. doi.org/10.1007/978-3-032-25156-5_4

Ciancarini, P., Lippolis, A. S., Nuzzolese, A. G., Presutti, V., & Ragagni, M. D. (2026). Bench4KE: Benchmarking Automated Competency Question Generation. In European Semantic Web Conference (pp. 212–231). Cham: Springer Nature Switzerland.

CQ4OE (Ontology Generation)

A benchmark for the systematic and reproducible evaluation of LLM-based ontology generation from competency questions, with CQ-aligned reference ontologies and fine-grained CQ-to-term and CQ-to-axiom provenance. It defines the CQ2Term (99 CQs) and CQ2Onto (118 CQs) tasks over six source ontologies.

Citation: Li, J., Wang, Z., Garijo, D., & Poveda-Villalón, M. (2026). CQ4OE: A benchmark for assessing LLM-assisted ontology generation from competency questions [Computer software]. https://doi.org/10.5281/zenodo.20080309