Home
Call for Participation
We invite researchers and developers to take part in the Knowledge Engineering Automation Challenge, an open and community-driven effort to evaluate automated knowledge engineering systems in a reproducible and comparable way. Participation is open and free to everyone. The challenge is co-located with the WOP workshop at ISWC 2026 in Bari, Italy. See Important Dates for the schedule.
Submissions are assigned a traffic-light status: Green for reproduced and verified results, Yellow for submissions under review or partially reproducible (for example if only the API is available), and Red for submissions that cannot be reproduced (for example if only the output is provided). Teams submit their report and artifact links through the Google Form. Accepted submissions will have the opportunity to be published through a fast-track process in a high-impact Semantic Web journal, with more details announced soon.
About the Challenge
Knowledge engineering turns human requirements into ontologies that machines can understand. A common approach is to start with requirements such as competency questions (CQs), the natural-language questions an ontology should answer, and use them to guide development and evaluation. Large language models promise to automate much of this process, but there is no common way to compare their performance, making results difficult to reproduce.
The Knowledge Engineering Automation Challenge addresses this gap by providing shared tasks, datasets, and evaluation rules. Participants choose a task, run their own systems, and submit their outputs, which are scored using common evaluation metrics. All tasks, metrics, and results are described in structured formats and linked to persistent W3IDs, making every submission traceable, reproducible, and citable.
The challenge runs as an annual series. The 2026 edition is the first; future editions build on it, with each edition kept under its own persistent identifier so that the benchmark and its history remain stable over time. Participation is subject to the challenge restrictions, including LLM size limits, task-specific input and output formats, and reproducibility requirements. See Results & Leaderboard for past and current editions.
Goals
- Assess the strengths and weaknesses of automated knowledge engineering systems
- Compare different methods on shared tasks
- Improve evaluation metrics and methodologies
- Encourage communication among developers and researchers
- Above all, advance reproducible and comparable research on knowledge engineering automation
Tasks at a Glance
CQ Generation
Systems generate competency questions from a variety of inputs, evaluated with Bench4KE against a gold standard of expert-authored CQs. See tasks →
Ontology Generation
Systems generate ontologies from competency questions, evaluated with CQ4OE against reference ontologies. See tasks →
How It Works
- Tasks and metrics are collected in the public challenge-catalog repository [coming soon].
- Each task and metric is assigned a persistent W3ID for long-term referencing.
- Participants submit their systems and results, where a reviewer verifies them before publication.
- Governance and decision-making follow the rules in governance.md.