Results & Leaderboard
Overview
Results are published after each evaluation phase. Each result is linked to the exact task, metric versions, and reproducibility artifacts it was produced from, and is assigned a persistent W3ID. Submissions are scored on the public benchmarks and on a held-out evaluation dataset that is kept private during the edition to prevent data contamination. Each result carries a traffic-light status, and only Green results enter the ranking.
Status Lights
Every published result carries a status light showing how far it could be verified.
- Green — the result was reproduced and verified. It is eligible for ranking and for prizes.
- Yellow — the submission is under review or only partially reproducible, for example if only the API is available.
- Red — the submission could not be reproduced, for example if only the output was provided.
How the Leaderboard Is Built
A result reaches the leaderboard through four steps:
- Submit. A participant submits their system outputs and the code needed to reproduce the run, together with a description of the method and how to run it.
- Evaluate. A reviewer runs the task's official metrics on the submitted outputs to compute the official scores. Any scores a participant reports are treated as a reference only, and the scores used for the leaderboard are the ones computed during review.
- Verify. If code is made available, the reviewer independently re-runs the submission to confirm the outputs and scores can be reproduced. For the Ontology Generation task (CQ4OE), this step also includes inspecting and, where needed, correcting the automatic term alignment before the downstream scores are confirmed. The same applies if an API is made available.
- Publish. Once the result is evaluated, it is assigned a persistent identifier (W3ID) and enters the leaderboard under the team name. Results that are submitted but not yet verified stay visible without a rank.
How Results Compare
Scores can be compared only within the same task, the same configuration, and the same edition. These terms are used as follows.
- Task. A main task of the challenge, either CQ Generation or Ontology Generation.
- Sub-task. A division within a task. Ontology Generation has two sub-tasks, CQ2Term and CQ2Onto.
- Configuration. The evaluation settings used to produce a result, such as thresholds and other parameters. Only runs with the same configuration are ranked together.
- Edition. One yearly instance of the challenge, identified by its year, for example 2026.
- Across tasks. CQ Generation and Ontology Generation use different benchmarks, inputs, and metrics, so their scores are not comparable. There is no combined or overall ranking across the two tasks; each task has its own leaderboard.
- Across configurations. Within a task, only runs that share the same evaluation settings are ranked against each other. A run with different settings forms a separate comparison group.
- Coverage. A system may be evaluated on a single domain or on several. Partial coverage is valid and is shown alongside the result, so that a result covering one domain is not read as if it covered all of them.
2026 Edition Current
CQ Generation Leaderboard
[The leaderboard will be populated once the 2026 evaluation phase runs.]
| Rank | Team | System | Score | Status |
|---|---|---|---|---|
| — | — | — | — | — |
Ontology Generation Leaderboard
| Rank | Team | System | Sub-task | Score | Status |
|---|---|---|---|---|---|
| — | — | — | — | — | — |
The Status column shows the traffic-light status (green, yellow, or red). Only green results are ranked.
Past Editions
As an annual series, completed editions and their results remain available here unchanged. The 2026 edition is the first; past editions will be listed here from 2027 onward.
Each edition is an independent snapshot. An edition uses the benchmark version and datasets fixed for that year, including held-out test data, so scores from different editions are not directly comparable. Every edition is preserved under its own persistent identifier, so the series builds a traceable history that shows how methods evolve over time rather than a single score to be compared year on year.