Submission

How to Submit

Submissions are made through a pull request to the challenge-catalog repository on Codeberg. Follow the seven steps below, you can submission your result to join the competition.

Fork the repository
Fork ke-automation-challenge/challenge-catalog. Everything for this edition lives in the folder keac-2026.
Pick your task folder
Inside keac-2026 there is one folder per task: CQ2Onto, CQ2Term and CQGen. Your submission for a task goes into that task's folder.
Create your submission folder
Under each task folder you take part in, add a folder named after your submission_name. Use the same name in every task.
Add one folder per submission
Each submission is an independent folder inside your submission_name folder. It must contain a metadata.yaml file that describes the submission (see Metadata description).
Lay out your results by domain
Inside the submission folder, create one folder for every domain you take part in, named exactly as the domain. A domain folder contains only your result file, in the format given in What is in your submission and in the task instructions.
Open the pull request
That is your submission. Challenge reviewer reviews the pull request to make sure everything is well-formatted; nothing is evaluated before this review.
Merge, evaluate, publish
Once the pull request is merged, the automatic evaluation is triggered and the results are added to this website.

What happens after you open the pull request

1 · Review

An organizer checks the folder layout, metadata.yaml and file formats.

2 · Merge

Merging the pull request triggers the automatic evaluation.

3 · Publish

Scores appear on the leaderboard of this website.

Rules at a glance

Any LLM used must be an open model with fewer than 30 billion parameters; give its Hugging Face ID in base_model.
Each team may open at most 3 pull requests.
You may update your submission before the deadline by pushing to the same pull request; only the latest version is evaluated.
Every submission is public from the moment the pull request is opened.

What Is in Your Submission

Submissions are organised by edition, task, submission_name and submissions. Here is layout of your forked folder before the pull request. The complete layout of a submitted folder looks like this:

challenge-catalog/
└── keac-2026/
    ├── CQ2Onto/                          one folder per task
    │   └── <submission_name>/            submission_name in metadata.yaml
    │       ├── <submission-1>/             one folder per submission
    │       │   ├── metadata.yaml
    │       │   ├── awo/
    │       │   │   └── awo_ontology.owl      only your result file
    │       │   └── wine/
    │       │       └── wine_ontology.owl
    │       └── <submission-2>/
    ├── CQ2Term/
    │   └── <submission_name>/
    │       └── <submission-1>/
    │           ├── metadata.yaml
    │           └── awo/
    │               └── awo_cq2terms_terms.json
    └── CQGen/
        └── <submission_name>/
            └── <submission-1>/
                ├── metadata.yaml
                └── submission_output.csv     one file, no domain folders

already in the repositorycreated by you

Result Requirement Per Task

CQ2Onto · requirement <domain>/<domain>_ontology.owl

An ontology written in the OWL 2 language (W3C OWL 2 Web Ontology Language), one file per domain, generated from the competency questions of that domain. Any RDF serialisation that rdflib can parse is accepted (RDF/XML as .owl, .rdf); the file name must start with the domain identifier.

CQ2Term · requirement <domain>/<domain>_cq2terms_terms.json

A JSON list with one object per competency question. Keep id and value exactly as given in the domain's competency-question file; fill class and property with the terms your method predicts (note the singular keys):

[
  {
    "id": "CQ1",
    "value": "Which animal eats which other animal?",
    "class": ["Animal"],
    "property": ["eats"]
  }
]

CQGen · requirement submission_output.csv

A single CSV file placed directly in the submission folder (no domain sub-folders), with one row per generated competency question. Here is what each column contains:

Column	Description	Example
`Project Name`	The project of the dataset row the CQ was generated from.	Polifonia
`Name`	The `Name` of that dataset row; may be empty.	Music Meta Ontology
`Scenario`	The scenario used as input, copied from the dataset row; empty if no scenario was used.	Amy wants to assess the developments of organ builders. This research includes looking into which organs an organ builder worked on …
`Dataset`	The dataset value used as input, copied from the dataset row; empty if no dataset was used.	codice_ente,ente_controllore,punto_prelievo,tipologia_punto_prelievo, …
`Link`	The ontology or PDF URL used as input, copied from the dataset row; the column can be left out when no such input was used.	https://raw.githubusercontent.com/D2KLab/llm4ke/…/swo_merged.owl
`Generated CQs`	One generated competency question per row.	Which organs did the organ builder work on?

Metadata Description

Here you will find the content of the metadata.yaml file that comes along with your submission. For each submission under a task, a metadata.yaml file is needed; otherwise, the submission will not be merged.

Field	Description	Data type
`submission_name`	The name of your submission, consistent across all your submissions regardless of the task you take part in.	String	required
`affiliation_name`	The name of your affiliation.	String	optional
`contact`	The contact email(s) of your team.	List of emails	required
`method_name`	The name of your proposed method.	String	optional
`base_model`	The Hugging Face reference ID of the LLM used in your submission.	String	required
`method_description`	A short description of the proposed method.	String	optional
`domain`	The domains covered by this submission, matching the domain folders it contains.	List of strings	optional
`reproducibility.type`	The type of reproducibility material you are providing.	`"source code"`, `"api"` or `"none"`	required
`reproducibility.documentation`	If reproducibility material is provided, a link to the file describing it (`README.md`).	URL	optional
`reproducibility.entry_point`	A link to the source code of your submission, or to the API.	URL	optional

A complete metadata.yaml looks like this:

submission_name: oeg
affiliation_name: Universidad Politécnica de Madrid
contact:
  - oeg@upm.es
method_name: maseo
base_model: Qwen/Qwen3.8-27B
method_description: >
  Generates the ontology one competency question at a time and merges
  the per-question fragments into a single OWL ontology per domain.
domain:
  - awo
  - wine
reproducibility:
  type: source code  
  documentation: https://github.com/oeg-upm/maseo/blob/main/README.md
  entry_point: https://github.com/oeg-upm/maseo

Evaluation Notice

Submissions are assigned a traffic-light status: Green for reproduced and verified results, Yellow for submissions under review or partially reproducible (for example if only the API is available), and Red for submissions that cannot be reproduced (for example if only the output is provided). Teams submit their report and artifact links through the Google Form. The results of the challenge will be published in the Journal of Web Semantics, and participants are also invited to submit to its Knowledge Engineering Automation special issue (deadline October 10). The best-performing teams will receive a monetary award.

Paper / Report and Author Guidelines

Each participating team submits a short system description or report.

Formatting. Reports must not exceed six pages excluding references, and must use the CEUR LaTeX or Word template. Annexes count towards the page limit. Each paper is reviewed by one or two challenge organisers.
Submission venue. Reports are submitted through the Google Form, with a field for the link to the paper and a link to the pull request.