Submission

How to Submit

Submissions are made in the open through a pull request so that anyone can inspect, reproduce, and build on them.

  1. Fork the challenge-catalog repository [coming soon].
  2. Add your submission under the path for the current edition, task, and team, following the structure below.
  3. Include your system outputs (if possible) and the resources needed for reproducibility.
  4. Open a pull request. A reviewer verifies your submission from the public artifacts before results are published under your team name.

Teams may update their submission at any time before the deadline. Only the latest version is evaluated.

The metadata.yml File

Each submission registers the following information in metadata.yml:

  • organization_id — the reference ID of the organization.
  • method_name — the name of your proposed method.
  • base_model — the reference ID of your selected LLM (open and at most 30B parameters).
  • contact — group name, participant names, and the contact email(s) of your team.
  • method_description — a short description of the method your team proposes.
  • tasks — the tasks you take part in (one or both).
  • domains — the ontology domains you take part in (one or several).
  • reproducibility — reproducibility details:
    • type — the type of reproducibility provided (none, source code, API, or both source code and API).
    • entry_point — the location of the code (repository).
  • documentation — in the README, provide environment setup, installation instructions, requirements, execution commands, how to configure secrets, and the license.

Submission Example

An example is available in the cq4oe-benchmark submissions folder.

  • The example-submission folder contains the submission template for Task 2.
  • The submission guideline is detailed in submission_guideline.md.
  • The pseudo_submission folder contains a test submission built as a dry run. Automated evaluation is implemented, and its output for the dry run is in the report and result folders.

What a Submission Contains

Submissions are organised by edition, task, and team, so that the challenge accumulates a complete, stable history year after year. Each submission contains:

  • code/ — a folder with the source code of the submission.
  • output/ — a folder with all generated results, organised by task and domain:
    • <task>/ — one folder per task, for example CQ2Onto/
    • <task>/<domain>/ — one folder per domain, for example CQ2Onto/awo/
    • <task>/<domain>/<result> — the result file, for example awo_ontology.owl
  • metadata.yml — the metadata for the submission (described above).
  • README — a more detailed description of the submission. (Optional)

Paper / Report and Author Guidelines

Each participating team submits a short system description or report.

  • Formatting. Reports must not exceed six pages excluding references, and must use the CEUR LaTeX or Word template. Annexes count towards the page limit. Each paper is reviewed by one or two challenge organisers.
  • Submission venue. Reports are submitted through the Google Form, with a field for the link to the paper and a link to the pull request.