Reproducibility and Replicability Papers

The annual SIGIR conference is the primary international forum for presenting new research results and demonstrating new systems and techniques in the broad information retrieval (IR) field. The 50th ACM SIGIR conference will be run in-person from July 18th to 24th, 2027, in San Jose, California, USA.

This year, the Reproducibility Track has been renamed the Reproducibility and Replicability Track (hereafter, this track). The new name reflects the complementary roles of reproducibility and replicability in strengthening the reliability and generalizability of scientific findings.

Following the current ACM badging terminology, reproducibility concerns whether a different team can obtain consistent results using the same experimental setup, while replicability concerns whether a different team can obtain consistent findings under different experimental conditions. This track particularly encourages studies that combine both perspectives: first establishing confidence in reported results and then investigating whether the resulting conclusions hold beyond the original setting. High-quality contributions focusing on either reproducibility or replicability are also welcome.

Papers will be peer-reviewed double-anonymously, ensuring that the identities of both authors and reviewers remain concealed. Papers are a maximum of 9 pages plus references; shorter papers are welcome. Accepted papers will be published in the main conference proceedings.

Important Dates (AoE)

What Do Reproducibility and Replicability Papers Look Like?

This track solicits papers that reproduce, replicate, generalise, or analyse prior work impacting information retrieval (as defined in the Full Paper call). Reproducibility (different team, same experimental setup) helps verify reported results, while replicability (different team, different experimental setup) examines whether findings hold beyond the original conditions.

We particularly encourage studies that combine both, while also welcoming high-quality work focusing on either aspect. The emphasis is on generating new research insights about established approaches: examining which assumptions hold up, explaining discrepancies, providing error analyses, or identifying the limits of prior conclusions.

To support future reproducibility and replicability, we expect authors to publicly share the resources associated with their paper, while at the same time acknowledge that this is not always feasible.

Review Criteria

Papers submitted to this track are expected to help establish whether prior research in IR is generalisable beyond the theoretical or experimental settings that the paper(s) being reproduced assume(s). Submissions are welcome on reproducibility and replicability in any area of IR.

Papers submitted to this track must explain:

• Which original work they are reproducing or replicating, clearly identified and cited; their motivation for selecting the original work, and the impact of the original work on the IR community.
• The experimental setup(s) used to reproduce or replicate the original work, including any directions in which the study departs from, generalises, or examines different aspects of the original work.
• The assumptions of the original work that they found to hold up, and those that could not be confirmed. The key aim is to share lessons about which findings from prior work remain reliable, under which conditions, and why.

The key review criteria are:

• Contribution: Does this work provide a novel angle on existing approaches and lead to novel insights for the IR community?
• Motivation: How relevant is the reproduced or replicated work for the IR community, and how impactful are the achieved conclusions?
• Soundness: Is the study sufficiently solid regarding methodology and evaluation?
• Quality of reproduction artefacts: Do the supplementary materials for this submission support ease of reproducibility?

Anonymity

The review process is double-anonymous, and authors are required to take all reasonable steps to preserve the anonymity of their submission. The submission must not include author information; citations or discussion of your own prior work should be written up in third person form.

Desk Rejection Policy

Any of the following may result in a desk rejection:

• Submissions that violate the preprint policy, length, or formatting requirements, or are determined to violate ACM policies on academic dishonesty, including plagiarism, author misrepresentation, falsification, and related misconduct.
• Figures, tables, proofs, appendices, acknowledgements, or any other content after page 9 of the submission.
• Formatting that is not in line with the provided guidelines.
• Addition of authors after the abstract-submission deadline.
• Content determined to have been copied from other sources.
• Any form of academic fraud or dishonesty.
• Lack of topical fit for SIGIR.
• Submissions that are not submitted anonymously.

Submission Guidelines

• Papers must be in English, and in PDF format.
• Length is at most 9 pages (including figures, tables, proofs, appendixes, acknowledgments, and any content except references). Authors are not expected to fill the entire 9 pages if not necessary.
• Unrestricted space for references in the current ACM two-column conference format.
• Suitable LaTeX, Word, and Overleaf templates are available from the ACM Website (use sigconf proceedings template for LaTeX and the Interim Template for Word).
• ACM’s CCS concepts and keywords are not required for review but may be required if accepted and published by the ACM.
• For LaTeX, the following should be used:

\documentclass[sigconf,natbib=true,review]{acmart}

Other Submission Requirements

Submitting to this track indicates your agreement to and compliance with the common ACM and SIGIR 2027 Policies and Requirements. These policies and requirements include (but are not limited to) authorship, ethics, the use of AI, ACM author open access page charges, and at least one author registering for and attending the conference.

Submission System

Submissions must be anonymous and should be submitted electronically via EasyChair (link to the submission site to come).

Reproducibility and Replicability Track Chairs

reproducibility@sigir2027.org

• Craig Macdonald, University of Glasgow, UK
• Zhijing Wu, Beijing Institute of Technology, China