What we mean by reproducible

Keep original records separate from derived files. Record their identifiers, versions or checksums, collection conditions, units, and data-use permissions. A figure should be traceable to the exact observations and transformations that produced it.

When a value is corrected or a record excluded, preserve the original and attach the change to a reason. This distinguishes an error in collection from one in processing and allows a later reviewer to reconstruct the decision.

The FAIR principles address findability, accessibility, interoperability, and reuse of research objects. Their attention to metadata and provenance supports this record. FAIR also permits access controls where needed; unrestricted publication and reusable documentation are separate decisions.

Source: Wilkinson, M. D., et al.

State the target comparison, independent units, inclusion rules, missing-data treatment, time alignment, and derived variables. Explain when each choice was made and whether it was revised after seeing a result.

“Poor-quality records were removed” is insufficient to reconstruct an analysis. A measurable rule, the affected identifiers, and the number of retained units at each stage make the exclusion inspectable.

A release should specify its inputs, software environment, entry point, and expected outputs. A clean rerun then tests whether the documented computation regenerates those outputs. Assessing a finding with new observations is a separate replication task.

Sensitivity analysis addresses another question: how much does the estimate depend on defensible analytical choices? Silberzahn and colleagues asked multiple teams to analyze the same dataset and found variation in estimated effects across their approaches. The implication for our practice is to expose consequential choices, rather than treating a single workflow as inevitable.

Choose sensitivity checks for a scientific reason and show the estimates they produce. Do not count correlated analyses as independent confirmations or select the variant with the most favorable result. Checks of model assumptions, measurement quality, and generalization address different vulnerabilities.

Sources: National Academies of Sciences, Engineering, and Medicine; Silberzahn, R., et al.

A useful release connects four records: a source manifest naming the original files; an observation table with identifiers, units, timestamps, and valid coverage; a decision log explaining exclusions and transformations; and an estimate table containing the values and uncertainty behind each figure. These records should agree about which experimental units remain in each comparison.

A clean rerun should rebuild those estimates from the declared inputs, with the documented environment and entry point. A separate audit should inspect whether the scoring rule, denominator, exclusions, and comparison answer the intended question. The first check tests the computation; the second tests the argument.

This is the release standard for our computational sleep research. If permissions restrict source data, the release should identify what can be shared and what a reader cannot verify. An undocumented collection decision stays visible as a limitation of the result, even when every later computational step is reproducible.