Tutorial Schedule

This 3-hour tutorial will be split into three parts:

Part I. Reproducibility and Provenance Primer

We start our tutorial by introducing a succinct terminology to characterize reproducibility based on a review of the various definitions provided in the literature. Specifically, we retained four terms that cover the different levels of reproducibility that can be achieved and/or desired, which we describe hereafter.

• Repeat. An experiment is said to be repeated when it is performed in the same lab (or computation environment) as the original experiment, that is, on the same scientific environment. The major goal of the repeat task is to check whether the initial experiment was correct and can be performed again.
• Replicate. An experiment is said to be replicated when it is performed in a different lab (or computational environment) than the original experiment. When replicated, a result has a high level of robustness compared with the repeat case: the result remains valid in a similar (even though different) setting.
• Reproduce. Reproduce is defined in the broadest possible sense of the term and denotes the situation where an experiment is performed within a different set-up but with the aim to validate the same scientific hypothesis. In other words, completely different approaches can be designed, completely different data sets can be used, as long as both experiments converge to the same scientific conclusion.
• Reuse. A last very important concept related to reproducibility is Reuse which denotes the case where a different experiment is performed, with similarities with an original experiment. A specific kind of reuse occurs when a single experiment is reused in a new context (and thus adapted to new needs), the experiment is then said to be repurposed.

In addition, we will introduce notions of provenance using the W7 model involving Why, Who, When, Where, Which, Who, How. We will conclude by presenting the W3C PROV model in details to demonstrate the critical role of provenance-based solutions in scientific reproducibility.

Part II. W3C PROV-Based Solutions for Facilitating Reproducibility

In the second part, we present provenance-based solutions that have been developed by the semantic web community for fostering the reproducibility of scientific research. Such solutions cover a large spectrum of uses that facilitate scientific reproducibility. For example, provenance can be used for partially automating the annotation, summarization and reporting of experiment results. Indeed, provenance has been used as a means for inferring annotations on the artifacts (data) used and generated by experiments, as well as partially automating the reporting the results of the experiment.

Provenance can be used also for comparison of experiments. In particular, provenance has been used to compare the executions of experiments that are instances of the same pipeline (workflows). This can be used to check (at least) the repeatability of experiments. Provenance can also be utilized for debugging experiments by helping users identifying the pinpointing the steps (modules) and/or datasets that are responsible for errors in computational experiments. Provenance has also been used in versioning experiments, in reducing the time and resources required for experiment re-execution using smart reruns, and to reuse experiments or part thereof.

We will present in the tutorial concrete examples that showcase the above uses of provenance, and discuss how they promote scientific reproducibility. As well as exposing the used of provenance for facilitating reproducibility, we will also discuss proposals/initiatives that seeks to automatically record provenance for reproducibility purposes, e.g., ReproZip, and ResearchObjects.

Part III. Where Are We Today? Challenges in Scientific Reproducibility

The last part of the tutorial will focus on drawing a map of the state of the art (presented in the second part), highlighting how these solutions cater for specific aspects of reproducibility. We will also discuss the role of provenance in international reproducibility-related efforts with a particular focus on FAIR, RDA, DataONE. Finally, we will discuss open issues (opportunities and challenges) that need to be tackled and the instrumental role that the semantic web community can play in this critical aspect of scientific research.