By Juliane Manitz, Yilong Zhang, and Andy Nicholls | January 7, 2022
There is a large variety of contributed R packages, which can be overwhelming when performing their accuracy assessment. These packages can be developed by anyone and may differ in accuracy. The white paper mentions the possibility to define “trusted resources” to simplify the assessment for some of the R packages.
The idea follows vendor assessments / audits to explore the internal validation practices of the vendor for proprietary software. For open-source software such audits are not logistically feasible. However, based on information available in the open-source domain, it may still be possible to perform a virtual audit of a vendor and their practices. In this context, we encourage the publication of software development life cycles (SDLC) documents, which support the process of risk assessment and provide evidence of software trustworthiness.
Early on, the R Foundation and R Core team have published information about their software development practices in R: Regulatory Compliance and Validation Issues. A Guidance Document for the Use of R in Regulated Clinical Trial Environments. This includes source code management, testing and validation, release cycles, version archiving, qualified personnel, security and disaster recovery. This SDLC document covers core and recommended packages.
Some encouraging examples of contributed packages are:
The tidyverse, which is a commercially supported collection of contributed R packages. Rstudio has provided validation guidance for various packages including tidyverse, tidymodels, r-lib, and gt as well as shiny and rmarkdown. If an organization feels comfortable to list the Rstudio development team as a trusted resource, these SDLC documents could be used to qualify respective packages.
Another early lighthouse project example is stan, which is broadly used in the pharmaceutical industry. The development team has published their Software Development Lifecycle practices. The layout and content of this document very closely follows the original publication by the R consortium.
The members of the R validation executive committee like this trend and would be happy to see more of this. Package developers could consider providing information on SDLC or validation reports as vignettes. Recent package developments like valtools can support such efforts.
Note that the presence of an SDLC document alone is insufficient to provide documented evidence as to why a sponsor trusts the software vendor / creator. As with a physical audit, it remains the responsibility of the sponsor to establish whether such SDLC are actually being followed. Typically, the sponsor would create their own summary document highlighting why they consider the organisation a ‘trusted resource’. Such a document would likely need to explain why they consider the SDLC documentation sufficient, and what evidence has been collected to demonstrate the adherence to the SDLC.