Risk Assessment

The R Validation Hub has develops & maintains useful tools that make the risk assessment approach proposed in our published white paper much easier to adopt for R packages.

Open-source tools

Though these tools likely won’t encapsulate every aspect of your organization’s end-to-end validation pipeline, we are constantly seeking to fill known gaps in the process. Thanks to partnerships with a sleuth of pharma organizations, these tools were designed to leverage industry consensus and provide flexibility for customization when needed. We’re proud that both {riskmetric} and {riskassessment} have claimed membership in the {pharmaverse} suite of packages.


riskmetric package riskassessment app pharmaverse suite


  • {riskmetric} is a framework to quantify an R package’s “risk” by assessing several meaningful metrics designed to evaluate package development best practices, code documentation, community engagement, and development sustainability. Users embrace an overall assessment of the package or rely solely on hand-picked metrics.

  • {riskassessment} is a full-fledged R package delivering a shiny front-end to augment the utility & adoption of {riskmetric}. The application’s goal is to provide a central hub for an organization to review and assess the risk of R packages, providing handy tools and guide rails along the way. Specifically useful features include the ability to manage reviewer privileges, explore package source files hands-on, automate decisions based on pre-set rules, and generate a handy summary report to share.

  • {riskscore} (hex logo not yet designed) is an experimental Github-only “data package” containing risk assessments & scores for every R package on CRAN or Bioconductor. This package exists to establish an easily retrievable trend of risk over time, useful for both riskmetric and riskassessment development workflows.


{riskmetric}

Contributed R packages are developed by anyone & everyone, and may differ in popularity and accuracy. As such, the R Validation Hub developed an R package titled riskmetric whose goal is to assess the risk of contributed R packages.

{riskmetric} has four groups of metric criteria:

  • Unit testing metrics - includes unit test coverage and composite coverage of dependencies
  • Documentation metrics - availability of vignettes, news tracking, example(s) and return object description for exported functions
  • Community engagement - number of downloads, availability of the code in a public repository, formal bug tracking and user interaction
  • Maintainability and reuse - number of active contributors, author / maintainer contacts, and type of license

Note: Even though the quality of software is sometimes measurable, sometimes it is not. For example, assessing the accuracy of a contributed open-source R package should be done outside of {riskmetric}. The term accuracy refers to the risk of an error in the code that, when used, could lead to an incorrect calculation. This incorrect calculation may lead to an incorrect decision during data analysis. The relative impact of an error should be determined by the individual organisation. Thus, impact is not a part of the risk assessment performed by {riskmetric}.

For a comprehensive list of metrics assessed via {riskmetric}, see the current state of our package reference guide or browse the Metric Development Progress GitHub project.

Interested in supporting package development?


{riskassessment}

The app’s main goal is to help those making “package inclusion” requests for validated GxP environments. So, the highest and best of {riskassessment} revolves around two things:

  • Empower members of your organization to embrace their responsibility to assess package risk themselves, prior to making uninformed IT requests like: “please add package xyz to our validated environment”.

  • Establish guide rails that adopt to your organizations validation strategy and use of {riskmetric} which culminates in a report for IT that summarizes each package’s adherence to those inclusion requirements.

The {riskassessment} app achieves that main goal with the following handy offerings:

  • Provides a platform for package exploration without the need to write any custom {riskmetric}
  • Runs {riskmetric} on the same machine with the same environment – creating a central hub for reproducibility
  • Maintains consistent, org-specific settings/options when producing risk outputs
  • Automates a risk-based “decision triage” based on an org-defined set of rules, saving time & effort
  • Manages who’s involved in the review process via user authentication & role management
  • Facilitates and stores user written summaries & communication, on certain packages and/or certain metrics
  • Generates risk summary reports, for sharing with the decision making parties

Below is a screenshot from the application’s current demo app, hosted on shinyapps.io. Feel free to give it a test ride and


riskassessment demo app

Interested in supporting package development?

We could always use extra help / feedback! Please consider one of the following options:


{riskscore}

A data package for cataloging riskmetric results across public repositories. WARNING: Right now, the {riskscore} is in a PoC stage that is not fully operational. With that said, there are several use cases that make the concept of {riskscore} valuable, including (but not limited to) the following: it …

  • Guides more effective discussion around how to summarize risk
  • Helps communicate changes to {riskmetric}’s summarizing algorithm or interpretations of assessment data
  • Aids the {riskmetric} dev team in identifying “edge cases” for analysis and code refinement.
  • Provides a channel to distribute handy tools for building {riskmetric} result data (ie, mimicking how our process for external packages could serve as a useful template for when comparing to internal or private repos).
  • Allows everyone to report risk scores in terms of a “CRAN percentile” instead of just some arbitrary numeric value.
  • Establishes a central repository for package scores, which can be used for many applications, like generating badge scores or trending in a package’s score over time to measure performance.

With this type of data at your finger tips, you can analyze package risk statistics with plots like the following (below), which allocates packages into different subgroups based on developers membership in the tidyverse / pharmaverse and groups defined by “most downloads”.


All CRAN scored


Interested in supporting package development?

We could always use extra help / feedback! Please consider one of the following options: