Introduction to the R Package `riskmetric`

Many contributed R packages lack documentation expected in software qualification, which is required within pharma and other regulated industries. For pharma, there are various regulations, which require documentation that demonstrates software is used appropriately and works as expected. Thus, industry needs to establish appropriate requirements for R packages using selected metadata and useful risk metrics.

In context of the R Validation Hub, the R package riskmetric has been developed, which seeks to take the first steps in identifying metrics and best practices to quantify the quality of R packages. It provides a framework for retrieving package metadata, assessing package metrics, and summarizing the risk that the package might not provide accurate results. A corresponding Shiny app, that can be used to generate package reports using riskmetric, is under development.

In this blog post, we want to illustrate the capabilities and usage of riskmetric and demonstrate how it could fit into an organizations validation process or its qualified environments.

The riskmetric package is not yet on CRAN. Until it is, it can be installed using devtools directly from GitHub:

devtools::install_github("pharmaR/riskmetric", force = TRUE)

Then, the package can be loaded:

library(riskmetric)

To illustrate how riskmetric works, a few packages with a wide range of popularity have been selected.

• riskmetric (Metrics to evaluate the risk of R packages): Not on CRAN yet
• utils (R utility functions): R core package
• ggplot2 (Create Elegant Data Visualisations Using the Grammar of Graphics): very popular package
• Hmisc (Harrell Miscellaneous functions): something more old school
• survminer (Drawing Survival Curves using ggplot2): less popular, but established package
• coxrobust (Robust Estimation in Cox Model): oldest R package on CRAN

When referencing a package, riskmetric first looks for installed packages but can also assess packages that have not been installed:

package_tbl <- pkg_ref(c("riskmetric", "utils", "ggplot2", "Hmisc", "survminer", "coxrobust")) 
package_tbl$survminer

## <pkg_install, pkg_ref> survminer v0.4.7
## $path
##   [1] "C:/Users/Andy/Documents/R/win-library/4.0/survminer"
## $source
##   [1] "pkg_install"
## $version
##   [1] '0.4.7'
## $name
##   [1] "survminer"
## $bug_reports...
## $bug_reports_host...
## $bug_reports_url...
## $description...
## $downloads...
## $help...
## $help_aliases...
## $maintainer...
## $news...
## $release_date...
## $source_control_url...
## $vignettes...
## $website_urls...

Note that many fields have a trailing ...; riskmetric will evaluate and cache the results of the queries later on. When we call the pkg_assess() function on each reference, the metrics will be stored and become available. In other words, the necessary package metadata is assessed and an atomic value is added for each assessment and package.

Then, the information is scored in order to estimate associated risk. This final score converts the assessment value into a single numeric score between 0 (poor) and 1 (great). Finally each package’s risk is summarized as a weigthed sum of assessment scores.

For more information, check out the riskmetric vignette.

res <- package_tbl %>%
  pkg_assess() %>%
  pkg_score() %>%
  mutate(risk = summarize_scores(.))

The function summarize_scores() serves as an example for how a risk score might be derived. Each organization should decide independently how to weight different assessments.

There are many good programming and package development practices that establish a package is well made and maintained:

• has_vignettes - Number of published vignettes
• has_news - Number of releases with a NEWS update
• has_bug_reports_url - Presence of a URL for users to report issues and bugs found in the package.

Community usage is determined based on the number of downloads. This is a useful proxy for community support and adhoc testing done by other developers.

• downloads_1yr – Number of downloads from CRAN, Bioconductor, and GitHub in the past year.

Furthermore, the test coverage of a package can provide well established insights on the package accuracy

• covr_coverage – Package unit test coverage percentage

Several other metrics are under active development that can interrogate package stability and complexity, e.g.

• Maturity – Package version and overall maturity
• Cyclomatic Complexity – Complexity of the code base itself

In addition to assessing the set of packages used to develop a project, riskmetric can also be used to assess a package before you introduce it into your development environment. Here is an example reviewing the number of downloads for the survminer package:

pkg_ref("survminer") %>%
  assess_downloads_1yr() %>% 
  metric_score()

## [1] 0.6030633
## attr(,"class")
## [1] "pkg_metric_downloads_1yr" "pkg_metric"              
## [3] "numeric"

Finally, this information can be used by a system administrator when evaluating the suitability of a package, or when writing a validation report:

survminer (v0.4.7)

Package survminer (v0.4.7) has 227894 downloads in the past year, which converts to a riskmetric score of 60.31%.

If you are interested in helping with development or the direction of the package, we are active on GitHub and welcome any contributions. More details can be found in the “Get Involved” section of the readme file for riskmetric GitHub page.