By Juliane Manitz, Douglas Kelkhoff, Eli Miller, and Yilong Zhang | June 9, 2020
Many contributed R packages lack documentation expected in software qualification, which is required within pharma and other regulated industries. For pharma, there are various regulations, which require documentation that demonstrates software is used appropriately and works as expected. Thus, industry needs to establish appropriate requirements for R packages using selected metadata and useful risk metrics.
In context of the R Validation Hub, the R package
riskmetric has been developed, which seeks to take the first steps in identifying metrics and best practices to quantify the quality of R packages. It provides a framework for retrieving package metadata, assessing package metrics, and summarizing the risk that the package might not provide accurate results. A corresponding Shiny app, that can be used to generate package reports using riskmetric, is under development.
In this blog post, we want to illustrate the capabilities and usage of
riskmetric and demonstrate how it could fit into an organizations validation process or its qualified environments.
riskmetric package is not yet on CRAN. Until it is, it can be installed using
devtools directly from GitHub:
devtools::install_github("pharmaR/riskmetric", force = TRUE)
Then, the package can be loaded:
To illustrate how
riskmetric works, a few packages with a wide range of popularity have been selected.
riskmetric(Metrics to evaluate the risk of R packages): Not on CRAN yet
utils(R utility functions): R core package
ggplot2(Create Elegant Data Visualisations Using the Grammar of Graphics): very popular package
Hmisc(Harrell Miscellaneous functions): something more old school
survminer(Drawing Survival Curves using
ggplot2): less popular, but established package
coxrobust(Robust Estimation in Cox Model): oldest R package on CRAN
When referencing a package, riskmetric first looks for installed packages but can also assess packages that have not been installed:
package_tbl <- pkg_ref(c("riskmetric", "utils", "ggplot2", "Hmisc", "survminer", "coxrobust")) package_tbl$survminer
## <pkg_install, pkg_ref> survminer v0.4.7 ## $path ##  "C:/Users/Andy/Documents/R/win-library/4.0/survminer" ## $source ##  "pkg_install" ## $version ##  '0.4.7' ## $name ##  "survminer" ## $bug_reports... ## $bug_reports_host... ## $bug_reports_url... ## $description... ## $downloads... ## $help... ## $help_aliases... ## $maintainer... ## $news... ## $release_date... ## $source_control_url... ## $vignettes... ## $website_urls...
Note that many fields have a trailing
...; riskmetric will evaluate and cache the results of the queries later on. When we call the
pkg_assess() function on each reference, the metrics will be stored and become available. In other words, the necessary package metadata is assessed and an atomic value is added for each assessment and package.
Then, the information is scored in order to estimate associated risk. This final score converts the assessment value into a single numeric score between 0 (poor) and 1 (great). Finally each package’s risk is summarized as a weigthed sum of assessment scores.
For more information, check out the
res <- package_tbl %>% pkg_assess() %>% pkg_score() %>% mutate(risk = summarize_scores(.))
summarize_scores() serves as an example for how a risk score might be derived. Each organization should decide independently how to weight different assessments.
There are many good programming and package development practices that establish a package is well made and maintained:
has_vignettes- Number of published vignettes
has_news- Number of releases with a NEWS update
has_bug_reports_url- Presence of a URL for users to report issues and bugs found in the package.
Community usage is determined based on the number of downloads. This is a useful proxy for community support and adhoc testing done by other developers.
downloads_1yr– Number of downloads from CRAN, Bioconductor, and GitHub in the past year.
Furthermore, the test coverage of a package can provide well established insights on the package accuracy
covr_coverage– Package unit test coverage percentage
Several other metrics are under active development that can interrogate package stability and complexity, e.g.
- Maturity – Package version and overall maturity
- Cyclomatic Complexity – Complexity of the code base itself
In addition to assessing the set of packages used to develop a project,
riskmetric can also be used to assess a package before you introduce it into your development environment. Here is an example reviewing the number of downloads for the
pkg_ref("survminer") %>% assess_downloads_1yr() %>% metric_score()
##  0.6030633 ## attr(,"class") ##  "pkg_metric_downloads_1yr" "pkg_metric" ##  "numeric"
Finally, this information can be used by a system administrator when evaluating the suitability of a package, or when writing a validation report:
Package survminer (v0.4.7) has 227894 downloads in the past year, which converts to a
riskmetricscore of 60.31%.
If you are interested in helping with development or the direction of the package, we are active on GitHub and welcome any contributions. More details can be found in the “Get Involved” section of the readme file for
riskmetric GitHub page.