How can a test script inform R CMD check that it should emit a custom message?

I'm writing a R package (delftfews) here at office. we are using svUnit for unit testing.

our process for describing new functionality: we define new unit tests, initially marked as DEACTIVATED; one block of tests at a time we activate them and implement the function described by the tests. almost all the time we have a small amount of DEACTIVATED tests, relative to functions that might be dropped or will be implemented.

my problem/question is: can I alter the doSvUnit.R so that R CMD check pkg emits a NOTE (i.e. a custom message "NOTE" instead of "OK") in case there are DEACTIVATED tests?

as of now, we see only that the active tests don't give error:

.
.
* checking for unstated dependencies in tests ... OK
* checking tests ...
  Running ‘doSvUnit.R’
 OK
* checking PDF version of manual ... OK

which is all right if all tests succeed, but less a开发者_StackOverflow中文版ll right if there are skipped tests and definitely wrong if there are failing tests. In this case, I'd actually like to see a NOTE or a WARNING like the following:

.
.
* checking for unstated dependencies in tests ... OK
* checking tests ...
  Running ‘doSvUnit.R’
 NOTE
6 test(s) were skipped.
 WARNING
1 test(s) are failing.
* checking PDF version of manual ... OK

As of now, we have to open the doSvUnit.Rout to check the real test results.

I contacted two of the maintainers at r-forge and CRAN and they pointed me to the sources of R, in particular the testing.R script.

if I understand it correctly, to answer this question we need patching the tools package:

• scripts in the tests directory are called using a system call,
• output (stdout and stderr) go to one single file,
• there are two possible outcomes: ok or not ok,

so I opened a change request on R, proposing something like bit-coding the return status, bit-0 for ERROR (as it is now), bit-1 for WARNING, bit-2 for NOTE.

with my modification, it would be easy producing this output:

.
.
* checking for unstated dependencies in tests ... OK
* checking tests ...
  Running ‘doSvUnit.R’
 NOTE - please check doSvUnit.Rout.
 WARNING - please check doSvUnit.Rout.
* checking PDF version of manual ... OK

Brian Ripley replied "There are however several packages with properly written unit tests that do signal as required. Please do take this discussion elsewhere: R-bugs is not the place to ask questions." and closed the change request.

anybody has hints?

You should be able to alter the doSvUnit.R script to at least emit warnings and errors as you describe. What you want to do is to run the tests and then inspect the return value of the test runner and have R code that calls, warning() or stop().

For an example of how this was done using RUnit, take a look at the codetoolsBioC package in the Bioconductor repository. The relevant code is in inst/templates and copied below:

.test <- function(dir, pattern = ".*_test\\.R$")
{
    .failure_details <- function(result) {
        res <- result[[1L]]
        if (res$nFail > 0 || res$nErr > 0) {
            Filter(function(x) length(x) > 0,
                   lapply(res$sourceFileResults,
                          function(fileRes) {
                              names(Filter(function(x) x$kind != "success",
                                           fileRes))
                          }))
        } else list()
    }

    if (missing(dir)) {
        dir <- system.file("unitTests", package="@PKG@")
        if (!nzchar(dir)) {
            dir <- system.file("UnitTests", package="@PKG@")
            if (!nzchar(dir))
                stop("unable to find unit tests, no 'unitTests' dir")
        }
    }

    ## Run unit tests from the directory containing the test files.
    ## This allows tests to refer to data files with relative paths
    cwd <- getwd()
    on.exit(setwd(cwd))
    setwd(dir)

    require("RUnit", quietly=TRUE) || stop("RUnit package not found")
    RUnit_opts <- getOption("RUnit", list())
    RUnit_opts$verbose <- 0L
    RUnit_opts$silent <- TRUE
    RUnit_opts$verbose_fail_msg <- TRUE
    options(RUnit = RUnit_opts)
    suite <- defineTestSuite(name="@PKG@ RUnit Tests", dirs=getwd(),
                             testFileRegexp=pattern,
                             rngKind="default",
                             rngNormalKind="default")
    result <- runTestSuite(suite)
    cat("\n\n")
    printTextProtocol(result, showDetails=FALSE)
    if (length(details <- .failure_details(result)) >0) {
        cat("\nTest files with failing tests\n")
        for (i in seq_along(details)) {
            cat("\n  ", basename(names(details)[[i]]), "\n")
            for (j in seq_along(details[[i]])) {
                cat("    ", details[[i]][[j]], "\n")
            }
        }
        cat("\n\n")
        stop("unit tests failed for package @PKG@")
    }
    result
}

I contacted two of the maintainers at r-forge and CRAN and they pointed me to the sources of R, in particular the check.R script.

if I understand it correctly:

• scripts in the tests directory are called using a system call,
• output (stdout and stderr) go to one single file,
• there are two possible outcomes: ok or not ok,
• to answer this question we need patching the library/tools package.

I opened a change request on R, my first guess is something like bit-coding the return status, bit-0 for ERROR (as it is now), bit-1 for WARNING, bit-2 for NOTE.

from doSvUnit.R, I would quit with status 2 in case of failures and 4 in case of skipped tests.

the patch would look like this:

Index: src/library/tools/R/testing.R
===================================================================
--- src/library/tools/R/testing.R   (revision 57214)
+++ src/library/tools/R/testing.R   (working copy)
@@ -352,10 +352,16 @@
         } else
             cmd <- paste("LANGUAGE=C", "R_TESTS=startup.Rs", cmd)
         res <- system(cmd)
-        if (res) {
+        if (res%/%4 %% 2) {
+            message("NOTE")
+        }
+        if (res%/%2 %% 2) {
+            message("WARNING")
+        }
+        if (res %% 2) {
             file.rename(outfile, paste(outfile, "fail", sep="."))
             return(1L)
         }
         savefile <- paste(outfile, "save", sep = "." )
         if (file.exists(savefile)) {
             message("  Comparing ", sQuote(outfile), " to ",

unpatched R sees anything different from 0 as ERROR.