##############################################################################
##############################################################################
###
### Running command:
###
###   F:\biocbuild\bbs-3.17-bioc\R\bin\R.exe CMD check --no-multiarch --install=check:goSorensen.install-out.txt --library=F:\biocbuild\bbs-3.17-bioc\R\library --no-vignettes --timings goSorensen_1.1.0.tar.gz
###
##############################################################################
##############################################################################


* using log directory 'F:/biocbuild/bbs-3.17-bioc-rtools43/meat/goSorensen.Rcheck'
* using R Under development (unstable) (2022-12-25 r83502 ucrt)
* using platform: x86_64-w64-mingw32 (64-bit)
* R was compiled by
    gcc.exe (GCC) 10.4.0
    GNU Fortran (GCC) 10.4.0
* running under: Windows Server x64 (build 20348)
* using session charset: UTF-8
* using option '--no-vignettes'
* checking for file 'goSorensen/DESCRIPTION' ... OK
* checking extension type ... Package
* this is package 'goSorensen' version '1.1.0'
* package encoding: UTF-8
* checking package namespace information ... OK
* checking package dependencies ... OK
* checking if this is a source package ... OK
* checking if there is a namespace ... OK
* checking for hidden files and directories ... OK
* checking for portable file names ... OK
* checking whether package 'goSorensen' can be installed ... OK
* checking installed package size ... OK
* checking package directory ... OK
* checking 'build' directory ... OK
* checking DESCRIPTION meta-information ... OK
* checking top-level files ... OK
* checking for left-over files ... OK
* checking index information ... OK
* checking package subdirectories ... OK
* checking R files for non-ASCII characters ... OK
* checking R files for syntax errors ... OK
* checking whether the package can be loaded ... OK
* checking whether the package can be loaded with stated dependencies ... OK
* checking whether the package can be unloaded cleanly ... OK
* checking whether the namespace can be loaded with stated dependencies ... OK
* checking whether the namespace can be unloaded cleanly ... OK
* checking dependencies in R code ... OK
* checking S3 generic/method consistency ... OK
* checking replacement functions ... OK
* checking foreign function calls ... OK
* checking R code for possible problems ... OK
* checking Rd files ... OK
* checking Rd metadata ... OK
* checking Rd cross-references ... OK
* checking for missing documentation entries ... OK
* checking for code/documentation mismatches ... OK
* checking Rd \usage sections ... OK
* checking Rd contents ... OK
* checking for unstated dependencies in examples ... OK
* checking contents of 'data' directory ... OK
* checking data for non-ASCII characters ... OK
* checking data for ASCII and uncompressed saves ... OK
* checking files in 'vignettes' ... OK
* checking examples ... OK
Examples with CPU (user + system) or elapsed time > 5s
                 user system elapsed
buildEnrichTable 14.5   1.93   16.62
* checking for unstated dependencies in 'tests' ... OK
* checking tests ...
  Running 'test_gosorensen_funcs.R'
  Running 'test_nonsense_genes.R'
 OK
* checking for unstated dependencies in vignettes ... OK
* checking package vignettes in 'inst/doc' ... OK
* checking running R code from vignettes ... SKIPPED
* checking re-building of vignette outputs ... SKIPPED
* checking PDF version of manual ... OK
* DONE

Status: OK

##############################################################################
##############################################################################
###
### Running command:
###
###   F:\biocbuild\bbs-3.17-bioc\R\bin\R.exe CMD INSTALL goSorensen
###
##############################################################################
##############################################################################


* installing to library 'F:/biocbuild/bbs-3.17-bioc/R/library'
* installing *source* package 'goSorensen' ...
** using staged installation
** R
** data
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (goSorensen)

R Under development (unstable) (2022-12-25 r83502 ucrt) -- "Unsuffered Consequences"
Copyright (C) 2022 The R Foundation for Statistical Computing
Platform: x86_64-w64-mingw32/x64 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(goSorensen)



Attaching package: 'goSorensen'

The following object is masked from 'package:utils':

    upgrade

> 
> # A contingency table of GO terms mutual enrichment
> # between gene lists "atlas" and "sanger":
> data(tab_atlas.sanger_BP3)
> tab_atlas.sanger_BP3
                 Enriched in sanger
Enriched in atlas TRUE FALSE
            TRUE    56    30
            FALSE    1   471
> ?tab_atlas.sanger_BP3
> class(tab_atlas.sanger_BP3)
[1] "table"
> 
> # Sorensen-Dice dissimilarity on this contingency table:
> ?dSorensen
> dSorensen(tab_atlas.sanger_BP3)
[1] 0.2167832
> 
> # Standard error of this Sorensen-Dice dissimilarity estimate:
> ?seSorensen
> seSorensen(tab_atlas.sanger_BP3)
[1] 0.03822987
> 
> # Upper 95% confidence limit for the Sorensen-Dice dissimilarity:
> ?duppSorensen
> duppSorensen(tab_atlas.sanger_BP3)
[1] 0.2796658
> # This confidence limit is based on an assimptotic normal N(0,1)
> # approximation to the distribution of (dSampl - d) / se, where
> # dSampl stands for the sample dissimilarity, d for the true dissimilarity
> # and se for the sample dissimilarity standard error estimate.
> 
> # Upper confidence limit but using a Student's t instead of a N(0,1)
> # (just as an example, not recommended -no theoretical justification)
> df <- sum(tab_atlas.sanger_BP3[1:3]) - 2
> duppSorensen(tab_atlas.sanger_BP3, z.conf.level = qt(1 - 0.95, df))
[1] 0.2803587
> 
> # Upper confidence limit but using a bootstrap approximation
> # to the sampling distribution, instead of a N(0,1)
> set.seed(123)
> duppSorensen(tab_atlas.sanger_BP3, boot = TRUE)
[1] 0.2871182
attr(,"eff.nboot")
[1] 10000
> 
> # Some computations on diverse data structures:
> badConti <- as.table(matrix(c(501, 27, 36, 12, 43, 15, 0, 0, 0),
+                             nrow = 3, ncol = 3,
+                             dimnames = list(c("a1","a2","a3"),
+                                             c("b1", "b2","b3"))))
> tryCatch(nice2x2Table(badConti), error = function(e) {return(e)})
<simpleError in nice2x2Table.table(badConti): Not a 2x2 table>
> 
> incompleteConti <- badConti[1,1:min(2,ncol(badConti)), drop = FALSE]
> incompleteConti
    b1  b2
a1 501  12
> tryCatch(nice2x2Table(incompleteConti), error = function(e) {return(e)})
<simpleError in nice2x2Table.table(incompleteConti): Not a 2x2 table>
> 
> contiAsVector <- c(32, 21, 81, 1439)
> nice2x2Table(contiAsVector)
[1] TRUE
> contiAsVector.mat <- matrix(contiAsVector, nrow = 2)
> contiAsVector.mat
     [,1] [,2]
[1,]   32   81
[2,]   21 1439
> contiAsVectorLen3 <- c(32, 21, 81)
> nice2x2Table(contiAsVectorLen3)
[1] TRUE
> 
> tryCatch(dSorensen(badConti), error = function(e) {return(e)})
<simpleError in nice2x2Table.table(x): Not a 2x2 table>
> 
> # Apparently, the next order works fine, but returns a wrong value!
> dSorensen(badConti, check.table = FALSE)
[1] 0.05915493
> 
> tryCatch(dSorensen(incompleteConti), error = function(e) {return(e)})
<simpleError in nice2x2Table.table(x): Not a 2x2 table>
> dSorensen(contiAsVector)
[1] 0.6144578
> dSorensen(contiAsVector.mat)
[1] 0.6144578
> dSorensen(contiAsVectorLen3)
[1] 0.6144578
> dSorensen(contiAsVectorLen3, check.table = FALSE)
[1] 0.6144578
> dSorensen(c(0,0,0,45))
[1] NaN
> 
> tryCatch(seSorensen(badConti), error = function(e) {return(e)})
<simpleError in nice2x2Table.table(x): Not a 2x2 table>
> tryCatch(seSorensen(incompleteConti), error = function(e) {return(e)})
<simpleError in nice2x2Table.table(x): Not a 2x2 table>
> seSorensen(contiAsVector)
[1] 0.04818012
> seSorensen(contiAsVector.mat)
[1] 0.04818012
> seSorensen(contiAsVectorLen3)
[1] 0.04818012
> seSorensen(contiAsVectorLen3, check.table = FALSE)
[1] 0.04818012
> tryCatch(seSorensen(contiAsVectorLen3, check.table = "not"), error = function(e) {return(e)})
<simpleError in seSorensen.numeric(contiAsVectorLen3, check.table = "not"): Argument 'check.table' must be logical>
> seSorensen(c(0,0,0,45))
[1] NaN
> 
> tryCatch(duppSorensen(badConti), error = function(e) {return(e)})
<simpleError in nice2x2Table.table(x): Not a 2x2 table>
> tryCatch(duppSorensen(incompleteConti), error = function(e) {return(e)})
<simpleError in nice2x2Table.table(x): Not a 2x2 table>
> duppSorensen(contiAsVector)
[1] 0.6937071
> duppSorensen(contiAsVector.mat)
[1] 0.6937071
> set.seed(123)
> duppSorensen(contiAsVector, boot = TRUE)
[1] 0.6922658
attr(,"eff.nboot")
[1] 10000
> set.seed(123)
> duppSorensen(contiAsVector.mat, boot = TRUE)
[1] 0.6922658
attr(,"eff.nboot")
[1] 10000
> duppSorensen(contiAsVectorLen3)
[1] 0.6937071
> # Bootstrapping requires full contingency tables (4 values)
> set.seed(123)
> tryCatch(duppSorensen(contiAsVectorLen3, boot = TRUE), error = function(e) {return(e)})
<simpleError in duppSorensen.numeric(contiAsVectorLen3, boot = TRUE): Bootstraping requires a numeric vector of 4 frequencies>
> duppSorensen(c(0,0,0,45))
[1] NaN
> 
> # Equivalence test, H0: d >= d0 vs  H1: d < d0 (d0 = 0.4444)
> ?equivTestSorensen
> equiv.atlas.sanger <- equivTestSorensen(tab_atlas.sanger_BP3)
> equiv.atlas.sanger

	Normal asymptotic test for 2x2 contingency tables based on the
	Sorensen-Dice dissimilarity

data:  tab_atlas.sanger_BP3
(d - d0) / se = -5.9551, p-value = 1.3e-09
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.2796658
sample estimates:
Sorensen dissimilarity 
             0.2167832 
attr(,"se")
standard error 
    0.03822987 

> getTable(equiv.atlas.sanger)
                 Enriched in sanger
Enriched in atlas TRUE FALSE
            TRUE    56    30
            FALSE    1   471
> getPvalue(equiv.atlas.sanger)
     p-value 
1.299869e-09 
> 
> tryCatch(equivTestSorensen(badConti), error = function(e) {return(e)})
<simpleError in nice2x2Table.table(x): Not a 2x2 table>
> tryCatch(equivTestSorensen(incompleteConti), error = function(e) {return(e)})
<simpleError in nice2x2Table.table(x): Not a 2x2 table>
> equivTestSorensen(contiAsVector)

	Normal asymptotic test for 2x2 contingency tables based on the
	Sorensen-Dice dissimilarity

data:  contiAsVector
(d - d0) / se = 3.5287, p-value = 0.9998
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.6937071
sample estimates:
Sorensen dissimilarity 
             0.6144578 
attr(,"se")
standard error 
    0.04818012 

> equivTestSorensen(contiAsVector.mat)

	Normal asymptotic test for 2x2 contingency tables based on the
	Sorensen-Dice dissimilarity

data:  contiAsVector.mat
(d - d0) / se = 3.5287, p-value = 0.9998
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.6937071
sample estimates:
Sorensen dissimilarity 
             0.6144578 
attr(,"se")
standard error 
    0.04818012 

> set.seed(123)
> equivTestSorensen(contiAsVector.mat, boot = TRUE)

	Bootstrap test for 2x2 contingency tables based on the Sorensen-Dice
	dissimilarity (10000 bootstrap replicates)

data:  contiAsVector.mat
(d - d0) / se = 3.5287, p-value = 0.9996
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.6922658
sample estimates:
Sorensen dissimilarity 
             0.6144578 
attr(,"se")
standard error 
    0.04818012 

> equivTestSorensen(contiAsVectorLen3)

	Normal asymptotic test for 2x2 contingency tables based on the
	Sorensen-Dice dissimilarity

data:  contiAsVectorLen3
(d - d0) / se = 3.5287, p-value = 0.9998
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.6937071
sample estimates:
Sorensen dissimilarity 
             0.6144578 
attr(,"se")
standard error 
    0.04818012 

> 
> tryCatch(equivTestSorensen(contiAsVectorLen3, boot = TRUE), error = function(e) {return(e)})
<simpleError in equivTestSorensen.numeric(contiAsVectorLen3, boot = TRUE): Bootstraping requires a numeric vector of 4 frequencies>
> 
> equivTestSorensen(c(0,0,0,45))

	No test performed due non finite (d - d0) / se statistic

data:  c(0, 0, 0, 45)
(d - d0) / se = NaN, p-value = NA
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
   0 NaN
sample estimates:
Sorensen dissimilarity 
                   NaN 
attr(,"se")
standard error 
           NaN 

> 
> # Sorensen-Dice computations from scratch, directly from gene lists
> data(allOncoGeneLists)
> ?allOncoGeneLists
> data(humanEntrezIDs)
> # First, the mutual GO node enrichment tables are built, then computations
> # proceed from these contingency tables.
> # Building the contingency tables is a slow process (many enrichment tests)
> normTest <- equivTestSorensen(allOncoGeneLists[["atlas"]], allOncoGeneLists[["sanger"]],
+                               listNames = c("atlas", "sanger"),
+                               onto = "BP", GOLevel = 5,
+                               geneUniverse = humanEntrezIDs, orgPackg = "org.Hs.eg.db")
> normTest

	Normal asymptotic test for 2x2 contingency tables based on the
	Sorensen-Dice dissimilarity

data:  tab
(d - d0) / se = -8.0329, p-value = 4.758e-16
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.3567572
sample estimates:
Sorensen dissimilarity 
             0.3341788 
attr(,"se")
standard error 
    0.01372669 

> 
> # To perform a bootstrap test from scratch would be even slower:
> # set.seed(123)
> # bootTest <- equivTestSorensen(allOncoGeneLists[["atlas"]], allOncoGeneLists[["sanger"]],
> #                               listNames = c("atlas", "sanger"),
> #                               boot = TRUE,
> #                               onto = "BP", GOLevel = 5,
> #                               geneUniverse = humanEntrezIDs, orgPackg = "org.Hs.eg.db")
> # bootTest
> 
> # It is much faster to upgrade 'normTest' to be a bootstrap test:
> set.seed(123)
> bootTest <- upgrade(normTest, boot = TRUE)
> bootTest

	Bootstrap test for 2x2 contingency tables based on the Sorensen-Dice
	dissimilarity (10000 bootstrap replicates)

data:  tab
(d - d0) / se = -8.0329, p-value = 9.999e-05
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.000000 0.357055
sample estimates:
Sorensen dissimilarity 
             0.3341788 
attr(,"se")
standard error 
    0.01372669 

> # To know the number of planned bootstrap replicates:
> getNboot(bootTest)
[1] 10000
> # To know the number of valid bootstrap replicates:
> getEffNboot(bootTest)
[1] 10000
> 
> # There are similar methods for dSorensen, seSorensen, duppSorensen, etc. to
> # compute directly from a pair of gene lists.
> # They are quite slow for the same reason as before (many enrichment tests).
> # dSorensen(allOncoGeneLists[["atlas"]], allOncoGeneLists[["sanger"]],
> #           listNames = c("atlas", "sanger"),
> #           onto = "BP", GOLevel = 5,
> #           geneUniverse = humanEntrezIDs, orgPackg = "org.Hs.eg.db")
> # seSorensen(allOncoGeneLists[["atlas"]], allOncoGeneLists[["sanger"]],
> #            listNames = c("atlas", "sanger"),
> #            onto = "BP", GOLevel = 5,
> #            geneUniverse = humanEntrezIDs, orgPackg = "org.Hs.eg.db")
> #
> # duppSorensen(allOncoGeneLists[["atlas"]], allOncoGeneLists[["sanger"]],
> #              listNames = c("atlas", "sanger"),
> #              onto = "BP", GOLevel = 5,
> #              geneUniverse = humanEntrezIDs, orgPackg = "org.Hs.eg.db")
> #
> # set.seed(123)
> # duppSorensen(allOncoGeneLists[["atlas"]], allOncoGeneLists[["sanger"]],
> #              boot = TRUE,
> #              listNames = c("atlas", "sanger"),
> #              onto = "BP", GOLevel = 5,
> #              geneUniverse = humanEntrezIDs, orgPackg = "org.Hs.eg.db")
> # etc.
> 
> # To build the contingency table first and then compute from it, may be a more flexible
> # and saving time strategy, in general:
> ?buildEnrichTable
> tab <- buildEnrichTable(allOncoGeneLists[["atlas"]], allOncoGeneLists[["sanger"]],
+                         listNames = c("atlas", "sanger"),
+                         onto = "BP", GOLevel = 5,
+                         geneUniverse = humanEntrezIDs, orgPackg = "org.Hs.eg.db")
> 
> tab
                 Enriched in sanger
Enriched in atlas TRUE FALSE
            TRUE   525   477
            FALSE   50  9096
> 
> # (Here, an obvious faster possibility would be to recover the enrichment contingency
> # table from the previous normal test result:)
> tab <- getTable(normTest)
> tab
                 Enriched in sanger
Enriched in atlas TRUE FALSE
            TRUE   525   477
            FALSE   50  9096
> 
> tst <- equivTestSorensen(tab)
> tst

	Normal asymptotic test for 2x2 contingency tables based on the
	Sorensen-Dice dissimilarity

data:  tab
(d - d0) / se = -8.0329, p-value = 4.758e-16
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.3567572
sample estimates:
Sorensen dissimilarity 
             0.3341788 
attr(,"se")
standard error 
    0.01372669 

> set.seed(123)
> bootTst <- equivTestSorensen(tab, boot = TRUE)
> bootTst

	Bootstrap test for 2x2 contingency tables based on the Sorensen-Dice
	dissimilarity (10000 bootstrap replicates)

data:  tab
(d - d0) / se = -8.0329, p-value = 9.999e-05
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.000000 0.357055
sample estimates:
Sorensen dissimilarity 
             0.3341788 
attr(,"se")
standard error 
    0.01372669 

> 
> dSorensen(tab)
[1] 0.3341788
> seSorensen(tab)
[1] 0.01372669
> # or:
> getDissimilarity(tst)
Sorensen dissimilarity 
             0.3341788 
attr(,"se")
standard error 
    0.01372669 
> 
> duppSorensen(tab)
[1] 0.3567572
> getUpper(tst)
   dUpper 
0.3567572 
> 
> set.seed(123)
> duppSorensen(tab, boot = TRUE)
[1] 0.357055
attr(,"eff.nboot")
[1] 10000
> getUpper(bootTst)
  dUpper 
0.357055 
> 
> # To perform from scratch all pairwise tests (or other Sorensen-Dice computations)
> # is even much slower. For example, all pairwise...
> # Dissimilarities:
> # # allPairDiss <- dSorensen(allOncoGeneLists,
> # #                          onto = "BP", GOLevel = 5,
> # #                          geneUniverse = humanEntrezIDs, orgPackg = "org.Hs.eg.db")
> # # allPairDiss
> #
> # # Still time consuming but faster: build all tables computing in parallel:
> # allPairDiss <- dSorensen(allOncoGeneLists,
> #                          onto = "BP", GOLevel = 5,
> #                          geneUniverse = humanEntrezIDs, orgPackg = "org.Hs.eg.db",
> #                          parallel = TRUE)
> # allPairDiss
> 
> # Standard errors:
> # seSorensen(allOncoGeneLists,
> #            onto = "BP", GOLevel = 5,
> #            geneUniverse = humanEntrezIDs, orgPackg = "org.Hs.eg.db")
> #
> # Upper confidence interval limits:
> # duppSorensen(allOncoGeneLists,
> #              onto = "BP", GOLevel = 5,
> #              geneUniverse = humanEntrezIDs, orgPackg = "org.Hs.eg.db")
> # All pairwise asymptotic normal tests:
> # allTests <- equivTestSorensen(allOncoGeneLists,
> #                               onto = "BP", GOLevel = 5,
> #                               geneUniverse = humanEntrezIDs, orgPackg = "org.Hs.eg.db")
> # getPvalue(allTests, simplify = FALSE)
> # getPvalue(allTests)
> # p.adjust(getPvalue(allTests), method = "holm")
> # To perform all pairwise bootstrap tests from scratch is (slightly)
> # even more time consuming:
> # set.seed(123)
> # allBootTests <- equivTestSorensen(allOncoGeneLists,
> #                                   boot = TRUE,
> #                                   onto = "BP", GOLevel = 5,
> #                                   geneUniverse = humanEntrezIDs, orgPackg = "org.Hs.eg.db")
> # Not all bootstrap replicates may conduct to finite statistics:
> # getNboot(allBootTests)
> 
> # Given the normal tests (object 'allTests'), it is much faster to upgrade
> # it to have the bootstrap tests:
> # set.seed(123)
> # allBootTests <- upgrade(allTests, boot = TRUE)
> # getPvalue(allBootTests, simplify = FALSE)
> 
> # Again, the faster and more flexible possibility may be:
> # 1) First, build all pairwise enrichment contingency tables (slow first step):
> # allTabsBP.4 <- buildEnrichTable(allOncoGeneLists,
> #                                 onto = "BP", GOLevel = 5,
> #                                 geneUniverse = humanEntrezIDs, orgPackg = "org.Hs.eg.db")
> # allTabsBP.4
> 
> # Better, directly use the dataset available at this package, goSorensen:
> data(allTabsBP.4)
> allTabsBP.4
$cangenes
$cangenes$atlas
                    Enriched in atlas
Enriched in cangenes TRUE FALSE
               TRUE     1     0
               FALSE  959  9397


$cis
$cis$atlas
               Enriched in atlas
Enriched in cis TRUE FALSE
          TRUE   181    11
          FALSE  779  9386

$cis$cangenes
               Enriched in cangenes
Enriched in cis  TRUE FALSE
          TRUE      1   191
          FALSE     0 10165


$miscellaneous
$miscellaneous$atlas
                         Enriched in atlas
Enriched in miscellaneous TRUE FALSE
                    TRUE   450    57
                    FALSE  510  9340

$miscellaneous$cangenes
                         Enriched in cangenes
Enriched in miscellaneous TRUE FALSE
                    TRUE     1   506
                    FALSE    0  9850

$miscellaneous$cis
                         Enriched in cis
Enriched in miscellaneous TRUE FALSE
                    TRUE   145   362
                    FALSE   47  9803


$sanger
$sanger$atlas
                  Enriched in atlas
Enriched in sanger TRUE FALSE
             TRUE   500    45
             FALSE  460  9352

$sanger$cangenes
                  Enriched in cangenes
Enriched in sanger TRUE FALSE
             TRUE     1   544
             FALSE    0  9812

$sanger$cis
                  Enriched in cis
Enriched in sanger TRUE FALSE
             TRUE   153   392
             FALSE   39  9773

$sanger$miscellaneous
                  Enriched in miscellaneous
Enriched in sanger TRUE FALSE
             TRUE   359   186
             FALSE  148  9664


$Vogelstein
$Vogelstein$atlas
                      Enriched in atlas
Enriched in Vogelstein TRUE FALSE
                 TRUE   542    76
                 FALSE  418  9321

$Vogelstein$cangenes
                      Enriched in cangenes
Enriched in Vogelstein TRUE FALSE
                 TRUE     1   617
                 FALSE    0  9739

$Vogelstein$cis
                      Enriched in cis
Enriched in Vogelstein TRUE FALSE
                 TRUE   163   455
                 FALSE   29  9710

$Vogelstein$miscellaneous
                      Enriched in miscellaneous
Enriched in Vogelstein TRUE FALSE
                 TRUE   374   244
                 FALSE  133  9606

$Vogelstein$sanger
                      Enriched in sanger
Enriched in Vogelstein TRUE FALSE
                 TRUE   512   106
                 FALSE   33  9706


$waldman
$waldman$atlas
                   Enriched in atlas
Enriched in waldman TRUE FALSE
              TRUE   641   138
              FALSE  319  9259

$waldman$cangenes
                   Enriched in cangenes
Enriched in waldman TRUE FALSE
              TRUE     1   778
              FALSE    0  9578

$waldman$cis
                   Enriched in cis
Enriched in waldman TRUE FALSE
              TRUE   171   608
              FALSE   21  9557

$waldman$miscellaneous
                   Enriched in miscellaneous
Enriched in waldman TRUE FALSE
              TRUE   467   312
              FALSE   40  9538

$waldman$sanger
                   Enriched in sanger
Enriched in waldman TRUE FALSE
              TRUE   446   333
              FALSE   99  9479

$waldman$Vogelstein
                   Enriched in Vogelstein
Enriched in waldman TRUE FALSE
              TRUE   488   291
              FALSE  130  9448


attr(,"class")
[1] "tableList" "list"     
> class(allTabsBP.4)
[1] "tableList" "list"     
> # 2) Then perform all required computatios from these enrichment contingency tables...
> # All pairwise tests:
> allTests <- equivTestSorensen(allTabsBP.4)
> allTests
$cangenes
$cangenes$atlas

	Normal asymptotic test for 2x2 contingency tables based on the
	Sorensen-Dice dissimilarity

data:  tab
(d - d0) / se = 266.22, p-value = 1
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0 1
sample estimates:
Sorensen dissimilarity 
             0.9979188 
attr(,"se")
standard error 
      0.002079 



$cis
$cis$atlas

	Normal asymptotic test for 2x2 contingency tables based on the
	Sorensen-Dice dissimilarity

data:  tab
(d - d0) / se = 13.583, p-value = 1
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.7149878
sample estimates:
Sorensen dissimilarity 
             0.6857639 
attr(,"se")
standard error 
    0.01776688 


$cis$cangenes

	Normal asymptotic test for 2x2 contingency tables based on the
	Sorensen-Dice dissimilarity

data:  tab
(d - d0) / se = 52.885, p-value = 1
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0 1
sample estimates:
Sorensen dissimilarity 
             0.9896373 
attr(,"se")
standard error 
      0.010309 



$miscellaneous
$miscellaneous$atlas

	Normal asymptotic test for 2x2 contingency tables based on the
	Sorensen-Dice dissimilarity

data:  tab
(d - d0) / se = -3.8685, p-value = 5.474e-05
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.000000 0.411139
sample estimates:
Sorensen dissimilarity 
             0.3865031 
attr(,"se")
standard error 
    0.01497758 


$miscellaneous$cangenes

	Normal asymptotic test for 2x2 contingency tables based on the
	Sorensen-Dice dissimilarity

data:  tab
(d - d0) / se = 140.39, p-value = 1
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0 1
sample estimates:
Sorensen dissimilarity 
              0.996063 
attr(,"se")
standard error 
   0.003929258 


$miscellaneous$cis

	Normal asymptotic test for 2x2 contingency tables based on the
	Sorensen-Dice dissimilarity

data:  tab
(d - d0) / se = 5.9904, p-value = 1
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.000000 0.623749
sample estimates:
Sorensen dissimilarity 
             0.5851216 
attr(,"se")
standard error 
    0.02348381 



$sanger
$sanger$atlas

	Normal asymptotic test for 2x2 contingency tables based on the
	Sorensen-Dice dissimilarity

data:  tab
(d - d0) / se = -7.738, p-value = 5.051e-15
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.3586962
sample estimates:
Sorensen dissimilarity 
             0.3355482 
attr(,"se")
standard error 
    0.01407301 


$sanger$cangenes

	Normal asymptotic test for 2x2 contingency tables based on the
	Sorensen-Dice dissimilarity

data:  tab
(d - d0) / se = 150.94, p-value = 1
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0 1
sample estimates:
Sorensen dissimilarity 
              0.996337 
attr(,"se")
standard error 
   0.003656295 


$sanger$cis

	Normal asymptotic test for 2x2 contingency tables based on the
	Sorensen-Dice dissimilarity

data:  tab
(d - d0) / se = 6.1374, p-value = 1
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.6224203
sample estimates:
Sorensen dissimilarity 
             0.5848033 
attr(,"se")
standard error 
    0.02286957 


$sanger$miscellaneous

	Normal asymptotic test for 2x2 contingency tables based on the
	Sorensen-Dice dissimilarity

data:  tab
(d - d0) / se = -7.701, p-value = 6.75e-15
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.3446065
sample estimates:
Sorensen dissimilarity 
             0.3174905 
attr(,"se")
standard error 
    0.01648539 



$Vogelstein
$Vogelstein$atlas

	Normal asymptotic test for 2x2 contingency tables based on the
	Sorensen-Dice dissimilarity

data:  tab
(d - d0) / se = -9.8173, p-value < 2.2e-16
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.3350683
sample estimates:
Sorensen dissimilarity 
             0.3130545 
attr(,"se")
standard error 
    0.01338347 


$Vogelstein$cangenes

	Normal asymptotic test for 2x2 contingency tables based on the
	Sorensen-Dice dissimilarity

data:  tab
(d - d0) / se = 171.22, p-value = 1
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0 1
sample estimates:
Sorensen dissimilarity 
              0.996769 
attr(,"se")
standard error 
   0.003225798 


$Vogelstein$cis

	Normal asymptotic test for 2x2 contingency tables based on the
	Sorensen-Dice dissimilarity

data:  tab
(d - d0) / se = 7.0238, p-value = 1
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.6333811
sample estimates:
Sorensen dissimilarity 
             0.5975309 
attr(,"se")
standard error 
    0.02179538 


$Vogelstein$miscellaneous

	Normal asymptotic test for 2x2 contingency tables based on the
	Sorensen-Dice dissimilarity

data:  tab
(d - d0) / se = -6.7191, p-value = 9.142e-12
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.3618762
sample estimates:
Sorensen dissimilarity 
             0.3351111 
attr(,"se")
standard error 
    0.01627199 


$Vogelstein$sanger

	Normal asymptotic test for 2x2 contingency tables based on the
	Sorensen-Dice dissimilarity

data:  tab
(d - d0) / se = -32.259, p-value < 2.2e-16
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.1360863
sample estimates:
Sorensen dissimilarity 
             0.1195185 
attr(,"se")
standard error 
     0.0100725 



$waldman
$waldman$atlas

	Normal asymptotic test for 2x2 contingency tables based on the
	Sorensen-Dice dissimilarity

data:  tab
(d - d0) / se = -15.308, p-value < 2.2e-16
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.2823131
sample estimates:
Sorensen dissimilarity 
             0.2627947 
attr(,"se")
standard error 
    0.01186635 


$waldman$cangenes

	Normal asymptotic test for 2x2 contingency tables based on the
	Sorensen-Dice dissimilarity

data:  tab
(d - d0) / se = 215.94, p-value = 1
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0 1
sample estimates:
Sorensen dissimilarity 
             0.9974359 
attr(,"se")
standard error 
   0.002560815 


$waldman$cis

	Normal asymptotic test for 2x2 contingency tables based on the
	Sorensen-Dice dissimilarity

data:  tab
(d - d0) / se = 10.327, p-value = 1
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.6801719
sample estimates:
Sorensen dissimilarity 
             0.6477858 
attr(,"se")
standard error 
    0.01968936 


$waldman$miscellaneous

	Normal asymptotic test for 2x2 contingency tables based on the
	Sorensen-Dice dissimilarity

data:  tab
(d - d0) / se = -12.16, p-value < 2.2e-16
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.2968117
sample estimates:
Sorensen dissimilarity 
              0.273717 
attr(,"se")
standard error 
    0.01404058 


$waldman$sanger

	Normal asymptotic test for 2x2 contingency tables based on the
	Sorensen-Dice dissimilarity

data:  tab
(d - d0) / se = -7.9582, p-value = 8.729e-16
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.3507062
sample estimates:
Sorensen dissimilarity 
              0.326284 
attr(,"se")
standard error 
    0.01484766 


$waldman$Vogelstein

	Normal asymptotic test for 2x2 contingency tables based on the
	Sorensen-Dice dissimilarity

data:  tab
(d - d0) / se = -10.211, p-value < 2.2e-16
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.3244082
sample estimates:
Sorensen dissimilarity 
             0.3013601 
attr(,"se")
standard error 
    0.01401229 



attr(,"class")
[1] "equivSDhtestList" "list"            
> class(allTests)
[1] "equivSDhtestList" "list"            
> set.seed(123)
> allBootTests <- equivTestSorensen(allTabsBP.4, boot = TRUE)
> allBootTests
$cangenes
$cangenes$atlas

	Bootstrap test for 2x2 contingency tables based on the Sorensen-Dice
	dissimilarity (6298 effective bootstrap replicates of 10000)

data:  tab
(d - d0) / se = 266.22, p-value = 1
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0 1
sample estimates:
Sorensen dissimilarity 
             0.9979188 
attr(,"se")
standard error 
      0.002079 



$cis
$cis$atlas

	Bootstrap test for 2x2 contingency tables based on the Sorensen-Dice
	dissimilarity (10000 bootstrap replicates)

data:  tab
(d - d0) / se = 13.583, p-value = 1
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.7143131
sample estimates:
Sorensen dissimilarity 
             0.6857639 
attr(,"se")
standard error 
    0.01776688 


$cis$cangenes

	Bootstrap test for 2x2 contingency tables based on the Sorensen-Dice
	dissimilarity (6342 effective bootstrap replicates of 10000)

data:  tab
(d - d0) / se = 52.885, p-value = 1
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0 1
sample estimates:
Sorensen dissimilarity 
             0.9896373 
attr(,"se")
standard error 
      0.010309 



$miscellaneous
$miscellaneous$atlas

	Bootstrap test for 2x2 contingency tables based on the Sorensen-Dice
	dissimilarity (10000 bootstrap replicates)

data:  tab
(d - d0) / se = -3.8685, p-value = 2e-04
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.4111369
sample estimates:
Sorensen dissimilarity 
             0.3865031 
attr(,"se")
standard error 
    0.01497758 


$miscellaneous$cangenes

	Bootstrap test for 2x2 contingency tables based on the Sorensen-Dice
	dissimilarity (6382 effective bootstrap replicates of 10000)

data:  tab
(d - d0) / se = 140.39, p-value = 1
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0 1
sample estimates:
Sorensen dissimilarity 
              0.996063 
attr(,"se")
standard error 
   0.003929258 


$miscellaneous$cis

	Bootstrap test for 2x2 contingency tables based on the Sorensen-Dice
	dissimilarity (10000 bootstrap replicates)

data:  tab
(d - d0) / se = 5.9904, p-value = 1
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.6245189
sample estimates:
Sorensen dissimilarity 
             0.5851216 
attr(,"se")
standard error 
    0.02348381 



$sanger
$sanger$atlas

	Bootstrap test for 2x2 contingency tables based on the Sorensen-Dice
	dissimilarity (10000 bootstrap replicates)

data:  tab
(d - d0) / se = -7.738, p-value = 9.999e-05
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.3588894
sample estimates:
Sorensen dissimilarity 
             0.3355482 
attr(,"se")
standard error 
    0.01407301 


$sanger$cangenes

	Bootstrap test for 2x2 contingency tables based on the Sorensen-Dice
	dissimilarity (6401 effective bootstrap replicates of 10000)

data:  tab
(d - d0) / se = 150.94, p-value = 1
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0 1
sample estimates:
Sorensen dissimilarity 
              0.996337 
attr(,"se")
standard error 
   0.003656295 


$sanger$cis

	Bootstrap test for 2x2 contingency tables based on the Sorensen-Dice
	dissimilarity (10000 bootstrap replicates)

data:  tab
(d - d0) / se = 6.1374, p-value = 1
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.6222571
sample estimates:
Sorensen dissimilarity 
             0.5848033 
attr(,"se")
standard error 
    0.02286957 


$sanger$miscellaneous

	Bootstrap test for 2x2 contingency tables based on the Sorensen-Dice
	dissimilarity (10000 bootstrap replicates)

data:  tab
(d - d0) / se = -7.701, p-value = 9.999e-05
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.3449603
sample estimates:
Sorensen dissimilarity 
             0.3174905 
attr(,"se")
standard error 
    0.01648539 



$Vogelstein
$Vogelstein$atlas

	Bootstrap test for 2x2 contingency tables based on the Sorensen-Dice
	dissimilarity (10000 bootstrap replicates)

data:  tab
(d - d0) / se = -9.8173, p-value = 9.999e-05
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.3357311
sample estimates:
Sorensen dissimilarity 
             0.3130545 
attr(,"se")
standard error 
    0.01338347 


$Vogelstein$cangenes

	Bootstrap test for 2x2 contingency tables based on the Sorensen-Dice
	dissimilarity (6276 effective bootstrap replicates of 10000)

data:  tab
(d - d0) / se = 171.22, p-value = 1
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0 1
sample estimates:
Sorensen dissimilarity 
              0.996769 
attr(,"se")
standard error 
   0.003225798 


$Vogelstein$cis

	Bootstrap test for 2x2 contingency tables based on the Sorensen-Dice
	dissimilarity (10000 bootstrap replicates)

data:  tab
(d - d0) / se = 7.0238, p-value = 1
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.6328101
sample estimates:
Sorensen dissimilarity 
             0.5975309 
attr(,"se")
standard error 
    0.02179538 


$Vogelstein$miscellaneous

	Bootstrap test for 2x2 contingency tables based on the Sorensen-Dice
	dissimilarity (10000 bootstrap replicates)

data:  tab
(d - d0) / se = -6.7191, p-value = 9.999e-05
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.3629782
sample estimates:
Sorensen dissimilarity 
             0.3351111 
attr(,"se")
standard error 
    0.01627199 


$Vogelstein$sanger

	Bootstrap test for 2x2 contingency tables based on the Sorensen-Dice
	dissimilarity (10000 bootstrap replicates)

data:  tab
(d - d0) / se = -32.259, p-value = 9.999e-05
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.000000 0.136948
sample estimates:
Sorensen dissimilarity 
             0.1195185 
attr(,"se")
standard error 
     0.0100725 



$waldman
$waldman$atlas

	Bootstrap test for 2x2 contingency tables based on the Sorensen-Dice
	dissimilarity (10000 bootstrap replicates)

data:  tab
(d - d0) / se = -15.308, p-value = 9.999e-05
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.2826499
sample estimates:
Sorensen dissimilarity 
             0.2627947 
attr(,"se")
standard error 
    0.01186635 


$waldman$cangenes

	Bootstrap test for 2x2 contingency tables based on the Sorensen-Dice
	dissimilarity (6278 effective bootstrap replicates of 10000)

data:  tab
(d - d0) / se = 215.94, p-value = 1
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0 1
sample estimates:
Sorensen dissimilarity 
             0.9974359 
attr(,"se")
standard error 
   0.002560815 


$waldman$cis

	Bootstrap test for 2x2 contingency tables based on the Sorensen-Dice
	dissimilarity (10000 bootstrap replicates)

data:  tab
(d - d0) / se = 10.327, p-value = 1
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.6796177
sample estimates:
Sorensen dissimilarity 
             0.6477858 
attr(,"se")
standard error 
    0.01968936 


$waldman$miscellaneous

	Bootstrap test for 2x2 contingency tables based on the Sorensen-Dice
	dissimilarity (10000 bootstrap replicates)

data:  tab
(d - d0) / se = -12.16, p-value = 9.999e-05
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.2973298
sample estimates:
Sorensen dissimilarity 
              0.273717 
attr(,"se")
standard error 
    0.01404058 


$waldman$sanger

	Bootstrap test for 2x2 contingency tables based on the Sorensen-Dice
	dissimilarity (10000 bootstrap replicates)

data:  tab
(d - d0) / se = -7.9582, p-value = 9.999e-05
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.3515402
sample estimates:
Sorensen dissimilarity 
              0.326284 
attr(,"se")
standard error 
    0.01484766 


$waldman$Vogelstein

	Bootstrap test for 2x2 contingency tables based on the Sorensen-Dice
	dissimilarity (10000 bootstrap replicates)

data:  tab
(d - d0) / se = -10.211, p-value = 9.999e-05
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
 0.0000000 0.3243615
sample estimates:
Sorensen dissimilarity 
             0.3013601 
attr(,"se")
standard error 
    0.01401229 



attr(,"class")
[1] "equivSDhtestList" "list"            
> class(allBootTests)
[1] "equivSDhtestList" "list"            
> getPvalue(allBootTests, simplify = FALSE)
                   atlas cangenes cis miscellaneous    sanger Vogelstein
atlas         0.00000000        1   1    0.00019998 9.999e-05  9.999e-05
cangenes      1.00000000        0   1    1.00000000 1.000e+00  1.000e+00
cis           1.00000000        1   0    1.00000000 1.000e+00  1.000e+00
miscellaneous 0.00019998        1   1    0.00000000 9.999e-05  9.999e-05
sanger        0.00009999        1   1    0.00009999 0.000e+00  9.999e-05
Vogelstein    0.00009999        1   1    0.00009999 9.999e-05  0.000e+00
waldman       0.00009999        1   1    0.00009999 9.999e-05  9.999e-05
                waldman
atlas         9.999e-05
cangenes      1.000e+00
cis           1.000e+00
miscellaneous 9.999e-05
sanger        9.999e-05
Vogelstein    9.999e-05
waldman       0.000e+00
> getEffNboot(allBootTests)
          cangenes.atlas                cis.atlas             cis.cangenes 
                    6298                    10000                     6342 
     miscellaneous.atlas   miscellaneous.cangenes        miscellaneous.cis 
                   10000                     6382                    10000 
            sanger.atlas          sanger.cangenes               sanger.cis 
                   10000                     6401                    10000 
    sanger.miscellaneous         Vogelstein.atlas      Vogelstein.cangenes 
                   10000                    10000                     6276 
          Vogelstein.cis Vogelstein.miscellaneous        Vogelstein.sanger 
                   10000                    10000                    10000 
           waldman.atlas         waldman.cangenes              waldman.cis 
                   10000                     6278                    10000 
   waldman.miscellaneous           waldman.sanger       waldman.Vogelstein 
                   10000                    10000                    10000 
> 
> # To adjust for testing multiplicity:
> p.adjust(getPvalue(allBootTests), method = "holm")
          cangenes.atlas.p-value                cis.atlas.p-value 
                      1.00000000                       1.00000000 
            cis.cangenes.p-value      miscellaneous.atlas.p-value 
                      1.00000000                       0.00239976 
  miscellaneous.cangenes.p-value        miscellaneous.cis.p-value 
                      1.00000000                       1.00000000 
            sanger.atlas.p-value          sanger.cangenes.p-value 
                      0.00209979                       1.00000000 
              sanger.cis.p-value     sanger.miscellaneous.p-value 
                      1.00000000                       0.00209979 
        Vogelstein.atlas.p-value      Vogelstein.cangenes.p-value 
                      0.00209979                       1.00000000 
          Vogelstein.cis.p-value Vogelstein.miscellaneous.p-value 
                      1.00000000                       0.00209979 
       Vogelstein.sanger.p-value            waldman.atlas.p-value 
                      0.00209979                       0.00209979 
        waldman.cangenes.p-value              waldman.cis.p-value 
                      1.00000000                       1.00000000 
   waldman.miscellaneous.p-value           waldman.sanger.p-value 
                      0.00209979                       0.00209979 
      waldman.Vogelstein.p-value 
                      0.00209979 
> 
> # If only partial statistics are desired:
> dSorensen(allTabsBP.4)
                  atlas  cangenes       cis miscellaneous    sanger Vogelstein
atlas         0.0000000 0.9979188 0.6857639     0.3865031 0.3355482  0.3130545
cangenes      0.9979188 0.0000000 0.9896373     0.9960630 0.9963370  0.9967690
cis           0.6857639 0.9896373 0.0000000     0.5851216 0.5848033  0.5975309
miscellaneous 0.3865031 0.9960630 0.5851216     0.0000000 0.3174905  0.3351111
sanger        0.3355482 0.9963370 0.5848033     0.3174905 0.0000000  0.1195185
Vogelstein    0.3130545 0.9967690 0.5975309     0.3351111 0.1195185  0.0000000
waldman       0.2627947 0.9974359 0.6477858     0.2737170 0.3262840  0.3013601
                waldman
atlas         0.2627947
cangenes      0.9974359
cis           0.6477858
miscellaneous 0.2737170
sanger        0.3262840
Vogelstein    0.3013601
waldman       0.0000000
> duppSorensen(allTabsBP.4)
                  atlas cangenes       cis miscellaneous    sanger Vogelstein
atlas         0.0000000        1 0.7149878     0.4111390 0.3586962  0.3350683
cangenes      1.0000000        0 1.0000000     1.0000000 1.0000000  1.0000000
cis           0.7149878        1 0.0000000     0.6237490 0.6224203  0.6333811
miscellaneous 0.4111390        1 0.6237490     0.0000000 0.3446065  0.3618762
sanger        0.3586962        1 0.6224203     0.3446065 0.0000000  0.1360863
Vogelstein    0.3350683        1 0.6333811     0.3618762 0.1360863  0.0000000
waldman       0.2823131        1 0.6801719     0.2968117 0.3507062  0.3244082
                waldman
atlas         0.2823131
cangenes      1.0000000
cis           0.6801719
miscellaneous 0.2968117
sanger        0.3507062
Vogelstein    0.3244082
waldman       0.0000000
> seSorensen(allTabsBP.4)
                   atlas    cangenes        cis miscellaneous      sanger
atlas         0.00000000 0.002079000 0.01776688   0.014977580 0.014073007
cangenes      0.00207900 0.000000000 0.01030900   0.003929258 0.003656295
cis           0.01776688 0.010309002 0.00000000   0.023483807 0.022869567
miscellaneous 0.01497758 0.003929258 0.02348381   0.000000000 0.016485388
sanger        0.01407301 0.003656295 0.02286957   0.016485388 0.000000000
Vogelstein    0.01338347 0.003225798 0.02179538   0.016271992 0.010072500
waldman       0.01186635 0.002560815 0.01968936   0.014040581 0.014847661
               Vogelstein     waldman
atlas         0.013383469 0.011866345
cangenes      0.003225798 0.002560815
cis           0.021795381 0.019689356
miscellaneous 0.016271992 0.014040581
sanger        0.010072500 0.014847661
Vogelstein    0.000000000 0.014012289
waldman       0.014012289 0.000000000
> 
> 
> # Tipically, in a real study it would be interesting to scan tests
> # along some ontologies and levels inside these ontologies:
> # (which obviously will be a quite slow process)
> # gc()
> # set.seed(123)
> # allBootTests_BP_MF_lev4to8 <- allEquivTestSorensen(allOncoGeneLists,
> #                                                    boot = TRUE,
> #                                                    geneUniverse = humanEntrezIDs, orgPackg = "org.Hs.eg.db",
> #                                                    ontos = c("BP", "MF"), GOLevels = 4:8)
> # getPvalue(allBootTests_BP_MF_lev4to8)
> # getEffNboot(allBootTests_BP_MF_lev4to8)
> 
> proc.time()
   user  system elapsed 
  93.31    9.43  102.70 

R Under development (unstable) (2022-12-25 r83502 ucrt) -- "Unsuffered Consequences"
Copyright (C) 2022 The R Foundation for Statistical Computing
Platform: x86_64-w64-mingw32/x64 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(goSorensen)



Attaching package: 'goSorensen'

The following object is masked from 'package:utils':

    upgrade

> 
> testError <- function(e) {return(e)}
> 
> tryCatch(dSorensen("Sec1", onto = "BP"), error = testError)
<simpleError in buildEnrichTable.character(x, y, check.table = check.table, ...): Argument 'y' is missing, 'x' and 'y' must be 'character' vectors of valid gene identifiers>
> 
> data(allOncoGeneLists)
> ?allOncoGeneLists
> data(humanEntrezIDs)
> 
> # Non-sense random gene lists. Generating Entrez-like gene identifiers, but random:
> set.seed(1234567)
> genList1 <- unique(as.character(sample.int(99999, size = 100)))
> genList2 <- unique(as.character(sample.int(99999, size = 100)))
> # Gene identifiers are numbers like Entrez identifiers at 'humanEntrezIDs', but random.
> dSorensen(genList1, genList2,
+           listNames = c("genList1", "genList2"),
+           onto = "BP", GOLevel = 5,
+           geneUniverse = humanEntrezIDs, orgPackg = "org.Hs.eg.db")
[1] NaN
> duppSorensen(genList1, genList2,
+              listNames = c("genList1", "genList2"),
+              onto = "BP", GOLevel = 5,
+              geneUniverse = humanEntrezIDs, orgPackg = "org.Hs.eg.db")
[1] NaN
> seSorensen(genList1, genList2,
+            listNames = c("genList1", "genList2"),
+            onto = "BP", GOLevel = 5,
+            geneUniverse = humanEntrezIDs, orgPackg = "org.Hs.eg.db")
[1] NaN
> nonSenseTst <- equivTestSorensen(genList1, genList2,
+                                  listNames = c("genList1", "genList2"),
+                                  onto = "BP", GOLevel = 5,
+                                  geneUniverse = humanEntrezIDs, orgPackg = "org.Hs.eg.db")
> nonSenseTst

	No test performed due not finite (d - d0) / se statistic

data:  tab
(d - d0) / se = NaN, p-value = NA
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
   0 NaN
sample estimates:
Sorensen dissimilarity 
                   NaN 
attr(,"se")
standard error 
           NaN 

> tab <- getTable(nonSenseTst)
> tab
                    Enriched in genList2
Enriched in genList1 TRUE FALSE
               TRUE     0     0
               FALSE    0 10148
> # Or, alternatively:
> tab <- buildEnrichTable(genList1, genList2,
+                         listNames = c("genList1", "genList2"),
+                         onto = "BP", GOLevel = 5,
+                         geneUniverse = humanEntrezIDs, orgPackg = "org.Hs.eg.db")
> tab
                    Enriched in genList2
Enriched in genList1 TRUE FALSE
               TRUE     0     0
               FALSE    0 10148
> dSorensen(tab)
[1] NaN
> duppSorensen(tab)
[1] NaN
> seSorensen(tab)
[1] NaN
> equivTestSorensen(tab)

	No test performed due not finite (d - d0) / se statistic

data:  tab
(d - d0) / se = NaN, p-value = NA
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
   0 NaN
sample estimates:
Sorensen dissimilarity 
                   NaN 
attr(,"se")
standard error 
           NaN 

> 
> # Even more non-sense, letters non numeric-style like those at 'humanEntrezIDs':
> set.seed(1234567)
> genList1 <- unique(vapply(seq_len(100), function(i) {
+   paste0(sample(c(letters, LETTERS), 6, replace = TRUE), collapse = "")
+ }, FUN.VALUE = character(1)))
> genList2 <- unique(vapply(seq_len(100), function(i) {
+   paste0(sample(c(letters, LETTERS), 6, replace = TRUE), collapse = "")
+ }, FUN.VALUE = character(1)))
> 
> # Gene identifiers incompatible with those at 'humanEntrezIDs':
> dSorensen(genList1, genList2,
+           listNames = c("genList1", "genList2"),
+           onto = "BP", GOLevel = 5,
+           geneUniverse = humanEntrezIDs, orgPackg = "org.Hs.eg.db")
--> No gene can be mapped....
--> Expected input gene ID: 3309,8546,222698,2692,284359,56776
--> return NULL...
--> No gene can be mapped....
--> Expected input gene ID: 54997,132243,3371,344018,84678,9184
--> return NULL...
[1] NaN
> duppSorensen(genList1, genList2,
+              listNames = c("genList1", "genList2"),
+              onto = "BP", GOLevel = 5,
+              geneUniverse = humanEntrezIDs, orgPackg = "org.Hs.eg.db")
--> No gene can be mapped....
--> Expected input gene ID: 3007,11144,221400,734,51207,374768
--> return NULL...
--> No gene can be mapped....
--> Expected input gene ID: 84225,9825,53405,5819,90780,84056
--> return NULL...
[1] NaN
> seSorensen(genList1, genList2,
+            listNames = c("genList1", "genList2"),
+            onto = "BP", GOLevel = 5,
+            geneUniverse = humanEntrezIDs, orgPackg = "org.Hs.eg.db")
--> No gene can be mapped....
--> Expected input gene ID: 100137049,151195,23617,5888,4956,128497
--> return NULL...
--> No gene can be mapped....
--> Expected input gene ID: 1235,440822,50487,57113,116369,6774
--> return NULL...
[1] NaN
> nonSenseTst <- equivTestSorensen(genList1, genList2,
+                                  listNames = c("genList1", "genList2"),
+                                  onto = "BP", GOLevel = 5,
+                                  geneUniverse = humanEntrezIDs, orgPackg = "org.Hs.eg.db")
--> No gene can be mapped....
--> Expected input gene ID: 2241,4323,7040,2072,8654,190
--> return NULL...
--> No gene can be mapped....
--> Expected input gene ID: 51087,5781,23492,84787,147700,7272
--> return NULL...
> nonSenseTst

	No test performed due not finite (d - d0) / se statistic

data:  tab
(d - d0) / se = NaN, p-value = NA
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
   0 NaN
sample estimates:
Sorensen dissimilarity 
                   NaN 
attr(,"se")
standard error 
           NaN 

> tab <- getTable(nonSenseTst)
> tab
                    Enriched in genList2
Enriched in genList1 TRUE FALSE
               TRUE     0     0
               FALSE    0 10148
> # Or, alternatively:
> tab <- buildEnrichTable(genList1, genList2,
+                         listNames = c("genList1", "genList2"),
+                         onto = "BP", GOLevel = 5,
+                         geneUniverse = humanEntrezIDs, orgPackg = "org.Hs.eg.db")
--> No gene can be mapped....
--> Expected input gene ID: 578,84464,23230,23626,90,406
--> return NULL...
--> No gene can be mapped....
--> Expected input gene ID: 18,132625,431707,84275,54890,2302
--> return NULL...
> tab
                    Enriched in genList2
Enriched in genList1 TRUE FALSE
               TRUE     0     0
               FALSE    0 10148
> dSorensen(tab)
[1] NaN
> duppSorensen(tab)
[1] NaN
> seSorensen(tab)
[1] NaN
> equivTestSorensen(tab)

	No test performed due not finite (d - d0) / se statistic

data:  tab
(d - d0) / se = NaN, p-value = NA
alternative hypothesis: true equivalence limit d0 is less than 0.4444444
95 percent confidence interval:
   0 NaN
sample estimates:
Sorensen dissimilarity 
                   NaN 
attr(,"se")
standard error 
           NaN 

> 
> proc.time()
   user  system elapsed 
 195.35    7.46  203.45