memory consumption and processing times

To measure a time taken by a process the following function can be useful

system.time() e.g.   

system.time(sum(seq(1,10000000,by=1)))     

   user  system elapsed
   0.44    0.08    0.53  

To display the output of the process as well, use print() inside

system.time(print(sum(seq(1,10000000,by=1))))  

[1] 5e+13
   user  system elapsed
   0.41    0.02    0.42  

   

To check the size of the object, use object_size() from 'pryr' package

library(pryr)   

object_size(iris)   

7.03 kB

To check how much memory has been used by the session,

mem_used()   

21 MB

To check how much memory change a particular process makes,

mem_used()   

21 MB

mem_change(1+100)  

10.6 kB   

mem_used()   

21.1 MB

Garbage collection gc() runs automatically in R to release memory when an object is removed. You can still run this command manually.

mem_used()   

21.1 MB 

dat<-iris[rep(1:nrow(iris),1000),]
mem_used()  

37.3 MB 

rm(dat)   

mem_used() 

24 MB 

gc()   

         used (Mb) gc trigger (Mb) max used  (Mb)
Ncells 343524 18.4     741108 39.6   707448  37.8
Vcells 591474  4.6    2028694 15.5 21067777 160.8 

mem_used()   

24 MB

For Windows, to check memory in use and change limits of memory allocation
 

library(utils)  

 
To check size of memory currently in use
 

memory.size(max=FALSE)  

 
To check maximum amount of memory obtained from OS
 

memory.size(max=TRUE)  

e.g.

#memory immediately after opening a new R session  

memory.size(max=TRUE)    

[1]  16.56 

memory.size(max=FALSE)   

 [1]  12.58 

dat<-matrix(1:2700, 30, 90)  #create a matrix 30 x 90  

memory.size(max=TRUE)  #maximum memory obtained remains same  

[1]  16.56  

memory.size(max=FALSE)  #memory currently in use increased after the matrix was created  

 [1]  12.61  

 

To change memory allocation limit to specific amount (in Mb)
 

memory.limit(size=8000)  

   

To check current memory allocation limit  
 

memory.limit(size=NA)  

e.g.

#at the opening of a new R session (32 bit)

memory.limit(size=NA)  

[1]  3583   

memory.limit(size=3600)  #increased size to 3600  

[1]  3600  

memory.limit(size=NA)   

[1]  3600   

memory.limit(size=3800)  #increased size to 3800  

[1]  3800   

memory.limit(size=NA)   

[1]  3800   

N.B. the memory allocation limit cannot be set to a lower limit (i.e. you can only increase it)

r updates and packages

The package 'installr' makes it simple to upgrade R to a new version with the following command

library(installr) 

To install the latest version of R:

install.R()

To install the associated program used with R:

The below lists only some of the examples

install.ImageMagick()  

install.Rtools() 

install.MikTex()  

install.pandoc()   

To update installed R to the latest version:

updateR()

To install packages, you can set the source by specifying 'repos=' and specify the path to the library by setting 'lib=' (in Windows, .libPaths() can be used to set up a default path to the library folder at the beginning for the session)

install.packages("dplyr", repos="http://cran.csiro.au/", lib="C:/Users/Documents/MyLibrary")

   
If you are installing packages from the local directory, provide path and name of the file to be installed.

          install.packages("[path to the directory]/[name of file (e.g. ada_2.0-5.zip)]", repos=NULL,  type="source")
 
 
The list of all repos url's can be found at http://cran.r-project.org/mirrors.html


To list all the packages installed at a specified library path

installed.packages(lib.loc="path")

To check if any packages in the library are old,   

old.packages()

To check for new packages,   

new.packages()

 

To update existing packages to the latest version  

update.packages()

 

For those work places using Windows where firewall prevents downloading of the packages, you may need to download zip files from the source and install them locally by using the tool bar options in the R console.  

To check what packages are available:

(.packages())

To check all available packages:

(.packages(all.available=TRUE))

alternatively,

library()

 

To check for all packages in the repository:

available.packages()

To remove packages:

remove.packages("name_of_package","library_path")

To download the package in its compressed form (e.g. *.tar.gz):

download.packages("name_of_package",destdir="destination_path")

To download multiple packages at once,

download.packages(c("name_of_package1","name_of_package2","name_of_package3"), destdir="destination_path")

A contributed package called 'tools' provides useful function to check for dependencies of the installed packages:

library(tools)   

inst_pack<-installed.packages()   

package.dependencies(inst_pack)
 



 



 

Intersection & Union

Let's set vectors

Group1<-c("A","R","t","Y","p","Q") 

Group2<-c("Z","R","t","O","p","X")

> Group1  

[1] "A" "R" "t" "Y" "p" "Q"

> Group2   

[1] "Z" "R" "t" "O" "p" "X"

Intersection: there are 2 ways

> intersect(Group1,Group2)
[1] "R" "t" "p"

> Group1[which(Group1%in%Group2)]
[1] "R" "t" "p"

Union:

> union(Group1,Group2)   

[1] "A" "R" "t" "Y" "p" "Q" "Z" "O" "X"

Elements exclusive to Group 1: there are 2 ways

> setdiff(Group1,Group2)  

[1] "A" "Y" "Q"   

> Group1[which(!Group1%in%Group2)]   

[1] "A" "Y" "Q"

Elements exclusive to Group 2: there are 2 ways

> setdiff(Group2,Group1)   

[1] "Z" "O" "X"

> Group2[which(!Group2%in%Group1)]   

[1] "Z" "O" "X"

Mutually exclusive elements

> union(setdiff(Group1,Group2),setdiff(Group2,Group1))   

[1] "A" "Y" "Q" "Z" "O" "X"

Working on Excel file from R

There are few packages available in R to let you work with (import/export) Excel files from R. This blog page will focus on 'xlsx' package.


'xlsx' package provides a command to import excel file, but this did not work well in my case. If the excel document is in Excel 97-2003 (.xls) format, the following works quite well.

Note: if ODBC installed in Windows is 32-bit, you need to use 32-bit R to do this.

library(RODBC) 

library(xlsx) 

excel.connect<-odbcConnectExcel("file_name.xls") 

Data<-sqlFetch(excel.connect,"Sheet_Name") 

odbcClose(excel.connect)

For exporting data into excel document, the following works well.

For a simple export:

library(xlsx)

wb<-createWorkbook() 

sheet<-createSheet(wb,sheetName="Sheet_Name") 

addDataFrame(dataframe,sheet,row.names=FALSE)  

saveWorkbook(wb,"file_name")

To create multiple worksheets:

wb<-createWorkbook() 

sheet<-createSheet(wb,sheetName="Sheet_Name_1") 

addDataFrame(dataframe_1,sheet,row.names=FALSE)  

saveWorkbook(wb,"file_name")

sheet<-createSheet(wb,sheetName="Sheet_Name_2") 

addDataFrame(dataframe_2,sheet,row.names=FALSE)  

saveWorkbook(wb,"file_name")

For exporting data with formatting:

wb<-createWorkbook() 

sheet<-createSheet(wb,sheetName="Sheet_Name") 

for(i in c(required row numbers){ 

ROW<-createRow(sheet,rowIndex=i)

#populate selected columns (selected respective to different formatting)

for(j in c(required column numbers)){ 

CELL<-createCell(ROW,colIndex=j)[[1,1]]  

#data format written as per the excel specification (go to cell format and custom format for other examples)

cs<-CellStyle(wb,dataFormat=DataFormat("#,##0;[Red]-#,##0"))+
Font(wb,isBold=FALSE)+
Alignment(h="ALIGN_RIGHT",wrapText=TRUE) 

setCellValue(CELL,dataframe[i,j],showNA=FALSE)  

setCellStyle(CELL,cs)
}

#populate remaining columns (selected respective to different formatting)

for(j in c(required column numbers)){ 

CELL<-createCell(ROW,colIndex=j)[[1,1]]   

cs<-CellStyle(wb,dataFormat=DataFormat("$#,##0.00;[Red]-$#,##0.00"))+
Font(wb,isBold=FALSE)+
Alignment(h="ALIGN_RIGHT",wrapText=TRUE) 

setCellValue(CELL,dataframe[i,j],showNA=FALSE)  

setCellStyle(CELL,cs)
}

Number formatting

For presentation of numeric tables in reports, one often has to use '000 separators and use equal spacing to make the results look neater and pretty.

formatC() in the base package is useful for this purpose.


Example 1:

Given the below input

data.frame(V1=c((-1:1)/0,c(1,100)*pi))
          V1
1       -Inf
2        NaN
3        Inf
4   3.141593
5 314.159265

converted to below

 data.frame(V1=formatC(c((-1:1)/0,c(1,100)*pi), width = 8, digits = 1))
        V1
1     -Inf
2      NaN
3      Inf
4        3
5    3e+02

Alternatively for 2 digits displayed

data.frame(V1=formatC(c((-1:1)/0,c(1,100)*pi), width = 8, digits = 2))
        V1
1     -Inf
2      NaN
3      Inf
4      3.1
5  3.1e+02

Example 2:

Given the below numbers

12344448.2 

3244149.9210001 

321.21 

78186271.32155

To only show integers

data.frame(V1=formatC(c(12344448.2,3244149.9210001,3217747.214,78186271.32155),format="d"))

        V1

1 12344448 

2  3244149 

3  321 

4 78186271

To show numbers with scientific format

data.frame(V1=formatC(c(12344448.2,3244149.9210001,321.21,78186271.32155),format="e"))

          V1 

1 1.2344e+07 

2 3.2441e+06 

3 3.2121e+02 

4 7.8186e+07

To show numbers with scientific format when the number is big

data.frame(V1=formatC(c(12344448.2,3244149.9210001,321.21,78186271.32155),format="g")) 

         V1
1 1.234e+07
2 3.244e+06
3     321.2
4 7.819e+07

To show numbers in its actual form (this is limited by the options setting in R)

data.frame(V1=formatC(c(12344448.2,3244149.9210001,321.21,78186271.32155),format="f")) 

             V1
1 12344448.2000
2  3244149.9210
3      321.2100
4 78186271.3215

To show remainder of digits in decimal points, specify 'digits' parameter (but format="f" will change the original input to comply with the new setting - so this should be used when you want to round off figures to smaller number of digits)

data.frame(V1=formatC(c(12344448.2,3244149.9210001,321.21,78186271.32155),format="f",digits=15)) 

                        V1
1 12344448.199999999254942
2  3244149.921000100206584
3      321.209999999999980
4 78186271.321549996733665

Use format="fg" in combination with 'digits' specification to display the actual input

data.frame(V1=formatC(c(12344448.2,3244149.9210001,321.21,78186271.32155),format="fg",digits=15)) 

                V1
1       12344448.2
2  3244149.9210001
3           321.21
4   78186271.32155

To insert '000 separator

data.frame(V1=formatC(c(12344448.2,3244149.9210001,321.21,78186271.32155),format="fg",digits=15,big.mark=",")) 

                 V1
1      12,344,448.2
2 3,244,149.9210001
3            321.21
4  78,186,271.32155

To make the decimal point more obvious

data.frame(V1=formatC(c(12344448.2,3244149.9210001,321.21,78186271.32155),format="fg",digits=15,big.mark=",",decimal.mark="\\")) 

                  V1
1      12,344,448\\2
2 3,244,149\\9210001
3            321\\21
4  78,186,271\\32155

To insert separators in the decimal points

data.frame(V1=formatC(c(12344448.2,3244149.9210001,321.21,78186271.32155),format="fg",digits=15,big.mark=",",decimal.mark="\\",small.mark=" ")) 

                   V1
1       12,344,448\\2
2 3,244,149\\92100 01
3             321\\21
4   78,186,271\\32155