readme.Rmd

---
title: "Javascript Pipeline"
output: html_notebook
---

The entire JavaScript pipeline is outlined in this readme file with links to detailed descriptions of respective sections. The workflow assumes that all commands are executed from the directory where the notebooks are stored so that the notebooks can be executed from within R studio. For every file, double click it, and then press `CTRL`+`ALT`+`R` to execute the code.

For more information about R notebooks in R Studio, see [R Notebooks in R Studio](http://rmarkdown.rstudio.com/r_notebooks.html).

What to do:

- if you are evaluation the VM artifact, skip steps 0 and 1 as they were already executed (using the dataset sample) and jump immediately to step 2.  
- if you are evaluating on the entire dataset on our server using the datasets provided there, you can only execute steps 8 and 9. 
- if you want to reproduce the entire pipeline, start with step 0. *Note that redoing the entire pipeline on entire github will take weeks*.

When time estimates are provided in the steps they usually concern the VM running time on the dataset sample. Running time on full dataset is in days and weeks. 

## 0 - Setup & Tools

*This step has already been executed on the VM*

An overview of the required tools and their setup. 

[Detailed Information](0-setup.nb.html) can be found in file `0-setup.Rmd`.

## 1 - Getting Github project urls

*This step has already been executed on the VM*

In this step, GHTorrent database snapshot is used to filter out list of available project urls on github, forks, deleted and duplicated projects are excluded. 

[Detailed Information](1-getting-projects.nb.html) can be found in file `1-getting-projects.Rmd`.

## 2 - Downloading and tokenizing the projects

This step details the download & tokenization techniques for the filtered projects. After this step the file & token hashes for all files are known. 

[Detailed Information](2-tokenizing.nb.html) in file `2-tokenizing.Rmd`.

## 3 - SourcererCC Clone Analysis

Tokenized files are fed into the SourcererCC and similar files are found. 

[Detailed Information](3-sourcerercc.nb.html) in file `3-sourcerercc.Rmd`.

## 4 - Database Import

The computed data is loaded into a MySQL database

[Detailed Information](4-dbimport.nb.html) in file `4-dbimport.Rmd`.

## 5 - Project Clones

Clone information for entire projects is calculated (this is the input data for the heatmaps). 

[Detailed Information](5-project-clones.nb.html) in file `5-project-clones.Rmd`.

## 6 - Metadata Acquisition

Dowloads metadata for the projects directy from github and from the ghtorrent snapshot.

[Detailed Information](6-metadata.nb.html) in file `6-metadata.Rmd`.

## 7 - Additional Processing

Additional processing of inputs to the reporting steps using both files on disk and the database itself. 

[Detailed Information](7-processing.nb.html) in file `7-processing.Rmd`.

## 8 - Data Analysis & Reporting

*This step can be executed on the machine with all data loaded to reproduce the results of the paper.*

These steps produces the graphs and tables in the paper. The graphs and tables in this section aggregate information for all four languages, but the notebook prints only JavaScript part for now. However, you can easily change the dataset used in the `config.R` to view reports of other languages as well. 

[Detailed Information](8-reports.nb.html) in file `8-reports.Rmd`.

## 9 - Language specific reporting

*This step can be executed on the machine with all data loaded to reproduce the results of the paper.*

These graphs from the paper apply only to the JavaScript since they either have no meaning (npm vs non-npm distinction), or the other languages do not provide the additional information required to construct them (time aggregation).  

[Detailed Information](9-reporting-other.nb.html) in file [`9-reporting-other.Rmd`](9-reporting-other.Rmd).