README.md

STA141B UC Davis

Spring 2023

Instructor:Duncan Temple Lang

Day 1 (4/4)

• Slides - Full Pages, 4-per page

• Federal Election Commission (FEC) data

  • https://www.fec.gov/data/browse-data/?tab=bulk-data.
  • "Contributions by individuals" panel
  • zip file
  • itcont.txt
  • Data/FEC/itcont100K.txt - smaller version - 100,000 lines

Day 2 (4/6)

• Exploring reading the FEC Data
• [R Session]((Lectures/Day2/Rsession2)

Day 3 (4/11)

• R Session
• Code in Data/NCBI
• Strategy
• Different versions of functions
  • test.R - run the functions
  • funs0.R, funs1.R, funs2.R
  • funs3.R - problems/bugs
  • funs4.R - improvements on funs3.R
• Debugging funs3.R

Day 4 (4/13)

• Day 4 R Session

• bob.R

  • The functions we wrote.
  • Also see funs3.R and funs4.R from Day3 (above)

Day 5 (4/18)

• Day 5 R Session
  • See Debugging funs3.R for the narrative of this debugging session.

Day 6 (4/20)

• slides on regular expressions
• R session Day 6
  • R commands from session Day 6

Day 7 (4/25)

• Recap of Regular Expression Language
• Case studies
• Web log case study
• getCaptures() function

Day 8 (Thu 4/27)

• Day 8 R Session

  • reading the web logs
    • using read.table() and fixing the results
    • start of reading via regular expressions and capture groups as described in the Web log case study.

• code for read.table() and cleanup approach

• code for regular expression & getCaptures() approach

Day 9 (Tue 5/2)

• Day 9 R Session
  • reading the web logs
    • regular expressions and capture groups
    • Variation of the approach in Web log case study.
      • specifically, getting the GET, file path and HTTP version at the same time as all of the other columns/capture groups.
  • Some code to get the data as data.frame and cleanup the variables

Day 10 (Thu 5/4)

• Day 10 Databases and SQL
  • slides on relational databases, SQL

Day 11 (Tue 5/9)

• slides 1
• slides 2
• FANG example database
• Day 11 Databases and SQL

Day 12 (Thu 5/11)

• Joins on Tables
• baseball database
• Baseball Case Studies
• Day 12 SQL Session

Day 13 (Tue 5/16)

• Day 13 SQL Session
• SQL Code for Self Join
• Example of Self Join to Compute Time/Year Difference
• Example of using SUM() to count the number of tuples in a particular category/logical condition
• Exploring, Debugging SQL and Joins

Day 14 (Thu 5/18)

• Web Scraping Introduction slides
• R Session
  • Reading HTML Table - Wikipedia's USA state populations

Day 15 (Thu 5/23)

github.com/search via HTML

• R Session
• Firefox Developer Tools
• Initial code to scrape github.com/search
• Brief notes on XPath

Day 16 (Thu 5/25)

github.com/search via HTML

• R Session
• Functions to scrape github.com/search
  • general approach code scraping search results and processing pages of results
  • results = ghSearch("R URL decode")
• XPath Axes

github.com/search

• Description of ghFuns.R and the steps involved

Day 17 Tue 5/30

• Notes and code from class
• JSON version of github searc ghFuns.R
• R Session

Day 18 Thu 6/1

• Call graphs for StackOverflow functions

  • Search page results only
  • Question/Answer pages only

• Writing functions

  • General guidelines and principles
  • email messages
    • outline of implementations
    • R functions
    • Data/examples
      • Individual email with body and attachment
      • Multiple emails in single file, no attachments from R-help mailing list archives.

Day 19 Tue 6/6

• slides
• R essentials
  • Vectors and lists
  • apply() functions
  • subsetting
  • preallocation/not concatenating
• Run times for URLdecode implementations
  • run time for utils::URLdecode() - quadratic function.
  • Extrapolation to 600K
  • run times for original, preallocated and vectorized versions
  • run times for preallocated and vectorized versions only for more detail.
  • Note, these were run on a slow, debugging version of R (compiled was not optimized.)
• Avoding redundant computations
  • riverdist package and the whoconnected() function.
  • See slides
• Start of riverdist package and removemicrosegs() function and removing for(jj ...) for(jjj ...) nested loops.
  • See slides
  • and also slides from Day 20.

Day 20 Thu 6/8

• riverdist package and removemicrosegs() function
  • remove for(jj ...) for(jjj ...) nested loops.
• slides on vectorization
• Example of combining pairs of words in data.frame based on alphabetical order
  • sample words in RDS files
• Vectorized creation of sample words
• Organizing Markdown
  • example of doing timing computations in a script and saving the timing results to file and then reading these in the Rmarkdown document in order to plot them, etc.
  • and having the functions in a separate file, e.g., URLdecodeFuns.R and using source() in the Rmarkdown to read and define the functins.
  • the Rmarkdown example