# Evolution of Code

Recently while scraping some data from the college football data warehouse site, I started to realize the evolution of my code. To preface this, I am definitely not a trained programmer, just a self taught junky who enjoys doing it when I have time. I’ve slowly evolved my programming skills from simply statistics languages like r or SPSS, to some other languages like LaTeX, HTML, CSS, Javascript, and I’ve started to work through some python.

Now back to my realization. As I mentioned, I was scraping some data from CFB Data Warehouse for a project that I’m working on with a colleague and was adapting some code that was written about 3 years ago. The problem was that my old code was broken. The original code was about 100 lines of code just to select the correct table and format it. Here is a chunk of the original code to select the correct table.

``````  ##Identifying correct tables
tb <- vector("list", length(tableNodes))
for(i in 1:length(tableNodes)){
}

##Tables that are the correct length
tabNum <- matrix(nrow=length(tableNodes), ncol=2)
tabNum[,1] <- sapply(tb, length)
tabNum[,2] <- 1:length(tableNodes)

Num <- subset(tabNum, tabNum[,1] == 7)[,2]

##Selecting and combining tables
if(length(Num) == 5){
tb1 <- tb[[Num[3]]]
tb1\$Other <- 0
tb2 <- tb[[Num[5]]]
tb2\$Other <- 1
tab <- rbind(tb1, tb2)
} else {
if(length(Num) ==3){
tab <- tb[[Num[3]]]
tab\$Other <- 1
} else {
tab <- matrix(NA, ncol= 8, nrow=1)
}
}``````

This code was looped over many different pages and was run once for every page. Essentially the code is complicated and inconsistent, but at the time 3 years ago the code ran and that was enough for me. Extract the data from the website no matter how much code was needed to do the work. This was back in an era when I was just becoming familiar with much or R, the XML package, and attempting to scrape data from a messy/complicated site.

My new code to extract the tables looks like this:

``````# extracting tables
tabs <- lapply(seq(3, length(Nodes), 1), function(x)