Course materials | Data Analytics: Learning from Data

Homework 1

Fri, 15 Oct 2021 00:00:00 +0000

Pre-course assignment

The aim of the pre-course assignment is to ensure that you successfully install the software, that you get some practice with markdown, and that you are able to knit an R Markdown (.Rmd) document into an HTML file.

Specifically, you need to:

Write a short biography using markdown
Fill out the code in the empty chunks provided, (you can definitely copy, paste, and adapt from tutorials!), and answer all questions.
Knit the Rmd to an HTML file

You can download pre-programme files (data, code, etc.) by pulling from course Github repo. Alternatively, please install package usethis. Once you have it, you can download, unzip, and open everything within an RStudio project by typing the following in the RStudio console

install.packages("usethis")
usethis::use_course("https://github.com/kostis-christodoulou/usi_EMBA_analytics/raw/master/pre_programme_assignment.zip")

Instructions

Please finish the assignment and submit your work as a knitted HTML file on iCorsi.

Also, please submit your work without any of the text I have written– just your code, plots, and explanations/stories.

Finance Data

Fri, 31 Jul 2020 00:00:00 +0000

“Data!data!data!” he cried impatiently. “I can’t make bricks without clay.”
–Arthur Conan Doyle, The Adventure of the Copper Beeches

The easiest way to download data is if someone makes available a CSV file and we can download it directly off the web with readr::read_csv()or with data.table::fread(). Alternatively, we can use the rio package to download many different types of files (Excel, SPSS, Stata, etc.)

In this section we will look at three packages that use wrapped Application Programming Interface (APIs) to get data off the web:

tidyquant to get finance data
wbstats to get data from the World Bank database, and
eurostat to get Eurostat data.

Finance data with the `tidyquant` package

The tidyquant package comes with a number of functions- utlities that allow us to download financial data off the web, as well as ways of handling all this data.

We begin by loading the data set into the R workspace. We create a collection of stocks with their ticker symbols and then use the piping operator %>% to use tidyquant’s tq_get to donwload historical data using Yahoo finance and, again, to group data by their ticker symbol.

library(tidyquant)
myStocks <- c("AAPL","JPM","DIS","DPZ","ANF","TSLA","XOM","SPY" ) %>%
  tq_get(get  = "stock.prices",
         from = "2011-01-01",
         to   = "2020-07-31") %>%
  group_by(symbol) 

glimpse(myStocks) # examine the structure of the resulting data frame

## Rows: 19,280
## Columns: 8
## Groups: symbol [8]
## $ symbol   <chr> "AAPL", "AAPL", "AAPL", "AAPL", "AAPL", "AAPL", "AAPL", "A...
## $ date     <date> 2011-01-03, 2011-01-04, 2011-01-05, 2011-01-06, 2011-01-0...
## $ open     <dbl> 46.5, 47.5, 47.1, 47.8, 47.7, 48.4, 49.3, 49.0, 49.3, 49.4...
## $ high     <dbl> 47.2, 47.5, 47.8, 47.9, 48.0, 49.0, 49.3, 49.2, 49.5, 49.8...
## $ low      <dbl> 46.4, 46.9, 47.1, 47.6, 47.4, 48.2, 48.5, 48.9, 49.1, 49.2...
## $ close    <dbl> 47.1, 47.3, 47.7, 47.7, 48.0, 48.9, 48.8, 49.2, 49.4, 49.8...
## $ volume   <dbl> 1.11e+08, 7.73e+07, 6.39e+07, 7.51e+07, 7.80e+07, 1.12e+08...
## $ adjusted <dbl> 40.8, 41.0, 41.3, 41.3, 41.6, 42.4, 42.3, 42.6, 42.8, 43.1...

For each ticker symbol, the data frame contains its symbol, the date, the prices for open,high, low and close, and the volume, or how many stocks were traded on that day. More importantly, the data frame contains the adjusted closing price, which adjusts for any stock splits and/or dividends paid and this is what we will be using for our analyses.

Calculating financial returns

Financial performance and CAPM analysis depend on returns and not on adjusted closing prices. So given the adjusted closing prices, our first step is to calculate daily and monthly returns.

#calculate daily returns
myStocks_returns_daily <- myStocks %>%
  tq_transmute(select     = adjusted, 
               mutate_fun = periodReturn, 
               period     = "daily", 
               type       = "log",
               col_rename = "daily.returns",
               cols = c(nested.col))  

#calculate monthly  returns
myStocks_returns_monthly <- myStocks %>%
  tq_transmute(select     = adjusted, 
               mutate_fun = periodReturn, 
               period     = "monthly", 
               type       = "arithmetic",
               col_rename = "monthly.returns",
               cols = c(nested.col)) 

#calculate yearly returns
myStocks_returns_annual <- myStocks %>%
  group_by(symbol) %>%
  tq_transmute(select     = adjusted, 
               mutate_fun = periodReturn, 
               period     = "yearly", 
               type       = "arithmetic",
               col_rename = "yearly.returns",
               cols = c(nested.col))

For yearly and monthly data, we assume discrete changes, so we the formula used to calculate the return for month (t+1) is

$Return(t+1)= \frac{Adj.Close(t+1)}{Adj.Close (t)}-1$

For daily data we use log returns, or $Return(t+1)= LN\frac{Adj.Close(t+1)}{Adj.Close (t)}$

The reason we use log returns are:

Compound interest interpretation; namely, that the log return can be interpreted as the continuously (rather than discretely) compounded rate of return
Log returns are assumed to follow a normal distribution
Log return over n periods is the sum of n log returns

Summarising the data set

Let us get quick summary statistics of daily returns for each stock, as well as a density plot whwre we use facet_grid to superimpose all the distributions in one plot.

symbol	min	median	max	mean	sd	annual_mean	annual_sd
AAPL	-0.138	0.001	0.113	0.001	0.017	0.233	0.276
ANF	-0.307	0.001	0.296	-0.001	0.034	-0.152	0.540
DIS	-0.139	0.001	0.135	0.001	0.015	0.129	0.241
DPZ	-0.106	0.001	0.228	0.001	0.018	0.344	0.287
JPM	-0.162	0.001	0.166	0.000	0.018	0.111	0.284
SPY	-0.116	0.001	0.087	0.000	0.011	0.117	0.172
TSLA	-0.215	0.001	0.218	0.002	0.034	0.417	0.535
XOM	-0.130	0.000	0.119	0.000	0.015	-0.026	0.232

Daily returns seem to follow a normal distribution with a mean close to zero. Since most people think of returns on an annual, rather than on a daily basis, we can calculate summary statistics of annual returns, a boxplot of annual returns, and a bar plot that shows return for each stock on a year-by-year basis.

myStocks_returns_annual %>% 
  group_by(symbol) %>% 
  mutate(median_return= median(yearly.returns)) %>% 

  # arrange stocks by median yearly return, so highest median return appears first, etc.   
  ggplot(aes(x=reorder(symbol, median_return), y=yearly.returns, colour=symbol)) +
  geom_boxplot()+
  coord_flip()+
  labs(x="Stock", 
       y="Returns", 
       title = "Boxplot of Annual Returns")+
  scale_y_continuous(labels = scales::percent_format(accuracy = 2))+
  guides(color=FALSE) +
  theme_bw()+
  NULL

ggplot(myStocks_returns_annual, aes(x=year(date), y=yearly.returns, fill=symbol)) +
  geom_col(position = "dodge")+
  labs(x="Year", y="Returns", title = "Annual Returns")+
  scale_y_continuous(labels = scales::percent)+
  guides(fill=guide_legend(title=NULL))+
  theme_bw()+
  NULL

Minimum and maximum price of each stock by quarter

What if we wanted to find out and visualise the min/max price by quarter?

Sharpe Ratio

The Sharpe ratio, introduced by William F. Sharpe, is used to understand the return of an investment compared to its risk. It is simply the return on an asset per unit of risk, with the unit of risk typically being the standard deviation of the returns of that particular asset.

Mathematically, the ratio is the average return earned in excess of the risk-free rate per unit of volatility. $Sharpe Ratio = \frac{R_{p}-R_{f}}{\sigma_{p}}$

Generally, the greater the value of the Sharpe ratio, the more attractive the risk-adjusted return.

myStocks_returns_monthly %>%
  tq_performance(Ra = monthly.returns, #the name of the variable containing the returns of the asset
                 Rb = NULL, 
                 performance_fun = SharpeRatio) %>% 
  kable() %>%
  kable_styling(c("striped", "bordered"))

symbol	ESSharpe(Rf=0%,p=95%)	StdDevSharpe(Rf=0%,p=95%)	VaRSharpe(Rf=0%,p=95%)
AAPL	0.163	0.296	0.211
JPM	0.068	0.166	0.102
DIS	0.104	0.203	0.147
DPZ	0.313	0.427	0.416
ANF	-0.010	-0.022	-0.014
TSLA	0.207	0.286	0.329
XOM	-0.002	-0.006	-0.003
SPY	0.119	0.280	0.193

Investment Growth

Finally, we may want to see what our investments would have grown to, if we had invested $1000 in each of the assets on Jan 1, 2011.

Scatterplots of individual stocks returns versus S&P500 Index returns

Besides these exploratory graphs of returns and price evolution, we also need to create scatterplots among the returns of different stocks. ggpairs from the GGally package creates a scattterplot matrix that shows the distribution of returns for each stock along the diagonal, and scatter plots and correlations for each pair of stocks. Running a ggpairs() correlation scatterplot-matrix typically takes a while to run.

#calculate daily returns
table_capm_returns <- myStocks_returns_daily %>%
            spread(key = symbol, value = daily.returns)  #just keep the period returns grouped by symbol

table_capm_returns[-1] %>% #exclude "Date", the first column, from the correlation matrix
  GGally::ggpairs() +
  theme_bw()+
    theme(axis.text.x = element_text(angle = 90, size=8),
         axis.title.x = element_blank())

Creating a portfolio of assets

DPZ may have been the best performing stock, but you believe that you can create a portfolio of technology stocks that will beat the relevant sector index, XLK. To create a portfolio, you need to choose a few stocks and then the weights, or how much of your total investment is allocated to each stock. To keep things simple we will assume you will choose among AAPL, GOOG, MSFT, NFLX, and NVDA and you will compare your performance against the sector index, XLK. We will also add two non-tech stocks, TSLA and DPZ so we can their position on the risk/return frontier.

ticker_symbols <- c("AAPL","GOOG","MSFT","NFLX","NVDA", "XLK", "TSLA", "DPZ") 

tech_stock_returns_monthly <- ticker_symbols %>%
    tq_get(get  = "stock.prices",
           from = "2011-01-01",
           to   = "2020-07-31") %>%
    group_by(symbol) %>%
    tq_transmute(select     = adjusted, 
                 mutate_fun = periodReturn, 
                 period     = "monthly", 
                 col_rename = "monthly_return")


baseline_returns_monthly <- "XLK" %>%
    tq_get(get  = "stock.prices",
           from = "2011-01-01",
           to   = "2020-07-31") %>%
    tq_transmute(select     = adjusted, 
                 mutate_fun = periodReturn, 
                 period     = "monthly", 
                 col_rename = "baseline_return")

# Summary Stats for individual Stocks
stocks_risk_return <- tech_stock_returns_monthly %>%
  tq_performance(Ra = monthly_return, Rb = NULL, performance_fun = table.Stats) %>% 
  select(symbol, ArithmeticMean, GeometricMean, Minimum,Maximum,Stdev, Quartile1, Quartile3) 



ggplot(stocks_risk_return, aes(x=Stdev, y = ArithmeticMean, colour= symbol, label= symbol))+
  geom_point(size = 4)+
  labs(title = 'Risk/Return profile of technology stocks', 
       x = 'Risk (stdev of monthly returns)', 
       y ="Average monthly return")+
  theme_bw()+
  scale_x_continuous(labels = scales::percent)+
  scale_y_continuous(labels = scales::percent)+
  geom_text_repel()+
  theme(legend.position = "none")

We have the monthly returns of the individual stocks and the relevenant sector index. To create a portfolio, we must specify the weights; as an example, suppose we only choose three stocks and invest 50% in AAPL, 35% in NFLX, and 15% in NVDA. To do this, we create a two-column tibble, with symbols in the first column and weights in the second; any symbol not specified by default gets a weight of zero.

weights_map <- tibble(
    symbols = c("AAPL", "NFLX", "NVDA"),
    weights = c(0.5, 0.35, 0.15)
)

tech_portfolio_returns <- tech_stock_returns_monthly %>%
    tq_portfolio(assets_col  = symbol, 
                 returns_col = monthly_return, 
                 weights     = weights_map, 
                 col_rename  = "monthly_portfolio_return")

tech_portfolio_returns %>%
    ggplot(aes(x = date, y = monthly_portfolio_return)) +
    geom_col() +
    scale_y_continuous(labels = scales::percent) +
    # geom_bar(stat = "identity", fill = palette_light()[[1]]) +
    labs(title = "Tech Portfolio Returns",
         subtitle = "50% AAPL, 35% NFLX, and 15% NVDA",
         x = "", y = "Monthly Returns") +
    theme_bw()

portfolio_growth_monthly <- tech_stock_returns_monthly %>%
    tq_portfolio(assets_col   = symbol, 
                 returns_col  = monthly_return, 
                 weights      = weights_map, 
                 col_rename   = "investment.growth",
                 wealth.index = TRUE) %>%
    mutate(investment.growth = investment.growth * 1000)

plot1 <- portfolio_growth_monthly %>%
    ggplot(aes(x = date, y = investment.growth)) +
    geom_line(size = 2) +
    labs(title = "Portfolio Growth",
         subtitle = "50% AAPL, 35% NFLX, and 15% NVDA",
         x = "", y = "Portfolio Value") +
    # geom_smooth(method = "loess", se = FALSE) +
    theme_bw() +
    scale_y_continuous(labels = scales::dollar)

Now that we have our portfolio returns and the baseline returns of the XLK index, we can merge to get our consolidated table of asset and baseline returns, create a scatter plot and fit a CAPM model.

tech_single_portfolio <- left_join(tech_portfolio_returns, 
                                   baseline_returns_monthly,
                                   by = "date")
tech_single_portfolio

## # A tibble: 115 x 3
##    date       monthly_portfolio_return baseline_return
##    <date>                        <dbl>           <dbl>
##  1 2011-01-31                  0.162           0.0204 
##  2 2011-02-28                 -0.00466         0.0219 
##  3 2011-03-31                  0.0122         -0.0155 
##  4 2011-04-29                  0.00601         0.0261 
##  5 2011-05-31                  0.0609         -0.0105 
##  6 2011-06-30                 -0.0586         -0.0248 
##  7 2011-07-29                  0.0592          0.00428
##  8 2011-08-31                 -0.0596         -0.0531 
##  9 2011-09-30                 -0.215          -0.0307 
## 10 2011-10-31                 -0.00422         0.102  
## # ... with 105 more rows

ggplot(tech_single_portfolio, aes(x = baseline_return, y= monthly_portfolio_return)) +
  geom_point()+
  geom_smooth(method="lm", se=FALSE) +
  scale_x_continuous(labels = scales::percent) +
  scale_y_continuous(labels = scales::percent) +
  labs(x = "Baseline returns (XLK)", 
       y= "Tech Portfolio Return", 
       title= "How do our tech fund returns compare to the the sector index XLK")

portfolio_CAPM <- lm(monthly_portfolio_return ~ baseline_return, data = tech_single_portfolio)
summary(portfolio_CAPM)

## 
## Call:
## lm(formula = monthly_portfolio_return ~ baseline_return, data = tech_single_portfolio)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -0.1841 -0.0375 -0.0016  0.0326  0.1618 
## 
## Coefficients:
##                 Estimate Std. Error t value Pr(>|t|)    
## (Intercept)      0.00836    0.00579    1.44     0.15    
## baseline_return  1.27904    0.12865    9.94   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.0586 on 113 degrees of freedom
## Multiple R-squared:  0.467,  Adjusted R-squared:  0.462 
## F-statistic: 98.8 on 1 and 113 DF,  p-value: <2e-16

autoplot(portfolio_CAPM, which = 1:3) +
  theme_bw()

Creating various portfolios by changing weights of assets

Suppose we wanted to examine a few more portfolios by varying the weights.

Naive portfolio: you split your investment equally among the five stocks, so each of them has a weight of 20%
Bitcoin mining: you invest 80-20 in NVDA and GOOG
Binge TV watching: you invest most (70%) in NFLX and 10% to AAPL, GOOG, and MSFT

ticker_symbols = c("AAPL", "GOOG", "MSFT", "NFLX", "NVDA")

weights <- c(
    0.2, 0.2, 0.2, 0.2, 0.2,
    0, 0.2, 0, 0, 0.8,
    0.1, 0.1, 0.1, 0, 0.7
)

weights_table <-  tibble(ticker_symbols) %>%
    tq_repeat_df(n = 3) %>%
    bind_cols(tibble(weights)) %>%
    group_by(portfolio)


stock_returns_monthly_multi <- tech_stock_returns_monthly %>%
    tq_repeat_df(n = 3)

# Calculate montly returns for all portfolios
portfolio_returns_monthly_multi <- stock_returns_monthly_multi %>%
    tq_portfolio(assets_col   = symbol, 
                 returns_col  = monthly_return, 
                 weights      = weights_table, 
                 col_rename   = "portfolio_return",
                 wealth.index = FALSE) 

# Calculate what an investment of 1000 will grow to 
portfolio_growth_monthly_multi <- stock_returns_monthly_multi %>%
    tq_portfolio(assets_col   = symbol, 
                 returns_col  = monthly_return, 
                 weights      = weights_table, 
                 col_rename   = "investment.growth",
                 wealth.index = TRUE) %>%
    mutate(investment.growth = investment.growth * 1000)

portfolio_growth_monthly_multi %>%
  ggplot(aes(x = date, y = investment.growth, colour = as.factor(portfolio))) +
  geom_line(size = 2) +
  labs(title = "Portfolio Growth",
       subtitle = "Comparing Multiple Portfolios",
       x = "", y = "Portfolio Value",
       color = "Portfolio") +
  theme_bw()+
  scale_y_continuous(labels = scales::dollar)+
  scale_colour_discrete(name="Portfolio",
                      labels=c("Naive", "Bitcoiners", "Binge Watchers"))

# Returns a basic set of statistics that match the period of the data passed in (e.g., monthly returns 
# will get monthly statistics, daily will be daily stats, and so on).

portfolio_risk_return <- portfolio_returns_monthly_multi %>%
  tq_performance(Ra = portfolio_return, Rb = NULL, performance_fun = table.Stats) %>% 
  select(portfolio, ArithmeticMean, GeometricMean, Minimum,Maximum,Stdev, Quartile1, Quartile3) 

portfolio_risk_return %>% 
  kable() %>%
  kable_styling(c("striped", "bordered"))

portfolio	ArithmeticMean	GeometricMean	Minimum	Maximum	Stdev	Quartile1	Quartile3
1	0.026	0.023	-0.176	0.232	0.069	-0.017	0.076
2	0.033	0.028	-0.242	0.408	0.104	-0.028	0.079
3	0.032	0.028	-0.232	0.360	0.098	-0.026	0.074

ggplot(portfolio_risk_return, 
       aes(x=Stdev, 
           y = ArithmeticMean,
           label= portfolio, 
           colour= as.factor(portfolio)))+
  geom_point(size = 4)+
  labs(title = 'Risk/Return profile of the three portfolios', 
       x = 'Risk (stdev of monthly returns)', 
       y ="Average monthly return")+
  theme_bw()+
  scale_x_continuous(labels = scales::percent)+
  scale_y_continuous(labels = scales::percent)+
  scale_colour_discrete(name="Portfolio",
                      labels=c("Naive", "Bitcoiners", "Binge Watchers"))+
  geom_text_repel()

Data from the Federal Reserve Economic Data with `tidyquant`

A lot of economic data can be extracted from the Federal Reserve Economic Data (FRED) database. For each data we are interested, we need to get its FRED symbol; for instance, if we cared about commodities, we can select the Henry Hub Natural Gas Spot Price and notice that its FRED symbol is DHHNGSP.

So, if we wanted to download this, as well as prices of WTI crude, gold, and USD:EUR, we first identify the FRED codes which are shown below

Henry Hub Natural Gas Spot Price: DHHNGSP
WTI Crude Oil Prices: DCOILWTICO
Gold Fixing Price:GOLDAMGBD228NLBM
U.S. / Euro Exchange Rate: DEXUSEU

To get the data and plot it

natgas_spot  <-   tq_get("DHHNGSP", get = "economic.data",
                       from = "2011-01-01",
                       to   = "2020-07-31")

ggplot(natgas_spot, aes(x=date, y=price)) +
  geom_line()+
  labs(x="Year", 
       y="NatGas Spot price", 
       title = "Henry Hub Natural Gas Spot Prices")+
  scale_y_continuous(labels = scales::dollar)+
  guides(fill=guide_legend(title=NULL))+
  theme_bw()+
  NULL

wti_price  <-   tq_get("DCOILWTICO", get = "economic.data",
                       from = "2011-01-01",
                       to   = "2020-07-31")

ggplot(wti_price, aes(x=date, y=price)) +
  geom_line()+
  labs(x="Year", 
       y="WTI price", 
       title = "West Texas Intermediate Crude Oil (WTI) Prices")+
  scale_y_continuous(labels = scales::dollar)+
  guides(fill=guide_legend(title=NULL))+
  theme_bw()+
  NULL

gold_price  <-   tq_get("GOLDAMGBD228NLBM", get = "economic.data",
                        from = "2011-01-01",
                        to   = "2020-07-31") 

ggplot(gold_price, aes(x=date, y=price)) +
  geom_line()+
  labs(x="Year", 
       y="Gold price", 
       title = "Gold Fixing Price 10:30 A.M. (London time) in London Bullion Market")+
  scale_y_continuous(labels = scales::dollar)+
  guides(fill=guide_legend(title=NULL))+
  theme_bw()+
  NULL

USDEUR_rate <-   tq_get("DEXUSEU", get = "economic.data",
                        from = "2011-01-01",
                        to   = "2020-07-31") 

ggplot(USDEUR_rate, aes(x=date, y=price)) +
  geom_line()+
  labs(x="Year", 
       y="Exchange rate", 
       title = "USD to EUR Exchange Rate")+
  scale_y_continuous(labels = scales::dollar)+
  guides(fill=guide_legend(title=NULL))+
  theme_bw()+
  NULL

Now suppose we wanted to check if there is any correlation between natgas spot prices, WTI, and Gold prices. We will download prices, then calculate returns, calculate statistics on daily returns, and visualise some of the returns.

commodities <- c("DHHNGSP", "DCOILWTICO", "GOLDAMGBD228NLBM")

commodities_prices  <- tq_get(commodities, get = "economic.data",
                              from = "2011-01-01",
                              to   = "2020-07-31") %>% 
  group_by(symbol) 


commodities_returns_daily <- commodities_prices %>% na.omit() %>% 
  tq_transmute(select     = price, 
               mutate_fun = periodReturn, 
               period     = "daily", 
               type       = "log",
               col_rename = "daily.returns")  

#calculate monthly  returns
commodities_returns_monthly <- commodities_prices %>%
  tq_transmute(select     = price, 
               mutate_fun = periodReturn, 
               period     = "monthly", 
               type       = "arithmetic",
               col_rename = "monthly.returns") 

favstats(daily.returns ~ symbol,  data=commodities_returns_daily) %>% 
  mutate(
    annual_mean = mean *250,
    annual_sd = sd * sqrt(250)
  ) %>% 
  select(symbol, min, median, max, mean, sd, annual_mean, annual_sd)  %>% 
  kable() %>%
  kable_styling(c("striped", "bordered"))

symbol	min	median	max	sd	annual_mean	annual_sd
DCOILWTICO	-0.281	0.001	0.426	0.030	-0.008	0.474
DHHNGSP	-0.476	0.000	0.525	0.042	-0.093	0.668
GOLDAMGBD228NLBM	-0.089	0.000	0.068	0.010	0.034	0.154

ggplot(commodities_returns_daily, aes(x=daily.returns, fill=symbol))+
  geom_density()+
  coord_cartesian(xlim=c(-0.05,0.05)) + 
  scale_x_continuous(labels = scales::percent_format(accuracy = 2))+
  facet_grid(rows = (vars(symbol))) + 
  theme_bw()+
  labs(x="Daily Returns", 
       y="Density", 
       title = "Charting the Distribution of Daily Log Returns")+
  guides(fill=FALSE) +
  NULL

ggplot(commodities_returns_daily, aes(x=symbol, y=daily.returns))+
  geom_boxplot(aes(colour=symbol))+
  coord_flip()+
  scale_y_continuous(labels = scales::percent_format(accuracy = 2))+
  theme_bw()+
  labs(x="Daily Returns", 
       y="", 
       title = "Boxplot of Daily Log Returns")+
  theme(legend.position="none") +
  NULL

commodities_returns_daily %>% 
  pivot_wider(names_from="symbol", values_from="daily.returns") %>% 
  na.omit() %>% 
  select(-date) %>% 
  dplyr::rename(
    "NatGas" = 'DHHNGSP',
    "WTI Oil" = 'DCOILWTICO',
    "Gold" = 'GOLDAMGBD228NLBM'
  ) %>% 
  ggpairs()+
  theme_bw()

Acknowledgments

This page is derived in part from Performance Analytics with tidyquant by Matt Dancho.

Confidence Intervals

Sat, 25 Jul 2020 00:00:00 +0000

Confidence Intervals

Sat, 25 Jul 2020 00:00:00 +0000

Exploratory Data Analysis for Modelling

Sat, 25 Jul 2020 00:00:00 +0000

Exploratory Data Analysis for Modelling

Sat, 25 Jul 2020 00:00:00 +0000

EDA. Is Y skewed? Does it need to be transformed? Why?

`GGally::ggpairs()` to get scatter plot + correlation matrix

R Syntax, Vectors, missing data

Sat, 25 Jul 2020 00:00:00 +0000

R Syntax

We can type commands in the command prompt and use R as a simple calculator. For instance, try typing 5 + 20, and hitting enter. When you do this, you’ve entered a command, and R will execute that command. However, it’s more interesting when we can create objects or variables and work with these beasts!

Assignmnent Operator `<-`

R treats everything (single numbers, lists, vectors, datasets) as objects. To create an object, we must use the assignment operator <-. For instance, if we had data on a student whose name is Alex, is 28 years old, and comes from Athens, we would create three objects, name, height, and city and assign the values of Alex, 28, and Athens respectively, we would type

name <- "Alex"
age <- 28
city <- "Athens"

The two objects have now been created; if we wanted to print out their values, we can use the print() function or just type the names of the objects.

print(name); print(age); print(city)

## [1] "Alex"

## [1] 28

## [1] "Athens"

name

## [1] "Alex"

age

## [1] 28

city

## [1] "Athens"

You can mentally read the command age <- 28 as object age becomes equal to the value 28. There is a keyboard shortcut Alt + - to get the assignment operator. We can do more interesting and useful things creating variables and assigning values to them. For instance, if we have the relevant dimensions and wanted to calculate the area and volume of a room, we could do it as follows:

room_length <- 5.63
room_width  <- 6.48
room_height <- 2.93
room_area <- room_length * room_width
room_volume <- room_length * room_width * room_height

room_area

## [1] 36.4824

room_volume

## [1] 106.8934

R is case sensitive

R is case sensitive and needs everything exactly as it was defined. age is different from AgE and Age. So if you type

age <- 28
AgE <- 34
Age <- 55

age; AgE; Age

## [1] 28

## [1] 34

## [1] 55

R will create three different objects.

Typos

R is a brilliant piece of software, but it cannot handle typos. Unlike Google’s search, “Did you mean…”, it takes it on faith that what you typed is exactly what you meant. For example, suppose that you forgot to hit the shift key when trying to type +, and as a result your command ended up being 5 = 20 rather than 5 + 20. Here’s what happens:

5 = 20

## Error in 5 = 20: invalid (do_set) left-hand side to assignment

R attempted to interpret 5 = 20 as a command, and spits out an error message because this makes no sense to it. Even more subtle is the fact that some typos won’t produce errors at all, because they happen to correspond to R commands. For instance, suppose that instead of 5 + 20, I mistakenly type command 5 - 20. Clearly, R has no way of knowing that you meant to add 20 to 5, not subtract 20 from 5, so what happens this time is this:

5 - 20

## [1] -15

In this case, R produces the right answer, but to the the wrong question.

R will always try to do exactly what you ask it to do. There is no autocorrect or equivalent to “Did you mean..” in R, and for good reason. When doing advanced stuff and even the simplest of statistics is pretty advanced in a lot of ways, it’s dangerous to let a mindless automaton like R try to overrule the human user. But because of this, it’s your responsibility to be careful. Always make sure you type exactly what you mean. When dealing with computers, it’s not enough to type approximately the right thing. In general, you absolutely must be precise in what you say to R … like all machines it is too stupid to be anything other than absurdly literal in its interpretation.

Comments

It is useful to put comments in your code, to make everything more readable. These comments could help others and you when you go back to your code in the future. R comments start with a hashtag sign #. Everything after the hashtag to the end of the line will be ignored by R. RStudio by default thinks that every line you write is a command; if you want to turn a line into a comment, place the cursor in the line and hit Ctrl + Shift + C in Windows or Cmd + Shift + C in a Mac.

# This line is a comment and will be ignored when run.
city # Text after the hashtag "#" is also ignored.

## [1] "Athens"

R knows you’re not finished

If you hit enter in a situation where it’s obvious to R that you haven’t actually finished typing the command, R is just smart enough to keep waiting. For example, if you wanted to calculate 15 - 4, and start by typing type 15 - and then press enter by mistake, R is smart enough to realise that you probably wanted to type in another number. So here’s what happens:

> 15 -
+

and there’s a blinking cursor next to the plus + sign. What this means is that R is still waiting for you to finish. It thinks you’re still typing your command, so it hasn’t tried to execute it yet. In other words, this plus sign is actually another command prompt. It’s different from the usual one (i.e., the > symbol) to remind you that R is going to add whatever you type now to what you typed last time. For example, if I then go on to type 4 and hit enter, what we get:

> 15 -
+ 4
[1] 11

And as far as R is concerned, this is exactly the same as if you had typed 15 - 4.

By the way, if after entering the 15 - you wanted to stop execution and cancel your command, just hit the escape key. R will return you to the normal command prompt (i.e. >) without attempting to execute the botched command.

Arithmetic Operations and Functions

R has the basic operators and you can use it as as simple calculator: addition is +, subtraction is -, multiplication is *, division is /, and ^ is the power operator:

2 + 3

## [1] 5

5 - 8

## [1] -3

13 * 21

## [1] 273

34 / 55

## [1] 0.6181818

(5 * 13)/4 - 7

## [1] 9.25

# ^ : to the power off
2^3

## [1] 8

# for exponentiation, you can also use **
2 ** 3

## [1] 8

# square root
sqrt(25)

## [1] 5

Besides the basic operations functions, you can use standard mathematical functions

Rounding -round(), floor(), ceiling(),
Logarithms and Exponentials -exp(), log(), log10(), log2()

# R knows pi = 3.1415926...

# round to 2 decimal places 
round(pi, digits = 2); round(pi,2)

## [1] 3.14

## [1] 3.14

#Round down to nearest integer
floor(pi)

## [1] 3

#Round up to nearest integer
ceiling(pi)

## [1] 4

Main Data types and Vectors

character: sometimes referred to as string data, tend to be surrounded by quotes <chr>
numeric: real or decimal numbers, sometimes referred to as “double” <dbl>
integer: a subset of numeric in which numbers are stored as integers <int>
factor: a categorical variables with different categories sorted alphabetically by default <fct>
logical: Boolean data (TRUE and FALSE) <lgl>

Your turn

Vectors

A vector is a collection of objects. There is a magical operator in R, c which we use to combine different elements.

# assign vector
ages <- c(20:30, 35, 50, 42, 72) 

# recall vector
ages

##  [1] 20 21 22 23 24 25 26 27 28 29 30 35 50 42 72

# how many things are in the vector 'ages'?
length(ages)

## [1] 15

# what type of object is 'ages?
class(ages)

## [1] "numeric"

R allows vectorized operations, so we can get the average, or median of ages by just typing

# performing functions with vectors
mean(ages)

## [1] 31.6

median(ages)

## [1] 27

We can also have a collection of strings, or characters

# vector of days of the week 
days <- c("Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday")

In this case, each word is encased in quotation marks, indicating they are characters rather than object names.

Please answer the following questions about days:

How many values are in days?
What type of data (class) is days?
Overview of days

Manipulating vectors

# add a value to end of vector
ages <- c(ages, 90)

# add value at the beginning
ages <- c(30, ages)

# extracting second value
days[2]

## [1] "Tuesday"

# excluding (dropping) second value
days[-2]

## [1] "Monday"    "Wednesday" "Thursday"  "Friday"    "Saturday"  "Sunday"

# extracting first and third values
days[c(1, 3)]

## [1] "Monday"    "Wednesday"

Your turn

R tends to handle interpreting data types in the background of most operations. Usually it tries to coerce data to fit the general pattern of the data given to it.

What type of data is each of the following objects? Anything unusual?

num_char <- c(1, 2, 3, "a")
num_logical <- c(1, 2, 3, TRUE)
char_logical <- c("a", "b", "c", TRUE)
tricky <- c(1, 2, 3, "4")

Factors

days is a character vector so R internally sorts it alphabetically and it thinks that Friday should be first. If we wanted to make into a categorical variable, we use

days <- factor(days)

We can reorder, or relevel the factor, using fct_relevel from the tidyverse package forcats, or using levels from baseR.

days_sorted <- forcats::fct_relevel(days, levels = c("Monday",
                                     "Tuesday",
                                     "Wednesday",
                                     "Thursday",
                                     "Friday",
                                     "Saturday",
                                     "Sunday"))

days_sorted2 <- factor(days, levels = c("Monday",
                                     "Tuesday",
                                     "Wednesday",
                                     "Thursday",
                                     "Friday",
                                     "Saturday",
                                     "Sunday"))

Your turn

Missing data, or `NA`

# create a vector with missing data
times <- c(2, 4, 4, NA, 6)

NA is not a character

# calculate mean and max on vector with missing data
mean(times)

## [1] NA

max(times)

## [1] NA

# add argument to remove NA
mean(times, na.rm = TRUE)

## [1] 4

max(times, na.rm = TRUE)

## [1] 6

# remove incomplete cases
na.omit(times)

## [1] 2 4 4 6
## attr(,"na.action")
## [1] 4
## attr(,"class")
## [1] "omit"

Your turn

Online Quiz: Variable Types - Vectors

R Syntax, Vectors, missing data

Sat, 25 Jul 2020 00:00:00 +0000

R Syntax

Assignmnent Operator `<-`

name <- "Alex"
age <- 28
city <- "Athens"

The two objects have now been created; if we wanted to print out their values, we can use the print() function or just type the names of the objects.

print(name); print(age); print(city)

## [1] "Alex"

## [1] 28

## [1] "Athens"

name

## [1] "Alex"

age

## [1] 28

city

## [1] "Athens"

room_length <- 5.63
room_width  <- 6.48
room_height <- 2.93
room_area <- room_length * room_width
room_volume <- room_length * room_width * room_height

room_area

## [1] 36.4824

room_volume

## [1] 106.8934

R is case sensitive

R is case sentitive and needs everything exactly as it was defined. age is different from AgE and Age. So if you type

age <- 28
AgE <- 34
Age <- 55

age; AgE; Age

## [1] 28

## [1] 34

## [1] 55

R will create three different objects.

Typos

5 = 20

## Error in 5 = 20: invalid (do_set) left-hand side to assignment

5 - 20

## [1] -15

In this case, R produces the right answer, but to the the wrong question.

Comments

# This line is a comment and will be ignored when run.
city # Text after the hashtag "#" is also ignored.

## [1] "Athens"

R knows you’re not finished

> 15 -
+

> 15 -
+ 4
[1] 11

And as far as R is concerned, this is exactly the same as if you had typed 15 - 4.

Arithmetic Operations and Functions

R has the basic operators and you can use it as as simple calculator: addition is +, subtraction is -, multiplication is *, division is /, and ^ is the power operator:

2 + 3

## [1] 5

5 - 8

## [1] -3

13 * 21

## [1] 273

34 / 55

## [1] 0.6181818

(5 * 13)/4 - 7

## [1] 9.25

# ^ : to the power off
2^3

## [1] 8

# for exponentiation, you can also use **
2 ** 3

## [1] 8

# square root
sqrt(25)

## [1] 5

Besides the basic operations functions, you can use standard mathematical functions

Rounding -round(), floor(), ceiling(),
Logarithms and Exponentials -exp(), log(), log10(), log2()

# R knows pi = 3.1415926...

# round to 2 decimal places 
round(pi, digits = 2); round(pi,2)

## [1] 3.14

## [1] 3.14

#Round down to nearest integer
floor(pi)

## [1] 3

#Round up to nearest integer
ceiling(pi)

## [1] 4

Main Data types and Vectors

character: sometimes referred to as string data, tend to be surrounded by quotes <chr>
numeric: real or decimal numbers, sometimes referred to as “double” <dbl>
integer: a subset of numeric in which numbers are stored as integers <int>
factor: a categorical variables with different categories sorted alphabetically by default <fct>
logical: Boolean data (TRUE and FALSE) <lgl>

Your turn

Vectors

A vector is a collection of objects. There is a magical operator in R, c which we use to combine different elements.

# assign vector
ages <- c(20:30, 35, 50, 42, 72) 

# recall vector
ages

##  [1] 20 21 22 23 24 25 26 27 28 29 30 35 50 42 72

# how many things are in the vector 'ages'?
length(ages)

## [1] 15

# what type of object is 'ages?
class(ages)

## [1] "numeric"

R allows vectorized operations, so we can get the average, or median of ages by just typing

# performing functions with vectors
mean(ages)

## [1] 31.6

median(ages)

## [1] 27

We can also have a collection of strings, or characters

# vector of days of the week 
days <- c("Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday")

In this case, each word is encased in quotation marks, indicating they are characters rather than object names.

Please answer the following questions about days:

How many values are in days?
What type of data (class) is days?
Overview of days

Manipulating vectors

# add a value to end of vector
ages <- c(ages, 90)

# add value at the beginning
ages <- c(30, ages)

# extracting second value
days[2]

## [1] "Tuesday"

# excluding (dropping) second value
days[-2]

## [1] "Monday"    "Wednesday" "Thursday"  "Friday"    "Saturday"  "Sunday"

# extracting first and third values
days[c(1, 3)]

## [1] "Monday"    "Wednesday"

Your turn

R tends to handle interpreting data types in the background of most operations. Usually it tries to coerce data to fit the general pattern of the data given to it.

What type of data is each of the following objects? Anything unusual?

num_char <- c(1, 2, 3, "a")
num_logical <- c(1, 2, 3, TRUE)
char_logical <- c("a", "b", "c", TRUE)
tricky <- c(1, 2, 3, "4")

Factors

days is a character vector so R internally sorts it alphabetically and it thinks that Friday should be first. If we wanted to make into a categorical variable, we use

days <- factor(days)

We can reorder, or relevel the factor, using fct_relevel from the tidyverse package forcats, or using levels from baseR.

days_sorted <- forcats::fct_relevel(days, levels = c("Monday",
                                     "Tuesday",
                                     "Wednesday",
                                     "Thursday",
                                     "Friday",
                                     "Saturday",
                                     "Sunday"))

days_sorted2 <- factor(days, levels = c("Monday",
                                     "Tuesday",
                                     "Wednesday",
                                     "Thursday",
                                     "Friday",
                                     "Saturday",
                                     "Sunday"))

Your turn

Missing data, or `NA`

# create a vector with missing data
times <- c(2, 4, 4, NA, 6)

NA is not a character

# calculate mean and max on vector with missing data
mean(times)

## [1] NA

max(times)

## [1] NA

# add argument to remove NA
mean(times, na.rm = TRUE)

## [1] 4

max(times, na.rm = TRUE)

## [1] 6

# remove incomplete cases
na.omit(times)

## [1] 2 4 4 6
## attr(,"na.action")
## [1] 4
## attr(,"class")
## [1] "omit"

Your turn

Online Quiz: Variable Types - Vectors

Visualise data

Sat, 25 Jul 2020 00:00:00 +0000

Introduction to `ggplot2` by visualising numeric data.

We will start with the gapminder data set. We look at its contents

glimpse(gapminder)

## Rows: 1,704
## Columns: 6
## $ country   <fct> Afghanistan, Afghanistan, Afghanistan, Afghanistan, Afgha...
## $ continent <fct> Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asi...
## $ year      <int> 1952, 1957, 1962, 1967, 1972, 1977, 1982, 1987, 1992, 199...
## $ lifeExp   <dbl> 28.801, 30.332, 31.997, 34.020, 36.088, 38.438, 39.854, 4...
## $ pop       <int> 8425333, 9240934, 10267083, 11537966, 13079460, 14880372,...
## $ gdpPercap <dbl> 779.4453, 820.8530, 853.1007, 836.1971, 739.9811, 786.113...

Animating changes

Racing bars! We will create a simple bar graph showing the evolution of GDP per capita for the top 8 countries

IMDB movie ratings: Scatterplots and relationships

For this section, we will use a sample of movies released since 2000 with data from IMDB. We have data on movies from the following six genres:

Action
Adventure
Comedy
Drama
Animation
Documentary

imdb <- read_csv(here::here("data", "movies.csv"))
imdb_short <- imdb %>% 
  filter(genre %in% c("Action", "Adventure", "Comedy", "Drama", "Animation", "Documentary"),
         year >= 2000)

glimpse(imdb_short)

## Rows: 1,762
## Columns: 11
## $ title               <chr> "Avatar", "Jurassic World", "The Avengers", "Th...
## $ genre               <chr> "Action", "Action", "Action", "Action", "Action...
## $ director            <chr> "James Cameron", "Colin Trevorrow", "Joss Whedo...
## $ year                <dbl> 2009, 2015, 2012, 2008, 2015, 2012, 2004, 2013,...
## $ duration            <dbl> 178, 124, 173, 152, 141, 164, 93, 146, 151, 103...
## $ gross               <dbl> 760505847, 652177271, 623279547, 533316061, 458...
## $ budget              <dbl> 2.37e+08, 1.50e+08, 2.20e+08, 1.85e+08, 2.50e+0...
## $ cast_facebook_likes <dbl> 4834, 8458, 87697, 57802, 92000, 106759, 1148, ...
## $ votes               <dbl> 886204, 418214, 995415, 1676169, 462669, 114433...
## $ reviews             <dbl> 3777, 1934, 2425, 5312, 1752, 3514, 688, 1208, ...
## $ rating              <dbl> 7.9, 7.0, 8.1, 9.0, 7.5, 8.5, 7.2, 7.6, 7.3, 8....

IMDB movie ratings: Boxplots, violin plots

Let us consider the rating movies got according to their genre. How can we visualise the distribution of ratings?

ggplot(imdb_short,
       aes(x=rating, y = genre, fill = genre,  alpha = 0.2))+
  geom_boxplot()+
  theme_minimal()+
  theme(legend.position = "none")

ggplot(imdb_short,
       aes(x=rating, y = genre, fill = genre,  alpha = 0.2))+
  geom_violin()+
  theme_minimal()+
  theme(legend.position = "none")

Multiple panels using `facet_wrap()` and `facet_grid()`

imdb_short %>% 
  filter(genre %in% c("Action", "Comedy", "Drama"),
         year >= 2010) %>% 
ggplot(aes(x=rating,  fill = genre,  alpha = 0.2))+
  geom_boxplot()+
  theme_minimal()+
  theme(legend.position = "none")+
  facet_grid(
    rows= vars(year),
    cols= vars(genre)
  )

imdb_short %>% 
  filter(genre %in% c("Action", "Comedy", "Drama"),
         year >= 2010) %>% 
ggplot(aes(x=rating,  fill = genre,  alpha = 0.2))+
  geom_boxplot()+
  theme_minimal()+
  theme(legend.position = "none")+
  facet_grid(
    rows= vars(cut(budget, 3)),
    cols= vars(genre)
  )

Visualise data

Sat, 25 Jul 2020 00:00:00 +0000

Introduction to `ggplot2` by visualising numeric data.

We will start with the gapminder data set. We look at its contents

glimpse(gapminder)

## Rows: 1,704
## Columns: 6
## $ country   <fct> Afghanistan, Afghanistan, Afghanistan, Afghanistan, Afgha...
## $ continent <fct> Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asi...
## $ year      <int> 1952, 1957, 1962, 1967, 1972, 1977, 1982, 1987, 1992, 199...
## $ lifeExp   <dbl> 28.801, 30.332, 31.997, 34.020, 36.088, 38.438, 39.854, 4...
## $ pop       <int> 8425333, 9240934, 10267083, 11537966, 13079460, 14880372,...
## $ gdpPercap <dbl> 779.4453, 820.8530, 853.1007, 836.1971, 739.9811, 786.113...

Animating changes

Racing bars! We will create a simple bar graph showing the evolution of GDP per capita for the top 8 countries

IMDB movie ratings: Scatterplots and relationships

For this section, we will use a sample of movies released since 2000 with data from IMDB. We have data on movies from the following six genres:

Action
Adventure
Comedy
Drama
Animation
Documentary

imdb <- read_csv(here::here("data", "movies.csv"))
imdb_short <- imdb %>% 
  filter(genre %in% c("Action", "Adventure", "Comedy", "Drama", "Animation", "Documentary"),
         year >= 2000)

glimpse(imdb_short)

## Rows: 1,762
## Columns: 11
## $ title               <chr> "Avatar", "Jurassic World", "The Avengers", "Th...
## $ genre               <chr> "Action", "Action", "Action", "Action", "Action...
## $ director            <chr> "James Cameron", "Colin Trevorrow", "Joss Whedo...
## $ year                <dbl> 2009, 2015, 2012, 2008, 2015, 2012, 2004, 2013,...
## $ duration            <dbl> 178, 124, 173, 152, 141, 164, 93, 146, 151, 103...
## $ gross               <dbl> 760505847, 652177271, 623279547, 533316061, 458...
## $ budget              <dbl> 2.37e+08, 1.50e+08, 2.20e+08, 1.85e+08, 2.50e+0...
## $ cast_facebook_likes <dbl> 4834, 8458, 87697, 57802, 92000, 106759, 1148, ...
## $ votes               <dbl> 886204, 418214, 995415, 1676169, 462669, 114433...
## $ reviews             <dbl> 3777, 1934, 2425, 5312, 1752, 3514, 688, 1208, ...
## $ rating              <dbl> 7.9, 7.0, 8.1, 9.0, 7.5, 8.5, 7.2, 7.6, 7.3, 8....

IMDB movie ratings: Boxplots, violin plots

Let us consider the rating movies got according to their genre. How can we visualise the distribution of ratings?

ggplot(imdb_short,
       aes(x=rating, y = genre, fill = genre,  alpha = 0.2))+
  geom_boxplot()+
  theme_minimal()+
  theme(legend.position = "none")

ggplot(imdb_short,
       aes(x=rating, y = genre, fill = genre,  alpha = 0.2))+
  geom_violin()+
  theme_minimal()+
  theme(legend.position = "none")

Multiple panels using `facet_wrap()` and `facet_grid()`

imdb_short %>% 
  filter(genre %in% c("Action", "Comedy", "Drama"),
         year >= 2010) %>% 
ggplot(aes(x=rating,  fill = genre,  alpha = 0.2))+
  geom_boxplot()+
  theme_minimal()+
  theme(legend.position = "none")+
  facet_grid(
    rows= vars(year),
    cols= vars(genre)
  )

imdb_short %>% 
  filter(genre %in% c("Action", "Comedy", "Drama"),
         year >= 2010) %>% 
ggplot(aes(x=rating,  fill = genre,  alpha = 0.2))+
  geom_boxplot()+
  theme_minimal()+
  theme(legend.position = "none")+
  facet_grid(
    rows= vars(cut(budget, 3)),
    cols= vars(genre)
  )

Installing R and RStudio

Sat, 25 Jul 2020 00:00:00 +0000

In this section we download and install R and R Studio, and then show you how to write R commands and navigate around the RStudio interface. The goal in this chapter is not to learn any statistical or programming concepts: we’re just trying to learn how R works and get comfortable interacting with the system. We’ll spend a bit of time using R as a simple calculator. Specifically, we will learn the basics of R and RStudio, namely

How to install R and RStudio interface
How to navigate around the RStudio interface; a free Integrated Development Environment (IDE) for R
How to install and load packages that provide extra functionality for R

Installing R & RStudio

An important distinction to remember is between the R programming language itself, and the software you use to interact with R. You could choose to interact with R directly from the terminal, but that’s painful, so most people use an integrated development environment (IDE), which takes care of a lot of boring tasks for you. To get started, make sure you have both R and RStudio installed on your computer. Both are free and open source, and for most people they should be straightforward to install.

Install `XCode` if you have a Mac

If you have a Mac make sure that before installing R and R studio you
- upgrade to the latest version of macOS
- install XCode through the appStore

Install R

First you need to install R itself (the engine). Go to the CRAN (Collective R Archive Network)– this is the site where R itself and most R packages live. Click on “Download R for XXX”, where XXX is either Mac or Windows:

Double click on the downloaded file. Click *Yes** through all the prompts to install like any other program. once finished, proceed to install R Studio.

Install RStudio IDE

Go to the R studio website, and follow the links to download. RStudio is a powerful user interface for programming in R. I suggest you install the preview version of R studio.

To get started, open the Rstudio application (i.e., RStudio.exe or RStudio.app), not the vanilla application (i.e., not R.exe or R.app). You should be looking at something like this:

The RStudio IDE is divided into 4 separate panes (one of which is hidden for now) which all serve specific functions. The Console starts with information about the R version number, license and contributors. The last line is a standard prompt > that indicates R is ready and expecting instructions to do something.

You edit scripts in the editor panel in R Studio and see results in the bottom right output panel.

For now, to make sure R and RStudio are setup correctly, type x <- 3 + 2 into the Console pane and execute it by pressing Enter/Return. You just created an object in R called x. What does this object contain? Type print(x) or just x into the console and press enter again. Your console should now contain the following output

x <- 3 + 2
print(x)

## [1] 5

Congratulations! You installed R and RStudio succesfully, created an object x to which you assigned the value 3+2 and managed to print the value of x

Change character encoding to `UTF-8`, and `UTF-8` only

This may seem like an overly technical issue, but please bear with me. Since LBS is a very international school, we always seem to have issues with the language, or character encoding (Chinese, Arabic, Greek, Cyrillic, Hebrew, Thai, French, German, etc.), that people use in their computers. By default, all base R functions use the system native language encoding which has to do with the different languages some of us may have on our computers. Chinese and Greek users, having a completely different alphabet, typically report issues/problems/errors related to character encodings.

UTF-8 is the best possible character encoding, it works everywhere and we shall ask R Studio to use UTF-8 encoding globally. Please go to Tools… Global Options… Code… Saving and and change the default text encoding to UTF-8 as shown below:

Exiting R & RStudio

When quitting RStudio you will be asked whether to Save workspace with two options:

Yes - Your current R workspace (containing the work that you have done) will be restored next time you open RStudio.
No - You will start with a fresh R session next time you open RStudio. For now select “No” to prevent errors being carried over from previous sessions.

In general, it’s good practice to always start with a fresh new session. If you want to do that, please go to Tools… Global Optionsand make sure that

Restore .RData into workspace at startup is NOT ticked
Save workspace to .RData on exit: select Never
Always save history (even when not saving .RData) is NOT ticked

as shown below

Updating R and RStudio

If you already installed R or RStudio for a previous course, update both to the most current version. Generally this entails downloading and installing the most recent version of both programs. When you update R, you don’t actually remove the old version - you have all versions on your computer and default to the most recent one. Sometimes this is useful when specific R libraries require an older version of R, however we will generally stick to the most recent versions of R and RStudio.
When you update R, make sure to update your packages as well. The following command should perform most of this work, update.packages(ask = FALSE, checkBuilt = TRUE) or you can go through the Packages tab in the bottom right panel of RStudio.

R commands

We have already seen how we can type commands in the command prompt and use R as a simple calculator. For instance, try typing 5 + 20, and hitting enter. When you do this, you’ve entered a command, and R will execute that command. What you see on screen now will be this:

5 + 20

## [1] 25

Assignmnent Operator `<-`

name <- "Alex"
age <- 28
city <- "Athens"

The two objects have now been created; if we wanted to print out their values, we can use the print() function or just type the names of the objects.

print(name); print(age); print(city)

## [1] "Alex"

## [1] 28

## [1] "Athens"

name

## [1] "Alex"

age

## [1] 28

city

## [1] "Athens"

room_length <- 5.63
room_width  <- 6.48
room_height <- 2.93
room_area <- room_length * room_width
room_volume <- room_length * room_width * room_height

room_area

## [1] 36.4824

room_volume

## [1] 106.8934

R is case sensitive

R is case sentitive and needs everything exactly as it was defined. age is different from AgE and Age. So if you type

age <- 28
AgE <- 34
Age <- 55

age; AgE; Age

## [1] 28

## [1] 34

## [1] 55

R will create three different objects.

Typos

5 = 20

## Error in 5 = 20: invalid (do_set) left-hand side to assignment

5 - 20

## [1] -15

In this case, R produces the right answer, but to the the wrong question.

Comments

# This line is a comment and will be ignored when run.
city # Text after the hashtag "#" is also ignored.

## [1] "Athens"

R knows you’re not finished

> 15 -
+

> 15 -
+ 4
[1] 11

And as far as R is concerned, this is exactly the same as if you had typed 15 - 4.

Arithmetic Operations and Functions

R has the basic operators and you can use it as as simple calculator: addition is +, subtraction is -, multiplication is *, division is /, and ^ is the power operator:

2 + 3

## [1] 5

5 - 8

## [1] -3

13 * 21

## [1] 273

34 / 55

## [1] 0.6181818

(5 * 13)/4 - 7

## [1] 9.25

# ^ : to the power off
2^3

## [1] 8

# square root
sqrt(25)

## [1] 5

Besides the basic operations functions, you can use standard mathematical functions

Rounding -round(), floor(), ceiling(),
Logarithms and Exponentials -exp(), log(), log10(), log2()

# R knows pi = 3.1415926...

# round to 2 decimal places 
round(pi, digits = 2); round(pi,2)

## [1] 3.14

## [1] 3.14

#Round down to nearest interger
floor(pi)

## [1] 3

#Round up to nearest interger
ceiling(pi)

## [1] 4

RStudio help

At this stage you know how to type in basic commands, including how to use some R functions. Few analysts bother to try to know or remember all the commands. What they really do is use tricks to make their lives easier. The first (and arguably most important one) is to use the internet. If you don’t know how a particular R function works, Google it. There is a lot of R documentation out there, and almost all of it is searchable! For the moment though, I want to call your attention to a couple of simple tricks that Rstudio makes available to you.

Tab autocomplete

The first thing I want to call your attention to is the autocomplete ability in Rstudio. Assume that what you want to do is to round a number. This time around, start typing the name of the function that you want, and then hit the Tab key. Rstudio will then display a little window like the one shown here:

In this figure, we have typed the letters rou at the command line, and then hit tab. The window has two panels. On the left, there’s a list of variables and functions that start with the letters typed shown in black text, and some grey text that tells you where that variable/function is stored. In our case, round is included in the {base} R, what is included in every new installation of R. There’s a few options there, and the one we want is round, but if you’re typing this yourself you’ll notice that when you hit the tab key the window pops up with the top entry highlighted. You can use the up and down arrow keys to select the one that you want. Or, if none of the options look right to you, you can hit the escape key (ESC) or the left arrow key to make the window go away.

In our case, the thing we want is the round option, and the panel on the right tells you a bit about how the function works. This display is really handy. The very first thing it says is round(x, digits = 0): what this is telling you is that the round function has two arguments. The first argument is called x, and it doesn’t have a default value. The second argument is digits, and it has a default value of 0. In a lot of situations, that’s all the information you need. But Rstudio goes a bit further, and provides some additional information about the function underneath. Sometimes that additional information is very helpful, sometimes it’s not: Rstudio pulls that text from the R help documentation, and my experience is that the helpfulness of that documentation varies wildly. Anyway, if you’ve decided that round is the function that you want to use, you can hit the enter key and Rstudio will finish typing the rest of the function name for you.

The history pane

One thing R does is keep track of your *command history**, i.e., it remembers all the commands previously typed. You can access this history in a few different ways. The simplest way is to use the up and down arrow keys. If you hit the up key, the R console will show you the most recent command that you’ve typed. Hit it again, and it will show you the command before that. If you want the text on the screen to go away, hit escape. Using the up and down keys can be handy if you’ve typed a long command that had one typo in it. Rather than having to type it again from scratch, you can use the up key to bring up the command and fix it.

The second way to get access to your command history is to look at the history panel in Rstudio. On the upper right panel of the Rstudio window, you’ll see a tab labelled History.

Click on that, and you’ ll see a list of all your recent commands displayed in that panel– double click on one of the commands, and it will be copied to the R console. You can achieve the same result by selecting the command you want with the mouse and then clicking the *“To Console”** button.

More resources

RStudio have a produced a great series of video tutorials RStudio Essentials Videos
RStudio IDE Cheatsheet

Acknowledgements

This page is derived in part from “R for Psychological Science”.

Textbooks and other resources

Sat, 25 Jul 2020 00:00:00 +0000

The following is a non-exhaustive list of free online textbooks and resources that use R

Textbooks/Readings

R Programming

R for Data Science – Garrett Grolemund and Hadley Wickham
- Open-source online version is available for free; Available for purchase online
- No official solution manual for the book exercises exists, but several can be found online, like this version by Jeffrey B. Arnold. Your exact solutions may vary, but these are a good starting point.
Hands-On Programming with R by Garrett Grolemund. This is a non-statistical introduction to R programming with many hands-on examples.
Advanced R – Hadley Wickham
- Hardcover available online, but the online version is free
- A deeper dive into R as a programming language, not just a tool for data science. Most of this material is best covered on your own after you are familiar with R.
R Programming for Data Science
Modern Data Science with R – Benjamin S. Baumer, Daniel T. Kaplan, Nicholas J. Horton

Statistics with R

Modern Dive: A moderndive into R and the tidyverse by Chester Ismay and Albert Y. Kim
Learning Statistics with R by Danielle Navarro
OpenIntro Statistics Open-source online version is available for free
An Introduction to Statistical Learning: with Applications in R – Gareth James, Daniela Witten, Trevor Hastie, and Robert Tibshirani
- Each chapter includes code that demonstrates how to implement different methods. Unfortunately, their code use a lot of base R functions and syntax, whereas our emphasis is on getting things done with the tidyverse collection of R packages. However, this is still a great book and the code provided is useful.
- You can download a free PDF of the entire book from the authors’ site
Broadening Your Statistical Horizons is an applied textbook on generalized linear models, with all of the examples / code in R.
Forecasting: Principles and Practice
Tidy Modeling with R The purpose of this book is to demonstrate how the tidyverse and tidymodels can be used to produce high quality models.
Text Mining with R by Julia Silge and David Robinson. What happens if your data is text, rather than numbers? What if you wanted to do sentiment analysis?

Visualisations

The de-facto standard for visualisations in R is the ggplot2 package. If you want to read Hadley Wickham’s paper that implemented the grammar of graphics into R, you can find it here

Data Visualization: A Practical Introduction by Kieran Healy.
Fundamentals of Data Visualization by Claus O. Wilke.
R Graphics Cookbook A practical guide by Winston Chang that provides any specific examples/ recipes to help you generate high-quality graphs quickly. I use it as quick reference to get my ggplot working.
The BBC Visual and Data Journalism cookbook for R graphics
Interactive web-based data visualization with R, plotly, and shiny

Spatial Visualisations

Online resources

Data science and statistical programming can be challenging. Computers are dumb and tiny errors in your code can cause hours of frustration (even if you’ve been doing this stuff for years!).

Fortunately, there are tons of online resources to help you with this. Two of the most important are StackOverflow (a Q&A site with thousands of answers to all sorts of statistical and programming questions) and RStudio Community (a forum specifically designed for people using RStudio and the tidyverse).

I highly recommend subscribing to the R Weekly newsletter which is sent every Monday and is full of helpful tutorials and ideas on how to do stuff with R.
RStudio Cheatsheets Printable cheat sheets for common R tasks and features

Software

Typora is a lightweight, stand alone editor for Markdown documents

Companies, Government Agencies, and NGOs Using R

Organisations using R

Podcasts

Tim Harford’s More or Less explains and debunks the numbers and statistics used in political debate, the news and everyday life. A great episode on sampling can be found here
Everything Hertz: A podcast by scientists, for scientists. Methodology, scientific life, and bad language

Using Markdown

Sat, 25 Jul 2020 00:00:00 +0000

Markdown is a special kind of markup language that lets you format text with simple syntax. You can then use a converter program like pandoc to convert Markdown into whatever format you want: HTML, PDF, Word, PowerPoint, etc. (see the full list of output types here)

Basic Markdown formatting

Type…	…or…	…to get
Some text in a paragraph. More text in the next paragraph. Always use empty lines between paragraphs.		Some text in a paragraph. More text in the next paragraph. Always use empty lines between paragraphs.
`Italic`	`_Italic_`	Italic
`Bold`	`__Bold__`	Bold
`# Heading 1`		Heading 1
`## Heading 2`		Heading 2
`### Heading 3`		Heading 3
(Go up to heading level 6 with `######`)
`[Link text](http://www.example.com)`		Link text
`![Image caption](/path/to/image.png)`
`Inline code` with backticks		`Inline code` with backticks
`> Blockquote`		Blockquote
- Things in - an unordered - list	* Things in * an unordered * list	Things in an unordered list
1. Things in 2. an ordered 3. list	1) Things in 2) an ordered 3) list	Things in an ordered list
Horizontal line ---	Horizontal line ***	Horizontal line

Mathematical formulas

Markdown uses LaTeX to create fancy mathematical equations. There are tons of little options and features available for math equations—you can find helpful examples of the the most common basic commands here.

You can use math in two different ways: inline or in a display block. To use math inline, wrap it in single dollar signs, like $y = mx + b$ :

Type…	…to get
Based on our regression model for estimating the effect of education on wages is $\hat{y} = \beta_0 + \beta_1 x_1 + \epsilon$, or $\text{Wages} = \beta_0 + \beta_1 \text{Education} + \epsilon$.	Based on our regression model for estimating the effect of education on wages is $\hat{y} = \beta_0 + \beta_1 x_1 + \epsilon$, or $\text{Wages} = \beta_0 + \beta_1 \text{Education} + \epsilon$.

To put an equation on its own line in a display block, wrap it in double dollar signs, like this:

Type…

The quadratic equation was an important part of high school math:

$$
x = \frac{-b \pm \sqrt{b^2 - 4ac}}{2a}
$$

But now we just use computers to solve for $x$.

…to get…

The quadratic equation was an important part of high school math:

\[ x = \frac{-b \pm \sqrt{b^2 - 4ac}}{2a} \]

But now we just use computers to solve for $x$.

Because dollar signs are used to indicate math equations, you can’t just use dollar signs like normal if you’re writing about actual dollars. For instance, if you write This book costs $5.75 and this other costs $40, Markdown will treat everything that comes between the dollar signs as math, like so: “This book costs $5.75 and this other costs $40”.

To get around that, put a backslash (\) in front of the dollar signs, so that This book costs \$5.75 and this other costs \$40 becomes “This book costs $5.75 and this other costs $40”.

Tables

There are 4 different ways to hand-create tables in Markdown—I say “hand-create” because it’s normally way easier to use R to generate these things with packages like pander (use pandoc.table()) or knitr (use kable()). The two most common are simple tables and pipe tables. You can find the full documentation here.

For simple tables, type…

  Right     Left     Center     Default
-------     ------ ----------   -------
     12     12        12            12
    123     123       123          123
      1     1          1             1

Table: Caption goes here

…to get…

Caption goes here
Right	Left	Center	Default
12	12	12	12
123	123	123	123
1	1	1	1

For pipe tables, type…

| Right | Left | Default | Center |
|------:|:-----|---------|:------:|
|   12  |  12  |    12   |    12  |
|  123  |  123 |   123   |   123  |
|    1  |    1 |     1   |     1  |

Table: Caption goes here

…to get…

Caption goes here
Right	Left	Default	Center
12	12	12	12
123	123	123	123
1	1	1	1

Front matter

You can include a special section at the top of a Markdown document that contains metadata (or data about your document) like the title, date, author, etc. This section uses a special simple syntax named YAML (or “YAML Ain’t Markup Language”) that follows this basic outline: setting: value for setting. Here’s an example YAML metadata section. Note that it must start and end with three dashes (---).

---
title: Title of your document
date: "January 13, 2020"
author: "Your name"
---

You can put the values inside quotes (like the date and name in the example above), or you can leave them outside of quotes (like the title in the example above). I typically use quotes just to be safe—if the value you’re using has a colon (:) in it, it’ll confuse Markdown since it’ll be something like title: My cool title: a subtitle, which has two colons. It’s better to do this:

---
title: "My cool title: a subtitle"
---

If you want to use quotes inside one of the values (e.g. your document is An evaluation of "scare quotes"), you can use single quotes instead:

---
title: 'An evaluation of "scare quotes"'
---

Other references

These websites have additional details and examples and practice tools:

CommonMark’s Markdown tutorial: A quick interactive Markdown tutorial.
Markdown tutorial: Another interactive tutorial to practice using Markdown.
Markdown cheatsheet: Useful one-page reminder of Markdown syntax.
The Plain Person’s Guide to Plain Text Social Science: A comprehensive explanation and tutorial about why you should write data-based reports in Markdown.

Part 2: Loading Data, data.frames, and ggplot2

Thu, 16 Apr 2020 00:00:00 +0000

Post-Class Survey

Please fill out the following survey and we will discuss the results during the next class. All responses will be anonymous. Remember, filling this out counts as attendance.

Pace
Clearest Point: What was the most clear part of the lecture?
Muddiest Point: What was the most unclear part of the lecture to you?
Anything Else: Is there something you’d like me to know?

https://ohsu.ca1.qualtrics.com/jfe/form/SV_6nc1ZLFMmRoE7nn

Muddiest Points

I’m not a programmer person so the concept of a programming environment is pretty mystifying. That said, it doesn’t stop me from applying what you’re teaching us, so no real worries there.

Stick with it!

The challenges since I lost where we were and had to back track all of the time to see where the instructor was.

Ok, I will do my best to announce which section we are on in the notebook.

na.omit

We will get to that today.

folder organization; projects vs notebooks?

We will talk about that today.

Part 1: Introduction to R/RStudio

Thu, 09 Apr 2020 00:00:00 +0000

Slides

Slides

Class Introduction

Open the class introduction slides in a separate window: https://ready4r.netlify.com/slides/01-introduction_slides#1

Post-Class Survey

Please fill out the following survey and we will discuss the results during the next class. All responses will be anonymous. Remember, filling this out counts as attendance.

Pace
Clearest Point: What was the most clear part of the lecture?
Muddiest Point: What was the most unclear part of the lecture to you?
Anything Else: Is there something you’d like me to know?

https://ohsu.ca1.qualtrics.com/jfe/form/SV_6nc1ZLFMmRoE7nn

Muddiest Points

I’m not a programmer person so the concept of a programming environment is pretty mystifying. That said, it doesn’t stop me from applying what you’re teaching us, so no real worries there.

Stick with it!

The challenges since I lost where we were and had to back track all of the time to see where the instructor was.

Ok, I will do my best to announce which section we are on in the notebook.

na.omit

We will get to that today.

folder organization; projects vs notebooks?

We will talk about that today.

Final Group Project: AirBnB analytics

Sun, 14 Nov 2021 00:00:00 +0000

In your final group assignment, you have to analyse data about Airbnb listings and fit a model to predict the total cost for two people staying 4 nights in an AirBnB in a city. You can download AirBnB data from insideairbnb.com; it was originally scraped from airbnb.com.

All of the listings are a GZ file, namely they are archive files compressed by the standard GNU zip (gzip) compression algorithm. You can download, save and extract the file if you wanted, but vroom::vroom() or readr::read_csv() can immediately read and extract this kind of a file. You should prefer vroom() as it is faster, but if vroom() is limited by a firewall, please use read_csv() instead.

listings <- vroom("http://data.insideairbnb.com/germany/bv/munich/2020-06-20/data/listings.csv.gz")

vroom will download the *.gz zipped file, unzip, and provide you with the dataframe.

Even though there are many variables in the dataframe, here is a quick description of some of the variables collected, with cost data typically expressed in US$

price = cost per night
cleaning_fee: cleaning fee
extra_people: charge for having more than 1 person
property_type: type of accommodation (House, Apartment, etc.)
room_type:
- Entire home/apt (guests have entire place to themselves)
- Private room (Guests have private room to sleep, all other rooms shared)
- Shared room (Guests sleep in room shared with others)
number_of_reviews: Total number of reviews for the listing
review_scores_rating: Average review score (0 - 100)
longitude , latitude: geographical coordinates to help us locate the listing
neighbourhood*: three variables on a few major neighbourhoods in each city

Exploratory Data Analysis (EDA)

In the R4DS Exploratory Data Analysis chapter, the authors state:

“Your goal during EDA is to develop an understanding of your data. The easiest way to do this is to use questions as tools to guide your investigation… EDA is fundamentally a creative process. And like most creative processes, the key to asking quality questions is to generate a large quantity of questions.”

Conduct a thorough EDA. Recall that an EDA involves three things:

Looking at the raw values.
- dplyr::glimpse()
Computing summary statistics of the variables of interest, or finding NAs
- mosaic::favstats()
- skimr::skim()
Creating informative visualizations.
- ggplot2::ggplot()
  - geom_histogram() or geom_density() for numeric continuous variables
  - geom_bar() or geom_col() for categorical variables
- GGally::ggpairs() for scaterrlot/correlation matrix
  - Note that you can add transparency to points/density plots in the aes call, for example: aes(colour = gender, alpha = 0.4)

You may wish to have a level 1 header (#) for your EDA, then use level 2 sub-headers (##) to make sure you cover all three EDA bases. At a minimum you should address these questions:

How many variables/columns? How many rows/observations?
Which variables are numbers?
Which are categorical or factor variables (numeric or character variables with variables that have a fixed and known set of possible values?
What are the correlations between variables? Does each scatterplot support a linear relationship between variables? Do any of the correlations appear to be conditional on the value of a categorical variable?

At this stage, you may also find you want to use filter, mutate, arrange, select, or count. Let your questions lead you!

In all cases, please think about the message your plot is conveying. Don’t just say “This is my X-axis, this is my Y-axis”, but rather what’s the so what of the plot. Tell some sort of story and speculate about the differences in the patterns in no more than a paragraph.

Data wrangling

Once you load the data, it’s always a good idea to use glimpse to see what kind of variables you have and what data type (chr, num, logical, date, etc) they are.

Notice that some of the price data (price, cleaning_fee, extra_people) is given as a character string, e.g., “$176.00”

Since price is a quantitative variable, we need to make sure it is stored as numeric data num in the dataframe. To do so, we will first use readr::parse_number() which drops any non-numeric characters before or after the frst number

listings <- listings %>% 
  mutate(price = parse_number(price))

Use typeof(listing$price) to confirm that price is now stored as a number.

Handling missing values (NAs)

Use skimr::skim() function to view a summary of the cleaning_fee data. This is also stored as a character, so you have to turn it into a number, as discussed earlier.

How many observations have missing values for cleaning_fee?
What do you think is the most likely reason for the missing observations of cleaning_fee? In other words, what does a missing value of cleaning_fee indicate?

cleaning_fee an example of data that is missing not at random, since there is a specific pattern/explanation to the missing data.

Fill in the code below to impute the missing values of cleaning_fee with an appropriate numeric value. Then use skimr::skim() function to confirm that there are no longer any missing values of cleaning_fee.

listings <- listings %>%
  mutate(cleaning_fee = case_when(
    is.na(cleaning_fee) ~ ______, 
    TRUE ~ cleaning_fee
  ))

Next, we look at the variable property_type. We can use the count function to determine how many categories there are their frequency. What are the top 4 most common property types? What proportion of the total listings do they make up?

Since the vast majority of the observations in the data are one of the top four or five property types, we would like to create a simplified version of property_type variable that has 5 categories: the top four categories and Other. Fill in the code below to create prop_type_simplified.

listings <- listings %>%
  mutate(prop_type_simplified = case_when(
    property_type %in% c("Apartment","______", "______","______") ~ property_type, 
    TRUE ~ "Other"
  ))

Use the code below to check that prop_type_simplified was correctly made.

listings %>%
  count(property_type, prop_type_simplified) %>%
  arrange(desc(n))

Airbnb is most commonly used for travel purposes, i.e., as an alternative to traditional hotels. We only want to include listings in our regression analysis that are intended for travel purposes:

What are the most common values for the variable minimum_nights?
Is ther any value among the common values that stands out?
What is the likely intended purpose for Airbnb listings with this seemingly unusual value for minimum_nights?

Filter the airbnb data so that it only includes observations with minimum_nights <= 4

Mapping

Visualisations of feature distributions and their relations are key to understanding a data set, and they can open up new lines of exploration. While we do not have time to go into all the wonderful geospatial visualisations one can do with R, you can use the following code to start with a map of your city, and overlay all AirBnB coordinates to get an overview of the spatial distribution of AirBnB rentals. For this visualisation we use the leaflet package, which includes a variety of tools for interactive maps, so you can easily zoom in-out, click on a point to get the actual AirBnB listing for that specific point, etc.

The following code, having created a dataframe listings with all AirbnB listings in Bordeaux, will plot on the map all AirBnBs where minimum_nights is less than equal to four (4). You could learn more about leaflet, by following the relevant Datacamp course on mapping with leaflet

leaflet(data = filter(listings, minimum_nights <= 4)) %>% 
  addProviderTiles("OpenStreetMap.Mapnik") %>% 
  addCircleMarkers(lng = ~longitude, 
                   lat = ~latitude, 
                   radius = 1, 
                   fillColor = "blue", 
                   fillOpacity = 0.4, 
                   popup = ~listing_url,
                   label = ~property_type)

Regression Analysis

For the target variable $Y$, we will use the cost for two people to stay at an Airbnb location for four (4) nights.

Create a new variable called price_4_nights that uses price, cleaning_fee, guests_included, and extra_people to calculate the total cost for two people to stay at the Airbnb property for 4 nights. This is the variable $Y$ we want to explain.

Use histograms or density plots to examine the distributions of price_4_nights and log(price_4_nights). Which variable should you use for the regression model? Why?

Fit a regression model called model1 with the following explanatory variables: prop_type_simplified, number_of_reviews, and review_scores_rating.

Interpret the coefficient review_scores_rating in terms of price_4_nights.
Interpret the coefficient of prop_type_simplified in terms of price_4_nights.

We want to determine if room_type is a significant predictor of the cost for 4 nights, given everything else in the model. Fit a regression model called model2 that includes all of the explanantory variables in model1 plus room_type.

Further variables/questions to explore on our own

Our dataset has many more variables, so here are some ideas on how you can extend your analysis

Are the number of bathrooms, bedrooms, beds, or size of the house (accomodates) significant predictors of price_4_nights?
Do superhosts (host_is_superhost) command a pricing premium, after controlling for other variables?
Most owners advertise the exact location of their listing (is_location_exact == TRUE), while a non-trivial proportion don’t. After controlling for other variables, is a listing’s exact location a significant predictor of price_4_nights?
For all cities, there are 3 variables that relate to neighbourhoods: neighbourhood, neighbourhood_cleansed, and neighbourhood_group_cleansed. There are typically more than 20 neighbourhoods in each city, and it wouldn’t make sense to include them all in your model. Use your city knowledge, or ask someone with city knowledge, and see whether you can group neighbourhoods together so the majority of listings falls in fewer (5-6 max) geographical areas. You would thus need to create a new categorical variabale neighbourhood_simplified and determine whether location is a predictor of price_4_nights
What is the effect of cancellation_policy on price_4_nights, after we control for other variables?

Diagnostics, collinearity, summary tables

As you keep building your models, it makes sense to:

Check the residuals, using autoplot(model_x)
As you start building models with more explanatory variables, make sure you use `car::vif(model_x)`` to calculate the Variance Inflation Factor (VIF) for your predictors and determine whether you have colinear variables. A general guideline is that a VIF larger than 5 or 10 is large, and your model may suffer from collinearity. Remove the variable in question and run your model again without it.
Create a summary table, using huxtable (https://bit-2021.netlify.app/example/modelling_side_by_side_tables/) that shows which models you worked on, which predictors are significant, the adjusted $R^2$, and the Residual Standard Error.
Finally, you must use the best model you came up with for prediction. Suppose you are planning to visit the city you have been assigned to over reading week, and you want to stay in an Airbnb. Find Airbnb’s that are apartment with a private room, have at least 10 reviews, and an average rating of at least 90. Use your best model to predict the total cost to stay at this Airbnb for 4 nights. Include the appropriate 95% interval with your prediction. Report the point prediction and interval in terms of price_4_nights.
If you used a log(price_4_nights) model, make sure you anti-log to convert the value in $. To interpret variables that are log-transformed, please have a look at FAQ HOW DO I INTERPRET A REGRESSION MODEL WHEN SOME VARIABLES ARE LOG TRANSFORMED?

Deliverables

By midnight on Monday 19 Oct 2020, you must upload on Canvas a short presentation (max 4-5 slides) with your findings, as some groups will be asked to present in class. You should present your Exploratory Data Analysis, as well as your best model. In addition, you must upload on Canvas your final report, written using R Markdown that incoprorates code and text to introduce, frame, and describe your story and findings.

Remember to follow R Markdown etiquette rules and style; don’t have the Rmd output extraneous messages or warnings, include summary tables in nice tables (use kableExtra), and remove any placeholder texts from past Rmd templates.

Rubric

We will use a basic checklist (adapted from the book Elements of Data Analytic Style) when reviewing data analyses. It can be used as a guide during the process of a data analysis, as a rubric for grading data analysis projects, or as a way to evaluate the quality of a reported data analysis. You don’t have to answer every one of these questions for every data analysis, but they are a useful set of ideas to keep in the back of your mind when reviewing a data analysis.

Answering the question

Did you specify the type of data analytic question (e.g. exploration, association causality) before touching the data?
Did you define the metric for success before beginning?
Did you understand the context for the question and the business application?
Did you consider whether the question could be answered with the available data?

Checking the data

Did you plot univariate and multivariate summaries of the data?
Did you check for outliers? How did you handle outliers?
Did you identify the missing data code?

Tidying the data

Is each variable one column?
Is each observation one row?
Did you record the steps for moving from raw to tidy data?
Did you record all parameters, units, and functions applied to the data?

Exploratory analysis

Did you identify missing values?
Did you make univariate plots (histograms, density plots, boxplots)?
Did you consider correlations between variables (scatterplots)?
Did you check the units of all data points to make sure they are in the right range?
Did you try to identify any errors or miscoding of variables?
Did you consider plotting on a log scale?
Would a scatterplot be more informative?

Inference

Did you identify what large population you are trying to describe?
Did you clearly identify the quantities of interest in your model?
Did you consider potential confounders?
Did you identify and model potential sources of correlation such as measurements over time or space?
Did you calculate a measure of uncertainty for each estimate?

Prediction

Did you identify in advance your error measure?
Did you chck for colinearity in your model(s)?

Written analyses

Did you describe the question of interest?
Did you describe the data set, experimental design, and question you are answering?
Did you specify the type of data analytic question you are answering?
Did you specify in clear notation the exact model you are fitting?
Did you explain on the scale of interest what each estimate and measure of uncertainty means?
Did you report a measure of uncertainty for each estimate?

Figures

Does each figure communicate an important piece of information or address a question of interest?
Do all your figures include plain language axis labels?
Is the font size large enough to read?
Does every figure have a detailed caption that explains all axes, legends, and trends in the figure?

Presentations

Did you lead with a brief, understandable to everyone statement of your problem?
Did you explain the data, measurement technology, and experimental design before you explained your model?
Did you explain the features you will use to model data before you explain the model?
Did you make sure all legends and axes were legible from the back of the room?

Reproducibility

Did you avoid doing calculations manually?
Did you create a script/Rmd that reproduces all your analyses?
Did you save the raw and processed versions of your data?
Did you record all versions of the software you used to process the data?
Did you try to have someone else run your analysis code to confirm they got the same answers?

Acknowledgements

The data from this lab is from insideairbnb.com
The rubric/data analysis guidelines is derived in part from Elements of Data Analytic Style

World Bank Data

Fri, 31 Jul 2020 00:00:00 +0000

The World Bank is one of the world’s largest producers of development data and research. It is a great source of global socio-economic data, spanning several decades and many topics. For example, you can read their 2018 Atlas of Sustainable Development Goals or a blog post on their all-new visual guide to data and development.

The wbstats package allows you to search for and download any open World Bank dataset. To identify the actual indicator you want, you have to find its code either in the World Bank datacatalog or, even better, through wbstats.

Population Growth 1970-2017

Suppose we wanted to get data on population growth. Manually, we would navigate to the World Bank datacatalog website, and search for population growth.

We get various results, but the more important ones are usually at the top with data on Population Growth (Annual %) with code SP.POP.GROW, on Rural Population Growth (Annual %) with code SP.RUR.TOTL.ZG, and on Urban Population Growth (Annual %) with code SP.URB.GROW.

Alternatively, we would load the wbstats package, and use pop_growth_codes <- wbsearch(pattern = "population growth") to get a dataframe with the codes that the search function returns.

library(wbstats)

pop_growth_codes <- wb_search(pattern = "population growth")
head(pop_growth_codes)

## # A tibble: 4 x 3
##   indicator_id    indicator            indicator_desc                           
##   <chr>           <chr>                <chr>                                    
## 1 IN.EC.POP.GRWT~ Decadal Growth of P~ Population growth rate over the 10 year ~
## 2 SP.POP.GROW     Population growth (~ Annual population growth rate for year t~
## 3 SP.RUR.TOTL.ZG  Rural population gr~ Rural population refers to people living~
## 4 SP.URB.GROW     Urban population gr~ Urban population refers to people living~

Either way, the indicator we are interested in is Population Growth Annual and its code = SP.POP.GROW. The next step is to download the data with the wbstats::wb_data() function.

The first argument the wb_data function takes is a list of countries; if left empty, is will download all data for individual countries and aggregate regions like Arab World, Euro area, etc. In our example, let us download data for individuals countries only starting at 1970 and ending in 2017.

# Download data for Population Growth Annual% SP.POP.GROW
pop_growth_data <- wb_data(country = "countries_only", 
                      indicator = "SP.POP.GROW", 
                      start_date = 1970, 
                      end_date = 2017,
                      return_wide=FALSE)

glimpse(pop_growth_data)

## Rows: 10,416
## Columns: 11
## $ indicator_id <chr> "SP.POP.GROW", "SP.POP.GROW", "SP.POP.GROW", "SP.POP.G...
## $ indicator    <chr> "Population growth (annual %)", "Population growth (an...
## $ iso2c        <chr> "AF", "AF", "AF", "AF", "AF", "AF", "AF", "AF", "AF", ...
## $ iso3c        <chr> "AFG", "AFG", "AFG", "AFG", "AFG", "AFG", "AFG", "AFG"...
## $ country      <chr> "Afghanistan", "Afghanistan", "Afghanistan", "Afghanis...
## $ date         <dbl> 2017, 2016, 2015, 2014, 2013, 2012, 2011, 2010, 2009, ...
## $ value        <dbl> 2.55, 2.78, 3.08, 3.36, 3.49, 3.41, 3.14, 2.75, 2.40, ...
## $ unit         <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA...
## $ obs_status   <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA...
## $ footnote     <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA...
## $ last_updated <date> 2020-08-18, 2020-08-18, 2020-08-18, 2020-08-18, 2020-...

The wb_cachelist is a cached version of useful information from the World Bank API and provides a snapshot of available countries, indicators, and other relevant information. The structure of wb_cachelist is as follows

glimpse(wb_cachelist, max.level = 1)

## List of 8
##  $ countries    : tibble [304 x 18] (S3: tbl_df/tbl/data.frame)
##  $ indicators   : tibble [16,607 x 8] (S3: tbl_df/tbl/data.frame)
##  $ sources      : tibble [61 x 9] (S3: tbl_df/tbl/data.frame)
##  $ topics       : tibble [21 x 3] (S3: tbl_df/tbl/data.frame)
##  $ regions      : tibble [48 x 4] (S3: tbl_df/tbl/data.frame)
##  $ income_levels: tibble [7 x 3] (S3: tbl_df/tbl/data.frame)
##  $ lending_types: tibble [4 x 3] (S3: tbl_df/tbl/data.frame)
##  $ languages    : tibble [23 x 3] (S3: tbl_df/tbl/data.frame)

and as we can see it contains data on countries and aggregate regions, well over 16,000 indicators, etc. If we wanted to see the data on countries, let us create a dataframecountries` and glimpse its contents.

countries <-  wb_cachelist$countries
glimpse(countries)

## Rows: 304
## Columns: 18
## $ iso3c              <chr> "ABW", "AFG", "AFR", "AGO", "ALB", "AND", "ANR",...
## $ iso2c              <chr> "AW", "AF", "A9", "AO", "AL", "AD", "L5", "1A", ...
## $ country            <chr> "Aruba", "Afghanistan", "Africa", "Angola", "Alb...
## $ capital_city       <chr> "Oranjestad", "Kabul", NA, "Luanda", "Tirane", "...
## $ longitude          <dbl> -70.02, 69.18, NA, 13.24, 19.82, 1.52, NA, NA, 5...
## $ latitude           <dbl> 12.52, 34.52, NA, -8.81, 41.33, 42.51, NA, NA, 2...
## $ region_iso3c       <chr> "LCN", "SAS", NA, "SSF", "ECS", "ECS", NA, NA, "...
## $ region_iso2c       <chr> "ZJ", "8S", NA, "ZG", "Z7", "Z7", NA, NA, "ZQ", ...
## $ region             <chr> "Latin America & Caribbean", "South Asia", "Aggr...
## $ admin_region_iso3c <chr> NA, "SAS", NA, "SSA", "ECA", NA, NA, NA, NA, "LA...
## $ admin_region_iso2c <chr> NA, "8S", NA, "ZF", "7E", NA, NA, NA, NA, "XJ", ...
## $ admin_region       <chr> NA, "South Asia", NA, "Sub-Saharan Africa (exclu...
## $ income_level_iso3c <chr> "HIC", "LIC", NA, "LMC", "UMC", "HIC", NA, NA, "...
## $ income_level_iso2c <chr> "XD", "XM", NA, "XN", "XT", "XD", NA, NA, "XD", ...
## $ income_level       <chr> "High income", "Low income", "Aggregates", "Lowe...
## $ lending_type_iso3c <chr> "LNX", "IDX", NA, "IBD", "IBD", "LNX", NA, NA, "...
## $ lending_type_iso2c <chr> "XX", "XI", NA, "XF", "XF", "XX", NA, NA, "XX", ...
## $ lending_type       <chr> "Not classified", "IDA", "Aggregates", "IBRD", "...

The dataframe contains the ISO country codes, the country name, its capital with its longitude and latitude, the region the country is in, the regions associated ISO code, as well as a classification on the income group, the country’s classification by income level, etc.

We can merge the dataframes pop_growth_data and countries with a left join, so we have a dataframe that contains data from both of them

countries <-  wb_cachelist$countries


# Merge with a left_join (a) country data with (b) population growth data
pop_growth <- 
  left_join(countries, pop_growth_data, by="iso3c") %>% 
              mutate(year = as.integer(date)) %>%  #make year an integer, rather than a character value
              select(iso3c, country.x, region, income_level, value, year=) %>% 
              na.omit()

Let us calculate and plot the average population growth for all countries between 1970 and 2017, faceted by region.

average_pop_growth <- pop_growth %>% 
              dplyr::group_by(region, income_level, country.x, iso3c) %>% 
              summarise(average_growth = mean(value)) %>% 
              arrange(average_growth) %>% 
              ungroup()

ggplot(data = average_pop_growth, 
       aes(x = reorder(country.x, average_growth), 
           y = average_growth, 
           fill = region))+
  geom_col()+
  coord_flip()+
  theme_minimal(7)+
  expand_limits(y=c(-1,8))+
  facet_wrap(~income_level, nrow=3, scales="free")+
  labs(title = 'Average annual population growth (%), 1970-2017',
       x = "",
       y = "Average Annual Population Growth (in %)",
       caption = 'Source: Worldbank') +
  # theme(legend.position="none")+
  NULL

World Happiness: how does it correlate with various indicators

Data from the UN’s World Happiness Report is available at Kaggle. We have downloaded the 2015 report in a CSV file, and have a quick glimpse at its structure.

world_happiness_2015 <- read_csv(here::here("data", "world_happiness_2015.csv"))
glimpse(world_happiness_2015)

As you notice, some of the variable names include a space, like Happiness Rank, all start with a capital letter, etc. We will use janitor::clean_names() to clean the variable names, so they are easier to deal with.

library(janitor)

world_happiness_2015 <- world_happiness_2015 %>%
  clean_names()

glimpse(world_happiness_2015)

## Rows: 158
## Columns: 12
## $ country                     <chr> "Switzerland", "Iceland", "Denmark", "N...
## $ region                      <chr> "Western Europe", "Western Europe", "We...
## $ happiness_rank              <dbl> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, ...
## $ happiness_score             <dbl> 7.59, 7.56, 7.53, 7.52, 7.43, 7.41, 7.3...
## $ standard_error              <dbl> 0.0341, 0.0488, 0.0333, 0.0388, 0.0355,...
## $ economy_gdp_per_capita      <dbl> 1.397, 1.302, 1.325, 1.459, 1.326, 1.29...
## $ family                      <dbl> 1.350, 1.402, 1.361, 1.331, 1.323, 1.31...
## $ health_life_expectancy      <dbl> 0.941, 0.948, 0.875, 0.885, 0.906, 0.88...
## $ freedom                     <dbl> 0.666, 0.629, 0.649, 0.670, 0.633, 0.64...
## $ trust_government_corruption <dbl> 0.4198, 0.1414, 0.4836, 0.3650, 0.3296,...
## $ generosity                  <dbl> 0.2968, 0.4363, 0.3414, 0.3470, 0.4581,...
## $ dystopia_residual           <dbl> 2.52, 2.70, 2.49, 2.47, 2.45, 2.62, 2.4...

First, we can look how happiness_score correlates with its of the variables the UN uses. We will use GGally:ggpairs() to get a correlation- scatterplot matrix. We do not want to include in our analyses the country name, its region, the happiness_rank and the standard error associated with the estimate of the happiness score.

world_happiness_2015 %>% 
  select(-country, -region, -happiness_rank, -standard_error) %>% 
  GGally::ggpairs()

We will now choose six (6) indicators form the World Bank data, downloads their values for 2015 and see how these correlate with the overall happiness score.

# Download data for the following indicators

indicators <- c("SE.PRM.NENR",     # School enrollment, primary (% net)
                "SP.DYN.LE00.IN",  # Life expectancy
                "SI.POV.DDAY",     # Extreme poverty (% earning less than $2/day)
                "EG.ELC.ACCS.ZS",  # Access to electricity
                "SI.POV.GINI",     # GINI Index
                "NY.GDP.PCAP.KD")  # GDP per capita


happiness_data_WB_long <- wb_data(country = "countries_only", 
                             indicator = indicators, 
                             start_date = 2015, 
                             end_date = 2015,
                             #since we have many indicators, we should get the data in long format
                             return_wide=FALSE) 

# look at the long dataframe
glimpse(happiness_data_WB_long)

## Rows: 1,302
## Columns: 11
## $ indicator_id <chr> "SE.PRM.NENR", "SE.PRM.NENR", "SE.PRM.NENR", "SE.PRM.N...
## $ indicator    <chr> "School enrollment, primary (% net)", "School enrollme...
## $ iso2c        <chr> "AF", "AL", "DZ", "AS", "AD", "AO", "AG", "AR", "AM", ...
## $ iso3c        <chr> "AFG", "ALB", "DZA", "ASM", "AND", "AGO", "ATG", "ARG"...
## $ country      <chr> "Afghanistan", "Albania", "Algeria", "American Samoa",...
## $ date         <dbl> 2015, 2015, 2015, 2015, 2015, 2015, 2015, 2015, 2015, ...
## $ value        <dbl> NA, 94.2, 97.5, NA, NA, NA, 94.2, 99.5, 92.7, NA, 97.0...
## $ unit         <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA...
## $ obs_status   <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA...
## $ footnote     <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, "Natio...
## $ last_updated <date> 2020-08-18, 2020-08-18, 2020-08-18, 2020-08-18, 2020-...

In order to get the two dataframes to combine into one, they have to have a shared column/ variable. We will merge the two datasets with a left_join() by “country”, and glimpse the structure of the resulting dataframe.

# Merge with a left_join (a) happiness data with all indicators and (b) the 2015 World Happiness index 

happiness <- 
  left_join(happiness_data_WB_long, world_happiness_2015, by="country") 

glimpse(happiness)

## Rows: 1,302
## Columns: 22
## $ indicator_id                <chr> "SE.PRM.NENR", "SE.PRM.NENR", "SE.PRM.N...
## $ indicator                   <chr> "School enrollment, primary (% net)", "...
## $ iso2c                       <chr> "AF", "AL", "DZ", "AS", "AD", "AO", "AG...
## $ iso3c                       <chr> "AFG", "ALB", "DZA", "ASM", "AND", "AGO...
## $ country                     <chr> "Afghanistan", "Albania", "Algeria", "A...
## $ date                        <dbl> 2015, 2015, 2015, 2015, 2015, 2015, 201...
## $ value                       <dbl> NA, 94.2, 97.5, NA, NA, NA, 94.2, 99.5,...
## $ unit                        <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,...
## $ obs_status                  <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,...
## $ footnote                    <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,...
## $ last_updated                <date> 2020-08-18, 2020-08-18, 2020-08-18, 20...
## $ region                      <chr> "Southern Asia", "Central and Eastern E...
## $ happiness_rank              <dbl> 153, 95, 68, NA, NA, 137, NA, 30, 127, ...
## $ happiness_score             <dbl> 3.58, 4.96, 5.61, NA, NA, 4.03, NA, 6.5...
## $ standard_error              <dbl> 0.0308, 0.0501, 0.0510, NA, NA, 0.0476,...
## $ economy_gdp_per_capita      <dbl> 0.320, 0.879, 0.939, NA, NA, 0.758, NA,...
## $ family                      <dbl> 0.303, 0.804, 1.078, NA, NA, 0.860, NA,...
## $ health_life_expectancy      <dbl> 0.3034, 0.8133, 0.6177, NA, NA, 0.1668,...
## $ freedom                     <dbl> 0.2341, 0.3573, 0.2858, NA, NA, 0.1038,...
## $ trust_government_corruption <dbl> 0.09719, 0.06413, 0.17383, NA, NA, 0.07...
## $ generosity                  <dbl> 0.3651, 0.1427, 0.0782, NA, NA, 0.1234,...
## $ dystopia_residual           <dbl> 1.95, 1.90, 2.43, NA, NA, 1.95, NA, 2.8...

We can create a histogram of happiness_score by region

ggplot(data = happiness, aes(x = happiness_score , fill=region))+
  geom_histogram()+
  theme_minimal()+
  facet_wrap(~region,nrow=5) +
  labs(title = '2015 World Happiness',
       x = "",
       y = "Total Happiness Score",
       caption = 'Source: Worldbank') +
  theme(legend.position="none")

We can also create a scatterplot of happiness_score against all the indicators we have downloaded.

ggplot(data = happiness, aes(x = value, y = happiness_score , colour=indicator))+
  geom_point()+
  geom_smooth(se=FALSE)+
  theme_minimal()+
  facet_wrap(~indicator,scales="free") +
  labs(title = '2015 World Happiness',
       x = "",
       y = "Total Happiness Score",
       caption = 'Source: Worldbank') +
  theme(legend.position="none")

Acknowledgments

This page is derived in part from Introduction to the wbstats R-package by Jesse Piburn.

Hypothesis Testing, A/B testing

Sat, 25 Jul 2020 00:00:00 +0000

Import, inspect, and clean data

Sat, 25 Jul 2020 00:00:00 +0000

Inspecting and Cleaning Data

Inspecting and Cleaning Data

Inspecting data using skimr::skim()

Import, inspect, and clean data

Sat, 25 Jul 2020 00:00:00 +0000

Inspecting and Cleaning Data

Inspecting and Cleaning Data

Inspecting data using skimr::skim()

Manipulate data

Sat, 25 Jul 2020 00:00:00 +0000

Select variables in a dataset using `select()` and sort using `arrange()`

The dataframe movies has been loaded into memory. It contains a sample of movies from IMDB, and its contents are shown below:

Filter rows in a dataset using `filter()`

Remember that select() allows us to choose columns, or variables, whereas filter() chooses rows, or cases, that conform to certain criteria

`mutate()` to change the data type of a variable and create new variables

`group_by()/summarise()` to get summary statistics, including counts, means, etc., within categories.

Putting it all together

You can put together all of your dplyr knowledge to work four genres of movies, namely action, adventure, comedy and drama and create the following plot.

For these genres, you have to

Make sure you do not have multiple entries of the same movie; use distinct(movie, _keep.all=TRUE)
Calculate a normalised metric for rating, where you adjust the movie’s rating by the number of votes it received out of the total votes in its genre, normalised_rating = rating * (votes / total votes in genre)
Arrange movies, so higher normalised_rating appears first.
Categorise and colour movies according to their budget cost
- cheap (<20m, or <20e6 as e6 is R shorthand for 1 million, or $10^6$,
- moderate (20-120m), and
- expensive (>120m)
Convert cost column to a factor variable and re-level in the correct order (cheap, moderate, expensive)
Change the labels in the x- and y-axis, and give appropriate titles, subtitles, etc
use theme minimal

Some tips:

for sorting columns within a a ggplot, check out the reorder() in x argument
if you perform dplyr on original dataframe make sure to overwrite dataframe otherwise all changes are done on the fly and are not saved
consider freeing the scales of the facet wrap

Manipulate data

Sat, 25 Jul 2020 00:00:00 +0000

Select variables in a dataset using `select()` and sort using `arrange()`

The dataframe movies has been loaded into memory. It contains a sample of movies from IMDB, and its contents are shown below:

Filter rows in a dataset using `filter()`

Remember that select() allows us to choose columns, or variables, whereas filter() chooses rows, or cases, that conform to certain criteria

`mutate()` to change the data type of a variable and create new variables

`group_by()/summarise()` to get summary statistics, including counts, means, etc., within categories.

Putting it all together

You can put together all of your dplyr knowledge to work four genres of movies, namely action, adventure, comedy and drama and create the following plot.

For these genres, you have to

Make sure you do not have multiple entries of the same movie; use distinct(movie, _keep.all=TRUE)
Calculate a normalised metric for rating, where you adjust the movie’s rating by the number of votes it received out of the total votes in its genre, normalised_rating = rating * (votes / total votes in genre)
Arrange movies, so higher normalised_rating appears first.
Categorise and colour movies according to their budget cost
- cheap (<20m, or <20e6 as e6 is R shorthand for 1 million, or $10^6$,
- moderate (20-120m), and
- expensive (>120m)
Convert cost column to a factor variable and re-level in the correct order (cheap, moderate, expensive)
Change the labels in the x- and y-axis, and give appropriate titles, subtitles, etc
use theme minimal

Some tips:

for sorting columns within a a ggplot, check out the reorder() in x argument
if you perform dplyr on original dataframe make sure to overwrite dataframe otherwise all changes are done on the fly and are not saved
consider freeing the scales of the facet wrap

Model Fitting

Sat, 25 Jul 2020 00:00:00 +0000

Model Fitting

Sat, 25 Jul 2020 00:00:00 +0000

Installing the tidyverse

Sat, 25 Jul 2020 00:00:00 +0000

Installing the `tidyverse`

R packages are easy to install with RStudio. Select the packages panel, click on “Install,” type the name of the package you want to install, and press enter.

This can sometimes be tedious when you’re installing lots of packages, though. The tidyverse ¹ for instance, consists of dozens of packages that all work together. Rather than install each package individually, you can install tidyverse, a meta-package if you wish, and get them all at the same time.

Go to the packages panel in RStudio, click on “Install,” type “tidyverse”, and press enter. You’ll see a bunch of output in the RStudio console as all the tidyverse packages are installed.

RStudio generates a line of code for you and run it: install.packages("tidyverse"). You can also just paste and run this instead of using the packages panel.

# install the major packages from the tidyverse
install.packages("tidyverse")

This will take a while as tidyverse is a collection of packages and R will have to install all dependencies.

Installing the `tidyverse` if you have a Mac

Unfortunately, installing the tidyverse isn’t quite always a straight-forward task with the current version of macOS 10.14, Mojave which was released on September 24, 2018.

To solve issues that may arise with missing xml2 library, please do the following:

Open Terminal (the tab right next to Console)
Type

xcode-select --install

Be careful as you do need two (2) dashes before the install. A software update popup window should appear that will ask if you want to install command line developer tools. Click on “Install” (you don’t need to click on “Get Xcode”)

Go to https://brew.sh and copy the long command under “Install Homebrew” (starts with /usr/bin/ruby -e "$(curl -fsSL.), paste it into Terminal, and press enter.

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

This installs Homebrew, which is special software that lets you install Unix-y programs from the terminal.

Type the following command line in Terminal to install libxml2

brew install libxml2

Then, within RStudio, type

install.packages("xml2")

Finally, you can now proceed with the installation of the tidyverse

install.packages("tidyverse")

Installing further packages

Once the tidyverse collection of packages installs and you get back to the R prompt >, you can install a series of packages that will be useful later in the course. You can copy/paste the code below; please note that this will take quite a while, so grab a coffee.

# install these packages as well
list_of_packages <- c(
  "moderndive",   # https://www.moderndive.com/
  "DT",           # Allows us to handle Data Tables and manipulate data faster 
  "unvotes",      # How countries have voted in UN resolutions
  "gridExtra",    # Miscellaneous Functions for "Grid" Graphics
  "GGally",       # Allows us to create a correlations/scatterplots matrix 
  "tidyquant",    # Download and manipulate financial data
  "wbstats",      # Download World Bank Data
  "eurostat",     # Download data from Eurostat
  "fpp2",         # Time Series and Forecasting fucntions, with data too 
  "car",          # Applied Regression- allows to calculate VIF, Variance Inflation Factor
  "gapminder",    # Data on life expectancy, GDP/capita, and population by country and year
  "nycflights13", # Data on all domestic flights through NYCs 3 airports (JFK, EWR, LGA) in 2013
  "fivethirtyeight", #Data used in articles that appeared in the fivethirtyeight.com website
  "corrr",        # correlation in R
  "plotly",       # interactive visualizations
  "sf",           # tidy geo-computing
  "cowplot",      # ggplot multiple figures addon
  "coefplot",     # plot coefficients from fitted models
  "interplot",    # plot effects of variables in interaction terms
  "scales",       # scale functions for visualisations 
  "ggridges",     # ridgeline plots in ggplot2
  "skimr",        # nice dataframe summaries
  "leaflet",      # interactive maps
  "ggrepel",      # geoms for ggplot2 to repel overlapping text labels
  "viridis",      # Colour Maps
  "rvest",        # scrape webpages
  "usethis",      # automation of package and project setup
  "remotes",      # installing packages from Github
  "tidytext",     # text mining
  "here",         # finding your files 
  "mosaic"        # summary stats, using mosaic::favstats()
)

install.packages(list_of_packages, dependencies=TRUE, repos = "https://cran.rstudio.com/")

Install from Github

Most of the time the packages that you’ll want to install have been made available on CRAN, the Comprehensive R Archive Network, so you use the install.packages("package_name") function. Sometimes people write packages that are not submitted to CRAN, and sometimes you might want to try out a package that is currently under development. In these situations, people who write packages will often make them available on GitHub. We can install packages directly from Github, using the remotes package.

The first thing you need to do is install remotes, which is easy because that package is available on CRAN and hopefully you installed it with all packages listed earlier. If not,

install.packages("remotes")

Once you install remotes, you must explicitly say to R you will be using it by typing library(devtools). Then, you can use the install_github command to install a package directly from a GitHub repository. For example, there’s an R data package featuring every Lego set from 1970 to 2015 put together by Sean Kross.

remotes::install_github("seankross/lego") #install the lego package directly from Github

R fetches and installs the package from Github, and we now have the new lego package to play with. To verify that everything worked properly, let’s load the lego package and look at its legosets dataframe:

library(lego)     #load the lego package into the computer's memory

legosets          #view the legosets dataframe

## # A tibble: 6,172 x 14
##    Item_Number Name    Year Theme Subtheme Pieces Minifigures Image_URL GBP_MSRP
##    <chr>       <chr>  <int> <chr> <chr>     <int>       <int> <chr>        <dbl>
##  1 10246       Detec~  2015 Adva~ "Modula~   2262           6 http://i~   133.  
##  2 10247       Ferri~  2015 Adva~ "Fairgr~   2464          10 http://i~   150.  
##  3 10248       Ferra~  2015 Adva~ "Vehicl~   1158          NA http://i~    70.0 
##  4 10249       Toy S~  2015 Adva~ "Winter~    898          NA http://i~    60.0 
##  5 10581       Ducks   2015 Duplo "Forest~     13           1 http://i~     9.99
##  6 10582       Anima~  2015 Duplo "Forest~     39           2 http://i~    17.0 
##  7 10583       Fishi~  2015 Duplo "Forest~     32           2 http://i~    20.0 
##  8 10584       Forest  2015 Duplo "Forest~    105           3 http://i~    50.0 
##  9 10585       Mom a~  2015 Duplo ""           13           2 http://i~     8.99
## 10 10586       Ice C~  2015 Duplo ""           11           2 http://i~    13.0 
## # ... with 6,162 more rows, and 5 more variables: USD_MSRP <dbl>,
## #   CAD_MSRP <dbl>, EUR_MSRP <dbl>, Packaging <chr>, Availability <chr>

glimpse(legosets) #examine the structure of the dataframe- variables, observations, type of variables, etc.

## Rows: 6,172
## Columns: 14
## $ Item_Number  <chr> "10246", "10247", "10248", "10249", "10581", "10582", "10~
## $ Name         <chr> "Detective's Office", "Ferris Wheel", "Ferrari F40", "Toy~
## $ Year         <int> 2015, 2015, 2015, 2015, 2015, 2015, 2015, 2015, 2015, 201~
## $ Theme        <chr> "Advanced Models", "Advanced Models", "Advanced Models", ~
## $ Subtheme     <chr> "Modular Buildings", "Fairground", "Vehicles", "Winter Vi~
## $ Pieces       <int> 2262, 2464, 1158, 898, 13, 39, 32, 105, 13, 11, 52, 13, 2~
## $ Minifigures  <int> 6, 10, NA, NA, 1, 2, 2, 3, 2, 2, 3, 1, NA, NA, NA, NA, 1,~
## $ Image_URL    <chr> "http://images.brickset.com/sets/images/10246-1.jpg", "ht~
## $ GBP_MSRP     <dbl> 132.99, 149.99, 69.99, 59.99, 9.99, 16.99, 19.99, 49.99, ~
## $ USD_MSRP     <dbl> 159.99, 199.99, 99.99, 79.99, 9.99, 19.99, 24.99, 59.99, ~
## $ CAD_MSRP     <dbl> 199.99, 229.99, 119.99, NA, 12.99, 24.99, 29.99, 69.99, 1~
## $ EUR_MSRP     <dbl> 149.99, 179.99, 89.99, 69.99, 9.99, 19.99, 24.99, 59.99, ~
## $ Packaging    <chr> "Box", "Box", "Box", "Box", "Box", "Box", "Box", "Box", "~
## $ Availability <chr> "Retail - limited", "Retail - limited", "LEGO exclusive",~

The dataframe has 14 variables (or columns) and 6,172 observations (rows). Besides the item number, year, theme/subtheme and the number of pieces and minifigures contained in each Lego box, we also have the recommeneded retail prices in GBP, USD, CAD, and EUR. While we are at it, let us have a quick look at how Lego prices (in GBP) have evolved over the years.

avg_price_per_year <- legosets %>% # create avg_price_year" by taking legosets, and then
  filter(!is.na(GBP_MSRP)) %>%    # filter out entries with no GBP prices, GBP_MSRP, and then
  group_by(Year) %>%              # group prices by year
  summarise(Price = mean(GBP_MSRP)) # create variable "Price" = yearly average of GBP_MSRP

ggplot(avg_price_per_year, 
       mapping = aes(x = Year, y = Price)) +  # time series plot: x=Year, y=Price
  geom_point(size = 0.5) +                    # simple scatterplot Y vs. X
  geom_line(size = 0.5) +                     # add the black line between points
  geom_smooth(se = FALSE) +                   # fit trend line,no error band around it "se = FALSE" 
  labs(x = "Year",   
       y = "Price (GBP)", 
       title = "Average price of LEGO sets",
       subtitle = "Amounts are reported in current GBP",
       caption = "Source: LEGO") +
  theme_bw()

There is a clear upward trend in average GBP prices.

And since we are talking about LEGOs, here is a fun application of creating LEGO mosaics from photos using R & the tidyverse

Updating packages

Every now and then the authors of packages release updated versions. The updated versions often add new functionality, fix bugs, and so on. It’s a good idea to update your packages periodically.

There’s an update.packages function, but it’s probably easier to stick with the RStudio tool. In the packages tab, click on the Update Packages button. This will bring up a window that looks like the one shown below:

In this window, each row refers to a package that needs to be updated. You can select which updates to install by checking the boxes on the left. If you feel lazy, click the Select All button, and then Install Updates. This might take a while to complete depending on how fast your internet connection is.

Updating R

About twice a year, a new version of R is released, and the features of all packages get changed to be compatible with the new version of R. The side effect of packages being compatible with the newest R version is that then you update to the newest version of R, you lose all the packages that you have downloaded and installed. Unfortuantely, you need to install the new versions of packages, even though they will typically behave just like the old ones.

Install `tinytex`

When you knit to PDF, R uses a special scientific typesetting program named LaTeX (pronounced “lay-tek” or “lah-tex”; for goofy nerdy reasons, the x is technically the “ch” sound in “Bach”, but most people just say it as “k”—saying “layteks” is frowned on for whatever reason).

LaTeX makes pretty documents, but it’s a huge program—the macOS version, for instance, is nearly 4 GB! To make life easier, there’s an R package named tinytex that installs a minimal LaTeX program and that automatically deals with differences between macOS and Windows.

Here’s how to install tinytex so you can knit to pretty PDFs:

Use the Packages in panel in RStudio to install tinytex like you did above with tidyverse. Alternatively, run install.packages("tinytex") in the console.
Run tinytex::install_tinytex() in the console.
Wait for a bit while R downloads and installs everything you need.
The end! You should now be able to knit to PDF.

A universe of packages centered around tidy data, including ggplot2↩︎

Using R Markdown

Sat, 25 Jul 2020 00:00:00 +0000

Reproducibility in scientific research

Reproducibility is the idea that data analyses, and more generally scientific claims, are published with their data and software code so that others may try to replicate the same work, get similar results, and build upon the works of others.

While this sounds obvious, it actually happens far less frequently than what it should.

For instance, scientists at the biotechnology company Amgen were unable to replicate the majority of published pre-clinical cancer research studies; as a matter of fact, only 6 out of 53 landmark results could be reproduced. Similarly, it has been argued that the great majority of preclinical results cannot be reproduced, leading to an annual estimate of the cost of irreproducibility on preclinical research industry to be equal to 28 Billion USD.

“You are always working with at least one collaborator: Future you.”
– Hadley Wickham

Suppose that your colleague sends you an Excel file with an analysis she has undertaken. The Excel file is likely to contain the raw data, but also graphs, results, etc. that were generated from the data. If you have ever received such an Excel analysis file, it takes a long time to navigate around it and try to understand the logic used to arrive at the results.

Data analysts who implement reproducibility in their projects can quickly and easily reproduce the original results and trace back to determine how they were derived. Literate programming, an idea from Donald Knuth, is a technique for mixing written text, where you write notes explaining what you did and why, and chunks of code that produce your graphs, analyses, etc.

This makes documentation of code easier, enables verification and replication, and allows the analyst to precisely replicate her analysis. This is extremely important when revisiting work done months later, because it’s highly likely you won’t remember how all the code/analysis works together when completing your work.

Let us change our traditional attitude to the construction of programs: Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to humans what we want the computer to do.
– Donald E. Knuth (1984), Literate Programming

Reproducibility is also key for communicating findings with other and decision makers; it allows them to follow your logic and verify your results, assess your assumptions, and understand how your answers were formed rather than solely relying on your claimed results. In the data science framework employed in R for Data Science, reproducibility is infused throughout the entire workflow.

Your reproducibility goals should be:

Are the results (tables and figures) reproducible from the code and data?
Does the code actually do what you think it does?
Is the code well documented so someone else can foolow your work?
In addition to what was done, is it clear why it was done? (e.g., how were parameter settings chosen?)
Can the code be used for other, or newer, data?
Can you generalise the code to do other things?

R Markdown = Markdown + R Code

R Markdown is regular Markdown with R code and output sprinkled in. You can do everything you can with regular Markdown, but you can incorporate graphs, tables, and other R output directly in your document. You can create HTML, PDF, and Word documents, PowerPoint and HTML presentations, websites, books, and even interactive dashboards with R Markdown. This whole course website is created with R Markdown (and a package named blogdown).

rmarkdown and knitr is a powerful combination of packages for literate programming, reproducible analysis, and document generation, which can:

Combine R code and Markdown syntax
Produce documents in PDF , Microsoft Word and various types of HTML documents
In HTML format, it can incorporate “extras” like interactive graphics

An R Markdown file is a plain text file that uses the extension .Rmd and contains three (3) major components:

A YAML header surrounded by ---s. This is the metadata of the document and it tells you how it is formed - what the title is, the author, date, output, and other control information.
Chunks of R code surounded by ```
Text mixed with simple text formatting using the Markdown syntax

Code chunks are interspersed with text throughout the document. To complete the document, you “Knit” or “render” the document. Most of you probably knit the document by clicking the “Knit” button in the script editor panel. You can also do this programmatically from the console by running the command rmarkdown::render("example.Rmd").

When you knit the document you send your .Rmd file to knitr, a package for R that executes all the code chunks and creates a second markdown document (.md). That markdown document is then passed onto pandoc, a document rendering software program independent from R. Pandoc allows users to convert back and forth between many different document formats such as HTML, $\LaTeX$, Microsoft Word, etc. By splitting the workflow up, you can convert your R Markdown document into a wide range of output formats.

The documentation for R Markdown is extremely comprehensive, and their tutorials and cheatsheets are excellent—rely on those.

Here are the most important things you’ll need to know about R Markdown in this class:

Key terms

Document: A Markdown file where you type stuff
Chunk: A piece of R code that is included in your document. It looks like this:
```
```{r chunk_name}
# Code goes here
```
```
There must be an empty line before and after the chunk. The final three backticks must be the only thing on the line—if you add more text, or if you forget to add the backticks, or accidentally delete the backticks, your document will not knit correctly.
Knit: When you “knit” a document, R runs each of the chunks sequentially and converts the output of each chunk into Markdown. R then runs the knitted document through pandoc to convert it to HTML or PDF or Word (or whatever output you’ve selected).

You can knit by clicking on the “Knit” button at the top of the editor window, or by pressing ⌘⇧K on macOS or control + shift + K on Windows.

Add chunks

There are three ways to insert chunks:

Press ⌘⌥I on macOS or control + alt + I on Windows
Click on the “Insert” button at the top of the editor window
Manually type all the backticks and curly braces (don’t do this)

Chunk names

You can add names to chunks to make it easier to navigate your document. If you click on the little dropdown menu at the bottom of your editor in RStudio, you can see a table of contents that shows all the headings and chunks. If you name chunks, they’ll appear in the list. If you don’t include a name, the chunk will still show up, but you won’t know what it does.

To add a name, include it immediately after the {r in the first line of the chunk. Names cannot contain spaces, but they can contain underscores and dashes. All chunk names in your document must be unique.

A word of caution: If you use the same chunk name more than once, knitr will give you an error message and refuse to knit your Rmd document. So ifyou copy/paste a named chunk, make sure you give them unique names.

```{r name-of-this-chunk}
# Code goes here
```

Chunk options

There are a bunch of different options you can set for each chunk. You can see a complete list in the RMarkdown Reference Guide or at knitr’s website.

Options go inside the {r} section of the chunk:

```{r name-of-this-chunk, warning=FALSE, message=FALSE}
# Code goes here
```

The most common chunk options are these:

fig.width=5 and fig.height=3 (or whatever number you want): Set the dimensions for figures
echo=FALSE: The code is not shown in the final document, but the results are.
include=FALSE: The chunk still runs, but the code and results are not included in the final document
message=FALSE: Any messages that R generates (like all the notes that appear after you load a package) are omitted
warning=FALSE: Any warnings that R generates are omitted
eval = FALSE - prevents code from being evaluated. I use this in my notes for class when I want to show how to write a specific function but don’t need to actually use it.
error = TRUE - causes the document to continue knitting and rendering even if the code generates a fatal error. If you’re debugging your code, you might want to use this option. However, for the final version of your work, you do not want to allow errors to pass through unnoticed.

You can also set chunk options by clicking on the little gear icon in the top right corner of any chunk:

Inline chunks

You can also include R output directly in your text, which is really helpful if you want to report numbers from your analysis. To do this, use `r r_code_here`.

It’s generally easiest to calculate numbers in a regular chunk beforehand and then use an inline chunk to display the value in your text. For instance, this document…

```{r find-avg-mpg, echo=FALSE}
avg_mpg <- mean(mtcars$mpg)
```

The average fuel efficiency for cars from 1974 was `r round(avg_mpg, 1)` miles per gallon.

… would knit into this:

The average fuel efficiency for cars from 1974 was 20.1 miles per gallon.

Caching

By default, every time you knit a document R starts anew and no previous results are saved.

If you have code chunks that run computationally intensive tasks, like running a ggpairs() correlation/scatterplot matrix in a large dataset, you might want to store these results to be more efficient and save time. If you use cache = TRUE, R will do exactly this. The output of the chunk will be saved to a specially named file on disk. Now, every time you knit the document the cached results will be used instead of running the code fresh.

Output formats

You can specify what kind of document you create when you knit in the YAML front matter.

title: "My document"
output:
  html_document: default
  pdf_document: default
  word_document: default

You can also click on the down arrow on the “Knit” button to choose the output and generate the appropriate YAML. If you click on the gear icon next to the “Knit” button and choose “Output options”, you change settings for each specific output type, like default figure dimensions or whether or not a table of contents is included.

The first output type listed under output: will be what is generated when you click on the “Knit” button or press the keyboard shortcut (⌘⇧K on macOS; control + shift + K on Windows). If you choose a different output with the “Knit” button menu, that output will be moved to the top of the output section.

The indentation of the YAML section matters, especially when you have settings nested under each output type. Here’s what a typical output section might look like:

---
title: "My document"
author: "My name"
date: "January 13, 2020"
output: 
  html_document: 
    toc: yes
    fig_caption: yes
    fig_height: 8
    fig_width: 10
  pdf_document: 
    latex_engine: xelatex  # More modern PDF typesetting engine
    toc: yes
  word_document: 
    toc: yes
    fig_caption: yes
    fig_height: 4
    fig_width: 5
---

Each output format has various options to customize the appearance of the final document. One option for HTML documents is to add a table of contents through the toc option. To add any option for an output format, just add it in a hierarchical format like this:

---
title: "My report"
author: "My Name"
date: 2020-07-26
output:  
  html_document:
    toc: true
    toc_depth: 2

You can explicitly set the number of levels included in the table of contents with toc_depth (the default is 3).

Appearance and style

There are several options that control the visual appearance of HTML documents.

theme specifies the Bootstrap theme to use for the page (themes are drawn from the Bootswatch theme library). Valid themes include default, cerulean, journal, flatly, readable, spacelab, united, cosmo, lumen, paper, sandstone, simplex, and yeti.
highlight specifies the syntax highlighting style for code chunks. Supported styles include default, tango, pygments, kate, monochrome, espresso, zenburn, haddock, and textmate.

Other references

Final Exam

Sun, 14 Nov 2021 00:00:00 +0000

The final exam will cover:

Programming in R and the tidyverse (about a third of total marks)
the core statistical tools of inferential statistics and linear models.

You final exam will be released at 2021-11-14 09:00 UTC and even though you can take it anytime before midnight, this will be a timed, 3-hour exam. You can use any notes and readings, but you must take the exam on your own and not talk to anyone about it.

Eurostat Data

Fri, 02 Oct 2020 00:00:00 +0000

Eurostat Data with the `eurostat` package

The eurostat package provides access to well over 9000 datasets from the Eurostat. It may seem a challenging task to find the correct dataset, but you are essentially looking for the code that describes the dataset. We an get a table of contents, namely all of th ecodes contained in the eurostat database.

library(eurostat)
library(fpp2) # for time series decomposition
library(seasonal)
library(tmap) #mapping eurostat data

# Get Eurostat data listing
# Function get_eurostat_toc() downloads a table of contents of eurostat datasets. 
# The values in column ‘code’ should be used to download a selected dataset.
toc <- get_eurostat_toc()

# Check the first 20 rows 
head(toc, 20) %>% 
  kable()

title	code	type	last update of data	last table structure change	data start	data end	values
Database by themes	data	folder	NA	NA	NA	NA	NA
General and regional statistics	general	folder	NA	NA	NA	NA	NA
European and national indicators for short-term analysis	euroind	folder	NA	NA	NA	NA	NA
Business and consumer surveys (source: DG ECFIN)	ei_bcs	folder	NA	NA	NA	NA	NA
Consumer surveys (source: DG ECFIN)	ei_bcs_cs	folder	NA	NA	NA	NA	NA
Consumers - monthly data	ei_bsco_m	dataset	29.09.2020	29.09.2020	1980M01	2020M09	NA
Consumers - quarterly data	ei_bsco_q	dataset	29.09.2020	30.07.2020	1990Q1	2020Q3	NA
Business surveys - NACE Rev. 2 activity (source: DG ECFIN)	ei_bcs_bs	folder	NA	NA	NA	NA	NA
Industry - monthly data	ei_bsin_m_r2	dataset	29.09.2020	29.09.2020	1980M01	2020M09	NA
Industry - quarterly data	ei_bsin_q_r2	dataset	29.09.2020	30.07.2020	1980Q1	2020Q3	NA
Construction - monthly data	ei_bsbu_m_r2	dataset	29.09.2020	29.09.2020	1980M01	2020M09	NA
Construction - quarterly data	ei_bsbu_q_r2	dataset	29.09.2020	30.07.2020	1981Q1	2020Q3	NA
Retail sale - monthly data	ei_bsrt_m_r2	dataset	29.09.2020	29.09.2020	1984M01	2020M09	NA
Sentiment indicators - monthly data	ei_bssi_m_r2	dataset	29.09.2020	29.09.2020	1980M01	2020M09	NA
Services - monthly data	ei_bsse_m_r2	dataset	29.09.2020	29.09.2020	1988M01	2020M09	NA
Services - quarterly data	ei_bsse_q_r2	dataset	29.09.2020	30.07.2020	2001Q2	2020Q3	NA
Euro-zone Business Climate Indicator - monthly data	ei_bsci_m_r2	dataset	29.09.2020	29.09.2020	1985M01	2020M09	NA
Financial services - monthly data	ei_bsfs_m	dataset	29.09.2020	29.09.2020	2006M04	2020M09	NA
Financial services - quarterly data	ei_bsfs_q	dataset	29.09.2020	30.07.2020	2007Q3	2020Q3	NA
Employment expectations indicator	ei_bsee_m_r2	dataset	29.09.2020	29.09.2020	1980M01	2020M09	NA

House Price Index (HPI)

The Eurostat House Price Index (HPI) measures price changes of all residential properties purchased by households (flats, detached houses, terraced houses, etc.), both new and existing, independently of their final use and their previous owners. First, we node that the code id for this dataset is teicp270. Once we know the relevant code id, we can download eurostat data using the get_eurostat(id) function.

hpi <- get_eurostat(id="teicp270")
glimpse(hpi)

## Rows: 1,257
## Columns: 5
## $ indic  <chr> "TOTAL", "TOTAL", "TOTAL", "TOTAL", "TOTAL", "TOTAL", "TOTAL...
## $ unit   <chr> "I15_NSA", "I15_NSA", "I15_NSA", "I15_NSA", "I15_NSA", "I15_...
## $ geo    <chr> "AT", "BE", "BG", "CY", "CZ", "DE", "DK", "EA", "EA19", "EE"...
## $ time   <date> 2017-04-01, 2017-04-01, 2017-04-01, 2017-04-01, 2017-04-01,...
## $ values <dbl> 114.2, 104.7, 115.4, 102.7, 119.1, 113.1, 110.5, 107.8, 107....

head(hpi,40) %>% 
  kable()

indic	unit	geo	time	values
TOTAL	I15_NSA	AT	2017-04-01	114.2
TOTAL	I15_NSA	BE	2017-04-01	104.7
TOTAL	I15_NSA	BG	2017-04-01	115.4
TOTAL	I15_NSA	CY	2017-04-01	102.7
TOTAL	I15_NSA	CZ	2017-04-01	119.1
TOTAL	I15_NSA	DE	2017-04-01	113.1
TOTAL	I15_NSA	DK	2017-04-01	110.5
TOTAL	I15_NSA	EA	2017-04-01	107.8
TOTAL	I15_NSA	EA19	2017-04-01	107.8
TOTAL	I15_NSA	EE	2017-04-01	108.4
TOTAL	I15_NSA	ES	2017-04-01	110.4
TOTAL	I15_NSA	EU	2017-04-01	109.0
TOTAL	I15_NSA	EU27_2020	2017-04-01	108.7
TOTAL	I15_NSA	EU28	2017-04-01	109.0
TOTAL	I15_NSA	FI	2017-04-01	103.0
TOTAL	I15_NSA	FR	2017-04-01	103.4
TOTAL	I15_NSA	HR	2017-04-01	104.5
TOTAL	I15_NSA	HU	2017-04-01	125.5
TOTAL	I15_NSA	IE	2017-04-01	115.9
TOTAL	I15_NSA	IS	2017-04-01	130.1
TOTAL	I15_NSA	IT	2017-04-01	99.6
TOTAL	I15_NSA	LT	2017-04-01	114.5
TOTAL	I15_NSA	LU	2017-04-01	112.2
TOTAL	I15_NSA	LV	2017-04-01	119.5
TOTAL	I15_NSA	MT	2017-04-01	108.9
TOTAL	I15_NSA	NL	2017-04-01	111.4
TOTAL	I15_NSA	NO	2017-04-01	115.5
TOTAL	I15_NSA	PL	2017-04-01	105.4
TOTAL	I15_NSA	PT	2017-04-01	115.5
TOTAL	I15_NSA	RO	2017-04-01	114.3
TOTAL	I15_NSA	SE	2017-04-01	116.0
TOTAL	I15_NSA	SI	2017-04-01	111.4
TOTAL	I15_NSA	SK	2017-04-01	113.1
TOTAL	I15_NSA	TR	2017-04-01	124.3
TOTAL	I15_NSA	UK	2017-04-01	111.2
TOTAL	PCH_Q1_NSA	AT	2017-04-01	2.4
TOTAL	PCH_Q1_NSA	BE	2017-04-01	-0.3
TOTAL	PCH_Q1_NSA	BG	2017-04-01	2.4
TOTAL	PCH_Q1_NSA	CY	2017-04-01	3.1
TOTAL	PCH_Q1_NSA	CZ	2017-04-01	2.5

Typically, the downloaded data has codes and abbreviations for all of the variables, but we can use label_eurostat to get a more verbose description.

house_price_index_data <-  hpi %>% 
  label_eurostat()

head(house_price_index_data,40) %>% 
  kable()

indic	unit	geo	time	values
Total	Index, 2015=100 (NSA)	Austria	2017-04-01	114.2
Total	Index, 2015=100 (NSA)	Belgium	2017-04-01	104.7
Total	Index, 2015=100 (NSA)	Bulgaria	2017-04-01	115.4
Total	Index, 2015=100 (NSA)	Cyprus	2017-04-01	102.7
Total	Index, 2015=100 (NSA)	Czechia	2017-04-01	119.1
Total	Index, 2015=100 (NSA)	Germany (until 1990 former territory of the FRG)	2017-04-01	113.1
Total	Index, 2015=100 (NSA)	Denmark	2017-04-01	110.5
Total	Index, 2015=100 (NSA)	Euro area (EA11-1999, EA12-2001, EA13-2007, EA15-2008, EA16-2009, EA17-2011, EA18-2014, EA19-2015)	2017-04-01	107.8
Total	Index, 2015=100 (NSA)	Euro area - 19 countries (from 2015)	2017-04-01	107.8
Total	Index, 2015=100 (NSA)	Estonia	2017-04-01	108.4
Total	Index, 2015=100 (NSA)	Spain	2017-04-01	110.4
Total	Index, 2015=100 (NSA)	European Union (EU6-1958, EU9-1973, EU10-1981, EU12-1986, EU15-1995, EU25-2004, EU27-2007, EU28-2013, EU27-2020)	2017-04-01	109.0
Total	Index, 2015=100 (NSA)	European Union - 27 countries (from 2020)	2017-04-01	108.7
Total	Index, 2015=100 (NSA)	European Union - 28 countries (2013-2020)	2017-04-01	109.0
Total	Index, 2015=100 (NSA)	Finland	2017-04-01	103.0
Total	Index, 2015=100 (NSA)	France	2017-04-01	103.4
Total	Index, 2015=100 (NSA)	Croatia	2017-04-01	104.5
Total	Index, 2015=100 (NSA)	Hungary	2017-04-01	125.5
Total	Index, 2015=100 (NSA)	Ireland	2017-04-01	115.9
Total	Index, 2015=100 (NSA)	Iceland	2017-04-01	130.1
Total	Index, 2015=100 (NSA)	Italy	2017-04-01	99.6
Total	Index, 2015=100 (NSA)	Lithuania	2017-04-01	114.5
Total	Index, 2015=100 (NSA)	Luxembourg	2017-04-01	112.2
Total	Index, 2015=100 (NSA)	Latvia	2017-04-01	119.5
Total	Index, 2015=100 (NSA)	Malta	2017-04-01	108.9
Total	Index, 2015=100 (NSA)	Netherlands	2017-04-01	111.4
Total	Index, 2015=100 (NSA)	Norway	2017-04-01	115.5
Total	Index, 2015=100 (NSA)	Poland	2017-04-01	105.4
Total	Index, 2015=100 (NSA)	Portugal	2017-04-01	115.5
Total	Index, 2015=100 (NSA)	Romania	2017-04-01	114.3
Total	Index, 2015=100 (NSA)	Sweden	2017-04-01	116.0
Total	Index, 2015=100 (NSA)	Slovenia	2017-04-01	111.4
Total	Index, 2015=100 (NSA)	Slovakia	2017-04-01	113.1
Total	Index, 2015=100 (NSA)	Turkey	2017-04-01	124.3
Total	Index, 2015=100 (NSA)	United Kingdom	2017-04-01	111.2
Total	Percentage change q/q-1 (NSA)	Austria	2017-04-01	2.4
Total	Percentage change q/q-1 (NSA)	Belgium	2017-04-01	-0.3
Total	Percentage change q/q-1 (NSA)	Bulgaria	2017-04-01	2.4
Total	Percentage change q/q-1 (NSA)	Cyprus	2017-04-01	3.1
Total	Percentage change q/q-1 (NSA)	Czechia	2017-04-01	2.5

We note that our dataframe contains both the value of the index (unit = I15_NSA), as well as the percentage change (unit = PCH_Q1_NSA). We will select the I15_NSA index, a few countries and the EU-28 index, and plot the evolution of house prices over time.

hpi_data <- hpi %>% 
  
  # choose the UK, France, Poland, Spain, Portugal, Germany, Italy, and the EU28
  filter(geo %in%  c("UK", "FR", "PL", "ES","PT", "DE","IT","EU28") ) %>%  
  
  # choose value of the index (unit =   `I15_NSA`) 
    filter(unit == "I15_NSA")

ggplot(hpi_data, aes(x=time, y=values, group=geo, colour=geo))+
  geom_point()+
  geom_line()+
  theme_bw()+
  labs(
    title= "House price index in the EU (2015 = 100)",
    x = "Time",
    y = "Housing Price Index", 
    caption = "Source: Eurostat, code id = teicp270"
  )

Tourism Seasonality in the Meditteranean

The eurostat database has a dedicated tourism section. I wanted to check monthly nights spent at hotels– the relevant code id = tour_occ_nim in the four Meditteranean countries, Portugal, Spain, Italy, and Greece since 2000.

The code below downloads the data and plots time series plots for all countries.

# create a dataframe tourism_data that contains the eurostat data for
# code id = "tour_occ_nim", namely value of monthly nights spent at hotels
tourism_data <- get_eurostat(id="tour_occ_nim")

med_tourism <-  tourism_data %>%   
  
  # choose Portugal, Spain, Italy, and Greece
  filter(geo %in%  c("PT", "ES", "IT", "EL" ) ) %>%
  
  #use label_eurostat to get verbose descriptions of codes
  label_eurostat() %>% 
  
  # choose number of total hotel accommodations since Jan 1, 2000
  filter (c_resid == "Total", 
          nace_r2 == "Hotels and similar accommodation", 
          unit == "Number",
          time >= "2000-01-01") %>% 
  
  # express values in million of nights
  mutate(values = values/1000000) 

ggplot(med_tourism, aes(x=time, y=values, group=geo, colour=geo))+
  geom_point()+
  geom_line()+
  geom_smooth(se=FALSE)+
  facet_wrap(~geo)+
  theme_bw()+
  labs(title="Hotel stays in the Medditeranean, 2000-present", 
       y= "Millions of nights spent in hotels",
       x = "Year",
       caption = "Source: Eurostat, code = tour_occ_nim")+
  theme(legend.position="none")

All countries exhibit the same seasonal pattern: there is a peak in July-August, and the minimum number is around December-January.

Look at the impact of Covid-19 on all countries!

#first define **ts** (time series ) objects; one for each country  

portugal_tourism <- med_tourism %>% 
  
  #select the country you are interested in, in this case Portugal
  filter (geo == "Portugal") %>% 
  
  #sort by time in ascending order, so  earliest observation is first
  arrange(time) %>%
  
  #we just want to keep the values 
  select(values) %>% 
  
  #time series (ts) starts Jan 2000 and has monthlyfrequency (12 months/yr)
  ts(start=2000, frequency = 12) 



spain_tourism <- med_tourism %>% 
  filter (geo == "Spain") %>% 
  arrange(time) %>% 
  select(values) %>% 
  ts(start=2000, frequency = 12)

italy_tourism <- med_tourism %>% 
  filter (geo == "Italy") %>% 
  arrange(time) %>% 
  select(values) %>%   
  ts(start=2000, frequency = 12)

greece_tourism <- med_tourism %>% 
  filter (geo == "Greece") %>% 
  arrange(time) %>% 
  select(values) %>%   
  ts(start=2000, frequency = 12)


#Season plot for Spain and Greece: the seasonal pattern is consistent since 2000
ggseasonplot(spain_tourism, year.labels=TRUE, year.labels.left=TRUE) +
  labs(
    title = "Seasonal plot: Hotel stays in Spain",
    y = "Millions of nights spent in hotels"
  )+
    theme_bw()

ggseasonplot(greece_tourism, year.labels=TRUE, year.labels.left=TRUE) +
  labs(
    title = "Seasonal plot: Hotel stays in Greece",
    y = "Millions of nights spent in hotels"
  )+
  theme_bw()

An interesting question is which country has the greatest seasonality distortion, namely, how much bigger is the summer peak from the winter bottom. For this we produce a subseries plot, one that emphasises the seasonal patterns and where the data for each season are collected together in separate mini time plots. The horizontal lines indicate the means for each month. This form of plot enables the underlying seasonal pattern to be seen clearly, and also shows the changes in seasonality over time. It is especially useful in identifying changes within particular seasons.

ggsubseriesplot(portugal_tourism)+
  labs(
    title = "Seasonal subseries plot: Hotel stays in Portugal 2000-present",
    subtitle = "Horizontal lines indicate monthly averages",
    y = "Millions of nights spent in hotels", 
    caption = "Source:Eurostat"
  )+
  theme_bw()

ggsubseriesplot(spain_tourism)+
  labs(
    title = "Seasonal subseries plot: Hotel stays in Spain 2000-present",
    subtitle = "Horizontal lines indicate monthly averages",
    y = "Millions of nights spent in hotels", 
    caption = "Source:Eurostat"
  )+
  theme_bw()

ggsubseriesplot(italy_tourism)+
  labs(
    title = "Seasonal subseries plot: Hotel stays in Italy 2000-present",
    subtitle = "Horizontal lines indicate monthly averages",
    y = "Millions of nights spent in hotels", 
        caption = "Source:Eurostat"
  )+
  theme_bw()

ggsubseriesplot(greece_tourism)+
  labs(
    title = "Seasonal subseries plot: Hotel stays in Greece 2000-present",
    subtitle = "Horizontal lines indicate monthly averages",
    y = "Millions of nights spent in hotels", 
    caption = "Source:Eurostat"
  )+
  theme_bw()

Visually, the approximate ratio of max:min averages for each of the four Mediterranean countries is as follows:

Portugal 6:2 = 3
Spain 39:13 = 3
Italy 43:10 = 4.3
Greece 13.5:1= 13.5

Disposable income of private households by NUTS 2 regions

Using the eurostat data, we can create maps of, e.g., disposable income at a regional level. NUTS or Nomenclature of Territorial Units for Statistics is a geocode standard for referencing subdvisions (regions, counties, districts, etc.) within a country.

We will work with the Disposable income of private households by NUTS 2 regions database

income_data <- get_eurostat(id="tgs00026") %>% 
  select(geo,time,values) %>% 
  dplyr::mutate(cat = cut_to_classes(values))


income_2016 <- income_data %>% 
  filter(time == "2016-01-01")

# Download geospatial data from GISCO
geodata <- get_eurostat_geospatial(output_class = "sf",
                                   resolution = "60",
                                   nuts_level = 2,
                                   year = 2016) 


map_data <- inner_join(geodata, income_2016)


ggplot(data=map_data) + geom_sf(aes(fill=cat),color="dim grey", size=.1) + 
  scale_fill_brewer(palette = "Accent") +
  guides(fill = guide_legend(reverse=T, title = "euro")) +
  labs(title="Disposable household income in 2016",
       caption="(C) EuroGeographics for the administrative boundaries 
                Map produced in R with a help from Eurostat-package <github.com/ropengov/eurostat/>") +
  theme_light() + theme(legend.position=c(.8,.8)) +
  coord_sf(xlim=c(-12,44), ylim=c(35,70))

Acknowledgments

This page is derived in part from Tutorial (vignette) for the eurostat R package.

Questions on Descriptive and Inferential Statistics

Wed, 26 Aug 2020 00:00:00 +0000

The questions listed below are designed for discussion and preparation. When reviewing these questions, try to illustrate your points with specific examples/cases from what we have seen in class.

Why do we make distinctions between samples and populations in statistics?
Discuss the use of exploratory data analysis. Illustrate with an example.
Why do we care about the distribution of their data?
What are outliers? What impact do they have on how you describe your data?
What is a robust statistic? Why would you choose to use it?
What is variability? Discuss measures of variability. What are their strengths and weaknesses? In a operations setting, what is the relation of variability to the quality of the product/service?
Why do we standardise data? What does a Z-score tell you, i.e., how do you interpret one? How do you convert a raw score to a Z-score?
What is the Normal distribution? What is the Standard Normal distribution?
How do you use the Normal distribution table to find the percentage of the population that is expected to fall between two points or beyond or below one point in the distribution?
What does the Central Limit Theorem tell us?
Discuss the differences between descriptive and inferential statistics. Is one better than the other? Are they competitive or complementary? Illustrate the kind of situation in which each approach is appropriate.
What are the steps involved in hypothesis testing?
What does specifying the null hypothesis mean? What about the alternative hypothesis? What is the benefit of being so specific about the hypotheses?
With inferential statistics, the goal is to reject the null hypothesis. What does this mean? Do we conclude that the alternative hypothesis is correct? Why or why not?
Why is the standard error of the mean, based on many samples, going to be smaller than the standard deviation of a single sample? In explaining your answer, be sure to describe the interpretation of a standard error of the mean.
What types of error can occur when making decisions based on test of hypothesis? Be specific.
Why are observations that are more than 3 or less than -3 standard deviations from the mean often considered outliers by some researchers?
What does it mean if a researcher sets her $\alpha = 0.01$, and rejects the null hypothesis? How does this differ from setting the alpha at $\alpha = 0.05$ and rejecting the null? In which case is the researcher going to be most likely to reject the null hypothesis?
What is the difference between a one-tailed (directional) and a two-tailed (non-directional) test? When would you use each of them?
What is meant when a researcher says that a finding is statistically significant?

Using Github

Tue, 25 Aug 2020 00:00:00 +0000

Introduction to Git/Github

When you engage in any kind of data science or programming, there comes a (frustrating) point that you need to understand how Git and GitHub work. Learning how to use Git and GitHub is especially important for keeping versions of your work (think something like Dropbox + MS Word’s Track Changes) and collaborating with others.

Git is essentially a boring time machine. Remember when you worked on a Word file and saved it by adding the date, or calling it “mywork-vesion1”, “mywork-final”, “mywork-final-final”, etc?

Git is organised around repositories; repos are folders where you keep a project with all necessary files (code, data, images, etc). So you first need to tell git which files/folders to keep track of for any changes you will be making.

As you keep adding code to your project/assignment/etc, you commit changes into your repository and you add an explanatory comment, or message to yourself briefly describing the changes/additions/new work you have done.

When you commit changes, it’s as though you take a snapshot of your work and write a short comment to yourself; it would be the same as saving your Word document adding today’s date in the filename, or v1, v2, final, final-final, etc.

After committing your changes, you need to pull first, so you get the latesr copy from git and then push them to git– this is when you actually upload changes, etc.

Git workflow

The following lists the main steps to create a repository and keep it updated

Create a repo on GitHub and initialize with a README.
Clone the repo to your local machine. You can either do it as an RStudio Project, or using a shell command: $ git clone REPOSITORY-URL
Add or Stage any changes you make: $ git add -A
Commit your changes: $ git commit -m "Helpful message to yourself/collaborators"
Pull from GitHub: $ git pull
Push your changes to GitHub: $ git push

Repeat steps 3—7, but especially steps 3-4, often.

Git keeps track of all the changes you have made in your repo, just in case you made a mistake and need to go back to an earlier version where things actually worked. GitHub is a website built on top of Git that allows you to collaborate on code with others, in helping with code fixes, documentation, and more.

Further resources

For R users, Jenny Bryan et al have created Happy Git with R, a brilliant resource that shows you how to use Git and GitHub in RStudio effectively.

One final thing: git can be confusing and frustrating as hell (ask me for details)– add git to the challenges of coding and you sometimes end up with people asking themselves interesting questions.

When things do go wrong (they will), have a look at https://ohshitgit.com/ and http://happygitwithr.com/burn.html

Handling dates/times

Mon, 27 Jul 2020 00:00:00 +0000

The lubridate package is one of the most useful packages to handle dates and times in R. Your data may contain dates as a string of characters (class <chr>), and you need to convert them to date objects before you can do any kind of analysis.

`lubridate` to convert strings to `Date` objects

Let us look at an example: We will define Christmas as a string in various formats and will then try to convert it to a Date object so we can manipulate it.

library(lubridate)

today <-  Sys.Date()
today

## [1] "2020-08-25"

class(today)

## [1] "Date"

date1 <-  "25-12-2020"
date2 <-  "12-25-2020"
date3 <-  "2000-12-25"
date4 <-  "Dec 25, 2020"

class(date1)

## [1] "character"

class(date4)

## [1] "character"

# dmy: day-month-year
xmas1 <- lubridate::dmy(date1)
class(xmas1)

## [1] "Date"

# mdy: month-day-year
xmas2 <- lubridate::mdy(date2)
class(xmas2)

## [1] "Date"

# ymd: year-month-date, ISO8601 standard
# https://en.wikipedia.org/wiki/ISO_8601
xmas3 <- ymd(date3)
class(xmas3)

## [1] "Date"

# mdy: month-day-year
xmas4 <- lubridate::mdy(date4)
class(xmas4)

## [1] "Date"

# once we have it as a Data object, we can do calculations...
xmas1 - today

## Time difference of 122 days

# ... but these calculations will not work if the date is a string (character) 
date1 - today

## Error in `-.Date`(date1, today): can only subtract from "Date" objects

On your own

We’ll use data from http://www.tfl.gov.uk to analyse usage of the London Bike Sharing scheme. This data has already been downloaded for you and exists in a CSV (Comma Separated Values), along with weather information.

bike <- read_csv(here::here("data", "londonBikes.csv"))

glimpse(bike)

## Rows: 3,439
## Columns: 14
## $ date          <chr> "01-01-11", "02-01-11", "03-01-11", "04-01-11", "05-0...
## $ bikes_hired   <dbl> 4555, 6250, 7262, 13430, 13757, 9595, 9294, 9338, 105...
## $ season        <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,...
## $ max_temp      <dbl> 7.2, 4.0, 2.9, NA, 7.1, NA, 10.8, 10.4, 7.2, 8.9, 8.3...
## $ min_temp      <dbl> NA, NA, NA, 0.3, 3.8, NA, 1.0, NA, NA, -1.9, NA, 3.7,...
## $ avg_temp      <dbl> 5.6, 2.9, 1.4, 2.7, 5.6, 4.1, 6.1, 6.9, 3.1, 4.3, 5.8...
## $ avg_humidity  <dbl> 84, 79, 80, 87, 84, 92, 92, 82, 79, 87, 82, 89, 89, 8...
## $ avg_pressure  <dbl> 1025, 1028, 1024, 1013, 1000, 996, 999, 997, 1012, 10...
## $ avg_windspeed <dbl> 10, 8, 6, 6, 19, 5, 11, 23, 16, 14, 16, 16, 23, 24, 2...
## $ rainfall_mm   <dbl> 0.0, 0.5, 0.0, 0.0, 0.0, 0.5, 11.4, 13.0, 1.0, 0.0, 7...
## $ rain          <lgl> TRUE, FALSE, FALSE, TRUE, TRUE, TRUE, TRUE, TRUE, FAL...
## $ fog           <lgl> FALSE, FALSE, FALSE, FALSE, FALSE, TRUE, TRUE, FALSE,...
## $ thunderstorm  <lgl> FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALS...
## $ snow          <lgl> FALSE, FALSE, TRUE, FALSE, FALSE, TRUE, FALSE, FALSE,...

date is a character string, and is given as 01-01-2011, 02-01-2011, 03-01-2011, meaning 1st, 2nd, 3rd of January, etc. In other words, the format of the string is dmy, or day-month-year.

Handling dates/times

Mon, 27 Jul 2020 00:00:00 +0000

`lubridate` to convert strings to `Date` objects

Let us look at an example: We will define Christmas as a string in various formats and will then try to convert it to a Date object so we can manipulate it.

library(lubridate)

today <-  Sys.Date()
today

## [1] "2020-08-05"

class(today)

## [1] "Date"

date1 <-  "25-12-2020"
date2 <-  "12-25-2020"
date3 <-  "2000-12-25"
date4 <-  "Dec 25, 2020"

class(date1)

## [1] "character"

class(date4)

## [1] "character"

# dmy: day-month-year
xmas1 <- lubridate::dmy(date1)
class(xmas1)

## [1] "Date"

# mdy: month-day-year
xmas2 <- lubridate::mdy(date2)
class(xmas2)

## [1] "Date"

# ymd: year-month-date, ISO8601 standard
# https://en.wikipedia.org/wiki/ISO_8601
xmas3 <- ymd(date3)
class(xmas3)

## [1] "Date"

# mdy: month-day-year
xmas4 <- lubridate::mdy(date4)
class(xmas4)

## [1] "Date"

# once we have it as a Data object, we can do calculations...
xmas1 - today

## Time difference of 142 days

# ... but these calculations will not work if the date is a string (character) 
date1 - today

## Error in `-.Date`(date1, today): can only subtract from "Date" objects

On your own

bike <- read_csv(here::here("data", "londonBikes.csv"))

glimpse(bike)

## Rows: 3,439
## Columns: 14
## $ date          <chr> "01-01-11", "02-01-11", "03-01-11", "04-01-11", "05-0...
## $ bikes_hired   <dbl> 4555, 6250, 7262, 13430, 13757, 9595, 9294, 9338, 105...
## $ season        <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,...
## $ max_temp      <dbl> 7.2, 4.0, 2.9, NA, 7.1, NA, 10.8, 10.4, 7.2, 8.9, 8.3...
## $ min_temp      <dbl> NA, NA, NA, 0.3, 3.8, NA, 1.0, NA, NA, -1.9, NA, 3.7,...
## $ avg_temp      <dbl> 5.6, 2.9, 1.4, 2.7, 5.6, 4.1, 6.1, 6.9, 3.1, 4.3, 5.8...
## $ avg_humidity  <dbl> 84, 79, 80, 87, 84, 92, 92, 82, 79, 87, 82, 89, 89, 8...
## $ avg_pressure  <dbl> 1025, 1028, 1024, 1013, 1000, 996, 999, 997, 1012, 10...
## $ avg_windspeed <dbl> 10, 8, 6, 6, 19, 5, 11, 23, 16, 14, 16, 16, 23, 24, 2...
## $ rainfall_mm   <dbl> 0.0, 0.5, 0.0, 0.0, 0.0, 0.5, 11.4, 13.0, 1.0, 0.0, 7...
## $ rain          <lgl> TRUE, FALSE, FALSE, TRUE, TRUE, TRUE, TRUE, TRUE, FAL...
## $ fog           <lgl> FALSE, FALSE, FALSE, FALSE, FALSE, TRUE, TRUE, FALSE,...
## $ thunderstorm  <lgl> FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALS...
## $ snow          <lgl> FALSE, FALSE, TRUE, FALSE, FALSE, TRUE, FALSE, FALSE,...

date is a character string, and is given as 01-01-2011, 02-01-2011, 03-01-2011, meaning 1st, 2nd, 3ed of January, etc. In other words, the format of the string is dmy, or day-month-year.

Model diagnostics

Sat, 25 Jul 2020 00:00:00 +0000

Model diagnostics

Sat, 25 Jul 2020 00:00:00 +0000

autoplot(model)

autoplot(model)

Reshape data

Sat, 25 Jul 2020 00:00:00 +0000

`pivot_longer()` - `pivot_wider` to make a wide dataset long and vice-versa

The United Nations have all sorts of data. For this example, we will work with data on tourist/visitor arrivals and tourism expenditure. The dataframe un_tourism_data has been loaded into memory, and contains data on tourist arrivals (in thousands) and tourism expenditure (in millions of US$). We would like to calculate spending per tourist and see how some of the top tourist destinations compare

You have to:

Convert un_tourism_data from long to wide format; you need to do this to create the new variable spending_per_tourist
clean column names
rename columns to “tourism_expenditure” and “tourist_arrivals”
remove rows where tourism expenditure or arrivals are NA
create a new column where you calculate spending per tourist (remember expenditure is in millions and arrival is in thousands)

You have successfully calculated spending per tourist. We are now faced with the challenge of producing a plot that looks like this

The best way to get this plot is to first reshape the dataframe from wide to long, and then apply your ggplot skills.

`left_join()` to join tables on columns

We have loaded into memory two dataframes countries and matches that contain matches played in various European football (soccer) leagues over a number of years. We want to join the two dataframes, so we can see the name, rather than an ID of the league. We also want to calculate the average number of goals per game in each league and plot those averages for all seasons.

`bind_rows()` to combine rows from two or more datasets

We have three distinct dataframes, england_matches, germany_matches, and italy_matches that contain data on each country. We need to combine these three datasets into one, and sort it in ascending order by date.

Reshape data

Sat, 25 Jul 2020 00:00:00 +0000

`pivot_longer()` - `pivot_wider` to make a wide dataset long and vice-versa

You have to:

Convert un_tourism_data from long to wide format; you need to do this to create the new variable spending_per_tourist
clean column names
rename columns to “tourism_expenditure” and “tourist_arrivals”
remove rows where tourism expenditure or arrivals are NA
create a new column where you calculate spending per tourist (remember expenditure is in millions and arrival is in thousands)

You have succesfully calculated spending per tourist. We are now faced with the challenge of producing a plot that looks like this

The best way to get this plot is to first reshape the dataframe from wide to long, and then apply your ggplot skills.

`left_join()` to join tables on columns

`bind_rows()` to combine rows from two or more datasets

H-E-L-P!

Sat, 25 Jul 2020 00:00:00 +0000

Overwiew

I recently got an ad on my phone trying to sell me a service to become a data scientist in a month. This may make you think that doing data science with R is an easy, straight-forward process.

It is not.

Any time I have been struggling to learn a new tool/technology/software … I go back to this short clip I cut out of @hadleywickham’s 2014 @user2014_ucla tutorial on Dplyr to motivate myself to keep pushing through … learning new tools is hard for everyone at the beginning!- Brain AMA
— Aliakbar Akbaritabar (Ali) (@Akbaritabar) July 25, 2018

You will stumble, get frustrated, lost, and confused, make (silly) mistakes even when you think you know stuff, not understand how to perform a task, not understand why your code is generating an error, etc. It still happens to me all the time, and I am still googling really basic stuff about R, even after quite a few years using it. But as Alfred so helpfully points out to Bruce Wayne in Batman Begins, do not fall to pieces when you fail. Instead, learn to pick yourself up, learn from experience, practice more, and get better.

Back in 2018, Hadley Wickham, one the major driving forces behind the tidyverse, recorded a video of his live analysis of a dataset with the goal of demonstrating his approach.

It’s great to see Hadley, a true expert, undertake data analysis; he makes quite a few mistakes in this video and he even forgets the arguments for routines/packages he has written! But it’s even more powerful that he shrugs off the mistake, corrects it, and moves forward.

Even more interesting is to see Hadley, the author of the ggplot2 package, admit to using Google to look things up in… ggplot2!

Error messages in R

Error messages are a normal part of working in R, not a sign you are bad. To make matters worse, R will alert you with red letters not just for errors, but for warnings, too. It helps to learn relatively early on how to decipher these messages and what common ones mean.

First, if after typing a command you see red letters, don’t panic– it may just be a warning, and most of the times you can ignore them or worry about them later.

But you will get errors (in red letters too!) As an example let me try to read a CSV file using the read_csv function

There are three main parts to an error:

The declaration that it is an Error, and not a Warning
The location of the error: it is in the read_csv("myfile.csv") line of my code
The issue my code caused: could not find function "read_csv", as I asked R to use a function from the readr package but forgot to load it.

Let me try again.

The error given now is again produced by the same read_csv function, but the error is that the CSV file does not exist in the working directory.

Failure, and the 15 minute rule

It’s good practice to follow the 15 minute rule. If you encounter a problem in your work, spend 15 minutes troubleshooting the problem on your own; Google, RStudio Support, and StackOverflow are good places to look for answers. So if you google your error message, you will find that 99% of the time someone has had the same error message and the solution is on stackoverflow.

However, if after 15 minutes you still cannot solve the problem, ask for help.

15 min rule: when stuck, you HAVE to try on your own for 15 min; after 15 min, you HAVE to ask for help.- Brain AMA
— Rachel Thomas (@math_rachel) August 15, 2016

The `reprex` package

How should you ask for help? You must provide enough information so others can understand what is the issue with your code and try to reproduce the issue on their own computer.Stackoverflow provides advice not only on technical questions but also on how to ask good questions! A very popular post addresses how to make a great R reproducible example:

The reprex package, written by Jenny Bryan, was developed to help create reproducible examples, so others can reproduce your code, run it, and see where the issue is.

`reprex` with copy-paste

Reprex works with whatever is currently saved on your clipboard. The easiest way to use reprex is to highlight with your mouse the part of code that gives you an error and copy it to your clipboard using Command+c (Mac) or Control+c(Windows)).

Now that the code has been highlighted, you can easily just type reprex() and the reprex code will now be on the clipboard, which means you can paste it directly into a new Rmd file

x <- 1:10
mean(x)
#> [1] 5.5

`reprex` directly with the reprex() command

Besides copy-and-paste which is the easiest way to use reprex, you can include the code you want to share ore debug directly into the reprex() command. Let us look at a few examples.

reprex(gapminder %>% summarise(lifeExp))
#> Error in gapminder %>% summarise(lifeExp): could not find function "%>%"

^{Created on 2019-07-16 by the reprex package (v0.3.0)}

The error message given is that it cannot find the pipe operator %>%, as we haven’t given the library(dplyr) command. reprex will ensure that all the necessary data and packages are loaded. The information above is now automatically stored on your clipboard, and you can paste it directly (with Ctrl/Cmd+c) as needed.

Let us load the library and try again.

reprex({library(dplyr); gapminder %>% summarise(lifeExp)})
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
#> Error in eval(lhs, parent, parent): object 'gapminder' not found

dplyr is ok now, and the pipe operator works, but we now realise that the gapminder package has not been loaded; let’s try again.

reprex({library(dplyr); library(gapminder); gapminder %>% summarise(lifeExp)}
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
#> Error: Column `lifeExp` must be length 1 (a summary value), not 1704

The error we get relates to the use of the summarise function; this function summarises many values into a single summary, like mean, min, median, etc. R tells us that lifeExp must be of length 1 (a single summary value) rather than 1704 values which is how many values lifeExp has.

Further Resources

Using SQL within R

Tue, 11 Aug 2020 00:00:00 +0000

SQL and `dbplyr`

This sort note will teach you the basics of using SQL databases with R. Sometimes, you have a massive dataset, made up of many different dataframes (or tables in SQL jargon), that would crash your computer’s memory if you try to load it. To interact with any database you typically use SQL, the Structured Query Language.

Rather than writing SQL commands, the dbplyr package automatically generates SQL commands from dplyr sequences. However, please keep in mind that SQL is a very large language, and dbplyr doesn’t do everything, but you can still get a lot out of it.

SQL commands vs dplyr verbs

One of the advantages of learning about tidy data and dplyr is that with dplyr verbs you can replicate a lot of what SQL does.

`SQL` command…	… translate to `dplyr` verb
SELECT	`select()` for columns `mutate()` for expressions `summarise()` for aggregates
FROM	which dataframe to use
WHERE	`filter()`
GROUP_BY	`group_by()`
ORDER_BY	`arrange()`
LIMIT	`head()`

Establish a connection with the SQLite database

We will use the European Soccer Database that has more than 25,000 matches and more than 11,000 players. We first need to establish a connection with the SQL database. Unlike dataframes that we just load into memory, the size of some SQL databases prohibits loading the entire database into memory. Although this soccer database is a locally saved file, we would use a similar connection into any SQL database over the internet

# set up a connection to sqlite database
football_db <- DBI::dbConnect(
  drv = RSQLite::SQLite(),
  dbname = "database.sqlite"
)

The general code for connecting to a remote database is:

connection_to_db <- DBI::dbConnect(
  drv = [database driver, eg odbc::odbc()],
  dbname = "database_name",
  user = "User_ID",
  password = "Password",
  host = "host name", (default=local connection)
  port = "port number" (default=local connection)
)

That’s pretty much it - R now has a direct connection to the database and you can start making queries.

Database objects or tibbles?

Now, an SQL database will typically contain multiple tables. You can think of these tables as R data frames (or tibbles). What are the tables in the soccer database? We can browse the tables in the database using DBI::dbListTables()

DBI::dbListTables(football_db)

## [1] "Country"           "League"            "Match"            
## [4] "Player"            "Player_Attributes" "Team"             
## [7] "Team_Attributes"   "sqlite_sequence"

We can easily set these tables up as database objects using dplyr

countries <- dplyr::tbl(football_db, "Country")
leagues <- dplyr::tbl(football_db, "League")
matches <- dplyr::tbl(football_db, "Match")
teams <- dplyr::tbl(football_db, "Team")
team_attributes <- dplyr::tbl(football_db, "Team_Attributes")
players <- dplyr::tbl(football_db, "Player")
player_attributes <- dplyr::tbl(football_db, "Player_Attributes")

Each of these tables are SQL database objects in your R session which you can manipulate in the same way as a dataframe.

class(countries)

## [1] "tbl_SQLiteConnection" "tbl_dbi"              "tbl_sql"             
## [4] "tbl_lazy"             "tbl"

When you define these tables, you are not physically downloading them, just creating a bare minimum extract to work with.

IF you wanted to handle these as normal dataframes or tibbles, you can simply pipe the database objects to as_tibble()

player_attributes_df <- player_attributes %>% as_tibble()
class(player_attributes)

## [1] "tbl_SQLiteConnection" "tbl_dbi"              "tbl_sql"             
## [4] "tbl_lazy"             "tbl"

class(player_attributes_df)

## [1] "tbl_df"     "tbl"        "data.frame"

Notice the difference between player_atributes, a database object, and player_atributes_df, a ‘regular’ dataframe/tibble.

Now that we have player attributes as a dataframe, we can handle it the usual way and, e.g., build a scatterplot/correlation matrix wth ggpairs()

player_attributes_df %>% 
  filter(!is.na(preferred_foot)) %>% 
  select(preferred_foot, ball_control, overall_rating) %>% 
  ggpairs(aes(colour=preferred_foot, alpha = 0.3))+
  scale_colour_manual(values = c("#67a9cf","#ef8a62"))+
  scale_fill_manual(values = c("#67a9cf","#ef8a62"))+
  theme_bw()

Querying the database with `dbplyr`

To create the ggpairs() plot we had to convert a database table to a dataframe, load it all in the computer’s memory, and then use ggplot. The beauty of working with databases is that we do NOT have to load everything into memory. Instead, all dplyr calls are evaluated lazily, generating SQL code that is only sent to the database when you request the data.

Let us look at an example. What if we wanted to calculate the average number of goals per league (country) per season and then plot those averages. This seems like a data wrangling exercise we would do with dplyr

# write dplyr code that will calculate average number of goals per country per season
goals_per_match <-  matches %>%
  group_by(country_id, season) %>%
  summarise(avg_goals = mean(home_team_goal + away_team_goal)) %>%
 
  #do a left_join, so we know the country's name rather than the country's ID
  left_join(countries, by = c("country_id"="id")) %>%
  arrange(desc(avg_goals)) %>% 
  ungroup()

What kind of an object is goals_per_match?

#what kind of an object is goals_per_match?
class(goals_per_match)

## [1] "tbl_SQLiteConnection" "tbl_dbi"              "tbl_sql"             
## [4] "tbl_lazy"             "tbl"

goals_per_match is not a dataframe (tibble), but rather a querty to an SQLite database table.

Generate the actual SQL commands

We are familiar with all the dplyr verbs (filter, select, group_by, summarise, arrange, etc.), but SQL has its own commands, all of which are written in capital letters (is SQL constantly angry and shouting? Who knew…). We can generate the actual SQL commands using dbplyr::sql_render() or dplyr::show_query()

# Generate actual SQL commands: We can either use dbplyr::sql_render() or dplyr::show_query()
dbplyr::sql_render(goals_per_match)

## <SQL> SELECT *
## FROM (SELECT `LHS`.`country_id` AS `country_id`, `LHS`.`season` AS `season`, `LHS`.`avg_goals` AS `avg_goals`, `RHS`.`name` AS `name`
## FROM (SELECT `country_id`, `season`, AVG(`home_team_goal` + `away_team_goal`) AS `avg_goals`
## FROM `Match`
## GROUP BY `country_id`, `season`) AS `LHS`
## LEFT JOIN `Country` AS `RHS`
## ON (`LHS`.`country_id` = `RHS`.`id`)
## )
## ORDER BY `avg_goals` DESC

goals_per_match %>% show_query()

## <SQL>
## SELECT *
## FROM (SELECT `LHS`.`country_id` AS `country_id`, `LHS`.`season` AS `season`, `LHS`.`avg_goals` AS `avg_goals`, `RHS`.`name` AS `name`
## FROM (SELECT `country_id`, `season`, AVG(`home_team_goal` + `away_team_goal`) AS `avg_goals`
## FROM `Match`
## GROUP BY `country_id`, `season`) AS `LHS`
## LEFT JOIN `Country` AS `RHS`
## ON (`LHS`.`country_id` = `RHS`.`id`)
## )
## ORDER BY `avg_goals` DESC

Run SQL query and retrieve results

Now that we have the SQL query we can retrieve the results into a local dataframe (tibble) using collect(). The main difference is that rather than loading all of the databases in memory, the goals_per_match goes to the SQL database, collects the necessary data, and only returns

# execute query and retrieve results in a tibble (dataframe). 
goals_match_tibble <- goals_per_match %>% 
  collect()

#have a look at the resulting dataframe with glimpse() and skim()
glimpse(goals_match_tibble)

## Rows: 88
## Columns: 4
## $ country_id <int> 24558, 13274, 13274, 13274, 7809, 13274, 24558, 13274, 2...
## $ season     <chr> "2009/2010", "2011/2012", "2010/2011", "2013/2014", "201...
## $ avg_goals  <dbl> 3.33, 3.26, 3.23, 3.20, 3.16, 3.15, 3.14, 3.08, 3.00, 2....
## $ name       <chr> "Switzerland", "Netherlands", "Netherlands", "Netherland...

skimr::skim(goals_match_tibble)

Table 1: Data summary
Name	goals_match_tibble
Number of rows	88
Number of columns	4
_______________________
Column type frequency:
character	2
numeric	2
________________________
Group variables	None

Variable type: character

skim_variable	n_missing	complete_rate	min	max	empty	n_unique	whitespace
season	0	1	9	9	0	8	0
name	0	1	5	11	0	11	0

Variable type: numeric

skim_variable	n_missing	complete_rate	mean	sd	p0	p25	p50	p75	p100	hist
country_id	0	1	12452.09	7877.88	1.00	4769.00	13274.00	19694.00	24558.00	▇▂▅▅▇
avg_goals	0	1	2.71	0.24	2.18	2.56	2.71	2.86	3.33	▂▆▇▃▂

The resulting dataframe only has the information we want: 4 variables (columns) and 88 rows (cases); for each country and season, we have the average number of goals scored per match. We can now use this smaller dataframe and plot the results.

# plot results, using goals_match_tibble
ggplot(goals_match_tibble) + 
  geom_point(aes(x=reorder(name, avg_goals),y=avg_goals, colour=name))+
  theme_bw(8)+ 
  facet_wrap(~season, nrow=4)+
  labs(
    title = "Which football leagues had the higest number of goals per game?",
    y = "Average Number of Goals per Match",
   x = "National League"
    ) + 
  coord_flip() +
  theme(legend.position = "none")

ggplot(goals_match_tibble, aes(x=reorder(name, avg_goals),y=avg_goals, colour=name)) + 
  geom_violin()+
  # geom_boxplot()+
  geom_jitter()+
  theme_bw()+
  labs(
    title = "Which football leagues had the higest number of goals per game?",
    subtitle="2008/2009 to 2015/2016",
    y = "Average Number of Goals per Match",
    x = "National League"
  ) + 
  coord_flip() +
  theme(legend.position = "none")

We cannow run queries on the database, collect the results in a local dataframe, and show the results of, e.g., the highest overall rating of all players in the database.

# Which are the top 20 players by overall rating (`overall_rating`)?
top_players <-  player_attributes %>%
  group_by(player_api_id) %>%
  summarise(max_rating = max(overall_rating)) %>% 
  arrange(desc(max_rating)) %>% 
  left_join(players, by = c("player_api_id"="player_api_id")) %>%
  collect

top_players %>% 
  head(20) %>% 
  kableExtra::kable()

player_api_id	max_rating	id	player_name	player_fifa_api_id	birthday	height	weight
30981	94	6176	Lionel Messi	158023	1987-06-24 00:00:00	170	159
30893	93	1995	Cristiano Ronaldo	20801	1985-02-05 00:00:00	185	176
30829	93	10749	Wayne Rooney	54050	1985-10-24 00:00:00	175	183
30717	93	3826	Gianluigi Buffon	1179	1978-01-28 00:00:00	193	201
39989	92	3994	Gregory Coupet	1747	1972-12-31 00:00:00	180	176
39854	92	10861	Xavi Hernandez	10535	1980-01-25 00:00:00	170	148
34520	91	3183	Fabio Cannavaro	1183	1973-09-13 00:00:00	175	165
30955	91	742	Andres Iniesta	41	1984-05-11 00:00:00	170	150
30743	91	9216	Ronaldinho	28130	1980-03-21 00:00:00	183	168
30723	91	388	Alessandro Nesta	1088	1976-03-19 00:00:00	188	174
30657	91	4366	Iker Casillas	5479	1981-05-20 00:00:00	185	185
30627	91	5120	John Terry	13732	1980-12-07 00:00:00	188	198
30626	91	10203	Thierry Henry	1625	1977-08-17 00:00:00	188	183
41044	90	5592	Kaka	138449	1982-04-22 00:00:00	185	183
40636	90	6377	Luis Suarez	176580	1987-01-24 00:00:00	183	187
38843	90	11039	Ze Roberto	28765	1974-07-06 00:00:00	173	159
35724	90	11057	Zlatan Ibrahimovic	41236	1981-10-03 00:00:00	196	209
30924	90	3514	Franck Ribery	156616	1983-04-07 00:00:00	170	159
30834	90	951	Arjen Robben	9014	1984-01-23 00:00:00	180	176
30728	90	2426	David Trezeguet	5984	1977-10-15 00:00:00	190	176

# Which are the top 20 goalkeepers by sum of all gk attributes (`gk_diving`,`gk_handling`, `gk_kicking`, etc)?
top_goalies <-  player_attributes %>%
  mutate(goalie_rating = gk_diving + gk_handling + gk_kicking + gk_positioning + gk_reflexes) %>% 
  group_by(player_api_id) %>%
  summarise(max_goalie_rating = max(goalie_rating)) %>% 
  arrange(desc(max_goalie_rating)) %>% 
  left_join(players, by = c("player_api_id"="player_api_id")) %>%
  collect
  

top_goalies %>% 
  head(20) %>% 
  kableExtra::kable()

player_api_id	max_goalie_rating	id	player_name	player_fifa_api_id	birthday	height	weight
30717	449	3826	Gianluigi Buffon	1179	1978-01-28 00:00:00	193	201
39989	447	3994	Gregory Coupet	1747	1972-12-31 00:00:00	180	176
30859	445	8580	Petr Cech	48940	1982-05-20 00:00:00	196	198
30657	442	4366	Iker Casillas	5479	1981-05-20 00:00:00	185	185
27299	440	6556	Manuel Neuer	167495	1986-03-27 00:00:00	193	203
30989	438	5536	Julio Cesar	48717	1979-09-03 00:00:00	185	174
24503	437	9579	Sebastian Frey	1289	1980-03-18 00:00:00	190	198
30726	436	2900	Edwin van der Sar	51539	1970-10-29 00:00:00	198	196
182917	429	2340	David De Gea	193080	1990-11-07 00:00:00	193	181
30660	428	8541	Pepe Reina	24630	1982-08-31 00:00:00	188	203
30622	426	8413	Paul Robinson	13914	1979-10-15 00:00:00	193	198
32657	425	10625	Victor Valdes	106573	1982-01-14 00:00:00	183	172
30742	425	7470	Mickael Landreau	3813	1979-05-14 00:00:00	183	185
26295	425	4272	Hugo Lloris	167948	1986-12-26 00:00:00	188	172
30841	424	6446	Maarten Stekelenburg	2147	1982-09-22 00:00:00	198	203
27341	424	9028	Robert Enke,30	158400	1977-08-24 00:00:00	185	172
30648	423	4832	Jens Lehmann	805	1969-11-10 00:00:00	190	192
33986	421	746	Andres Palop	8247	1973-10-22 00:00:00	183	165
31293	421	10009	Steve Mandanda	163705	1985-03-28 00:00:00	185	181
30380	420	1345	Brad Friedel	11983	1971-05-18 00:00:00	188	203

Other references

Other Data Sources

Fri, 31 Jul 2020 00:00:00 +0000

The web is a vast source of datasets on almost any subject, such as demographics, disease, economics, finance, geography, entertainment, science, etc. You can always start with Google’s Dataset Search that indexes thousands of public datasets.

Here are some more suggestions:

Kaggle: Kaggle hosts machine learning competitions and contains a large number of datasets that are generally free and open to the public.
Awesome Public Datasets: Collection of public datasets, arranged by area
Reddit
UK data and UK Office for National Statistics
U.S. Government’s open data with many datasets on a range of issues
Data is plural: a weekly newsletter that has collected over a thousand useful/curious datasets. This may well be one of my favourite dataset collections!
Our World in Data contains time series of demographic and global development data. Their collection of Covid-19 data is among the best.
TidyTuesday: A weekly data project in R where they release a new dataset every week and emphasis is placed on understanding how to summarise and arrange data to make meaningful charts with ggplot2, tidyr, dplyr, and other tools in the tidyverse ecosystem.
fivethirtyeight.com is a data-driven journalism site that share the data on most of their stories
In terms of investigative journalism, The Markup and ProPublica are both data-driven and share their data; All Markup data is freely available and ProPublica provides many of their datasets for free
Erik Gahner’s list of political science datasets: Datasets divided by topic (governance, elections, policy, political elites, etc.), geography (country, region), etc.
BigQuery public datasets Google has set up BigQuery which is a data warehouse for some large datasets that you really need to access with SQL. There is even an R package bigrquery that allows you to easily talk with BigQuery’s database.

Exploratory Data Analysis for Modelling

Tue, 28 Jul 2020 00:00:00 +0000

You may remember the Happy Feet movie and here is an Adélie penguin singing My Way. We shall analyse data that were collected and made available by Dr. Kristen Gorman and the Palmer Station, Antarctica LTER. The dataset contains data for 344 penguins on 3 different species of penguins (Adélie, Chinstrap, and Gentoo), collected from 3 islands in the Palmer Archipelago, Antarctica.

This great artwork was made by [@allison_horst](https://twitter.com/allison_horst)

and if you are a Happy Feet fan, these are the penguins we have data on

Penguin Bill dimensions

The culmen is the upper ridge of a bird’s bill. In the simplified penguins data, culmen length and depth are renamed as variables bill_length_mm and bill_depth_mm to be more intuitive.

For this penguin data, the culmen (bill) length and depth are measured as shown below:

Exploratory Data Analysis (EDA).

The variable of interest is penguin body mass. We want to see whether body mass is related to any of the other variables included in the dataframe. So let us start by looking at the data

library(palmerpenguins)
glimpse(penguins)

## Rows: 344
## Columns: 8
## $ species           <fct> Adelie, Adelie, Adelie, Adelie, Adelie, Adelie, A...
## $ island            <fct> Torgersen, Torgersen, Torgersen, Torgersen, Torge...
## $ bill_length_mm    <dbl> 39.1, 39.5, 40.3, NA, 36.7, 39.3, 38.9, 39.2, 34....
## $ bill_depth_mm     <dbl> 18.7, 17.4, 18.0, NA, 19.3, 20.6, 17.8, 19.6, 18....
## $ flipper_length_mm <int> 181, 186, 195, NA, 193, 190, 181, 195, 193, 190, ...
## $ body_mass_g       <int> 3750, 3800, 3250, NA, 3450, 3650, 3625, 4675, 347...
## $ sex               <fct> male, female, female, NA, female, male, female, m...
## $ year              <int> 2007, 2007, 2007, 2007, 2007, 2007, 2007, 2007, 2...

skimr::skim(penguins)

Table 1: Data summary
Name	penguins
Number of rows	344
Number of columns	8
_______________________
Column type frequency:
factor	3
numeric	5
________________________
Group variables	None

Variable type: factor

skim_variable	n_missing	complete_rate	ordered	n_unique	top_counts
species	0	1.00	FALSE	3	Ade: 152, Gen: 124, Chi: 68
island	0	1.00	FALSE	3	Bis: 168, Dre: 124, Tor: 52
sex	11	0.97	FALSE	2	mal: 168, fem: 165

Variable type: numeric

skim_variable	n_missing	complete_rate	mean	sd	p0	p25	p50	p75	p100	hist
bill_length_mm	2	0.99	43.92	5.46	32.1	39.23	44.45	48.5	59.6	▃▇▇▆▁
bill_depth_mm	2	0.99	17.15	1.97	13.1	15.60	17.30	18.7	21.5	▅▅▇▇▂
flipper_length_mm	2	0.99	200.92	14.06	172.0	190.00	197.00	213.0	231.0	▂▇▃▅▂
body_mass_g	2	0.99	4201.75	801.95	2700.0	3550.00	4050.00	4750.0	6300.0	▃▇▆▃▂
year	0	1.00	2008.03	0.82	2007.0	2007.00	2008.00	2009.0	2009.0	▇▁▇▁▇

`GGally::ggpairs()` to get scatter plot + correlation matrix

GGally::ggpairs() is a useful tool for exploring distributions and correlations.

ggpairs1 <- penguins %>% 
  na.omit() %>% 
  select( body_mass_g, flipper_length_mm, bill_length_mm, bill_depth_mm, species, sex, island) %>% 
  rename(`Flipper length(mm)` = flipper_length_mm, 
         `Body mass (g)` = body_mass_g, 
         `Bill length (mm)` = bill_length_mm, 
         `Bill depth (mm)` = bill_depth_mm) %>% 
  GGally::ggpairs() +
  theme_minimal() 

ggpairs1

ggpairs() provides a lot of information, so let us spend some time deciphering this chart.

Along the diagonal we get a density plot of each variable of interest. For instance, body mass seems to be right skewed, and the rest of the variables seem bimodal.
The upper part of the matrix, shows correlation coefficients– to determine between which variables, read off the corresponding row and column header
The bottom part provides a scatterplot between any two variables which you can again determine by looking at the row/column they correspond.
For categorical variables (species, sex, island), we do not get any numerical values, but rather histograms and boxplots that show the distribution of outcomes. If we wanted to get a numerical correlation value, we would use the polycor package and polycor::hetcor() that calculates polyserial correlations between numeric and ordinal variables, and polychoric correlations between ordinal variables, but this is outside the scope of this class.

Which correlation is the highest? That between body mass and flipper length, or 0.873. We can see the scatterplot of these two variables in the upper left, just underneath the density plot of body mass, whereas to the right of the body mass density plot we can see the correlation value of 0.873. We could somehow “see” a line through these points. In building a model for body mass, surely flipper length will be the first variable to consider and the one that would explain a fair bit of the variation in body mass.

What about the lowest correlation? That seems to be the one between bill length and bill depth, with a correlation value of -0.229. On the face of it, there seems to be very weak relationship between these two variables.

What about the second lowest correlation? Numerically, it is the one between body mass and bill depth (-0.472). However, if you look at the corresponding scatterplot in the lower left corner you see two clusters of points, so we need to dig a bit deeper.

Dummy variables

Dummy, or categorical, variables allows us to incorporate non-numeric data into our analysis. In this example we have two categorical variables (species, sex) and we can use ggpairs() to dig a bit deeper and then use these categorical variables in our modelling. The first scatterplot matrix does not take into consideration the species or the sex of the penguins, variables that may help explain differences in body mass.

ggpairs2 <- penguins %>% 
  na.omit() %>% 
  select( species, sex, body_mass_g, flipper_length_mm, bill_length_mm, bill_depth_mm) %>% 
  rename(`Flipper lenght(mm)` = flipper_length_mm, 
         `Body mass (g)` = body_mass_g, 
         `Bill length (mm)` = bill_length_mm, 
         `Bill depth (mm)` = bill_depth_mm) %>% 
  GGally::ggpairs(aes(colour=species, shape=species),
                  alpha = 0.4) +
  scale_colour_manual(values = c("darkorange","purple","cyan4")) +
  scale_fill_manual(values = c("darkorange","purple","cyan4")) +
  theme_minimal()
  

ggpairs2

When we include the categorical (factor) variables of species and sex, and since these are no longer numerical values, we get boxplots and histograms. For instance to explore the relationship between body mass and species, look at the upper left of the table; yo can see that the green penguins (Gentoo) seem to be heavier than the other two species which may have no difference between them.

Since this plot may be confusing, let us just concentrate on the scatterplot matrix of numerical values coloured by species.

ggpairs3 <- penguins %>% 
  na.omit() %>% 
  select( species, sex, body_mass_g, flipper_length_mm, bill_length_mm, bill_depth_mm) %>% 
  rename(`Flipper lenght(mm)` = flipper_length_mm, 
         `Body mass (g)` = body_mass_g, 
         `Bill length (mm)` = bill_length_mm, 
         `Bill depth (mm)` = bill_depth_mm) %>% 
  GGally::ggpairs(aes(colour=species, shape=species),
                  alpha = 0.4,
                  columns = c(3:6)) + 
  scale_colour_manual(values = c("darkorange","purple","cyan4")) +
  scale_fill_manual(values = c("darkorange","purple","cyan4")) +
  theme_minimal()
  

ggpairs3

Modelling considerations of numerical variables

Let us examine the relationship between body mass and flipper length, and body mass and bill depth.

Penguin body mass vs flipper length

mass_flipper <- ggplot(data = penguins,
                       aes(x = flipper_length_mm,
                           y = body_mass_g)) +
  geom_point(aes(colour = species,
                 shape = species),
             size = 3,
             alpha = 0.6) +
  theme_minimal() +
  geom_smooth(method = "lm", se=FALSE, aes(colour = species)) +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  labs(title = "Penguin size, Palmer Station LTER",
       subtitle = "Flipper length and body mass for Adelie, Chinstrap and Gentoo Penguins",
       x = "Flipper length (mm)",
       y = "Body mass (g)",
       color = "Penguin species",
       shape = "Penguin species") +
  theme(legend.position = c(0.2, 0.7),
        legend.background = element_rect(fill = "white", colour = NA),
        plot.title.position = "plot",
        plot.caption = element_text(hjust = 0, face= "italic"),
        plot.caption.position = "plot")

mass_flipper

Penguin body mass vs bill depth

mass_bill_depth <- ggplot(data = penguins,
                       aes(x = bill_depth_mm,
                           y = body_mass_g)) +
  geom_point(aes(colour = species,
                 shape = species),
             size = 3,
             alpha = 0.6) +
  theme_minimal() +
  geom_smooth(method = "lm", se=FALSE, aes(colour = species)) +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  labs(title = "Penguin size, Palmer Station LTER",
       subtitle = "Bill depth and body mass for Adelie, Chinstrap and Gentoo Penguins",
       x = "Bill depth (mm)",
       y = "Body mass (g)",
       color = "Penguin species",
       shape = "Penguin species") +
  theme(legend.position = c(0.8, 0.8),
        legend.background = element_rect(fill = "white", colour = NA),
        plot.title.position = "plot",
        plot.caption = element_text(hjust = 0, face= "italic"),
        plot.caption.position = "plot")

mass_bill_depth

This shows that there is very little difference between Adelie and Chinstrap, but Gentoo is markedly different. All three species have a positive relationship (as bill depth increases, so does body mass), which is not what the numerical correlation coefficient of -0.472 ould have us believe.

Penguin body mass and flipper size, faceted by sex

ggplot(penguins, aes(x = flipper_length_mm,
                            y = body_mass_g)) +
  geom_point(aes(colour = sex)) +
  theme_minimal() +
  scale_color_manual(values = c("darkorange","cyan4"), na.translate = FALSE) +
  labs(title = "Penguin flipper and body mass",
       subtitle = "Dimensions for male and female Adelie, Chinstrap and Gentoo Penguins at Palmer Station LTER",
       x = "Flipper length (mm)",
       y = "Body mass (g)",
       color = "Penguin sex") +
  theme(legend.position = "bottom",
        legend.background = element_rect(fill = "white", color = NA),
        plot.title.position = "plot",
        plot.caption = element_text(hjust = 0, face= "italic"),
        plot.caption.position = "plot") +
  facet_wrap(~species)

On average, male penguins seem to be heavier than female ones, and this is consistent along all three species.

Simpson’s Paradox: Penguin bill length vs bill depth

In the original scatterplot matrix without species, the lowest correlation coefficient of -0.235 was between bill length and bill depth. The following is just the scatterplot with the line of best fit added.

bill_no_species <- ggplot(data = penguins,
                         aes(x = bill_length_mm,
                             y = bill_depth_mm)) +
  geom_point() +
  theme_minimal() +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  labs(title = "Penguin bill dimensions (omit species)",
       subtitle = "Palmer Station LTER",
       x = "Bill length (mm)",
       y = "Bill depth (mm)") +
  theme(plot.title.position = "plot",
        plot.caption = element_text(hjust = 0, face= "italic"),
        plot.caption.position = "plot") +
  geom_smooth(method = "lm", se = FALSE, color = "blue")

bill_no_species

However, if we plot the same scatterplot colouring points by species, we get a completely different story.

bill_len_dep <- ggplot(data = penguins,
                         aes(x = bill_length_mm,
                             y = bill_depth_mm,
                             group = species)) +
  geom_point(aes(colour = species,
                 shape = species),
             size = 3,
             alpha = 0.8) +
  geom_smooth(method = "lm", se = FALSE, aes(colour = species)) +
  theme_minimal() +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  labs(title = "Penguin bill dimensions",
       subtitle = "Bill length and depth for Adelie, Chinstrap and Gentoo Penguins at Palmer Station LTER",
       x = "Bill length (mm)",
       y = "Bill depth (mm)",
       color = "Penguin species",
       shape = "Penguin species") +
  theme(legend.position = c(0.85, 0.10),
        legend.background = element_rect(fill = "white", color = NA),
        plot.title.position = "plot",
        plot.caption = element_text(hjust = 0, face= "italic"),
        plot.caption.position = "plot")

bill_len_dep

Simpson’s paradox

Is Y heavily skewed? Does it need to be transformed?

THe variable we are interested to explain, body mass, does not appear to be heavily skewed so there is no need for any transformation.

Acknowledgements

This page is adapted from the Palmer Palmer Penguins package Vignette

Exploratory Data Analysis for Modelling

Tue, 28 Jul 2020 00:00:00 +0000

This great artwork was made by [@allison_horst](https://twitter.com/allison_horst)

and if you are a Happy Feet fan, these are the penguins we have data on

Penguin Bill dimensions

The culmen is the upper ridge of a bird’s bill. In the simplified penguins data, culmen length and depth are renamed as variables bill_length_mm and bill_depth_mm to be more intuitive.

For this penguin data, the culmen (bill) length and depth are measured as shown below:

Exploratory Data Analysis (EDA).

The variable of interest is penguin body mass. We want to see whether body mass is related to any of the other variables included in the dataframe. So let us start by looking at the data

library(palmerpenguins)
glimpse(penguins)

## Rows: 344
## Columns: 8
## $ species           <fct> Adelie, Adelie, Adelie, Adelie, Adelie, Adelie, Adel~
## $ island            <fct> Torgersen, Torgersen, Torgersen, Torgersen, Torgerse~
## $ bill_length_mm    <dbl> 39.1, 39.5, 40.3, NA, 36.7, 39.3, 38.9, 39.2, 34.1, ~
## $ bill_depth_mm     <dbl> 18.7, 17.4, 18.0, NA, 19.3, 20.6, 17.8, 19.6, 18.1, ~
## $ flipper_length_mm <int> 181, 186, 195, NA, 193, 190, 181, 195, 193, 190, 186~
## $ body_mass_g       <int> 3750, 3800, 3250, NA, 3450, 3650, 3625, 4675, 3475, ~
## $ sex               <fct> male, female, female, NA, female, male, female, male~
## $ year              <int> 2007, 2007, 2007, 2007, 2007, 2007, 2007, 2007, 2007~

skimr::skim(penguins)

Table 1: Data summary
Name	penguins
Number of rows	344
Number of columns	8
_______________________
Column type frequency:
factor	3
numeric	5
________________________
Group variables	None

Variable type: factor

skim_variable	n_missing	complete_rate	ordered	n_unique	top_counts
species	0	1.00	FALSE	3	Ade: 152, Gen: 124, Chi: 68
island	0	1.00	FALSE	3	Bis: 168, Dre: 124, Tor: 52
sex	11	0.97	FALSE	2	mal: 168, fem: 165

Variable type: numeric

skim_variable	n_missing	complete_rate	mean	sd	p0	p25	p50	p75	p100	hist
bill_length_mm	2	0.99	43.92	5.46	32.1	39.23	44.45	48.5	59.6	▃▇▇▆▁
bill_depth_mm	2	0.99	17.15	1.97	13.1	15.60	17.30	18.7	21.5	▅▅▇▇▂
flipper_length_mm	2	0.99	200.92	14.06	172.0	190.00	197.00	213.0	231.0	▂▇▃▅▂
body_mass_g	2	0.99	4201.75	801.95	2700.0	3550.00	4050.00	4750.0	6300.0	▃▇▆▃▂
year	0	1.00	2008.03	0.82	2007.0	2007.00	2008.00	2009.0	2009.0	▇▁▇▁▇

`GGally::ggpairs()` to get scatter plot + correlation matrix

GGally::ggpairs() is a useful tool for exploring distributions and correlations.

ggpairs1 <- penguins %>% 
  na.omit() %>% 
  select( body_mass_g, flipper_length_mm, bill_length_mm, bill_depth_mm, species, sex, island) %>% 
  rename(`Flipper length(mm)` = flipper_length_mm, 
         `Body mass (g)` = body_mass_g, 
         `Bill length (mm)` = bill_length_mm, 
         `Bill depth (mm)` = bill_depth_mm) %>% 
  GGally::ggpairs() +
  theme_minimal() 

ggpairs1

ggpairs() provides a lot of information, so let us spend some time deciphering this chart.

Along the diagonal we get a density plot of each variable of interest. For instance, body mass seems to be right skewed, and the rest of the variables seem bimodal.
The upper part of the matrix, shows correlation coefficients– to determine between which variables, read off the corresponding row and column header
The bottom part provides a scatterplot between any two variables which you can again determine by looking at the row/column they correspond.
For categorical variables (species, sex, island), we do not get any numerical values, but rather histograms and boxplots that show the distribution of outcomes. If we wanted to get a numerical correlation value, we would use the polycor package and polycor::hetcor() that calculates polyserial correlations between numeric and ordinal variables, and polychoric correlations between ordinal variables, but this is outside the scope of this class.

Dummy variables

ggpairs2 <- penguins %>% 
  na.omit() %>% 
  select( species, sex, body_mass_g, flipper_length_mm, bill_length_mm, bill_depth_mm) %>% 
  rename(`Flipper lenght(mm)` = flipper_length_mm, 
         `Body mass (g)` = body_mass_g, 
         `Bill length (mm)` = bill_length_mm, 
         `Bill depth (mm)` = bill_depth_mm) %>% 
  GGally::ggpairs(aes(colour=species, shape=species),
                  alpha = 0.4) +
  scale_colour_manual(values = c("darkorange","purple","cyan4")) +
  scale_fill_manual(values = c("darkorange","purple","cyan4")) +
  theme_minimal()
  

ggpairs2

Since this plot may be confusing, let us just concentrate on the scatterplot matrix of numerical values coloured by species.

ggpairs3 <- penguins %>% 
  na.omit() %>% 
  select( species, sex, body_mass_g, flipper_length_mm, bill_length_mm, bill_depth_mm) %>% 
  rename(`Flipper lenght(mm)` = flipper_length_mm, 
         `Body mass (g)` = body_mass_g, 
         `Bill length (mm)` = bill_length_mm, 
         `Bill depth (mm)` = bill_depth_mm) %>% 
  GGally::ggpairs(aes(colour=species, shape=species),
                  alpha = 0.4,
                  columns = c(3:6)) + 
  scale_colour_manual(values = c("darkorange","purple","cyan4")) +
  scale_fill_manual(values = c("darkorange","purple","cyan4")) +
  theme_minimal()
  

ggpairs3

Modelling considerations of numerical variables

Let us examine the relationship between body mass and flipper length, and body mass and bill depth.

Penguin body mass vs flipper length

mass_flipper <- ggplot(data = penguins,
                       aes(x = flipper_length_mm,
                           y = body_mass_g)) +
  geom_point(aes(colour = species,
                 shape = species),
             size = 3,
             alpha = 0.6) +
  theme_minimal() +
  geom_smooth(method = "lm", se=FALSE, aes(colour = species)) +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  labs(title = "Penguin size, Palmer Station LTER",
       subtitle = "Flipper length and body mass for Adelie, Chinstrap and Gentoo Penguins",
       x = "Flipper length (mm)",
       y = "Body mass (g)",
       color = "Penguin species",
       shape = "Penguin species") +
  theme(legend.position = c(0.2, 0.7),
        legend.background = element_rect(fill = "white", colour = NA),
        plot.title.position = "plot",
        plot.caption = element_text(hjust = 0, face= "italic"),
        plot.caption.position = "plot")

mass_flipper

Penguin body mass vs bill depth

mass_bill_depth <- ggplot(data = penguins,
                       aes(x = bill_depth_mm,
                           y = body_mass_g)) +
  geom_point(aes(colour = species,
                 shape = species),
             size = 3,
             alpha = 0.6) +
  theme_minimal() +
  geom_smooth(method = "lm", se=FALSE, aes(colour = species)) +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  labs(title = "Penguin size, Palmer Station LTER",
       subtitle = "Bill depth and body mass for Adelie, Chinstrap and Gentoo Penguins",
       x = "Bill depth (mm)",
       y = "Body mass (g)",
       color = "Penguin species",
       shape = "Penguin species") +
  theme(legend.position = c(0.8, 0.8),
        legend.background = element_rect(fill = "white", colour = NA),
        plot.title.position = "plot",
        plot.caption = element_text(hjust = 0, face= "italic"),
        plot.caption.position = "plot")

mass_bill_depth

Penguin body mass and flipper size, faceted by sex

ggplot(penguins, aes(x = flipper_length_mm,
                            y = body_mass_g)) +
  geom_point(aes(colour = sex)) +
  theme_minimal() +
  scale_color_manual(values = c("darkorange","cyan4"), na.translate = FALSE) +
  labs(title = "Penguin flipper and body mass",
       subtitle = "Dimensions for male and female Adelie, Chinstrap and Gentoo Penguins at Palmer Station LTER",
       x = "Flipper length (mm)",
       y = "Body mass (g)",
       color = "Penguin sex") +
  theme(legend.position = "bottom",
        legend.background = element_rect(fill = "white", color = NA),
        plot.title.position = "plot",
        plot.caption = element_text(hjust = 0, face= "italic"),
        plot.caption.position = "plot") +
  facet_wrap(~species)

On average, male penguins seem to be heavier than female ones, and this is consistent along all three species.

Simpson’s Paradox: Penguin bill length vs bill depth

bill_no_species <- ggplot(data = penguins,
                         aes(x = bill_length_mm,
                             y = bill_depth_mm)) +
  geom_point() +
  theme_minimal() +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  labs(title = "Penguin bill dimensions (omit species)",
       subtitle = "Palmer Station LTER",
       x = "Bill length (mm)",
       y = "Bill depth (mm)") +
  theme(plot.title.position = "plot",
        plot.caption = element_text(hjust = 0, face= "italic"),
        plot.caption.position = "plot") +
  geom_smooth(method = "lm", se = FALSE, color = "blue")

bill_no_species

However, if we plot the same scatterplot colouring points by species, we get a completely different story.

bill_len_dep <- ggplot(data = penguins,
                         aes(x = bill_length_mm,
                             y = bill_depth_mm,
                             group = species)) +
  geom_point(aes(colour = species,
                 shape = species),
             size = 3,
             alpha = 0.8) +
  geom_smooth(method = "lm", se = FALSE, aes(colour = species)) +
  theme_minimal() +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  labs(title = "Penguin bill dimensions",
       subtitle = "Bill length and depth for Adelie, Chinstrap and Gentoo Penguins at Palmer Station LTER",
       x = "Bill length (mm)",
       y = "Bill depth (mm)",
       color = "Penguin species",
       shape = "Penguin species") +
  theme(legend.position = c(0.85, 0.10),
        legend.background = element_rect(fill = "white", color = NA),
        plot.title.position = "plot",
        plot.caption = element_text(hjust = 0, face= "italic"),
        plot.caption.position = "plot")

bill_len_dep

Simpson’s paradox

Is Y heavily skewed? Does it need to be transformed?

The variable we are interested to explain, body mass, does not appear to be heavily skewed so there is no need for any transformation.

Acknowledgements

This page is adapted from the Palmer Palmer Penguins package Vignette

Import data

Tue, 21 Jul 2020 00:00:00 +0000

Learning Objectives
1. Load external data from a .csv file into a data frame.
2. Describe what a data frame is.
3. Use indexing to subset specific portions of data frames.
4. Describe what a factor is.
5. Reorder and rename factors.
6. Format dates.

Overview

One of the things that I found strange when I started working with R was that, unlike other software like Excel, Stata, SPSS, etc., you couldn’t just double click on an .xls, .dta, or .sav file, load the data and look at its contents. In R, we must use a command to explicitly import the data into memory.

While there are many possible data formats, we will concentrate on CSV files, namely Comma Separated Values files that are a common way to save the raw data from spreadsheets, without any of the formatting, etc. The readr R package contains functions for importing data saved as flat file documents; readr is a core member of the tidyverse and is loaded everytime you call library(tidyverse).

CSV file names end with a .csv and if you opened one inside Excel, it would look like a regular Excel file. NASA provides an estimate of global surface temperature change which allows us to calculate weather anomalies. The data is available at https://data.giss.nasa.gov/gistemp/tabledata_v3/NH.Ts+dSST.csv as a CSV file which you open inside Excel looks something like this:

However, this is what a CSV file looks like on the inside: a bunch of values separated with commas.

By the way, if you look at the data closely, you will notice that the values in the D-N (December-November) and DJF (December-January-February) columns for the year 1880 are ***. These *** denote a missing value, in the same way that R uses the NA (or not available) value.

If you’d like R to treat these *** values as missing, you will need to convert them to NAs. One way to do this is to ask read_csv() to parse *** values as NA values when it reads in the data.

Importing CSV files: `read_csv()`

Importing CSV is part of base R using the read.csv() command. However, we will use the readr package and its read_csv() command that allows us to read flat data. read_csv() is significantly (8-10 times) faster and more user friendly than the base R command, with no need to define rownames, no stringsAsFactors = TRUE.

Even though we only concentrate on CSV files, readr has several functions that allow you to import a specific flat file format.

Function	Reads
`read_csv()`	Comma separated values
`read_csv2()`	Semi-colon separate values
`read_delim()`	General delimited files
`read_fwf()`	Fixed width files
`read_log()`	Apache log files
`read_table()`	Space separated files
`read_tsv()`	Tab delimited values

Just as you can import data, readr allows you to export data and save it locally. These functions are similar to the read_ functions and each save a tibble (or data frame) in the specific file format.

Function	Writes
`write_csv()`	Comma separated values
`write_excel_csv()`	CSV that you plan to open in Excel
`write_delim()`	General delimited files
`write_file()`	A single string, written as is
`write_lines()`	A vector of strings, one string per line
`write_tsv()`	Tab delimited values

To use a write_ function, first give it the name of the data frame to save, then give it a filename to save in your working directory.

Importing CSV files directly off the internet

If the CSV exists on the internet and you have the URL address, you don’t have to download it to your local machine and then import it; you can import it directly off the web using the URL link.

url <- "https://data.giss.nasa.gov/gistemp/tabledata_v3/NH.Ts+dSST.csv"
weather <- read_csv(url, skip = 1, na = "***")

When using the read_csv() function, we added two options:

skip = 1 option is there as the real data table only starts in Row 2, so we need to skip one row.
na = "***"option informs R how missing observations in the spreadsheet are coded. As discussed earlier, it is best to specify NA values here, as otherwise some of the data may not be recognized as numeric data.

Notice that the code above saves the output to an object named weather. You must save the output of read_csv() to an object if you wish to use it later; otherwise, read_csv() will just print the contents of the data set at the command line.

Also, the assignment statement doesn’t produce any output for weather because assignments don’t display anything. If we want to check that our data has been loaded, we can see glimpse the structure of the dataframe using glimpse() or see its contents by just typing its name: weather.

glimpse(weather) shows us the number of observations and variables, and then, for each variable, shows the variable type; in our case, all of the variables are dbl or double, namely numeric variables.
weather, just invoking the name of the dataframe, shows the contents of the dataframe in the tabular form it is saved in

glimpse(weather)

## Rows: 140
## Columns: 19
## $ Year  <dbl> 1880, 1881, 1882, 1883, 1884, 1885, 1886, 1887, 1888, 1889, 1...
## $ Jan   <dbl> -0.54, -0.19, 0.22, -0.59, -0.23, -1.00, -0.68, -1.07, -0.53,...
## $ Feb   <dbl> -0.38, -0.25, 0.22, -0.67, -0.11, -0.37, -0.68, -0.58, -0.59,...
## $ Mar   <dbl> -0.26, 0.02, 0.00, -0.16, -0.65, -0.21, -0.57, -0.36, -0.58, ...
## $ Apr   <dbl> -0.37, -0.02, -0.36, -0.27, -0.62, -0.53, -0.34, -0.42, -0.24...
## $ May   <dbl> -0.11, -0.06, -0.32, -0.32, -0.42, -0.55, -0.34, -0.27, -0.16...
## $ Jun   <dbl> -0.22, -0.36, -0.38, -0.26, -0.52, -0.47, -0.43, -0.20, -0.04...
## $ Jul   <dbl> -0.23, -0.06, -0.37, -0.09, -0.48, -0.39, -0.20, -0.23, 0.04,...
## $ Aug   <dbl> -0.24, -0.03, -0.14, -0.26, -0.50, -0.44, -0.47, -0.52, -0.19...
## $ Sep   <dbl> -0.26, -0.23, -0.17, -0.33, -0.45, -0.32, -0.34, -0.17, -0.12...
## $ Oct   <dbl> -0.32, -0.40, -0.53, -0.21, -0.41, -0.30, -0.31, -0.40, 0.04,...
## $ Nov   <dbl> -0.37, -0.42, -0.32, -0.40, -0.48, -0.28, -0.45, -0.19, -0.03...
## $ Dec   <dbl> -0.48, -0.28, -0.42, -0.25, -0.40, 0.00, -0.17, -0.43, -0.26,...
## $ `J-D` <dbl> -0.32, -0.19, -0.21, -0.32, -0.44, -0.40, -0.42, -0.40, -0.22...
## $ `D-N` <dbl> NA, -0.21, -0.20, -0.33, -0.43, -0.44, -0.40, -0.38, -0.24, -...
## $ DJF   <dbl> NA, -0.31, 0.06, -0.56, -0.20, -0.59, -0.45, -0.61, -0.52, -0...
## $ MAM   <dbl> -0.24, -0.02, -0.22, -0.25, -0.56, -0.43, -0.42, -0.35, -0.33...
## $ JJA   <dbl> -0.23, -0.15, -0.30, -0.20, -0.50, -0.44, -0.37, -0.32, -0.06...
## $ SON   <dbl> -0.32, -0.35, -0.34, -0.32, -0.44, -0.30, -0.37, -0.25, -0.04...

weather

## # A tibble: 140 x 19
##     Year   Jan    Feb    Mar   Apr   May   Jun   Jul   Aug   Sep   Oct   Nov
##    <dbl> <dbl>  <dbl>  <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
##  1  1880 -0.54 -0.38  -0.26  -0.37 -0.11 -0.22 -0.23 -0.24 -0.26 -0.32 -0.37
##  2  1881 -0.19 -0.25   0.02  -0.02 -0.06 -0.36 -0.06 -0.03 -0.23 -0.4  -0.42
##  3  1882  0.22  0.22   0     -0.36 -0.32 -0.38 -0.37 -0.14 -0.17 -0.53 -0.32
##  4  1883 -0.59 -0.67  -0.16  -0.27 -0.32 -0.26 -0.09 -0.26 -0.33 -0.21 -0.4 
##  5  1884 -0.23 -0.11  -0.65  -0.62 -0.42 -0.52 -0.48 -0.5  -0.45 -0.41 -0.48
##  6  1885 -1    -0.37  -0.21  -0.53 -0.55 -0.47 -0.39 -0.44 -0.32 -0.3  -0.28
##  7  1886 -0.68 -0.68  -0.570 -0.34 -0.34 -0.43 -0.2  -0.47 -0.34 -0.31 -0.45
##  8  1887 -1.07 -0.580 -0.36  -0.42 -0.27 -0.2  -0.23 -0.52 -0.17 -0.4  -0.19
##  9  1888 -0.53 -0.59  -0.580 -0.24 -0.16 -0.04  0.04 -0.19 -0.12  0.04 -0.03
## 10  1889 -0.31  0.35   0.07   0.15 -0.05 -0.12 -0.1  -0.16 -0.26 -0.34 -0.61
## # ... with 130 more rows, and 7 more variables: Dec <dbl>, `J-D` <dbl>,
## #   `D-N` <dbl>, DJF <dbl>, MAM <dbl>, JJA <dbl>, SON <dbl>

When you use read_csv(), read_csv() tries to match each column of input to one of the basic data types in R. In our case, read_csv() misidentified the contents of the Year column to a real number, rather than an integer. You can correct this with R’s as.integer() function, or you can read the data in again, this time instructing read_csv() to parse the column as integers.

To do this:

add the argument col_types to read_csv()
set equal the col_types arguent to a list.
add a named element to the list for each column you would like to manually parse; in our case, we want to make column ‘Year’ an integer.

weather <- read_csv(url, skip = 1, na = "***",
                   col_types = list(Year = col_integer()))

To complete the code, set Year equal to one of the functions below, each function instructs read_csv() to parse Year as a specific type of data.

Type function	Data Type
`col_character()`	character
`col_date()`	Date
`col_datetime()`	POSIXct (date-time)
`col_double()`	double (numeric)
`col_factor()`	factor
`col_guess()`	let readr geuss (default)
`col_integer()`	integer
`col_logical()`	logical
`col_number()`	numbers mixed with non-number characters
`col_numeric()`	double or integer
`col_skip()`	do not read this column
`col_time()`	time

Importing CSV files saved locally

If you want to read a CSV file you have saved locally in your computer, you must let RStudio know which folder the file lives in; in more technical terms, you have to set the Working Directory. You can determine the location of your working directory by running getwd(). You can change the location of your working directory by going to Session > Set Working Directory in the RStudio IDE menu bar or use the Ctrl+Shift+H shortcut in Windows, Cmd+Shift+H in Mac to browse for the folder where the file resides.

Need for speed: Enter `data.table::fread()`

If you have a file which is fairly large, the fread() function from the data.table package, data.table::fread(), can make your life easier. You use it in a similar way to read_csv(), but it is faster. The following table compares how long it takes to read 1.1 million rows from CDC’s COVID-19 Case Surveillance Public Use Data. We downloaded the CSV locally and then read it with base R read.csv(), readr::read_csv(), and data.table::fread()

library(microbenchmark)
mbm = microbenchmark(
  baseR =   read.csv("COVID-19_Case_Surveillance_Public_Use_Data.csv"),
  readr =   read_csv("COVID-19_Case_Surveillance_Public_Use_Data.csv"),
  data.table =   fread("COVID-19_Case_Surveillance_Public_Use_Data.csv"),
  times=10
)
mbm

Unit: seconds
expr	min	lq	mean	median	uq	max	neval	cld
baseR	6.314878	7.598240	9.930234	9.556545	11.353675	15.014476	10	c
readr	4.450299	5.270310	6.300699	6.317381	7.489372	8.341003	10	b
data.table	1.080079	1.950533	2.240901	2.251069	2.729169	3.030911	10	a

The diffenrence in speed is remarkarble; base R is the slowest with an avearge loading time of 9.93 seconds. read_csv() yields an average of 6.3 seconds, and data.table:fread() reduces loading time to 2.24 seconds.

Other data formats

The readr package provides efficient functions for reading and saving common flat file data formats. For other data types, consider using :

Package	Reads
readxl	Excel files (.xls, .xlsx)
haven	SPSS, Stata, and SAS files
jsonlite	json
xml2	xml
httr	web API’s
rvest	web pages (web scraping)
DBI	databases
sparklyr	data loaded into spark

`rio`: a swiss-army knife for data input-output

A really neat package to handle importing- exporing data is rio whose authors call it A Swiss-Army knife for data input-output. It works by determining the data structure from the file extension, uses reasonable defaults for data import and export (e.g., ‘stringsAsFactors=FALSE’), supports web-based import (including from SSL/HTTPS). It also has a useful function, ‘convert()’, that provides a simple method for converting between file types. You can read more about rio here

Never work directly on the raw data

In 2012 Cecilia Giménez, an 83-year-old widow and amateur painter, attempted to restore a century-old fresco of Jesus crowned with thorns in her local church in Borja, Spain. The restoration didn’t go very well, but, surprisingly, the botched restoration of Jesus fresco miraculously saved the Spanish Town.

As a most important rule, please do not work on the raw data; it’s unlikely you will have Cecilia Giménez’s good fortune to become (in)famous for your not-so-brilliant work. Make sure you import the data in R, leave the raw data aside, and if you make any changes tidying and wrangling your data, save it using write_csv() with a different file name.

Importing data

Tue, 21 Jul 2020 00:00:00 +0000

Learning Objectives
1. Load external data from a .csv file into a data frame.
2. Describe what a data frame is.
3. Use indexing to subset specific portions of data frames.
4. Describe what a factor is.
5. Reorder and rename factors.
6. Format dates.

Overview

However, this is what a CSV file looks like on the inside: a bunch of values separated with commas.

Importing CSV files: `read_csv()`

Even though we only concentrate on CSV files, readr has several functions that allow you to import a specific flat file format.

Function	Reads
`read_csv()`	Comma separated values
`read_csv2()`	Semi-colon separate values
`read_delim()`	General delimited files
`read_fwf()`	Fixed width files
`read_log()`	Apache log files
`read_table()`	Space separated files
`read_tsv()`	Tab delimited values

Function	Writes
`write_csv()`	Comma separated values
`write_excel_csv()`	CSV that you plan to open in Excel
`write_delim()`	General delimited files
`write_file()`	A single string, written as is
`write_lines()`	A vector of strings, one string per line
`write_tsv()`	Tab delimited values

To use a write_ function, first give it the name of the data frame to save, then give it a filename to save in your working directory.

Importing CSV files directly off the internet

If the CSV exists on the internet and you have the URL address, you don’t have to download it to your local machine and then import it; you can import it directly off the web using the URL link.

url <- "https://data.giss.nasa.gov/gistemp/tabledata_v3/NH.Ts+dSST.csv"
weather <- read_csv(url, skip = 1, na = "***")

When using the read_csv() function, we added two options:

skip = 1 option is there as the real data table only starts in Row 2, so we need to skip one row.
na = "***"option informs R how missing observations in the spreadsheet are coded. As discussed earlier, it is best to specify NA values here, as otherwise some of the data may not be recognized as numeric data.

glimpse(weather) shows us the number of observations and variables, and then, for each variable, shows the variable type; in our case, all of the variables are dbl or double, namely numeric variables.
weather, just invoking the name of the dataframe, shows the contents of the dataframe in the tabular form it is saved in

glimpse(weather)

## Rows: 140
## Columns: 19
## $ Year  <dbl> 1880, 1881, 1882, 1883, 1884, 1885, 1886, 1887, 1888, 1889, 1...
## $ Jan   <dbl> -0.54, -0.19, 0.22, -0.59, -0.23, -1.00, -0.68, -1.07, -0.53,...
## $ Feb   <dbl> -0.38, -0.25, 0.22, -0.67, -0.11, -0.37, -0.68, -0.58, -0.59,...
## $ Mar   <dbl> -0.26, 0.02, 0.00, -0.16, -0.65, -0.21, -0.57, -0.36, -0.58, ...
## $ Apr   <dbl> -0.37, -0.02, -0.36, -0.27, -0.62, -0.53, -0.34, -0.42, -0.24...
## $ May   <dbl> -0.11, -0.06, -0.32, -0.32, -0.42, -0.55, -0.34, -0.27, -0.16...
## $ Jun   <dbl> -0.22, -0.36, -0.38, -0.26, -0.52, -0.47, -0.43, -0.20, -0.04...
## $ Jul   <dbl> -0.23, -0.06, -0.37, -0.09, -0.48, -0.39, -0.20, -0.23, 0.04,...
## $ Aug   <dbl> -0.24, -0.03, -0.14, -0.26, -0.50, -0.44, -0.47, -0.52, -0.19...
## $ Sep   <dbl> -0.26, -0.23, -0.17, -0.33, -0.45, -0.32, -0.34, -0.17, -0.12...
## $ Oct   <dbl> -0.32, -0.40, -0.53, -0.21, -0.41, -0.30, -0.31, -0.40, 0.04,...
## $ Nov   <dbl> -0.37, -0.42, -0.32, -0.40, -0.48, -0.28, -0.45, -0.19, -0.03...
## $ Dec   <dbl> -0.48, -0.28, -0.42, -0.25, -0.40, 0.00, -0.17, -0.43, -0.26,...
## $ `J-D` <dbl> -0.32, -0.19, -0.21, -0.32, -0.44, -0.40, -0.42, -0.40, -0.22...
## $ `D-N` <dbl> NA, -0.21, -0.20, -0.33, -0.43, -0.44, -0.40, -0.38, -0.24, -...
## $ DJF   <dbl> NA, -0.31, 0.06, -0.56, -0.20, -0.59, -0.45, -0.61, -0.52, -0...
## $ MAM   <dbl> -0.24, -0.02, -0.22, -0.25, -0.56, -0.43, -0.42, -0.35, -0.33...
## $ JJA   <dbl> -0.23, -0.15, -0.30, -0.20, -0.50, -0.44, -0.37, -0.32, -0.06...
## $ SON   <dbl> -0.32, -0.35, -0.34, -0.32, -0.44, -0.30, -0.37, -0.25, -0.04...

weather

## # A tibble: 140 x 19
##     Year   Jan    Feb    Mar   Apr   May   Jun   Jul   Aug   Sep   Oct   Nov
##    <dbl> <dbl>  <dbl>  <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
##  1  1880 -0.54 -0.38  -0.26  -0.37 -0.11 -0.22 -0.23 -0.24 -0.26 -0.32 -0.37
##  2  1881 -0.19 -0.25   0.02  -0.02 -0.06 -0.36 -0.06 -0.03 -0.23 -0.4  -0.42
##  3  1882  0.22  0.22   0     -0.36 -0.32 -0.38 -0.37 -0.14 -0.17 -0.53 -0.32
##  4  1883 -0.59 -0.67  -0.16  -0.27 -0.32 -0.26 -0.09 -0.26 -0.33 -0.21 -0.4 
##  5  1884 -0.23 -0.11  -0.65  -0.62 -0.42 -0.52 -0.48 -0.5  -0.45 -0.41 -0.48
##  6  1885 -1    -0.37  -0.21  -0.53 -0.55 -0.47 -0.39 -0.44 -0.32 -0.3  -0.28
##  7  1886 -0.68 -0.68  -0.570 -0.34 -0.34 -0.43 -0.2  -0.47 -0.34 -0.31 -0.45
##  8  1887 -1.07 -0.580 -0.36  -0.42 -0.27 -0.2  -0.23 -0.52 -0.17 -0.4  -0.19
##  9  1888 -0.53 -0.59  -0.580 -0.24 -0.16 -0.04  0.04 -0.19 -0.12  0.04 -0.03
## 10  1889 -0.31  0.35   0.07   0.15 -0.05 -0.12 -0.1  -0.16 -0.26 -0.34 -0.61
## # ... with 130 more rows, and 7 more variables: Dec <dbl>, `J-D` <dbl>,
## #   `D-N` <dbl>, DJF <dbl>, MAM <dbl>, JJA <dbl>, SON <dbl>

To do this:

add the argument col_types to read_csv()
set equal the col_types arguent to a list.
add a named element to the list for each column you would like to manually parse; in our case, we want to make column ‘Year’ an integer.

weather <- read_csv(url, skip = 1, na = "***",
                   col_types = list(Year = col_integer()))

To complete the code, set Year equal to one of the functions below, each function instructs read_csv() to parse Year as a specific type of data.

Type function	Data Type
`col_character()`	character
`col_date()`	Date
`col_datetime()`	POSIXct (date-time)
`col_double()`	double (numeric)
`col_factor()`	factor
`col_guess()`	let readr geuss (default)
`col_integer()`	integer
`col_logical()`	logical
`col_number()`	numbers mixed with non-number characters
`col_numeric()`	double or integer
`col_skip()`	do not read this column
`col_time()`	time

Importing CSV files saved locally

Need for speed: Enter `data.table::fread()`

library(microbenchmark)
mbm = microbenchmark(
  baseR =   read.csv("COVID-19_Case_Surveillance_Public_Use_Data.csv"),
  readr =   read_csv("COVID-19_Case_Surveillance_Public_Use_Data.csv"),
  data.table =   fread("COVID-19_Case_Surveillance_Public_Use_Data.csv"),
  times=10
)
mbm

Unit: seconds
expr	min	lq	mean	median	uq	max	neval	cld
baseR	6.314878	7.598240	9.930234	9.556545	11.353675	15.014476	10	c
readr	4.450299	5.270310	6.300699	6.317381	7.489372	8.341003	10	b
data.table	1.080079	1.950533	2.240901	2.251069	2.729169	3.030911	10	a

Other data formats

The readr package provides efficient functions for reading and saving common flat file data formats. For other data types, consider using :

Package	Reads
readxl	Excel files (.xls, .xlsx)
haven	SPSS, Stata, and SAS files
jsonlite	json
xml2	xml
httr	web API’s
rvest	web pages (web scraping)
DBI	databases
sparklyr	data loaded into spark

`rio`: a swiss-army knife for data input-output

Never work directly on the raw data

Sampling and confidence intervals (CI)

Wed, 29 Jul 2020 00:00:00 +0000

We are back to dealing with penguins, and we want to explore body mass and flipper length across the three different species.

Summary statistics

penguins %>%
  group_by(species) %>%
  summarize(across(c( body_mass_g, flipper_length_mm),
                   mean, na.rm = TRUE)) %>% 
  kable()

species	body_mass_g	flipper_length_mm
Adelie	3701	190
Chinstrap	3733	196
Gentoo	5076	217

Boxplots

body_mass_plot <- ggplot(data = penguins, aes(y = species, x= body_mass_g)) +
  geom_boxplot(aes(color = species), width = 0.3, show.legend = FALSE) +
  geom_jitter(aes(color = species), alpha = 0.5, show.legend = FALSE, position = position_jitter(width = 0.2, seed = 0)) +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  theme_minimal() +
  labs(title = "Penguin size, Palmer Station LTER",
       subtitle = "Body mass (in grams) for Adelie, Chinstrap and Gentoo Penguins",
       y = "Species",
       x = "Body mass (grams)")

body_mass_plot

flipper_plot <- ggplot(data = penguins, aes(y = species, x = flipper_length_mm)) +
  geom_boxplot(aes(color = species), width = 0.3, show.legend = FALSE) +
  geom_jitter(aes(color = species), alpha = 0.5, show.legend = FALSE, position = position_jitter(width = 0.2, seed = 0)) +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  theme_minimal() +
  labs(title = "Penguin size, Palmer Station LTER",
       subtitle = "Flipper length for Adelie, Chinstrap and Gentoo Penguins",
       y = "Species",
       x = "Flipper length (mm)")



flipper_plot

Confidence Intervals (CI)

CIs for body mass

formula_ci_body_mass <- penguins %>%
  group_by(species) %>%
  summarise( mean_body_mass = mean(body_mass_g, na.rm = TRUE), 
             sd_mass = sd(body_mass_g, na.rm = TRUE), 
             count = n(), 
             
             # get t-critical value with (n-1) degrees of freedom
             t_critical = qt(0.975, count-1),
             se = sd_mass/sqrt(count),
             margin_of_error = t_critical * se,
             ci_low = mean_body_mass - margin_of_error,
             ci_high = mean_body_mass + margin_of_error
  )


formula_ci_body_mass %>% 
  kable()

species	mean_body_mass	sd_mass	count	t_critical	se	margin_of_error	ci_low	ci_high
Adelie	3701	459	152	1.98	37.2	73.5	3627	3774
Chinstrap	3733	384	68	2.00	46.6	93.0	3640	3826
Gentoo	5076	504	124	1.98	45.3	89.6	4986	5166

#visualise  CIs for all species 
ggplot(formula_ci_body_mass, 
       aes(x=reorder(species, mean_body_mass), 
           y=mean_body_mass, 
           colour=species)) +
  geom_point() +
  scale_colour_manual(values = c("darkorange","purple","cyan4")) +
  geom_errorbar(width=.2, aes(ymin=ci_low, ymax=ci_high)) + 
  labs(x=" ",
       y= "Mean body mass (grams)", 
       title="Which species has the highest mean weight?") + 
  theme_minimal()+
  coord_flip()+
  theme(legend.position = "none")+
  NULL

# we will draw a violin plot and then use position="jitter" or geom_jitter() 
# to see how spread out the actual points are

ggplot(data = penguins, aes(y = species, x= body_mass_g)) +
  geom_violin(aes(colour = species), width = 0.3, show.legend = FALSE) +
  geom_jitter(aes(colour = species), alpha = 0.5, show.legend = FALSE, position = position_jitter(width = 0.2, seed = 0)) +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  
 # superimpose  the mean as a big orange dot
  geom_point(data = formula_ci_body_mass,
             aes(x=mean_body_mass, y = species), colour = "orange", size = 8)+

  
  theme_minimal() +
  labs(title = "Penguin size, Palmer Station LTER",
       subtitle = "Body mass (in grams) for Adelie, Chinstrap and Gentoo Penguins",
       y = "Species",
       x = "Body mass (grams)")

CIs for flipper length

formula_ci_flipper_length <- penguins %>%
  group_by(species) %>%
  summarise( mean_flipper_length = mean(flipper_length_mm, na.rm = TRUE), 
             sd_flipper_length = sd(flipper_length_mm, na.rm = TRUE), 
             count = n(), 
             
             # get t-critical value with (n-1) degrees of freedom
             t_critical = qt(0.975, count-1),
             se = sd_flipper_length/sqrt(count),
             margin_of_error = t_critical * se,
             ci_low = mean_flipper_length - margin_of_error,
             ci_high = mean_flipper_length + margin_of_error
  )


formula_ci_flipper_length %>% 
  kable()

species	mean_flipper_length	sd_flipper_length	count	t_critical	se	margin_of_error	ci_low	ci_high
Adelie	190	6.54	152	1.98	0.530	1.05	189	191
Chinstrap	196	7.13	68	2.00	0.865	1.73	194	198
Gentoo	217	6.49	124	1.98	0.582	1.15	216	218

#visualise  CIs for all species 
ggplot(formula_ci_flipper_length, 
       aes(x=reorder(species, mean_flipper_length), 
           y=mean_flipper_length, 
           colour=species)) +
  geom_point() +
  scale_colour_manual(values = c("darkorange","purple","cyan4")) +
  geom_errorbar(width=.2, aes(ymin=ci_low, ymax=ci_high)) + 
  labs(x=" ",
       y= "Mean flipper length (mm)", 
       title="Which species has the longest mean flipper?") + 
  theme_minimal()+
  coord_flip()+
  theme(legend.position = "none")+
  NULL

# we will draw a violin plot and then use position="jitter" or geom_jitter() 
# to see how spread out the actual points are

ggplot(data = penguins, aes(y = species, x= flipper_length_mm)) +
  geom_violin(aes(colour = species), width = 0.3, show.legend = FALSE) +
  geom_jitter(aes(colour = species), alpha = 0.5, show.legend = FALSE, position = position_jitter(width = 0.2, seed = 0)) +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  
 # superimpose  the mean as a big orange dot
  geom_point(data = formula_ci_flipper_length,
             aes(x=mean_flipper_length, y = species), colour = "orange", size = 8)+

  theme_minimal() +
  labs(title = "Penguin size, Palmer Station LTER",
       subtitle = "Flipper length (in mm) for Adelie, Chinstrap and Gentoo Penguins",
       y = "Species",
       x = "Flipper length (mm)")

Remember in the penguins data, we saw that Gentoo penguins are very unlike the other two; however, what if we wanted to compare Adelie and Chinstrap both in terms of body mass and flipper length? By looking at the confidence intervals, we already have an indication as to whether there is a difference or not. We will use a t-Test to check if the group means are different.

Briefly, a t-Test should be used when we want to assess whether the mean between two groups are similar or not. The null hypothesis for a t-test is that the two means are equal, and the alternative is that they are not.

t-Test assuming unequal variance

R’s built-in function for running a t-test is t.test() and by default R assumes that the variance in the two groups’ populations is not equal.

t-Test for body mass

Remember that in our plots, body mass seemed to be fairly similar. While there was variability between the two species, the two average values were fairly similar and the two Confidence Intervals ovelapped quite a bit.

When we run our hypothesis test, we must first set up the hypotheses.

Null Hypothesis, $H_0$: There is no difference in mean body mass measurements between the two species (Adelie and Chinstrap). In other words $\mu_1 = \mu_2$, or their difference is equal to 0.
Altenative Hypothesis, $H_1$: There is a difference in mean body mass measurements between the two species.

Does the data provide enough evidence to reject the null hypothesis, or could the variation be due to luck? Typically, we wanr the p-value to be less than 5%, or equivalently the t-stat to be roughly more than 2, as fairly strong evidence to reject the null hypothesis.

#select only Adelie and Chinstrap penguins
adelie_chinstrap_test_data <- penguins %>%
  filter(species %in% c("Adelie", "Chinstrap"))


test1 <- t.test(body_mass_g ~ species, 
        data = adelie_chinstrap_test_data) 

test1

## 
##  Welch Two Sample t-test
## 
## data:  body_mass_g by species
## t = -0.5, df = 152, p-value = 0.6
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -150.4   85.5
## sample estimates:
##    mean in group Adelie mean in group Chinstrap 
##                    3701                    3733

In our case, the t-test confirms what we already knew. First, the t-value is -0.5 and the p-value=0.6. Another way to look at it, is that the CI for the difference between the two means is [-150.4, 85.5] which contains zero indicating that we do not have strong evidence to reject the null hypothesis.

We can use broom:tidy() to convert these t-test results to a nice data frame.

test1_tidy <- tidy(test1) %>% 
  # Calculate difference in means, since t.test() doesn't actually do that
  mutate(estimate = estimate1 - estimate2) %>%
  # Rearrange columns
  select(starts_with("estimate"), everything())

test1_tidy

## # A tibble: 1 x 10
##   estimate estimate1 estimate2 statistic p.value parameter conf.low conf.high
##      <dbl>     <dbl>     <dbl>     <dbl>   <dbl>     <dbl>    <dbl>     <dbl>
## 1    -32.4     3701.     3733.    -0.543   0.588      152.    -150.      85.5
## # ... with 2 more variables: method <chr>, alternative <chr>

A much cleaner output! The estimated average difference in body mass is -32.4g (we subtracted Adelie - Chinstrap, 3701-3733), the t-statistic = -0.543 and the p-value = 0.588, way greater than 0.05.

t-Test for flipper length

Null Hypothesis, $H_0$: There is no difference in mean flipper length measurements between the two species (Adelie and Chinstrap).
Altenative Hypothesis, $H_1$: There is a difference in mean flipper length measurements between the two species.

test2 <- t.test(flipper_length_mm ~ species, 
        data = adelie_chinstrap_test_data) 

test2_tidy <- tidy(test2) %>% 
  # Calculate difference in means, since t.test() doesn't actually do that
  mutate(estimate = estimate1 - estimate2) %>%
  # Rearrange columns
  select(starts_with("estimate"), everything())

test2_tidy

## # A tibble: 1 x 10
##   estimate estimate1 estimate2 statistic p.value parameter conf.low conf.high
##      <dbl>     <dbl>     <dbl>     <dbl>   <dbl>     <dbl>    <dbl>     <dbl>
## 1    -5.87      190.      196.     -5.78 6.05e-8      120.    -7.88     -3.86
## # ... with 2 more variables: method <chr>, alternative <chr>

In our case, the t-test confirms what we already knew. the t-value is well above 2 and the p-value well below 0.05, indicating that we have strong evidence to reject the null hypothesis and therefore determine that there is a difference in mean flipper length.

The estimated average difference in flipper length is -5.9mm, the t-statistic is t-stat = -5.78 and the p-value = 6.05e-08 = $6.05*10^{-8} = 0.00000605$, a tiny number which is way less than 0.05.

So where does this leave us?

In terms of body mass even though we measured an average difference of 32 grams, this is not statistically significant, as its t-statistic was less than 2 and, equivalently, its p-value is >> 0.05
In terms of lfipper length, he measured average difference of 5.87mm is statistically significant as the t-statistic is 5.78 and the p-vaue <<< 0.0.5

t-Test assuming equal variance

We can run t.test() assuming the two groups have equal variance.

test1_equal_variance <- t.test(body_mass_g ~ species, 
        data = adelie_chinstrap_test_data,
        var.equal = TRUE) # assume equal variance 

test1_tidy_equal_variance <- tidy(test1_equal_variance) %>% 
  # Calculate difference in means, since t.test() doesn't actually do that
  mutate(estimate = estimate1 - estimate2) %>%
  # Rearrange columns
  select(starts_with("estimate"), everything())

test1_tidy_equal_variance

## # A tibble: 1 x 10
##   estimate estimate1 estimate2 statistic p.value parameter conf.low conf.high
##      <dbl>     <dbl>     <dbl>     <dbl>   <dbl>     <dbl>    <dbl>     <dbl>
## 1    -32.4     3701.     3733.    -0.508   0.612       217    -158.      93.4
## # ... with 2 more variables: method <chr>, alternative <chr>

test2_equal_variance <- t.test(flipper_length_mm ~ species, 
        data = adelie_chinstrap_test_data,
        var.equal = TRUE) # assume equal variance 

test2_tidy_equal_variance <- tidy(test2_equal_variance) %>% 
  # Calculate difference in means, since t.test() doesn't actually do that
  mutate(estimate = estimate1 - estimate2) %>%
  # Rearrange columns
  select(starts_with("estimate"), everything())

test2_tidy_equal_variance

## # A tibble: 1 x 10
##   estimate estimate1 estimate2 statistic p.value parameter conf.low conf.high
##      <dbl>     <dbl>     <dbl>     <dbl>   <dbl>     <dbl>    <dbl>     <dbl>
## 1    -5.87      190.      196.     -5.97 9.38e-9       217    -7.81     -3.93
## # ... with 2 more variables: method <chr>, alternative <chr>

How do we test whether the two groups have equal variance?

There are several ways to check if the two groups have equal variance. For all these tests, the null hypothesis is that the two groups have equal variances.

As in all hypothesis tests, if the p-value is less than 0.05, we can assume that they have unequal variances.

Bartlett test: Check equality of variances based on the mean

body_mass_variance <- bartlett.test(body_mass_g ~ species, 
        data = adelie_chinstrap_test_data)
body_mass_variance

## 
##  Bartlett test of homogeneity of variances
## 
## data:  body_mass_g by species
## Bartlett's K-squared = 3, df = 1, p-value = 0.1

flipper_variance <- bartlett.test(flipper_length_mm ~ species, 
        data = adelie_chinstrap_test_data)
flipper_variance

## 
##  Bartlett test of homogeneity of variances
## 
## data:  flipper_length_mm by species
## Bartlett's K-squared = 0.7, df = 1, p-value = 0.4

In both cases, since the p-value is greater than 0.05, we cannot reject the null hypothesis so we assume that the two groups have equal variances.

Levene test Check equality of variances based on the median

Levene’s test also checks for homogeneity of variance and can based either on the mean or on the median. The median is a robust statistic, as it’s not influenced by outliers as much as the mean can be.

car::leveneTest(body_mass_g ~ species, 
                center = mean,
                data = adelie_chinstrap_test_data)

## Levene's Test for Homogeneity of Variance (center = mean)
##        Df F value Pr(>F)  
## group   1    4.63  0.032 *
##       217                 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

car::leveneTest(flipper_length_mm ~ species, 
                  center = mean,
                  data = adelie_chinstrap_test_data)

## Levene's Test for Homogeneity of Variance (center = mean)
##        Df F value Pr(>F)
## group   1    0.62   0.43
##       217

car::leveneTest(body_mass_g ~ species, 
                center = median,
                data = adelie_chinstrap_test_data)

## Levene's Test for Homogeneity of Variance (center = median)
##        Df F value Pr(>F)  
## group   1    4.82  0.029 *
##       217                 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

car::leveneTest(flipper_length_mm ~ species, 
                  center = median,
                  data = adelie_chinstrap_test_data)

## Levene's Test for Homogeneity of Variance (center = median)
##        Df F value Pr(>F)
## group   1    0.62   0.43
##       217

Checking for homogeneity of variance based on the median, we can reject the null hypothesis for body mass (p-value = 0.029 < 0.05), but not for flipper length.

Fligner-Killeen test: Check homogeneity of variances based on the median, so it’s more robust to outliers

fligner.test(body_mass_g ~ species, 
             data = adelie_chinstrap_test_data)

## 
##  Fligner-Killeen test of homogeneity of variances
## 
## data:  body_mass_g by species
## Fligner-Killeen:med chi-squared = 4, df = 1, p-value = 0.04

fligner.test(flipper_length_mm ~ species, 
              data = adelie_chinstrap_test_data)

## 
##  Fligner-Killeen test of homogeneity of variances
## 
## data:  flipper_length_mm by species
## Fligner-Killeen:med chi-squared = 0.5, df = 1, p-value = 0.5

Let us summarise all the p-values from these tests

Test	Body Mass	Flipper Length
Bartlett	0.1	0.4
Levene (mean)	0.032	0.43
Levene (median)	0.029	0.43
Fligner-Killeen	0.04	0.4

In all of the Body mass tests, with the exception of the Bartlett tests, the p-value is less than 0.05. In other words, we sem to have enough evidence to conclude that the variances are different.

However, in all of the flipper length tests,, all of the p-values are > 0.0.5, which means we cannot reject the null hypothesis so we’re probably safe assuming the variances are equal and leaving var.equal = TRUE on.

Acknowledgements

This page is adapted from the Palmer Penguins package Vignette

janitor::clean_names()

Fri, 24 Jul 2020 00:00:00 +0000

`janitor` package for cleaning variable names

When we create data files, we frequently use variable names and formats that are easily readable for humans, but no so for computers.

Data scientists, according to interviews and expert estimates, spend from 50 percent to 80 percent of their time mired in this more mundane labor of collecting and preparing unruly digital data, before it can be explored for useful nuggets. – For Big-Data Scientists, ‘Janitor Work’ Is Key Hurdle to Insights The New York Times, 2014

janitor has many functions, but its core function is clean_names() which will make your life easier if you call it whenever you load data into R. The following example is taken from janitor’s documentation page

Let us read an Excel file with a roster of teachers at a fictional American high school, stored in the Microsoft Excel file dirty_data.xlsx.

Some of the variable names, e.g., First Name, Last Name, are not only capitalised, but also contain a space in the variable name. Let us read in the file and have a glimpse inside it.

roster <- readxl::read_excel(here("data", "dirty_data.xlsx"))

glimpse(roster)

## Rows: 13
## Columns: 11
## $ `First Name`        <chr> "Jason", "Jason", "Alicia", "Ada", "Desus", "Ch...
## $ `Last Name`         <chr> "Bourne", "Bourne", "Keys", "Lovelace", "Nice",...
## $ `Employee Status`   <chr> "Teacher", "Teacher", "Teacher", "Teacher", "Ad...
## $ Subject             <chr> "PE", "Drafting", "Music", NA, "Dean", "Physics...
## $ `Hire Date`         <dbl> 39690, 39690, 37118, 27515, 41431, 11037, 11037...
## $ `% Allocated`       <dbl> 0.75, 0.25, 1.00, 1.00, 1.00, 0.50, 0.50, NA, 0...
## $ `Full time?`        <chr> "Yes", "Yes", "Yes", "Yes", "Yes", "Yes", "Yes"...
## $ `do not edit! --->` <lgl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA
## $ Certification...9   <chr> "Physical ed", "Physical ed", "Instr. music", "...
## $ Certification...10  <chr> "Theater", "Theater", "Vocal music", "Computers...
## $ Certification...11  <lgl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA

We notice that if we wanted to refer to the variable for a first name (1st in the list) or percent allocated (6th in the list), we would need to refer to them as the string “First Name” and “% Allocated” respectively. To avoid this, we can use janitor::clean_names()

roster_clean <- roster %>% 
  clean_names()

glimpse(roster_clean)

## Rows: 13
## Columns: 11
## $ first_name        <chr> "Jason", "Jason", "Alicia", "Ada", "Desus", "Chie...
## $ last_name         <chr> "Bourne", "Bourne", "Keys", "Lovelace", "Nice", "...
## $ employee_status   <chr> "Teacher", "Teacher", "Teacher", "Teacher", "Admi...
## $ subject           <chr> "PE", "Drafting", "Music", NA, "Dean", "Physics",...
## $ hire_date         <dbl> 39690, 39690, 37118, 27515, 41431, 11037, 11037, ...
## $ percent_allocated <dbl> 0.75, 0.25, 1.00, 1.00, 1.00, 0.50, 0.50, NA, 0.5...
## $ full_time         <chr> "Yes", "Yes", "Yes", "Yes", "Yes", "Yes", "Yes", ...
## $ do_not_edit       <lgl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA
## $ certification_9   <chr> "Physical ed", "Physical ed", "Instr. music", "PE...
## $ certification_10  <chr> "Theater", "Theater", "Vocal music", "Computers",...
## $ certification_11  <lgl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA

Now, the variable names contain no spaces, are all lower case, and we can explicitly refer to them rather than using a string of characters– it all makes life a bit easier!

Code that works is not necessarily good code

According to Phil Karlton, there are only two hard things in Computer Science: cache invalidation and naming things. It is good practice to use meaningful names for variables and data frames, use spacing, comments, etc. Both Google and Hadley Wickham have great style guides for programming in R and the janitor package helps in creating variable names with a consistent style.

Testing for differences in mean values

Wed, 29 Jul 2020 00:00:00 +0000

We are back to dealing with penguins, and we want to explore body mass and flipper length across the three different species.

Summary statistics

penguins %>%
  group_by(species) %>%
  summarize(across(c( body_mass_g, flipper_length_mm),
                   mean, na.rm = TRUE)) %>% 
  kable()

species	body_mass_g	flipper_length_mm
Adelie	3701	190
Chinstrap	3733	196
Gentoo	5076	217

Boxplots

body_mass_plot <- ggplot(data = penguins, aes(y = species, x= body_mass_g)) +
  geom_boxplot(aes(color = species), width = 0.3, show.legend = FALSE) +
  geom_jitter(aes(color = species), alpha = 0.5, show.legend = FALSE, position = position_jitter(width = 0.2, seed = 0)) +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  theme_minimal() +
  labs(title = "Penguin size, Palmer Station LTER",
       subtitle = "Body mass (in grams) for Adelie, Chinstrap and Gentoo Penguins",
       y = "Species",
       x = "Body mass (grams)")

body_mass_plot

flipper_plot <- ggplot(data = penguins, aes(y = species, x = flipper_length_mm)) +
  geom_boxplot(aes(color = species), width = 0.3, show.legend = FALSE) +
  geom_jitter(aes(color = species), alpha = 0.5, show.legend = FALSE, position = position_jitter(width = 0.2, seed = 0)) +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  theme_minimal() +
  labs(title = "Penguin size, Palmer Station LTER",
       subtitle = "Flipper length for Adelie, Chinstrap and Gentoo Penguins",
       y = "Species",
       x = "Flipper length (mm)")



flipper_plot

Confidence Intervals (CI)

CIs for body mass

formula_ci_body_mass <- penguins %>%
  group_by(species) %>%
  summarise( mean_body_mass = mean(body_mass_g, na.rm = TRUE), 
             sd_mass = sd(body_mass_g, na.rm = TRUE), 
             count = n(), 
             
             # get t-critical value with (n-1) degrees of freedom
             t_critical = qt(0.975, count-1),
             se = sd_mass/sqrt(count),
             margin_of_error = t_critical * se,
             ci_low = mean_body_mass - margin_of_error,
             ci_high = mean_body_mass + margin_of_error
  )


formula_ci_body_mass %>% 
  kable()

species	mean_body_mass	sd_mass	count	t_critical	se	margin_of_error	ci_low	ci_high
Adelie	3701	459	152	1.98	37.2	73.5	3627	3774
Chinstrap	3733	384	68	2.00	46.6	93.0	3640	3826
Gentoo	5076	504	124	1.98	45.3	89.6	4986	5166

#visualise  CIs for all species 
ggplot(formula_ci_body_mass, 
       aes(x=reorder(species, mean_body_mass), 
           y=mean_body_mass, 
           colour=species)) +
  geom_point() +
  scale_colour_manual(values = c("darkorange","purple","cyan4")) +
  geom_errorbar(width=.2, aes(ymin=ci_low, ymax=ci_high)) + 
  labs(x=" ",
       y= "Mean body mass (grams)", 
       title="Which species has the highest mean weight?") + 
  theme_minimal()+
  coord_flip()+
  theme(legend.position = "none")+
  NULL

# we will draw a violin plot and then use position="jitter" or geom_jitter() 
# to see how spread out the actual points are

ggplot(data = penguins, aes(y = species, x= body_mass_g)) +
  geom_violin(aes(colour = species), width = 0.3, show.legend = FALSE) +
  geom_jitter(aes(colour = species), alpha = 0.5, show.legend = FALSE, position = position_jitter(width = 0.2, seed = 0)) +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  
 # superimpose  the mean as a big orange dot
  geom_point(data = formula_ci_body_mass,
             aes(x=mean_body_mass, y = species), colour = "orange", size = 8)+

  
  theme_minimal() +
  labs(title = "Penguin size, Palmer Station LTER",
       subtitle = "Body mass (in grams) for Adelie, Chinstrap and Gentoo Penguins",
       y = "Species",
       x = "Body mass (grams)")

CIs for flipper length

formula_ci_flipper_length <- penguins %>%
  group_by(species) %>%
  summarise( mean_flipper_length = mean(flipper_length_mm, na.rm = TRUE), 
             sd_flipper_length = sd(flipper_length_mm, na.rm = TRUE), 
             count = n(), 
             
             # get t-critical value with (n-1) degrees of freedom
             t_critical = qt(0.975, count-1),
             se = sd_flipper_length/sqrt(count),
             margin_of_error = t_critical * se,
             ci_low = mean_flipper_length - margin_of_error,
             ci_high = mean_flipper_length + margin_of_error
  )


formula_ci_flipper_length %>% 
  kable()

species	mean_flipper_length	sd_flipper_length	count	t_critical	se	margin_of_error	ci_low	ci_high
Adelie	190	6.54	152	1.98	0.530	1.05	189	191
Chinstrap	196	7.13	68	2.00	0.865	1.73	194	198
Gentoo	217	6.49	124	1.98	0.582	1.15	216	218

#visualise  CIs for all species 
ggplot(formula_ci_flipper_length, 
       aes(x=reorder(species, mean_flipper_length), 
           y=mean_flipper_length, 
           colour=species)) +
  geom_point() +
  scale_colour_manual(values = c("darkorange","purple","cyan4")) +
  geom_errorbar(width=.2, aes(ymin=ci_low, ymax=ci_high)) + 
  labs(x=" ",
       y= "Mean flipper length (mm)", 
       title="Which species has the longest mean flipper?") + 
  theme_minimal()+
  coord_flip()+
  theme(legend.position = "none")+
  NULL

# we will draw a violin plot and then use position="jitter" or geom_jitter() 
# to see how spread out the actual points are

ggplot(data = penguins, aes(y = species, x= flipper_length_mm)) +
  geom_violin(aes(colour = species), width = 0.3, show.legend = FALSE) +
  geom_jitter(aes(colour = species), alpha = 0.5, show.legend = FALSE, position = position_jitter(width = 0.2, seed = 0)) +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  
 # superimpose  the mean as a big orange dot
  geom_point(data = formula_ci_flipper_length,
             aes(x=mean_flipper_length, y = species), colour = "orange", size = 8)+

  theme_minimal() +
  labs(title = "Penguin size, Palmer Station LTER",
       subtitle = "Flipper length (in mm) for Adelie, Chinstrap and Gentoo Penguins",
       y = "Species",
       x = "Flipper length (mm)")

t-Test assuming unequal variance

R’s built-in function for running a t-test is t.test() and by default R assumes that the variance in the two groups’ populations is not equal.

t-Test for body mass

When we run our hypothesis test, we must first set up the hypotheses.

Null Hypothesis, $H_0$: There is no difference in mean body mass measurements between the two species (Adelie and Chinstrap). In other words $\mu_1 = \mu_2$, or their difference is equal to 0.
Altenative Hypothesis, $H_1$: There is a difference in mean body mass measurements between the two species.

#select only Adelie and Chinstrap penguins
adelie_chinstrap_test_data <- penguins %>%
  filter(species %in% c("Adelie", "Chinstrap"))


test1 <- t.test(body_mass_g ~ species, 
        data = adelie_chinstrap_test_data) 

test1

## 
##  Welch Two Sample t-test
## 
## data:  body_mass_g by species
## t = -0.5, df = 152, p-value = 0.6
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -150.4   85.5
## sample estimates:
##    mean in group Adelie mean in group Chinstrap 
##                    3701                    3733

We can use broom:tidy() to convert these t-test results to a nice data frame.

test1_tidy <- tidy(test1) %>% 
  # Calculate difference in means, since t.test() doesn't actually do that
  mutate(estimate = estimate1 - estimate2) %>%
  # Rearrange columns
  select(starts_with("estimate"), everything())

test1_tidy

## # A tibble: 1 x 10
##   estimate estimate1 estimate2 statistic p.value parameter conf.low conf.high
##      <dbl>     <dbl>     <dbl>     <dbl>   <dbl>     <dbl>    <dbl>     <dbl>
## 1    -32.4     3701.     3733.    -0.543   0.588      152.    -150.      85.5
## # ... with 2 more variables: method <chr>, alternative <chr>

A much cleaner output! The estimated average difference in body mass is -32.4g (we subtracted Adelie - Chinstrap, 3701-3733), the t-statistic = -0.543 and the p-value = 0.588, way greater than 0.05.

t-Test for flipper length

Null Hypothesis, $H_0$: There is no difference in mean flipper length measurements between the two species (Adelie and Chinstrap).
Altenative Hypothesis, $H_1$: There is a difference in mean flipper length measurements between the two species.

test2 <- t.test(flipper_length_mm ~ species, 
        data = adelie_chinstrap_test_data) 

test2_tidy <- tidy(test2) %>% 
  # Calculate difference in means, since t.test() doesn't actually do that
  mutate(estimate = estimate1 - estimate2) %>%
  # Rearrange columns
  select(starts_with("estimate"), everything())

test2_tidy

## # A tibble: 1 x 10
##   estimate estimate1 estimate2 statistic p.value parameter conf.low conf.high
##      <dbl>     <dbl>     <dbl>     <dbl>   <dbl>     <dbl>    <dbl>     <dbl>
## 1    -5.87      190.      196.     -5.78 6.05e-8      120.    -7.88     -3.86
## # ... with 2 more variables: method <chr>, alternative <chr>

The estimated average difference in flipper length is -5.9mm, the t-statistic is t-stat = -5.78 and the p-value = 6.05e-08 = $6.05*10^{-8} = 0.00000605$, a tiny number which is way less than 0.05.

So where does this leave us?

In terms of body mass even though we measured an average difference of 32 grams, this is not statistically significant, as its t-statistic was less than 2 and, equivalently, its p-value is >> 0.05
In terms of lfipper length, he measured average difference of 5.87mm is statistically significant as the t-statistic is 5.78 and the p-vaue <<< 0.0.5

t-Test assuming equal variance

We can run t.test() assuming the two groups have equal variance.

test1_equal_variance <- t.test(body_mass_g ~ species, 
        data = adelie_chinstrap_test_data,
        var.equal = TRUE) # assume equal variance 

test1_tidy_equal_variance <- tidy(test1_equal_variance) %>% 
  # Calculate difference in means, since t.test() doesn't actually do that
  mutate(estimate = estimate1 - estimate2) %>%
  # Rearrange columns
  select(starts_with("estimate"), everything())

test1_tidy_equal_variance

## # A tibble: 1 x 10
##   estimate estimate1 estimate2 statistic p.value parameter conf.low conf.high
##      <dbl>     <dbl>     <dbl>     <dbl>   <dbl>     <dbl>    <dbl>     <dbl>
## 1    -32.4     3701.     3733.    -0.508   0.612       217    -158.      93.4
## # ... with 2 more variables: method <chr>, alternative <chr>

test2_equal_variance <- t.test(flipper_length_mm ~ species, 
        data = adelie_chinstrap_test_data,
        var.equal = TRUE) # assume equal variance 

test2_tidy_equal_variance <- tidy(test2_equal_variance) %>% 
  # Calculate difference in means, since t.test() doesn't actually do that
  mutate(estimate = estimate1 - estimate2) %>%
  # Rearrange columns
  select(starts_with("estimate"), everything())

test2_tidy_equal_variance

## # A tibble: 1 x 10
##   estimate estimate1 estimate2 statistic p.value parameter conf.low conf.high
##      <dbl>     <dbl>     <dbl>     <dbl>   <dbl>     <dbl>    <dbl>     <dbl>
## 1    -5.87      190.      196.     -5.97 9.38e-9       217    -7.81     -3.93
## # ... with 2 more variables: method <chr>, alternative <chr>

How do we test whether the two groups have equal variance?

There are several ways to check if the two groups have equal variance. For all these tests, the null hypothesis is that the two groups have equal variances.

As in all hypothesis tests, if the p-value is less than 0.05, we can assume that they have unequal variances.

Bartlett test: Check equality of variances based on the mean

body_mass_variance <- bartlett.test(body_mass_g ~ species, 
        data = adelie_chinstrap_test_data)
body_mass_variance

## 
##  Bartlett test of homogeneity of variances
## 
## data:  body_mass_g by species
## Bartlett's K-squared = 3, df = 1, p-value = 0.1

flipper_variance <- bartlett.test(flipper_length_mm ~ species, 
        data = adelie_chinstrap_test_data)
flipper_variance

## 
##  Bartlett test of homogeneity of variances
## 
## data:  flipper_length_mm by species
## Bartlett's K-squared = 0.7, df = 1, p-value = 0.4

In both cases, since the p-value is greater than 0.05, we cannot reject the null hypothesis so we assume that the two groups have equal variances.

Levene test Check equality of variances based on the median

car::leveneTest(body_mass_g ~ species, 
                center = mean,
                data = adelie_chinstrap_test_data)

## Levene's Test for Homogeneity of Variance (center = mean)
##        Df F value Pr(>F)  
## group   1    4.63  0.032 *
##       217                 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

car::leveneTest(flipper_length_mm ~ species, 
                  center = mean,
                  data = adelie_chinstrap_test_data)

## Levene's Test for Homogeneity of Variance (center = mean)
##        Df F value Pr(>F)
## group   1    0.62   0.43
##       217

car::leveneTest(body_mass_g ~ species, 
                center = median,
                data = adelie_chinstrap_test_data)

## Levene's Test for Homogeneity of Variance (center = median)
##        Df F value Pr(>F)  
## group   1    4.82  0.029 *
##       217                 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

car::leveneTest(flipper_length_mm ~ species, 
                  center = median,
                  data = adelie_chinstrap_test_data)

## Levene's Test for Homogeneity of Variance (center = median)
##        Df F value Pr(>F)
## group   1    0.62   0.43
##       217

Checking for homogeneity of variance based on the median, we can reject the null hypothesis for body mass (p-value = 0.029 < 0.05), but not for flipper length.

Fligner-Killeen test: Check homogeneity of variances based on the median, so it’s more robust to outliers

fligner.test(body_mass_g ~ species, 
             data = adelie_chinstrap_test_data)

## 
##  Fligner-Killeen test of homogeneity of variances
## 
## data:  body_mass_g by species
## Fligner-Killeen:med chi-squared = 4, df = 1, p-value = 0.04

fligner.test(flipper_length_mm ~ species, 
              data = adelie_chinstrap_test_data)

## 
##  Fligner-Killeen test of homogeneity of variances
## 
## data:  flipper_length_mm by species
## Fligner-Killeen:med chi-squared = 0.5, df = 1, p-value = 0.5

Let us summarise all the p-values from these tests

Test	Body Mass	Flipper Length
Bartlett	0.1	0.4
Levene (mean)	0.032	0.43
Levene (median)	0.029	0.43
Fligner-Killeen	0.04	0.4

In all of the Body mass tests, with the exception of the Bartlett tests, the p-value is less than 0.05. In other words, we sem to have enough evidence to conclude that the variances are different.

Acknowledgements

This page is adapted from the Palmer Penguins package Vignette

Testing for differences in mean values

Wed, 29 Jul 2020 00:00:00 +0000

We are back to dealing with penguins, and we want to explore body mass and flipper length across the three different species.

Summary statistics

penguins %>%
  group_by(species) %>%
  summarize(across(c( body_mass_g, flipper_length_mm),
                   mean, na.rm = TRUE)) %>% 
  kable()

species	body_mass_g	flipper_length_mm
Adelie	3701	190
Chinstrap	3733	196
Gentoo	5076	217

Boxplots

body_mass_plot <- ggplot(data = penguins, aes(y = species, x= body_mass_g)) +
  geom_boxplot(aes(color = species), width = 0.3, show.legend = FALSE) +
  geom_jitter(aes(color = species), alpha = 0.5, show.legend = FALSE, position = position_jitter(width = 0.2, seed = 0)) +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  theme_minimal() +
  labs(title = "Penguin size, Palmer Station LTER",
       subtitle = "Body mass (in grams) for Adelie, Chinstrap and Gentoo Penguins",
       y = "Species",
       x = "Body mass (grams)")

body_mass_plot

flipper_plot <- ggplot(data = penguins, aes(y = species, x = flipper_length_mm)) +
  geom_boxplot(aes(color = species), width = 0.3, show.legend = FALSE) +
  geom_jitter(aes(color = species), alpha = 0.5, show.legend = FALSE, position = position_jitter(width = 0.2, seed = 0)) +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  theme_minimal() +
  labs(title = "Penguin size, Palmer Station LTER",
       subtitle = "Flipper length for Adelie, Chinstrap and Gentoo Penguins",
       y = "Species",
       x = "Flipper length (mm)")



flipper_plot

Confidence Intervals (CI)

CIs for body mass

formula_ci_body_mass <- penguins %>%
  group_by(species) %>%
  summarise( mean_body_mass = mean(body_mass_g, na.rm = TRUE), 
             sd_mass = sd(body_mass_g, na.rm = TRUE), 
             count = n(), 
             
             # get t-critical value with (n-1) degrees of freedom
             t_critical = qt(0.975, count-1),
             se = sd_mass/sqrt(count),
             margin_of_error = t_critical * se,
             ci_low = mean_body_mass - margin_of_error,
             ci_high = mean_body_mass + margin_of_error
  )


formula_ci_body_mass %>% 
  kable()

species	mean_body_mass	sd_mass	count	t_critical	se	margin_of_error	ci_low	ci_high
Adelie	3701	459	152	1.98	37.2	73.5	3627	3774
Chinstrap	3733	384	68	2.00	46.6	93.0	3640	3826
Gentoo	5076	504	124	1.98	45.3	89.6	4986	5166

#visualise  CIs for all species 
ggplot(formula_ci_body_mass, 
       aes(x=reorder(species, mean_body_mass), 
           y=mean_body_mass, 
           colour=species)) +
  geom_point() +
  scale_colour_manual(values = c("darkorange","purple","cyan4")) +
  geom_errorbar(width=.2, aes(ymin=ci_low, ymax=ci_high)) + 
  labs(x=" ",
       y= "Mean body mass (grams)", 
       title="Which species has the highest mean weight?") + 
  theme_minimal()+
  coord_flip()+
  theme(legend.position = "none")+
  NULL

# we will draw a violin plot and then use position="jitter" or geom_jitter() 
# to see how spread out the actual points are

ggplot(data = penguins, aes(y = species, x= body_mass_g)) +
  geom_violin(aes(colour = species), width = 0.3, show.legend = FALSE) +
  geom_jitter(aes(colour = species), alpha = 0.5, show.legend = FALSE, position = position_jitter(width = 0.2, seed = 0)) +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  
 # superimpose  the mean as a big orange dot
  geom_point(data = formula_ci_body_mass,
             aes(x=mean_body_mass, y = species), colour = "orange", size = 8)+

  
  theme_minimal() +
  labs(title = "Penguin size, Palmer Station LTER",
       subtitle = "Body mass (in grams) for Adelie, Chinstrap and Gentoo Penguins",
       y = "Species",
       x = "Body mass (grams)")

CIs for flipper length

formula_ci_flipper_length <- penguins %>%
  group_by(species) %>%
  summarise( mean_flipper_length = mean(flipper_length_mm, na.rm = TRUE), 
             sd_flipper_length = sd(flipper_length_mm, na.rm = TRUE), 
             count = n(), 
             
             # get t-critical value with (n-1) degrees of freedom
             t_critical = qt(0.975, count-1),
             se = sd_flipper_length/sqrt(count),
             margin_of_error = t_critical * se,
             ci_low = mean_flipper_length - margin_of_error,
             ci_high = mean_flipper_length + margin_of_error
  )


formula_ci_flipper_length %>% 
  kable()

species	mean_flipper_length	sd_flipper_length	count	t_critical	se	margin_of_error	ci_low	ci_high
Adelie	190	6.54	152	1.98	0.530	1.05	189	191
Chinstrap	196	7.13	68	2.00	0.865	1.73	194	198
Gentoo	217	6.49	124	1.98	0.582	1.15	216	218

#visualise  CIs for all species 
ggplot(formula_ci_flipper_length, 
       aes(x=reorder(species, mean_flipper_length), 
           y=mean_flipper_length, 
           colour=species)) +
  geom_point() +
  scale_colour_manual(values = c("darkorange","purple","cyan4")) +
  geom_errorbar(width=.2, aes(ymin=ci_low, ymax=ci_high)) + 
  labs(x=" ",
       y= "Mean flipper length (mm)", 
       title="Which species has the longest mean flipper?") + 
  theme_minimal()+
  coord_flip()+
  theme(legend.position = "none")+
  NULL

# we will draw a violin plot and then use position="jitter" or geom_jitter() 
# to see how spread out the actual points are

ggplot(data = penguins, aes(y = species, x= flipper_length_mm)) +
  geom_violin(aes(colour = species), width = 0.3, show.legend = FALSE) +
  geom_jitter(aes(colour = species), alpha = 0.5, show.legend = FALSE, position = position_jitter(width = 0.2, seed = 0)) +
  scale_color_manual(values = c("darkorange","purple","cyan4")) +
  
 # superimpose  the mean as a big orange dot
  geom_point(data = formula_ci_flipper_length,
             aes(x=mean_flipper_length, y = species), colour = "orange", size = 8)+

  theme_minimal() +
  labs(title = "Penguin size, Palmer Station LTER",
       subtitle = "Flipper length (in mm) for Adelie, Chinstrap and Gentoo Penguins",
       y = "Species",
       x = "Flipper length (mm)")

t-Test assuming unequal variance

R’s built-in function for running a t-test is t.test() and by default R assumes that the variance in the two groups’ populations is not equal.

t-Test for body mass

When we run our hypothesis test, we must first set up the hypotheses.

Null Hypothesis, $H_0$: There is no difference in mean body mass measurements between the two species (Adelie and Chinstrap). In other words $\mu_1 = \mu_2$, or their difference is equal to 0.
Altenative Hypothesis, $H_1$: There is a difference in mean body mass measurements between the two species.

#select only Adelie and Chinstrap penguins
adelie_chinstrap_test_data <- penguins %>%
  filter(species %in% c("Adelie", "Chinstrap"))


test1 <- t.test(body_mass_g ~ species, 
        data = adelie_chinstrap_test_data) 

test1

## 
##  Welch Two Sample t-test
## 
## data:  body_mass_g by species
## t = -0.5, df = 152, p-value = 0.6
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -150.4   85.5
## sample estimates:
##    mean in group Adelie mean in group Chinstrap 
##                    3701                    3733

We can use broom:tidy() to convert these t-test results to a nice data frame.

test1_tidy <- tidy(test1) %>% 
  # Calculate difference in means, since t.test() doesn't actually do that
  mutate(estimate = estimate1 - estimate2) %>%
  # Rearrange columns
  select(starts_with("estimate"), everything())

test1_tidy

## # A tibble: 1 x 10
##   estimate estimate1 estimate2 statistic p.value parameter conf.low conf.high
##      <dbl>     <dbl>     <dbl>     <dbl>   <dbl>     <dbl>    <dbl>     <dbl>
## 1    -32.4     3701.     3733.    -0.543   0.588      152.    -150.      85.5
## # ... with 2 more variables: method <chr>, alternative <chr>

A much cleaner output! The estimated average difference in body mass is -32.4g (we subtracted Adelie - Chinstrap, 3701-3733), the t-statistic = -0.543 and the p-value = 0.588, way greater than 0.05.

t-Test for flipper length

Null Hypothesis, $H_0$: There is no difference in mean flipper length measurements between the two species (Adelie and Chinstrap).
Altenative Hypothesis, $H_1$: There is a difference in mean flipper length measurements between the two species.

test2 <- t.test(flipper_length_mm ~ species, 
        data = adelie_chinstrap_test_data) 

test2_tidy <- tidy(test2) %>% 
  # Calculate difference in means, since t.test() doesn't actually do that
  mutate(estimate = estimate1 - estimate2) %>%
  # Rearrange columns
  select(starts_with("estimate"), everything())

test2_tidy

## # A tibble: 1 x 10
##   estimate estimate1 estimate2 statistic p.value parameter conf.low conf.high
##      <dbl>     <dbl>     <dbl>     <dbl>   <dbl>     <dbl>    <dbl>     <dbl>
## 1    -5.87      190.      196.     -5.78 6.05e-8      120.    -7.88     -3.86
## # ... with 2 more variables: method <chr>, alternative <chr>

The estimated average difference in flipper length is -5.9mm, the t-statistic is t-stat = -5.78 and the p-value = 6.05e-08 = $6.05*10^{-8} = 0.00000605$, a tiny number which is way less than 0.05.

So where does this leave us?

In terms of body mass even though we measured an average difference of 32 grams, this is not statistically significant, as its t-statistic was less than 2 and, equivalently, its p-value is >> 0.05
In terms of lfipper length, he measured average difference of 5.87mm is statistically significant as the t-statistic is 5.78 and the p-vaue <<< 0.0.5

t-Test assuming equal variance

We can run t.test() assuming the two groups have equal variance.

test1_equal_variance <- t.test(body_mass_g ~ species, 
        data = adelie_chinstrap_test_data,
        var.equal = TRUE) # assume equal variance 

test1_tidy_equal_variance <- tidy(test1_equal_variance) %>% 
  # Calculate difference in means, since t.test() doesn't actually do that
  mutate(estimate = estimate1 - estimate2) %>%
  # Rearrange columns
  select(starts_with("estimate"), everything())

test1_tidy_equal_variance

## # A tibble: 1 x 10
##   estimate estimate1 estimate2 statistic p.value parameter conf.low conf.high
##      <dbl>     <dbl>     <dbl>     <dbl>   <dbl>     <dbl>    <dbl>     <dbl>
## 1    -32.4     3701.     3733.    -0.508   0.612       217    -158.      93.4
## # ... with 2 more variables: method <chr>, alternative <chr>

test2_equal_variance <- t.test(flipper_length_mm ~ species, 
        data = adelie_chinstrap_test_data,
        var.equal = TRUE) # assume equal variance 

test2_tidy_equal_variance <- tidy(test2_equal_variance) %>% 
  # Calculate difference in means, since t.test() doesn't actually do that
  mutate(estimate = estimate1 - estimate2) %>%
  # Rearrange columns
  select(starts_with("estimate"), everything())

test2_tidy_equal_variance

## # A tibble: 1 x 10
##   estimate estimate1 estimate2 statistic p.value parameter conf.low conf.high
##      <dbl>     <dbl>     <dbl>     <dbl>   <dbl>     <dbl>    <dbl>     <dbl>
## 1    -5.87      190.      196.     -5.97 9.38e-9       217    -7.81     -3.93
## # ... with 2 more variables: method <chr>, alternative <chr>

How do we test whether the two groups have equal variance?

There are several ways to check if the two groups have equal variance. For all these tests, the null hypothesis is that the two groups have equal variances.

As in all hypothesis tests, if the p-value is less than 0.05, we can assume that they have unequal variances.

Bartlett test: Check equality of variances based on the mean

body_mass_variance <- bartlett.test(body_mass_g ~ species, 
        data = adelie_chinstrap_test_data)
body_mass_variance

## 
##  Bartlett test of homogeneity of variances
## 
## data:  body_mass_g by species
## Bartlett's K-squared = 3, df = 1, p-value = 0.1

flipper_variance <- bartlett.test(flipper_length_mm ~ species, 
        data = adelie_chinstrap_test_data)
flipper_variance

## 
##  Bartlett test of homogeneity of variances
## 
## data:  flipper_length_mm by species
## Bartlett's K-squared = 0.7, df = 1, p-value = 0.4

In both cases, since the p-value is greater than 0.05, we cannot reject the null hypothesis so we assume that the two groups have equal variances.

Levene test Check equality of variances based on the median

car::leveneTest(body_mass_g ~ species, 
                center = mean,
                data = adelie_chinstrap_test_data)

## Levene's Test for Homogeneity of Variance (center = mean)
##        Df F value Pr(>F)  
## group   1    4.63  0.032 *
##       217                 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

car::leveneTest(flipper_length_mm ~ species, 
                  center = mean,
                  data = adelie_chinstrap_test_data)

## Levene's Test for Homogeneity of Variance (center = mean)
##        Df F value Pr(>F)
## group   1    0.62   0.43
##       217

car::leveneTest(body_mass_g ~ species, 
                center = median,
                data = adelie_chinstrap_test_data)

## Levene's Test for Homogeneity of Variance (center = median)
##        Df F value Pr(>F)  
## group   1    4.82  0.029 *
##       217                 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

car::leveneTest(flipper_length_mm ~ species, 
                  center = median,
                  data = adelie_chinstrap_test_data)

## Levene's Test for Homogeneity of Variance (center = median)
##        Df F value Pr(>F)
## group   1    0.62   0.43
##       217

Checking for homogeneity of variance based on the median, we can reject the null hypothesis for body mass (p-value = 0.029 < 0.05), but not for flipper length.

Fligner-Killeen test: Check homogeneity of variances based on the median, so it’s more robust to outliers

fligner.test(body_mass_g ~ species, 
             data = adelie_chinstrap_test_data)

## 
##  Fligner-Killeen test of homogeneity of variances
## 
## data:  body_mass_g by species
## Fligner-Killeen:med chi-squared = 4, df = 1, p-value = 0.04

fligner.test(flipper_length_mm ~ species, 
              data = adelie_chinstrap_test_data)

## 
##  Fligner-Killeen test of homogeneity of variances
## 
## data:  flipper_length_mm by species
## Fligner-Killeen:med chi-squared = 0.5, df = 1, p-value = 0.5

Let us summarise all the p-values from these tests

Test	Body Mass	Flipper Length
Bartlett	0.1	0.4
Levene (mean)	0.032	0.43
Levene (median)	0.029	0.43
Fligner-Killeen	0.04	0.4

In all of the Body mass tests, with the exception of the Bartlett tests, the p-value is less than 0.05. In other words, we sem to have enough evidence to conclude that the variances are different.

Acknowledgements

This page is adapted from the Palmer Penguins package Vignette

Simulation-based tests

Wed, 29 Jul 2020 00:00:00 +0000

There is only one test

We can use t.test() to check whether two means are equal or not. Instead of dealing with the assumptions of the data and finding the appropriate statistical test to test for equality of variance, we can use the power of bootstrapping, permutation, and simulation to construct a null distribution, calculate confidence intervals and run ny kind of test for inference.

The beauty and power of simulation is that we can run hypothesis test not just on differences of means that we have fomuals for, but also on other parameters, like medians

According to Allen Downey, there is only one statistical test and that all statistical tests follow the same pattern:

Step 1: Calculate a sample statistic from the observed data, or $\delta^{*}$. This is the measure you care about: the difference in means, the mean, the median, the proportion, the difference in proportions, anything you want really!
Step 2: Use simulation to invent a world where $\delta$ is null. Simulate what the world would look like if there was no difference between two groups, or if there was no difference in medians or proportions, or where the average value is a specific number.
Step 3: Look at $\delta^{*}$ in the null world. Put the observed sample statistic in the null world and see if it fits well.
Step 4: Calculate the probability that $\delta^{*}$ could exist in null world. This is the p-value, or the probability that you’d see a $\delta^{*}$ at least that high in a world where there’s no difference.
Step 5: Decide if $\delta^{*}$ is statistically significant. Choose some threshold, cutoff value (e.g., 0.05) for deciding if there’s sufficient proof for rejecting the null world.

Bootstrap simulation, using `infer` package

CI for median

Inpecting data

Fri, 24 Jul 2020 00:00:00 +0000

Learning Objectives
1. Glimpse the structure of the dataframe
3. Summarize the structure of a dataframe

Overview

Once you have loaded your data set into R, you have to inspect and get a feel for the data. We are typically interested in the following:

The dimensions of the data set; how many rows (cases) and how many columns does the data frame have.
The types of variables we have; are they integer, character, logical, factor (categorical) etc.
The number of missing, or NA, values in the dataframe.
A quick look at some summary statistics

First, if we wanted to look at the dataset in a spreadsheet-style data viewer, we can just invoke View(gapminder) (View with a capital V).

While this is nice, it is not very useful, as we cannot dig deeper and see what kind of variables we have, whether there are any missing values, etc.

There are two functions that we will talk about, dplyr::glimpse() and skimr::skim().

`dplyr::glimpse()`

glimpse() is like a transposed version of print(): It first gives you the dimensions (rows and columns) and then gives us the dataframe’s columns (or variables), the variable type (, , ), and then gives us the first few values of each variable. Let us look at the outcome of

glimpse(gapminder)

## Rows: 1,704
## Columns: 6
## $ country   <fct> Afghanistan, Afghanistan, Afghanistan, Afghanistan, Afgha...
## $ continent <fct> Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asi...
## $ year      <int> 1952, 1957, 1962, 1967, 1972, 1977, 1982, 1987, 1992, 199...
## $ lifeExp   <dbl> 28.801, 30.332, 31.997, 34.020, 36.088, 38.438, 39.854, 4...
## $ pop       <int> 8425333, 9240934, 10267083, 11537966, 13079460, 14880372,...
## $ gdpPercap <dbl> 779.4453, 820.8530, 853.1007, 836.1971, 739.9811, 786.113...

We see that we have 1704 rows, or cases. We also have 6 columns, or variables and right underneath we see each column individually:

country is a categorical, or factor variable of type <fct>. and the first few cases are all Afghanistan, just because it is the first one alphabetically
continent is also a factor variable, and the first values of this categorical variable are “Asia”, "Europe’, etc.
year is an integer variable of type <int>. This is the year for which we have data for each country, between 1952 and 2007 in 5-year intervals.
lifeExp is a double precision, or real number, of type <dbl> that refers to life expectancy
pop is an integer variable of type <int> that refers to the population
gdpPercap is a double precision, or real number, of type <dbl> that refers to GDP per capita

`skimr::skim()`

While glimpse() allows us to look at the contents of the dataframe, skimr::skim() is more useful and I always use it in my workflow.

skimr::skim(gapminder)

Table 1: Data summary
Name	gapminder
Number of rows	1704
Number of columns	6
_______________________
Column type frequency:
factor	2
numeric	4
________________________
Group variables	None

Variable type: factor

skim_variable	n_missing	complete_rate	ordered	n_unique	top_counts
country	0	1	FALSE	142	Afg: 12, Alb: 12, Alg: 12, Ang: 12
continent	0	1	FALSE	5	Afr: 624, Asi: 396, Eur: 360, Ame: 300

Variable type: numeric

skim_variable	complete_rate	mean	sd	p0	p25	p50	p75	p100	hist
year	1	1979.50	17.27	1952.00	1965.75	1979.50	1993.25	2007.0	▇▅▅▅▇
lifeExp	1	59.47	12.92	23.60	48.20	60.71	70.85	82.6	▁▆▇▇▇
pop	1	29601212.32	106157896.74	60011.00	2793664.00	7023595.50	19585221.75	1318683096.0	▇▁▁▁▁
gdpPercap	1	7215.33	9857.45	241.17	1202.06	3531.85	9325.46	113523.1	▇▁▁▁▁

skimr::skim() give us a data summary with the dimensions (rows and columns) of the dataframe, and the type of columns; in this case, we have 2 factor and 2 numeric columns (variables).

For all variable types, it gives us the number of missing values (n_missing) and the complete_rate; in the gapminder data, there are no missing values, so n_mising = 0 and complete_rate = 1.

For factor variables, skim() provides information on
- whether it is an ordered factor; if false, the default ordering is alphabetical, otherwise one has to explicitly specify the order of the categories.
- the n_unique, or distinct instances of each country; in gapminder we have data on 142 distinct countries and 5 continents
- the top_counts shows the top number of instances for each factor; each country has 12 observations, but in continent Africa has 624 observations, Asia 396, etc.
For numeric variables, skim() provides summary statistics; mean, standard deviation and the 0th (min), 25th, 50, 75th and 100th (max) percentile. It also gives us a rough histogram to get an idea on the shape of the distribution (normal, skewed, uniform)

On you own

The following dataframe has data on London’d cycle hire scheme, Santander Cycles. Besides the number of bikes rented out, the dataframe also contains weather information.

bikes <- read_csv(here("data", "londonBikes.csv"))

skimr::skim(bikes)

Table 2: Data summary
Name	bikes
Number of rows	3439
Number of columns	14
_______________________
Column type frequency:
character	1
logical	4
numeric	9
________________________
Group variables	None

Variable type: character

skim_variable	n_missing	complete_rate	min	max	empty	n_unique	whitespace
date	0	1	8	8	0	3439	0

Variable type: logical

skim_variable	n_missing	complete_rate	mean	count
rain	851	0.75	0.62	TRU: 1595, FAL: 993
fog	851	0.75	0.07	FAL: 2403, TRU: 185
thunderstorm	851	0.75	0.03	FAL: 2512, TRU: 76
snow	851	0.75	0.02	FAL: 2533, TRU: 55

Variable type: numeric

skim_variable	n_missing	complete_rate	mean	sd	p0	p25	p50	p75	p100	hist
bikes_hired	0	1.00	26158.95	9135.13	3531.0	19626.00	26022.0	32759.0	73094.0	▃▇▅▁▁
season	0	1.00	2.46	1.12	1.0	1.00	2.0	3.0	4.0	▇▇▁▇▇
max_temp	1877	0.45	16.48	6.19	-1.2	11.93	16.7	20.9	36.7	▁▆▇▃▁
min_temp	1929	0.44	7.62	5.14	-8.2	3.90	7.9	11.8	20.0	▁▅▇▇▂
avg_temp	27	0.99	11.70	5.41	-4.1	7.60	11.6	15.9	28.6	▁▆▇▅▁
avg_humidity	745	0.78	74.91	10.84	37.0	67.00	76.0	83.0	100.0	▁▂▆▇▂
avg_pressure	773	0.78	1015.10	10.24	979.0	1009.00	1016.0	1022.0	1044.0	▁▂▇▆▁
avg_windspeed	745	0.78	14.01	6.10	3.0	10.00	13.0	18.0	47.0	▇▇▂▁▁
rainfall_mm	51	0.99	1.67	3.68	0.0	0.00	0.0	1.5	48.0	▇▁▁▁▁

A couple of graded learnr interactive exercices?

What kind of variable is date? What kind of variable is season?
How often does it rain in London?
What is the average annual temperature (in degrees C)?
What is the maximum rainfall?

Inpect data

Fri, 24 Jul 2020 00:00:00 +0000

Learning Objectives
1. Glimpse the structure of the dataframe
3. Summarize the structure of a dataframe

Overview

Once you have loaded your data set into R, you have to inspect and get a feel for the data. We are typically interested in the following:

The dimensions of the data set; how many rows (cases) and how many columns does the data frame have.
The types of variables we have; are they integer, character, logical, factor (categorical) etc.
The number of missing, or NA, values in the dataframe.
A quick look at some summary statistics

First, if we wanted to look at the dataset in a spreadsheet-style data viewer, we can just invoke View(gapminder) (View with a capital V).

While this is nice, it is not very useful, as we cannot dig deeper and see what kind of variables we have, whether there are any missing values, etc.

There are two functions that we will talk about, dplyr::glimpse() and skimr::skim().

`dplyr::glimpse()`

glimpse(gapminder)

## Rows: 1,704
## Columns: 6
## $ country   <fct> Afghanistan, Afghanistan, Afghanistan, Afghanistan, Afgha...
## $ continent <fct> Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asi...
## $ year      <int> 1952, 1957, 1962, 1967, 1972, 1977, 1982, 1987, 1992, 199...
## $ lifeExp   <dbl> 28.801, 30.332, 31.997, 34.020, 36.088, 38.438, 39.854, 4...
## $ pop       <int> 8425333, 9240934, 10267083, 11537966, 13079460, 14880372,...
## $ gdpPercap <dbl> 779.4453, 820.8530, 853.1007, 836.1971, 739.9811, 786.113...

We see that we have 1704 rows, or cases. We also have 6 columns, or variables and right underneath we see each column individually:

country is a categorical, or factor variable of type <fct>. and the first few cases are all Afghanistan, just because it is the first one alphabetically
continent is also a factor variable, and the first values of this categorical variable are “Asia”, "Europe’, etc.
year is an integer variable of type <int>. This is the year for which we have data for each country, between 1952 and 2007 in 5-year intervals.
lifeExp is a double precision, or real number, of type <dbl> that refers to life expectancy
pop is an integer variable of type <int> that refers to the population
gdpPercap is a double precision, or real number, of type <dbl> that refers to GDP per capita

`skimr::skim()`

While glimpse() allows us to look at the contents of the dataframe, skimr::skim() is more useful and I always use it in my workflow.

skimr::skim(gapminder)

Table 1: Data summary
Name	gapminder
Number of rows	1704
Number of columns	6
_______________________
Column type frequency:
factor	2
numeric	4
________________________
Group variables	None

Variable type: factor

skim_variable	n_missing	complete_rate	ordered	n_unique	top_counts
country	0	1	FALSE	142	Afg: 12, Alb: 12, Alg: 12, Ang: 12
continent	0	1	FALSE	5	Afr: 624, Asi: 396, Eur: 360, Ame: 300

Variable type: numeric

skim_variable	complete_rate	mean	sd	p0	p25	p50	p75	p100	hist
year	1	1979.50	17.27	1952.00	1965.75	1979.50	1993.25	2007.0	▇▅▅▅▇
lifeExp	1	59.47	12.92	23.60	48.20	60.71	70.85	82.6	▁▆▇▇▇
pop	1	29601212.32	106157896.74	60011.00	2793664.00	7023595.50	19585221.75	1318683096.0	▇▁▁▁▁
gdpPercap	1	7215.33	9857.45	241.17	1202.06	3531.85	9325.46	113523.1	▇▁▁▁▁

skimr::skim() give us a data summary with the dimensions (rows and columns) of the dataframe, and the type of columns; in this case, we have 2 factor and 2 numeric columns (variables).

For all variable types, it gives us the number of missing values (n_missing) and the complete_rate; in the gapminder data, there are no missing values, so n_mising = 0 and complete_rate = 1.

For factor variables, skim() provides information on
- whether it is an ordered factor; if false, the default ordering is alphabetical, otherwise one has to explicitly specify the order of the categories.
- the n_unique, or distinct instances of each country; in gapminder we have data on 142 distinct countries and 5 continents
- the top_counts shows the top number of instances for each factor; each country has 12 observations, but in continent Africa has 624 observations, Asia 396, etc.
For numeric variables, skim() provides summary statistics; mean, standard deviation and the 0th (min), 25th, 50, 75th and 100th (max) percentile. It also gives us a rough histogram to get an idea on the shape of the distribution (normal, skewed, uniform)

On you own

The following dataframe has data on London’d cycle hire scheme, Santander Cycles. Besides the number of bikes rented out, the dataframe also contains weather information.

bikes <- read_csv(here("data", "londonBikes.csv"))

skimr::skim(bikes)

Table 2: Data summary
Name	bikes
Number of rows	3439
Number of columns	14
_______________________
Column type frequency:
character	1
logical	4
numeric	9
________________________
Group variables	None

Variable type: character

skim_variable	n_missing	complete_rate	min	max	empty	n_unique	whitespace
date	0	1	8	8	0	3439	0

Variable type: logical

skim_variable	n_missing	complete_rate	mean	count
rain	851	0.75	0.62	TRU: 1595, FAL: 993
fog	851	0.75	0.07	FAL: 2403, TRU: 185
thunderstorm	851	0.75	0.03	FAL: 2512, TRU: 76
snow	851	0.75	0.02	FAL: 2533, TRU: 55

Variable type: numeric

skim_variable	n_missing	complete_rate	mean	sd	p0	p25	p50	p75	p100	hist
bikes_hired	0	1.00	26158.95	9135.13	3531.0	19626.00	26022.0	32759.0	73094.0	▃▇▅▁▁
season	0	1.00	2.46	1.12	1.0	1.00	2.0	3.0	4.0	▇▇▁▇▇
max_temp	1877	0.45	16.48	6.19	-1.2	11.93	16.7	20.9	36.7	▁▆▇▃▁
min_temp	1929	0.44	7.62	5.14	-8.2	3.90	7.9	11.8	20.0	▁▅▇▇▂
avg_temp	27	0.99	11.70	5.41	-4.1	7.60	11.6	15.9	28.6	▁▆▇▅▁
avg_humidity	745	0.78	74.91	10.84	37.0	67.00	76.0	83.0	100.0	▁▂▆▇▂
avg_pressure	773	0.78	1015.10	10.24	979.0	1009.00	1016.0	1022.0	1044.0	▁▂▇▆▁
avg_windspeed	745	0.78	14.01	6.10	3.0	10.00	13.0	18.0	47.0	▇▇▂▁▁
rainfall_mm	51	0.99	1.67	3.68	0.0	0.00	0.0	1.5	48.0	▇▁▁▁▁

A couple of graded learnr interactive exercices?

What kind of variable is date? What kind of variable is season?
How often does it rain in London?
What is the average annual temperature (in degrees C)?
What is the maximum rainfall?

Simulation-based tests

Wed, 29 Jul 2020 00:00:00 +0000

There is only one test

The beauty and power of simulation is that we can run hypothesis test not just on differences of means that we have fomuals for, but also on other parameters, like medians

According to Allen Downey, there is only one statistical test and that all statistical tests follow the same pattern:

Step 1: Calculate a sample statistic from the observed data, or $\delta^{*}$. This is the measure you care about: the difference in means, the mean, the median, the proportion, the difference in proportions, anything you want really!
Step 2: Use simulation to invent a world where $\delta$ is null. Simulate what the world would look like if there was no difference between two groups, or if there was no difference in medians or proportions, or where the average value is a specific number.
Step 3: Look at $\delta^{*}$ in the null world. Put the observed sample statistic in the null world and see if it fits well.
Step 4: Calculate the probability that $\delta^{*}$ could exist in null world. This is the p-value, or the probability that you’d see a $\delta^{*}$ at least that high in a world where there’s no difference.
Step 5: Decide if $\delta^{*}$ is statistically significant. Choose some threshold, cutoff value (e.g., 0.05) for deciding if there’s sufficient proof for rejecting the null world.

Bootstrap simulation, using `infer` package

CI for median

Clean data

Fri, 24 Jul 2020 00:00:00 +0000

`janitor` package for cleaning variable names

When we create data files, we frequently use variable names and formats that are easily readable for humans, but no so for computers.

Data scientists, according to interviews and expert estimates, spend from 50 percent to 80 percent of their time mired in this more mundane labor of collecting and preparing unruly digital data, before it can be explored for useful nuggets. – For Big-Data Scientists, ‘Janitor Work’ Is Key Hurdle to Insights The New York Times, 2014

Let us read an Excel file with a roster of teachers at a fictional American high school, stored in the Microsoft Excel file dirty_data.xlsx.

Some of the variable names, e.g., First Name, Last Name, are not only capitalised, but also contain a space in the variable name. Let us read in the file and have a glimpse inside it.

roster <- readxl::read_excel(here("data", "dirty_data.xlsx"))

glimpse(roster)

## Rows: 13
## Columns: 11
## $ `First Name`        <chr> "Jason", "Jason", "Alicia", "Ada", "Desus", "Ch...
## $ `Last Name`         <chr> "Bourne", "Bourne", "Keys", "Lovelace", "Nice",...
## $ `Employee Status`   <chr> "Teacher", "Teacher", "Teacher", "Teacher", "Ad...
## $ Subject             <chr> "PE", "Drafting", "Music", NA, "Dean", "Physics...
## $ `Hire Date`         <dbl> 39690, 39690, 37118, 27515, 41431, 11037, 11037...
## $ `% Allocated`       <dbl> 0.75, 0.25, 1.00, 1.00, 1.00, 0.50, 0.50, NA, 0...
## $ `Full time?`        <chr> "Yes", "Yes", "Yes", "Yes", "Yes", "Yes", "Yes"...
## $ `do not edit! --->` <lgl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA
## $ Certification...9   <chr> "Physical ed", "Physical ed", "Instr. music", "...
## $ Certification...10  <chr> "Theater", "Theater", "Vocal music", "Computers...
## $ Certification...11  <lgl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA

roster_clean <- roster %>% 
  clean_names()

glimpse(roster_clean)

## Rows: 13
## Columns: 11
## $ first_name        <chr> "Jason", "Jason", "Alicia", "Ada", "Desus", "Chie...
## $ last_name         <chr> "Bourne", "Bourne", "Keys", "Lovelace", "Nice", "...
## $ employee_status   <chr> "Teacher", "Teacher", "Teacher", "Teacher", "Admi...
## $ subject           <chr> "PE", "Drafting", "Music", NA, "Dean", "Physics",...
## $ hire_date         <dbl> 39690, 39690, 37118, 27515, 41431, 11037, 11037, ...
## $ percent_allocated <dbl> 0.75, 0.25, 1.00, 1.00, 1.00, 0.50, 0.50, NA, 0.5...
## $ full_time         <chr> "Yes", "Yes", "Yes", "Yes", "Yes", "Yes", "Yes", ...
## $ do_not_edit       <lgl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA
## $ certification_9   <chr> "Physical ed", "Physical ed", "Instr. music", "PE...
## $ certification_10  <chr> "Theater", "Theater", "Vocal music", "Computers",...
## $ certification_11  <lgl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA

Now, the variable names contain no spaces, are all lower case, and we can explicitly refer to them rather than using a string of characters– it all makes life a bit easier!

Code that works is not necessarily good code

Linear Model Fitting

Tue, 28 Jul 2020 00:00:00 +0000

We will be using the Palmer penguins data to understand body mass.

Fit a model using `lm(Y ~ X1 + X2 +..., data = dataframe)`

The function to fit a linear regression model in R is lm(Y ~ X1 + X2 +..., data = mydataframe). lm, as many other functions in R, uses the formula interface The tilde (~) can be translated as is a function of. We are saying that $Y$ is a function of $X1$, $X2$, etc., and the data for our analysis comes from mydataframe.

Back to our penguins, we want to see whether body mass is a function of flipper length. We create an object called model1 that holds the results of this linear regression model.

model1 <- lm(body_mass_g ~ flipper_length_mm, data = penguins)

Look at model output

We will be using the broom package to make modelling easier to work with. There are 3 main functions in broom:

tidy() - This is where you get most of the output you want, including coefficients and p-values
glance() - additional measures on your model, including R-squared, log likelihood, and AIC/BIC
augment() - make predictions with your model using new data

For now, we will use broom::tidy() and broom::glance() to get model results.

model1 %>% broom::tidy()

term	estimate	std.error	statistic	p.value
(Intercept)	-5.78e+03	306	-18.9	5.59e-55
flipper_length_mm	49.7	1.52	32.7	4.37e-107

model1 %>% broom::glance()

r.squared	adj.r.squared	sigma	statistic	p.value	df	logLik	AIC	BIC	deviance	df.residual	nobs
0.759	0.758	394	1.07e+03	4.37e-107	1	-2.53e+03	5.06e+03	5.07e+03	5.29e+07	340	342

Write down the equation for model1

\[ \text{body_mass_g} = \alpha + \beta_{1}(\text{flipper_length_mm}) + \epsilon \] \[ \text{body_mass_g} = -5780.83 + 49.69(\text{flipper_length_mm}) + \epsilon \]

Plot scatterplot and residuals

Add more explanatory variables

model2 <- lm(body_mass_g ~ flipper_length_mm + species , data = penguins)

model3 <- lm(body_mass_g ~ flipper_length_mm + species + sex , data = penguins)

model4 <- lm(body_mass_g ~ flipper_length_mm + species + sex + bill_length_mm, data = penguins)

model5 <- lm(body_mass_g ~ flipper_length_mm + species + sex + bill_length_mm + bill_depth_mm , data = penguins)

# Fit a model with all explanatory variables (~ .)
model6 <- lm(body_mass_g ~ . , data = penguins)

Check collinearity

With so many explanatory variables, we need to worry about colinearity, i.e., whether the explanatory variables (all of the $X$’s) are highly correlated among themselves.

#model2 <- lm(body_mass_g ~ flipper_length_mm + species , data = penguins)
car::vif(model2)

##                   GVIF Df GVIF^(1/(2*Df))
## flipper_length_mm 4.51  1            2.12
## species           4.51  2            1.46

# model3 <- lm(body_mass_g ~ flipper_length_mm + species + sex , data = penguins)
car::vif(model3)

##                   GVIF Df GVIF^(1/(2*Df))
## flipper_length_mm 6.05  1            2.46
## species           5.65  2            1.54
## sex               1.36  1            1.17

# model4 <- lm(body_mass_g ~ flipper_length_mm + species + sex + bill_length_mm, data = penguins)
car::vif(model4)

##                    GVIF Df GVIF^(1/(2*Df))
## flipper_length_mm  6.44  1            2.54
## species           18.16  2            2.06
## sex                1.81  1            1.35
## bill_length_mm     5.95  1            2.44

# model5 <- lm(body_mass_g ~ flipper_length_mm + species + sex + bill_length_mm + bill_depth_mm , data = penguins)
car::vif(model5)

##                    GVIF Df GVIF^(1/(2*Df))
## flipper_length_mm  6.69  1            2.59
## species           41.07  2            2.53
## sex                2.31  1            1.52
## bill_length_mm     6.07  1            2.46
## bill_depth_mm      6.08  1            2.47

# model6 <- lm(body_mass_g ~ . , data = penguins)
car::vif(model6)

##                    GVIF Df GVIF^(1/(2*Df))
## species           71.20  2            2.90
## island             3.76  2            1.39
## bill_length_mm     6.12  1            2.47
## bill_depth_mm      6.27  1            2.50
## flipper_length_mm  7.78  1            2.79
## sex                2.34  1            1.53
## year               1.17  1            1.08

model7 <- lm(body_mass_g ~ flipper_length_mm +  sex + bill_depth_mm , data = penguins)
car::vif(model7)

## flipper_length_mm               sex     bill_depth_mm 
##              2.44              1.89              2.65

Summary model comparison table using `huxtable::huxreg()`

Which of the six models we have fit seems to be the best one? Let us compare them on one table.

huxreg(model1, model2, model3, model4, model5, model6, model7,
                 statistics = c('#observations' = 'nobs', 
                                'R squared' = 'r.squared', 
                                'Adj. R Squared' = 'adj.r.squared', 
                                'Residual SE' = 'sigma'), 
                 bold_signif = 0.05
) %>% 
  set_caption('Comparison of models')

Comparison of models
	(1)	(2)	(3)	(4)	(5)	(6)	(7)
(Intercept)	-5780.831 ***	-4031.477 ***	-365.817	-759.064	-1460.995 *	84087.945 *	-2246.829 ***
	(305.815)	(584.151)	(532.050)	(541.377)	(571.308)	(41912.019)	(625.286)
flipper_length_mm	49.686 ***	40.705 ***	20.025 ***	17.847 ***	15.950 ***	18.504 ***	38.190 ***
	(1.518)	(3.071)	(2.846)	(2.902)	(2.910)	(3.128)	(2.084)
speciesChinstrap		-206.510 ***	-87.634	-291.711 ***	-251.477 **	-282.539 **
		(57.731)	(46.347)	(81.502)	(81.079)	(88.790)
speciesGentoo		266.810 **	836.260 ***	707.028 ***	1014.627 ***	890.958 ***
		(95.264)	(85.185)	(94.359)	(129.561)	(144.563)
sexmale			530.381 ***	465.395 ***	389.892 ***	378.977 ***	538.080 ***
			(37.810)	(43.081)	(47.848)	(48.074)	(51.310)
bill_length_mm				21.633 **	18.204 *	18.964 **
				(7.148)	(7.106)	(7.112)
bill_depth_mm					67.218 ***	60.798 **	-86.947 ***
					(19.742)	(20.002)	(15.456)
islandDream						-21.180
						(58.390)
islandTorgersen						-58.777
						(60.852)
year						-42.785 *
						(20.949)
#observations	342	342	333	333	333	333	333
R squared	0.759	0.783	0.867	0.871	0.875	0.877	0.823
Adj. R Squared	0.758	0.781	0.865	0.869	0.873	0.873	0.821
Residual SE	394.278	375.535	295.565	291.955	287.338	286.524	340.427
* p < 0.001; p < 0.01; * p < 0.05.

The best model seems to be model 7, so we will use broom::tidy() and broom::glance() to get model results.

model7 %>% broom::tidy()

term	estimate	std.error	statistic	p.value
(Intercept)	-2.25e+03	625	-3.59	0.000376
flipper_length_mm	38.2	2.08	18.3	3.47e-52
sexmale	538	51.3	10.5	2.17e-22
bill_depth_mm	-86.9	15.5	-5.63	3.96e-08

model7 %>% broom::glance()

r.squared	adj.r.squared	sigma	statistic	p.value	df	logLik	AIC	BIC	deviance	df.residual	nobs
0.823	0.821	340	509	2.9e-123	3	-2.41e+03	4.83e+03	4.85e+03	3.81e+07	329	333

Let us write down its equation

\[ \begin{aligned} \text{body_mass_g} &= \alpha + \beta_{1}(\text{flipper_length_mm}) + \beta_{2}(\text{sex}_{\text{male}})\ + \beta_{3}(\text{bill_depth_mm}) + \epsilon \end{aligned} \]\[ \begin{aligned} \text{body_mass_g} &= -2246.83 + 38.19(\text{flipper_length_mm}) + 538.08(\text{sex}_{\text{male}})\ - 86.95(\text{bill_depth_mm}) + \epsilon \end{aligned} \]

Fitting multiple regression models in one go

Let us recall the relationship between body mass and bill depth and have a look at the scatteplot.

We could run three separate regression, but we can estimate three regression models with a few lines of code.

penguins %>%
  na.omit() %>% 
  group_by(species) %>%
  summarise(
    broom::tidy(lm( body_mass_g ~ bill_depth_mm))
  )

## # A tibble: 6 x 6
## # Groups:   species [3]
##   species   term          estimate std.error statistic  p.value
##   <fct>     <chr>            <dbl>     <dbl>     <dbl>    <dbl>
## 1 Adelie    (Intercept)     -297.      469.    -0.634  5.27e- 1
## 2 Adelie    bill_depth_mm    218.       25.5    8.55   1.67e-14
## 3 Chinstrap (Intercept)      -36.2     613.    -0.0591 9.53e- 1
## 4 Chinstrap bill_depth_mm    205.       33.2    6.16   4.79e- 8
## 5 Gentoo    (Intercept)     -422.      488.    -0.864  3.89e- 1
## 6 Gentoo    bill_depth_mm    368.       32.5   11.3    1.64e-20

What we see is BLAH…

What if we add sex? First, let us facet_wrap() our scatter plot to see what it looks like

penguins %>%
  na.omit() %>% 
  group_by(species) %>%
  summarise(
    broom::tidy(lm( body_mass_g ~ bill_depth_mm + sex))
  )

## # A tibble: 9 x 6
## # Groups:   species [3]
##   species   term          estimate std.error statistic  p.value
##   <fct>     <chr>            <dbl>     <dbl>     <dbl>    <dbl>
## 1 Adelie    (Intercept)     1931.      452.      4.28  3.47e- 5
## 2 Adelie    bill_depth_mm     81.6      25.6     3.19  1.74e- 3
## 3 Adelie    sexmale          556.       62.1     8.96  1.63e-15
## 4 Chinstrap (Intercept)      830.      861.      0.964 3.39e- 1
## 5 Chinstrap bill_depth_mm    153.       48.9     3.14  2.55e- 3
## 6 Chinstrap sexmale          156.      110.      1.42  1.60e- 1
## 7 Gentoo    (Intercept)     2741.      579.      4.73  6.37e- 6
## 8 Gentoo    bill_depth_mm    136.       40.6     3.35  1.08e- 3
## 9 Gentoo    sexmale          604.       79.8     7.57  9.94e-12

Simpson’s paradox

Recall from our EDA, we saw no relationship between bill length and depth.

bill_no_species

If we fit a simple regression model, we get the following

simpsons_model <- lm( bill_depth_mm ~ bill_length_mm, 
                      data = penguins %>% na.omit())

simpsons_model %>% broom::tidy()

term	estimate	std.error	statistic	p.value
(Intercept)	20.8	0.854	24.3	1.03e-75
bill_length_mm	-0.0823	0.0193	-4.27	2.53e-05

simpsons_model %>% broom::glance()

r.squared	adj.r.squared	sigma	statistic	p.value	df	logLik	AIC	BIC	deviance	df.residual	nobs
0.0523	0.0494	1.92	18.3	2.53e-05	1	-689	1.38e+03	1.39e+03	1.22e+03	331	333

The slope is significant, but the model r.squared (R2) explains only 5% of the overall variability.

However, when we plotted the same scatterplot colouring points by species, we got a completely different story.

bill_len_dep

We can again fit three individual models in one go

penguins %>%
  na.omit() %>% 
  group_by(species) %>%
  summarise(
    broom::tidy(lm( bill_depth_mm ~ bill_length_mm )),
    broom::glance(lm( bill_depth_mm ~ bill_length_mm ))
  )

## # A tibble: 6 x 16
## # Groups:   species [3]
##   species term  estimate std.error statistic  p.value r.squared adj.r.squared
##   <fct>   <chr>    <dbl>     <dbl>     <dbl>    <dbl>     <dbl>         <dbl>
## 1 Adelie  (Int~   11.5      1.37        25.2 1.51e- 6     0.149         0.143
## 2 Adelie  bill~    0.177    0.0352      25.2 1.51e- 6     0.149         0.143
## 3 Chinst~ (Int~    7.57     1.55        49.2 1.53e- 9     0.427         0.418
## 4 Chinst~ bill~    0.222    0.0317      49.2 1.53e- 9     0.427         0.418
## 5 Gentoo  (Int~    5.12     1.06        87.5 7.34e-16     0.428         0.423
## 6 Gentoo  bill~    0.208    0.0222      87.5 7.34e-16     0.428         0.423
## # ... with 8 more variables: sigma <dbl>, df <dbl>, logLik <dbl>, AIC <dbl>,
## #   BIC <dbl>, deviance <dbl>, df.residual <int>, nobs <int>

Alternatively, we can run a multiple regression model and the adjusted R2 = 76.5%

simpsons_model2 <- lm( bill_depth_mm ~ bill_length_mm + species, 
                      data = penguins %>% na.omit())

simpsons_model2 %>% broom::tidy()

term	estimate	std.error	statistic	p.value
(Intercept)	10.6	0.691	15.3	2.98e-40
bill_length_mm	0.2	0.0177	11.3	2.26e-25
speciesChinstrap	-1.93	0.226	-8.56	4.26e-16
speciesGentoo	-5.1	0.194	-26.3	1.04e-82

simpsons_model2 %>% broom::glance()

r.squared	adj.r.squared	sigma	statistic	p.value	df	logLik	AIC	BIC	deviance	df.residual	nobs
0.767	0.765	0.954	362	8.88e-104	3	-455	920	939	300	329	333

Linear Model Fitting

Tue, 28 Jul 2020 00:00:00 +0000

We will be using the Palmer penguins data to understand body mass.

Fit a model using `lm(Y ~ X1 + X2 +..., data = dataframe)`

Back to our penguins, we want to see whether body mass is a function of flipper length. We create an object called model1 that holds the results of this linear regression model.

model1 <- lm(body_mass_g ~ flipper_length_mm, data = penguins)

Look at model output

We will be using the broom package to make modelling easier to work with. There are 3 main functions in broom:

tidy() - This is where you get most of the output you want, including coefficients and p-values
glance() - additional measures on your model, including R-squared, log likelihood, and AIC/BIC
augment() - make predictions with your model using new data

For now, we will use broom::tidy() and broom::glance() to get model results.

model1 %>% broom::tidy()

term	estimate	std.error	statistic	p.value
(Intercept)	-5.78e+03	306	-18.9	5.59e-55
flipper_length_mm	49.7	1.52	32.7	4.37e-107

model1 %>% broom::glance()

r.squared	adj.r.squared	sigma	statistic	p.value	df	logLik	AIC	BIC	deviance	df.residual	nobs
0.759	0.758	394	1.07e+03	4.37e-107	1	-2.53e+03	5.06e+03	5.07e+03	5.29e+07	340	342

Write down the equation for model1

\[ \text{body_mass_g} = \alpha + \beta_{1}(\text{flipper_length_mm}) + \epsilon \] \[ \text{body_mass_g} = -5780.83 + 49.69(\text{flipper_length_mm}) + \epsilon \]

Plot scatterplot and residuals

Add more explanatory variables

model2 <- lm(body_mass_g ~ flipper_length_mm + species , data = penguins)

model3 <- lm(body_mass_g ~ flipper_length_mm + species + sex , data = penguins)

model4 <- lm(body_mass_g ~ flipper_length_mm + species + sex + bill_length_mm, data = penguins)

model5 <- lm(body_mass_g ~ flipper_length_mm + species + sex + bill_length_mm + bill_depth_mm , data = penguins)

# Fit a model with all explanatory variables (~ .)
model6 <- lm(body_mass_g ~ . , data = penguins)

Check collinearity

With so many explanatory variables, we need to worry about colinearity, i.e., whether the explanatory variables (all of the $X$’s) are highly correlated among themselves.

#model2 <- lm(body_mass_g ~ flipper_length_mm + species , data = penguins)
car::vif(model2)

##                   GVIF Df GVIF^(1/(2*Df))
## flipper_length_mm 4.51  1            2.12
## species           4.51  2            1.46

# model3 <- lm(body_mass_g ~ flipper_length_mm + species + sex , data = penguins)
car::vif(model3)

##                   GVIF Df GVIF^(1/(2*Df))
## flipper_length_mm 6.05  1            2.46
## species           5.65  2            1.54
## sex               1.36  1            1.17

# model4 <- lm(body_mass_g ~ flipper_length_mm + species + sex + bill_length_mm, data = penguins)
car::vif(model4)

##                    GVIF Df GVIF^(1/(2*Df))
## flipper_length_mm  6.44  1            2.54
## species           18.16  2            2.06
## sex                1.81  1            1.35
## bill_length_mm     5.95  1            2.44

# model5 <- lm(body_mass_g ~ flipper_length_mm + species + sex + bill_length_mm + bill_depth_mm , data = penguins)
car::vif(model5)

##                    GVIF Df GVIF^(1/(2*Df))
## flipper_length_mm  6.69  1            2.59
## species           41.07  2            2.53
## sex                2.31  1            1.52
## bill_length_mm     6.07  1            2.46
## bill_depth_mm      6.08  1            2.47

# model6 <- lm(body_mass_g ~ . , data = penguins)
car::vif(model6)

##                    GVIF Df GVIF^(1/(2*Df))
## species           71.20  2            2.90
## island             3.76  2            1.39
## bill_length_mm     6.12  1            2.47
## bill_depth_mm      6.27  1            2.50
## flipper_length_mm  7.78  1            2.79
## sex                2.34  1            1.53
## year               1.17  1            1.08

model7 <- lm(body_mass_g ~ flipper_length_mm +  sex + bill_depth_mm , data = penguins)
car::vif(model7)

## flipper_length_mm               sex     bill_depth_mm 
##              2.44              1.89              2.65

Summary model comparison table using `huxtable::huxreg()`

Which of the six models we have fit seems to be the best one? Let us compare them on one table.

huxreg(model1, model2, model3, model4, model5, model6, model7,
                 statistics = c('#observations' = 'nobs', 
                                'R squared' = 'r.squared', 
                                'Adj. R Squared' = 'adj.r.squared', 
                                'Residual SE' = 'sigma'), 
                 bold_signif = 0.05
) %>% 
  set_caption('Comparison of models')

Comparison of models
	(1)	(2)	(3)	(4)	(5)	(6)	(7)
(Intercept)	-5780.831 ***	-4031.477 ***	-365.817	-759.064	-1460.995 *	84087.945 *	-2246.829 ***
	(305.815)	(584.151)	(532.050)	(541.377)	(571.308)	(41912.019)	(625.286)
flipper_length_mm	49.686 ***	40.705 ***	20.025 ***	17.847 ***	15.950 ***	18.504 ***	38.190 ***
	(1.518)	(3.071)	(2.846)	(2.902)	(2.910)	(3.128)	(2.084)
speciesChinstrap		-206.510 ***	-87.634	-291.711 ***	-251.477 **	-282.539 **
		(57.731)	(46.347)	(81.502)	(81.079)	(88.790)
speciesGentoo		266.810 **	836.260 ***	707.028 ***	1014.627 ***	890.958 ***
		(95.264)	(85.185)	(94.359)	(129.561)	(144.563)
sexmale			530.381 ***	465.395 ***	389.892 ***	378.977 ***	538.080 ***
			(37.810)	(43.081)	(47.848)	(48.074)	(51.310)
bill_length_mm				21.633 **	18.204 *	18.964 **
				(7.148)	(7.106)	(7.112)
bill_depth_mm					67.218 ***	60.798 **	-86.947 ***
					(19.742)	(20.002)	(15.456)
islandDream						-21.180
						(58.390)
islandTorgersen						-58.777
						(60.852)
year						-42.785 *
						(20.949)
#observations	342	342	333	333	333	333	333
R squared	0.759	0.783	0.867	0.871	0.875	0.877	0.823
Adj. R Squared	0.758	0.781	0.865	0.869	0.873	0.873	0.821
Residual SE	394.278	375.535	295.565	291.955	287.338	286.524	340.427
* p < 0.001; p < 0.01; * p < 0.05.

The best model seems to be model 7, so we will use broom::tidy() and broom::glance() to get model results.

model7 %>% broom::tidy()

term	estimate	std.error	statistic	p.value
(Intercept)	-2.25e+03	625	-3.59	0.000376
flipper_length_mm	38.2	2.08	18.3	3.47e-52
sexmale	538	51.3	10.5	2.17e-22
bill_depth_mm	-86.9	15.5	-5.63	3.96e-08

model7 %>% broom::glance()

r.squared	adj.r.squared	sigma	statistic	p.value	df	logLik	AIC	BIC	deviance	df.residual	nobs
0.823	0.821	340	509	2.9e-123	3	-2.41e+03	4.83e+03	4.85e+03	3.81e+07	329	333

Let us write down its equation

Fitting multiple regression models in one go

Let us recall the relationship between body mass and bill depth and have a look at the scatteplot.

We could run three separate regression, but we can estimate three regression models with a few lines of code.

penguins %>%
  na.omit() %>% 
  group_by(species) %>%
  summarise(
    broom::tidy(lm( body_mass_g ~ bill_depth_mm))
  )

## # A tibble: 6 x 6
## # Groups:   species [3]
##   species   term          estimate std.error statistic  p.value
##   <fct>     <chr>            <dbl>     <dbl>     <dbl>    <dbl>
## 1 Adelie    (Intercept)     -297.      469.    -0.634  5.27e- 1
## 2 Adelie    bill_depth_mm    218.       25.5    8.55   1.67e-14
## 3 Chinstrap (Intercept)      -36.2     613.    -0.0591 9.53e- 1
## 4 Chinstrap bill_depth_mm    205.       33.2    6.16   4.79e- 8
## 5 Gentoo    (Intercept)     -422.      488.    -0.864  3.89e- 1
## 6 Gentoo    bill_depth_mm    368.       32.5   11.3    1.64e-20

What we see is BLAH…

What if we add sex? First, let us facet_wrap() our scatter plot to see what it looks like

penguins %>%
  na.omit() %>% 
  group_by(species) %>%
  summarise(
    broom::tidy(lm( body_mass_g ~ bill_depth_mm + sex))
  )

## # A tibble: 9 x 6
## # Groups:   species [3]
##   species   term          estimate std.error statistic  p.value
##   <fct>     <chr>            <dbl>     <dbl>     <dbl>    <dbl>
## 1 Adelie    (Intercept)     1931.      452.      4.28  3.47e- 5
## 2 Adelie    bill_depth_mm     81.6      25.6     3.19  1.74e- 3
## 3 Adelie    sexmale          556.       62.1     8.96  1.63e-15
## 4 Chinstrap (Intercept)      830.      861.      0.964 3.39e- 1
## 5 Chinstrap bill_depth_mm    153.       48.9     3.14  2.55e- 3
## 6 Chinstrap sexmale          156.      110.      1.42  1.60e- 1
## 7 Gentoo    (Intercept)     2741.      579.      4.73  6.37e- 6
## 8 Gentoo    bill_depth_mm    136.       40.6     3.35  1.08e- 3
## 9 Gentoo    sexmale          604.       79.8     7.57  9.94e-12

Simpson’s paradox

Recall from our EDA, we saw no relationship between bill length and depth.

bill_no_species

If we fit a simple regression model, we get the following

simpsons_model <- lm( bill_depth_mm ~ bill_length_mm, 
                      data = penguins %>% na.omit())

simpsons_model %>% broom::tidy()

term	estimate	std.error	statistic	p.value
(Intercept)	20.8	0.854	24.3	1.03e-75
bill_length_mm	-0.0823	0.0193	-4.27	2.53e-05

simpsons_model %>% broom::glance()

r.squared	adj.r.squared	sigma	statistic	p.value	df	logLik	AIC	BIC	deviance	df.residual	nobs
0.0523	0.0494	1.92	18.3	2.53e-05	1	-689	1.38e+03	1.39e+03	1.22e+03	331	333

The slope is significant, but the model r.squared (R2) explains only 5% of the overall variability.

However, when we plotted the same scatterplot colouring points by species, we got a completely different story.

bill_len_dep

We can again fit three individual models in one go

penguins %>%
  na.omit() %>% 
  group_by(species) %>%
  summarise(
    broom::tidy(lm( bill_depth_mm ~ bill_length_mm )),
    broom::glance(lm( bill_depth_mm ~ bill_length_mm ))
  )

## # A tibble: 6 x 16
## # Groups:   species [3]
##   species term  estimate std.error statistic  p.value r.squared adj.r.squared
##   <fct>   <chr>    <dbl>     <dbl>     <dbl>    <dbl>     <dbl>         <dbl>
## 1 Adelie  (Int~   11.5      1.37        25.2 1.51e- 6     0.149         0.143
## 2 Adelie  bill~    0.177    0.0352      25.2 1.51e- 6     0.149         0.143
## 3 Chinst~ (Int~    7.57     1.55        49.2 1.53e- 9     0.427         0.418
## 4 Chinst~ bill~    0.222    0.0317      49.2 1.53e- 9     0.427         0.418
## 5 Gentoo  (Int~    5.12     1.06        87.5 7.34e-16     0.428         0.423
## 6 Gentoo  bill~    0.208    0.0222      87.5 7.34e-16     0.428         0.423
## # ... with 8 more variables: sigma <dbl>, df <dbl>, logLik <dbl>, AIC <dbl>,
## #   BIC <dbl>, deviance <dbl>, df.residual <int>, nobs <int>

Alternatively, we can run a multiple regression model and the adjusted R2 = 76.5%

simpsons_model2 <- lm( bill_depth_mm ~ bill_length_mm + species, 
                      data = penguins %>% na.omit())

simpsons_model2 %>% broom::tidy()

term	estimate	std.error	statistic	p.value
(Intercept)	10.6	0.691	15.3	2.98e-40
bill_length_mm	0.2	0.0177	11.3	2.26e-25
speciesChinstrap	-1.93	0.226	-8.56	4.26e-16
speciesGentoo	-5.1	0.194	-26.3	1.04e-82

simpsons_model2 %>% broom::glance()

r.squared	adj.r.squared	sigma	statistic	p.value	df	logLik	AIC	BIC	deviance	df.residual	nobs
0.767	0.765	0.954	362	8.88e-104	3	-455	920	939	300	329	333

Data Visualisation

Tue, 21 Jul 2020 00:00:00 +0000

Learning Objectives
1. Produce scatter plots, boxplots, and time series plots using ggplot.
2. Set universal plot settings
3. Describe what faceting is and apply faceting in ggplot.
4. Modify the aesthetics of an existing ggplot plot (including axis labels and colour).
5. Build complex and customized plots from data in a data frame.

Overview

Above all else show the data.
–Edward Tufte, The Visual Display of Quantitative Information, 2001

ggplot2 has become the de facto standard for visualising data in R. The ggplot system moves away from a defined set of graphs (e.g., scatterplot, bar chart, etc) and instead breaks graphics down to their basic components and allows you to build plots layer by layer.

“In brief… a statistical graphic is a mapping from data to aesthetic attributes (colour, shape, size) of geometric objects (points, lines, bars). The plot may also contain statistical transformations of the data and is drawn on a specific coordinates system”
– Hadley Wickham (ggplot2 creator)

It may seem verbose and unwieldy, but the idea of building a plot on a layer-by-layer basis is very powerful.

You begin a plot by defining the dataset we will use.
Then, we specify aesthetics, namely (x,y) coordinates, colour, size, etc.
Finally, we choose what geom (or geometric shape) we want to use to represent our data.

We can then add more layers, like legends, labels, facets. etc.

In the following examples, we will use the gapminder dataset with data on life expectancy lifeExp, population pop, and GDP per capita gdpPerCap for a number of countries between 1952 and 2007. We want to build a graph that shows the relationship between GDP per capita and life expectancy.

As we said, first we define the dataset we are using

library(gapminder) #load the package gapminder that contains the data

ggplot(data=gapminder)

We just get an empty canvas, as we haven’t done much with our dataset.

The next thing is to map aesthetics. In our case, we will map gdpPercap to the x-axis, and lifeExp to the y-axis.

ggplot(
  data = gapminder,
  mapping = aes(
    x = gdpPercap,
    y = lifeExp))

This is an improvement over the blank canvas we got earlier, as we have mapped the x- and y- axes and we see the likely ranges of both variables. However, to see the scatter plot we want, we must add a geometry; as scatter plots are a bunch of points, the relevant geometry is geom_point().

ggplot(
  data = gapminder,
  mapping = aes(
    x = gdpPercap,
    y = lifeExp)) +
  geom_point()

What if we wanted to colour the points by the continent each country is in? This is a change of the aesthetic properties, so we just add colour = continent.

ggplot(
  data = gapminder,
  mapping = aes(
    x = gdpPercap,
    y = lifeExp, 
    colour = continent)) +
  geom_point()

What if instead of a scatter plot we wanted to create a line plot? It would be the same code as before, but now the relevant geometry we should is geom_line insrtead of geom_point.

ggplot(
  data = gapminder,
  mapping = aes(
    x = gdpPercap,
    y = lifeExp, 
    colour = continent)) +
  geom_line()

However, this is not a particularly useful plot, so let us go back to our scatter plot.

What if we wanted to have the size of each point correspond to the population of the country? This is not a geometry, but an aesthetic property. If we add size = pop, the points produced will be proportional to the country’s population, and we still have the aesthetic property colour = continent that will colour its point with the continent the country is in.

ggplot(
  data = gapminder,
  mapping = aes(
    x = gdpPercap,
    y = lifeExp, 
    colour = continent,
    size = pop)) +
  geom_point()

This is a more interesting graph, but given the non-linear pattern we see, we can perhaps improve it by taking the logarithm of the x-axis, GDP per capita. At the end of the commands, or layers, that make up our graph we add scale_x_log10(). This will take the logarithm of the values in the x-axis and should produce a scatterplot with a linear pattern.

ggplot(
  data = gapminder,
  mapping = aes(
    x = gdpPercap,
    y = lifeExp, 
    colour = continent,
    size = pop)) +
  geom_point()+
  scale_x_log10()

If we wanted to change the labels on the x-axis to dollars, we add labels = scales::dollar to the function scale_x_log10().

ggplot(
  data = gapminder,
  mapping = aes(
    x = gdpPercap,
    y = lifeExp, 
    colour = continent,
    size = pop)) +
  geom_point()+
  scale_x_log10(labels = scales::dollar)

Any graph should be properly labelled, and we can add labels by adding another layer: labs will add the relevant labels (title, subtitle, x- and y-axes, and a caption) as shown below.

ggplot(
  data = gapminder,
  mapping = aes(
    x = gdpPercap,
    y = lifeExp, 
    colour = continent,
    size = pop)) +
  geom_point() +
  scale_x_log10(labels = scales::dollar) +
  labs(title = "Life Expectancy vs GDP per capita",
       subtitle = "1952-2007", 
       x = "GDP per capita", 
       y = "Life Expectancy",
       caption = "Source: Gapminder"  
  )+
  NULL

Style advice: While you can have the entire code for a ggplot in one single line, please don’t!
First, it makes it very hard to read and understand.
Secondly, you build a ggplot in layers; by having each layer in a separate line, you can easily comment out a line (just add a hashtag # at the beginning of the line) and see what is the effect of removing that layer.
What about the final NULL? Well, it’s there to ensure that no matter how many lines you comment out, you have no orphan +s and your code will run fine.

Finally, we can change the default theme which is a plot on a grey background; for this graph, we have chosen theme_minimal().

ggplot(
  data = gapminder,
  mapping = aes(
    x = gdpPercap,
    y = lifeExp, 
    colour = continent,
    size = pop)) +
  geom_point() +
  scale_x_log10(labels = scales::dollar) +
  labs(title = "Life Expectancy vs GDP per capita",
       subtitle = "1952-2007", 
       x = "GDP per capita", 
       y = "Life Expectancy",
       caption = "Source: Gapminder"  
      ) +
  theme_minimal()+
  NULL

Your turn:
Try exprimenting with different themes.
1. Change theme_minimal() to theme_bw(). What’s the difference?
2. Now use theme_void() which is an even more minimal theme!

Let us revisit our simple scatter plot. Because we have too may data points, we can add alpha = 0.4 in geom_point() to make some of the points more transparent; alpha = 1 means solid colour and opaque data points, whereas lower values of alpha make some points more transparent.

ggplot(
  data = gapminder,
  mapping = aes(
    x = gdpPercap,
    y = lifeExp, 
    colour = continent,
    size = pop, 
    )) +
  geom_point(alpha = 0.4) +
  scale_x_log10(labels = scales::dollar) +
  labs(title = "Life Expectancy vs GDP per capita",
       subtitle = "1952-2007", 
       x = "GDP per capita", 
       y = "Life Expectancy",
       caption = "Source: Gapminder"  
      ) +
  theme_minimal()

Layers

ggplot create graphics in layers. Once you define your data and the aesthetics [(x,y) coordinates, colour, size, fill, etc.], you can then add add more layers in that you keep on ‘doing’ things to the data.

In essence, each geom layer specifies

A geom: the graphical object to be drawn (histogram, boxplot, density plot, etc.)
A stat: what “statistic” it is applied to
A position: how it is placed; identity, jitter, dodge, stack, fill

Unfortunately, due to an early design mistake I called these either stat_() or geom_(). A better decision would have been to call them layer_() functions: that’s a more accurate description because every layer involves a stat and a geom. – Hadley Wickham

Position adjustments are used, as the name says, to adjust the position of each geom. The following position adjustments and their defaults are shown below:

position_identity - default of most geoms– Doesn’t adjust position
position_jitter - default of geom_jitter. Adding random noise to a plot can sometimes make it easier to read. Jittering is particularly useful for small datasets with at least one discrete position.
position_dodge - default of geom_boxplot. Dodging preserves the vertical position of an geom while adjusting the horizontal position.
position_stack - default of geom_bar==geom_histogram and geom_area– it stacks bars on top of each other
position_fill - useful for geom_bar==geom_histogram and geom_area– stacks bars and standardises each stack to have constant height

Let us create a base plot of life expectancy, coloured by continent

life_exp_plot <- 
  ggplot(
    data = gapminder,
    mapping = aes(
      x = lifeExp,
      fill = continent)
  )

Nothing much happens, as we have just defined the base plot. Let us now plot a geom_histogram(), which uses position_fill as its default.

life_exp_plot + 
  geom_histogram()

Is this a useful graph? position_stack, the deafult for geom_histogram(), stacks bars on top of each other. Look at the bar that appears right after the 70 year life expectancy. Right at the bottomw, we have a a few observations from, followed by the blue European one, the green Asia, etc. all the way to the top where you see the few red observations that correspond to Africa.

We can improve on this by using position = "identity" that doesn’t adjust position. We also use alpha = 0.3 to make the bars more transparent. We also plot a density plot, a smoothed version of a histogram using geom_density; its default position is identity and both plots are equivalent.

life_exp_plot + 
  geom_histogram(
    position = "identity",
    alpha = 0.3
  )

life_exp_plot + 
  geom_density( alpha = 0.3)

If we again think what each layer specifies

A geom: density plot
A stat: density
A position: identity

What if we change the position and we use stack for the position layer?

life_exp_plot + 
  geom_histogram(
    position = "stack",
    alpha = 0.3
  )

life_exp_plot + 
  geom_histogram(alpha = 0.3)

Both plots are identical; in the second one, we didn’t specify what position should be, so ggplot used the default position for a histogram, which is position = stack.

Finallt, we can also use position = "fill" which stacks bars and standardises each stack to have constant height, or position = "dodge" (to separate each continent) for the position layer

life_exp_plot + 
  geom_histogram(
    position = "fill",
    alpha = 0.3
  )

life_exp_plot + 
  geom_histogram(
    position = "dodge",
    alpha = 0.5
  )

Facetting

One of the nice features of ggplot2 is a special technique called faceting that allows us to split one plot into multiple plots based on a factor included in the dataset. In the gapminder scatterplot example, we can use facetting and produce one scatter plot for each continent separately by using facet_wrap and facet_grid as shown below.

Before proceeding, we will define an object gapminder_scatterplot with the sequence of layers that gives us the ‘core’ life expectancy vs GDP scatterplot. Having stored the ‘core’ plot into an object, we can then add layers to it as needed, something which is useful for programming, as it saves you from retyping things.

facet_wrap() allows us to get the same graph, but looking at by changing another variable; in our case, we will look at the core scatterplot first by continent, and then by `year.

#define the core gapminder scatterplot of life expectancy vs GDP
# store it in an obect called `gapminder_scatterplot`
gapminder_scatterplot <-  
  ggplot(
    data = gapminder,
    mapping = aes(
      x = gdpPercap,
      y = lifeExp, 
      colour = continent, 
      alpha = 0.2)) +
  geom_point() +
  scale_x_log10(labels = scales::dollar) +
  labs(title = "Life Expectancy vs GDP per capita, 1952-2007", 
       x = "GDP per capita", 
       y = "Life Expectancy",
       caption = "Source: Gapminder"  
  ) +
  theme_minimal()


# We now add a new layer to our base plot: facet_wrap(~x), 
# where x is the variable you want to facet by

# first, facet the scatterplot by continent
gapminder_scatterplot +
  facet_wrap(~continent)

# then, facet the scatterplot by year
gapminder_scatterplot +
  facet_wrap(~year)

We can use the nrow argument to manually control the number of rows in the faceting. We will consider the faceting by continent plot and want to have the output in 3 rows, so nrow = 3. Also, we do not want any legends for the colours used, as ggplot will explicitly name the continents. To remove the legends, we add theme(legend.position="none") to our ggplot.

gapminder_scatterplot +
  facet_wrap(
    facets = vars(continent),
         nrow = 3) +
  theme(legend.position="none")

If you want to facet a plot and have its results appear in grid, we can use facet_grid(). You can define what the row and the columns in your grid should correspond to.

# use facet_grid(), where rows refer to continents
gapminder_scatterplot +
  facet_grid(vars(rows=continent)) +
  theme(legend.position="none")

# we can create a facet_grid where you can define *both* rows and columns
# in our scatterplot, we add a facet_grid() layer where columns = continents and rows =  year
gapminder_scatterplot+
  theme_minimal(8) + # just make the font size smaller
  facet_grid(
    cols = vars(continent), 
    rows = vars(year)
    ) +
  theme(legend.position="none")

Finally, if instead of a scatter plot we wanted to create a boxplot of life expectancy by continent, we use similar aesthetics, but the relevant geometry is geom_boxplot().

ggplot(
  data = gapminder,
  mapping = aes(
    x = continent,
    y = lifeExp, 
    fill = continent)) +
  geom_boxplot() +
  labs(title = "Life Expectancy among the continents, 1952-2007", 
       x = " ", # Empty, as the levels of the x-variable are the continets
       y = "Life Expectancy",
       caption = "Source: Gapminder"  
      ) +
  theme_minimal()

Tweaking graphics for publication quality

ggplot comes with many other options for tweaking plots to get them just the way you want for publication. These can be a bit hard to remember, but I usually look them up in R graphics cookbook and the BBC Visual and Data Journalism cookbook for R graphics, both of which have example code to cover most use cases!

In the example below, we select only those observations between 1997 and 2007, calculate the average life expectancy, average GDP per capita, and average population. We then create a new object, gapminder9707_plot which is the series of commands that make up our plot. To actually see the plot, we either use print(gapminder9707_plot) or just gapminder9707_plot.

gapminder9707 <- gapminder %>% 
  group_by(continent, country) %>%
  filter(year %in% c(1997, 2002, 2007)) %>%
  summarise(avg_life = mean(lifeExp, na.rm = TRUE),
            avg_gdp = mean(gdpPercap, na.rm = TRUE),
            avg_population_millions = mean(pop/1000000, na.rm = TRUE)) %>% 
  ungroup()          
            
gapminder9707_plot <- ggplot(data = gapminder9707,
       mapping = aes(x = avg_gdp,
                     y = avg_life,
                     colour = continent,
                     size = avg_population_millions,
                     label = country)) +
  geom_point() +
  scale_x_log10(labels = scales::dollar) +
  theme_bw() +
  labs(title = "Life Expectancy vs GDP per capita, 1997-2007", 
       x = "Average GDP per capita", 
       y = "Average Life Expectancy",
       caption = "Source: Gapminder") +
  geom_text(nudge_y = -.8, size = 2.2, check_overlap = TRUE)+
  theme(legend.position="none") 


gapminder9707_plot

Making plots interactive using `plotly`

We can make our plots interactive using the plotly package, which allows us to look at each point, zoon in/out, etc. Once you load the plotly library, it is simply a matter or using the ggplotly command. Move your cursor on the graph and see what happens!

plotly::ggplotly(gapminder9707_plot)

Animated Graphs

Animated graphs have recently become popular. The internet is full of tutorials and code-throughs where people explain how to do something interesting with R, so here is one if you wanted to know more about animations in R. You have to install the gganimate package and the animated graphs usually take some time to produce, as R needs to generates a number of GIF files and then create the animation, so please be patient!

Gapminder Animations - `transition_time()`

First we look at an animated boxplot of life expectancy by continent over time. The code to produce the plot is fairly straight-forward ggplot, but the last couple of lines ( transition_time(year) + ease_aes("linear")) are the ones that produce the animation.

library(gganimate)

boxplot_animation <- ggplot(data = gapminder,
       mapping = aes(x = continent,
                     y = lifeExp,
                     fill = continent)) +
  geom_boxplot() +
  theme_bw() +
  theme(legend.position="none") +
  labs(title = "Year: {frame_time}", 
       x = "Continent", 
       y = "Life Expectancy") +  
  transition_time(year) +
  ease_aes("linear")


animate(boxplot_animation, height=600, width = 600)

If we want to animate the evolution of the relationship between life expectancy and GDP, similar to Hans Rosling’s 200 Countries, 200 Years, 4 Minutes, we can use the code below

animation1 <- ggplot(data = gapminder,
       mapping = aes(x = gdpPercap,
                     y = lifeExp,
                     colour = continent,
                     size = pop)) +
  geom_point(alpha = 0.5) +
  scale_x_log10(labels = scales::dollar) +
  theme_bw() +
  theme(legend.position="none") +
  labs(title = "Year: {frame_time}", 
       x = "GDP per capita", 
       y = "Life Expectancy") +    
  transition_time(year)+
  ease_aes("linear")

animate(animation1, height=600, width = 600)

Finally, instead of one scatter plot, if we wanted to facet our animation by continent, we just add the facet_wrap(~continent) line of code as shown below

faceted_animation <- ggplot(data = gapminder,
       mapping = aes(x = gdpPercap,
                     y = lifeExp,
                     colour = continent,
                     size = pop)) +
  geom_point(alpha = 0.5) +
  scale_x_log10(labels = scales::dollar) +
  theme_bw() +
  theme(legend.position="none") +
  facet_wrap(~continent) +
  labs(title = "Year: {frame_time}", 
       x = "GDP per capita", 
       y = "Life Expectancy") +    
  transition_time(year)+
  ease_aes("linear")

animate(faceted_animation, height=800, width = 800)

Why you should always plot your data

We have touched on the basics of ggplot visualisations, but in this section we wanted to discuss why one should always plot the data and not just rely on tables of summary statistics.

Let us consider thirteen datasets all of which have 142 observations of (x,y) values. The table below shows the average value of X and Y, the standard deviation of X and Y, as well as the correlation coefficient between X and Y.

id	n	mean_x	mean_y	sd_x	sd_y	correlation
1	142	54.3	47.8	16.8	26.9	-0.064
2	142	54.3	47.8	16.8	26.9	-0.069
3	142	54.3	47.8	16.8	26.9	-0.068
4	142	54.3	47.8	16.8	26.9	-0.064
5	142	54.3	47.8	16.8	26.9	-0.060
6	142	54.3	47.8	16.8	26.9	-0.062
7	142	54.3	47.8	16.8	26.9	-0.069
8	142	54.3	47.8	16.8	26.9	-0.069
9	142	54.3	47.8	16.8	26.9	-0.069
10	142	54.3	47.8	16.8	26.9	-0.063
11	142	54.3	47.8	16.8	26.9	-0.069
12	142	54.3	47.8	16.8	26.9	-0.067
13	142	54.3	47.8	16.8	26.9	-0.066

Since our datasets contain values for X and Y, we can estimate 13 regression models and plot the values for each of the 13 intercepts and slope for X.

If we just looked at either the summary statistics table, or the plots of intercepts and slopes, we may be tempted to conclude that the 13 datasets are either identical or very much alike. However, this is far from the truth, as this is what the 13 individual datasets look like.

What data patterns can lie behind a correlation coefficient?

Jan Vanhove has written about the data patterns that can lie behind a correlation coefficient and why you should always plot and visualise a scatter plot; he has created a package, cannoball, where you specify a correlation coefficient r and a sample size n, and you get multiple scatterplots of the same correlation value, but fairly different in their scatter.

We will visualise 16 different datasets, all of which have a correlation of 0.50, and a sample of size n = 100.

RStudio’s primers for ggplot2

You can work through RStudio’s introductory primers for ggplot2; these are fairly short once you get used to the syntax of ggplot().

RStudios’s primers on visualising data

Further resources

Model diagnostics

Wed, 29 Jul 2020 00:00:00 +0000

Linear Regression Assumptions: L-I-N-E

L: Linear relationship between (Y) and the explanatory variable (X)
I: Independence of errors—there’s no connection between how far any two points lie from the regression line _ N: Normal distribution of Y at each level of X
E: equality of variance of the errors – variability remains the same for all levels of X.

In other words, the residuals (errors) should satisfy the following:

L: The mean value for Y at each level of X lies on regression line.
I: There is no clear pattern in the errors
N: At each level of X, the values for Y are normally distributed.
E: The variability in the Y’s for each level of X is the same

Figure 1: Assumptions for linear ordinary least squares (OLS) regression

Regression diagnostic plots with `ggfortify::autoplot()`

Let us see what is happening in our models. We will use the ggfortify package and its autoplot() command to get the following regression diagnostic plots:

Residuals vs. Fitted: checks Linearity assumption. Residuals should be random, with no pattern, and around Y = 0; if not, there is a pattern in the data that is currently unaccounted for.
Normal Q-Q: checks residual Normality assumption. Deviations from a straight line indicate that residuals do not follow a Normal distribution.
Scale-Location: checks whether residuals have equal/constant variance or not. Positive or negative trends across the fitted values indicate variability that is not constant.
Residuals vs. Leverage: check for influential points. Points with high leverage (having unusual values of the predictors) and/or high absolute residuals can have an undue influence on estimates of model parameters.

model1 <- lm(body_mass_g ~ flipper_length_mm, data = penguins)

model2 <- lm(body_mass_g ~ flipper_length_mm + species , data = penguins)

model3 <- lm(body_mass_g ~ flipper_length_mm + species + sex , data = penguins)

model4 <- lm(body_mass_g ~ flipper_length_mm + species + sex , data = penguins)

model5 <- lm(body_mass_g ~ flipper_length_mm + species + sex + bill_length_mm + bill_depth_mm , data = penguins)

model6 <- lm(body_mass_g ~ . , data = penguins)


library(ggfortify)

autoplot(model1) +
  theme_minimal() + 
  labs (title = "Model 1 Diagnostic Plots")

autoplot(model2) +
  theme_minimal() + 
  labs (title = "Model 2 Diagnostic Plots")

autoplot(model3) +
  theme_minimal() + 
  labs (title = "Model 3 Diagnostic Plots")

autoplot(model4) +
  theme_minimal() + 
  labs (title = "Model 4 Diagnostic Plots")

autoplot(model5) +
  theme_minimal() + 
  labs (title = "Model 5 Diagnostic Plots")

autoplot(model6) +
  theme_minimal() + 
  labs (title = "Model 6 Diagnostic Plots")

Model diagnostics

Wed, 29 Jul 2020 00:00:00 +0000

Linear Regression Assumptions: L-I-N-E

L: Linear relationship between (Y) and the explanatory variable (X)
I: Independence of errors—there’s no connection between how far any two points lie from the regression line _ N: Normal distribution of Y at each level of X
E: equality of variance of the errors – variability remains the same for all levels of X.

In other words, the residuals (errors) should satisfy the following:

L: The mean value for Y at each level of X lies on regression line.
I: There is no clear pattern in the errors
N: At each level of X, the values for Y are normally distributed.
E: The variability in the Y’s for each level of X is the same

Figure 1: Assumptions for linear ordinary least squares (OLS) regression

Regression diagnostic plots with `ggfortify::autoplot()`

Let us see what is happening in our models. We will use the ggfortify package and its autoplot() command to get the following regression diagnostic plots:

Residuals vs. Fitted: checks Linearity assumption. Residuals should be random, with no pattern, and around Y = 0; if not, there is a pattern in the data that is currently unaccounted for.
Normal Q-Q: checks residual Normality assumption. Deviations from a straight line indicate that residuals do not follow a Normal distribution.
Scale-Location: checks whether residuals have equal/constant variance or not. Positive or negative trends across the fitted values indicate variability that is not constant.
Residuals vs. Leverage: check for influential points. Points with high leverage (having unusual values of the predictors) and/or high absolute residuals can have an undue influence on estimates of model parameters.

model1 <- lm(body_mass_g ~ flipper_length_mm, data = penguins)

model2 <- lm(body_mass_g ~ flipper_length_mm + species , data = penguins)

model3 <- lm(body_mass_g ~ flipper_length_mm + species + sex , data = penguins)

model4 <- lm(body_mass_g ~ flipper_length_mm + species + sex , data = penguins)

model5 <- lm(body_mass_g ~ flipper_length_mm + species + sex + bill_length_mm + bill_depth_mm , data = penguins)

model6 <- lm(body_mass_g ~ . , data = penguins)


library(ggfortify)

autoplot(model1) +
  theme_minimal() + 
  labs (title = "Model 1 Diagnostic Plots")

autoplot(model2) +
  theme_minimal() + 
  labs (title = "Model 2 Diagnostic Plots")

autoplot(model3) +
  theme_minimal() + 
  labs (title = "Model 3 Diagnostic Plots")

autoplot(model4) +
  theme_minimal() + 
  labs (title = "Model 4 Diagnostic Plots")

autoplot(model5) +
  theme_minimal() + 
  labs (title = "Model 5 Diagnostic Plots")

autoplot(model6) +
  theme_minimal() + 
  labs (title = "Model 6 Diagnostic Plots")

Data manipulation

Tue, 21 Jul 2020 00:00:00 +0000

Learning Objectives
1. Select certain variables (or columns) in a dataframe with the dplyr function select dplyr::select()
2. Select certain cases (or rows) in a dataframe according to filtering conditions with the dplyr function filter dplyr::filter()
3. Pass the output of one dplyr function to the input of another function with the ‘pipe’ operator %>%
4. Create new variables (columns) in a dataframe that are functions of existing columns with dplyr::mutate()
5. Use dplyr::group_by(), dplyr::summarise(), and dplyr::count() to split a dataframe into groups of observations, calculate summary statistics for each group, and also count the number of total observations in each group

When working on a real project, data will seldom (if ever!) arrive in exactly the format you would like to have it in in order to analyse it. We need to manipulate and transform data and just as we have a grammar for generating graphics (the layered grammar of graphics in ggplot), we also have a syntax for data transformation.

dplyr is a package that contains useful functions for transforming and manipulating data frames. You can think of these functions as verbs, that do something to the data. All of the dplyr verbs (or functions), and in fact pretty much everything in the tidyverse, works in the following fashion:

The first argument is a data frame
Subsequent arguments describe what to do with the data frame
The result is a new data frame

The `pipe` operator, or `%>%`

The pipe operator, this strange %>% thing, takes the value to the left of it and passes it through to the thing to the right of it. Let us create a couple of lists and a simple function to see an example

# A list (or vector) of multiple values too:
my_first_list <- c(1, 2, 3, 5, 8, 13, 21, 34, 55, 89)
my_second_list <- c(1, 1, 2, 3, 5, 8, 13, 21, 34, 55)

# Define a function that takes X and adds 100
my_function <- function(x) {
  new_x <- x + 100
  return(new_x)
}

Functions work on single values and on lists (or vectors):

# call my_function with x=14 as an argument
my_function(14)

## [1] 114

# call my_function with x=my_first_list as an argument; this is a 
# vectorised operation, as it will add 100 to each value in my_first_list
my_function(my_first_list)

##  [1] 101 102 103 105 108 113 121 134 155 189

# call my_function with my_first_list+my_second_list as argument; this is a 
# vectorised operation, as it will first add my_first_list+my_second_list 
# and then add 100 to each value 
my_function(my_first_list+my_second_list)

##  [1] 102 103 105 108 113 121 134 155 189 244

We can nest functions inside each other and use mean(my_function(my_first_list))

mean(my_function(my_first_list))

## [1] 123

But this can get really hard to read, since you have to read from the inside out. In English, this nested mess reads “Calculate the mean of the results of my_function applied to my_first_list.” We can simplify this by reversing the nested chain and using the pipe operator

my_first_list %>% 
  my_function() %>% 
  mean()

## [1] 123

Here we start with the data and then describe the actions/verbs to do something to the data. We can read this chain as "Take my_first_list, pass it through my_function, and calculate the mean of that.

The %>% is called a pipe and you can also read or think of the pipe operator as the words “and then.” There’s also a keyboard shortcut for this too, since typing %>% all the time can be tedious: In Windows you would use Ctrl + Shift + M and in Mac wou would use ⌘ /Cmd+ shift + M

Similarly, we frequently need to perform a series of intermediate steps to transform data for analysis. If we write each step as a discrete command and store their contents as new objects, our code becomes difficult to read and understand.

When speaking or writing, we never start with a sentence with a verb, but rather with a noun (subject). It is good practice to start with a dataframe/object and then use verbs (or functions) to describe what you want to do.

Suppose we wanted to look at the first few rows of life expectancy values, using the head() function, of the gapminder dataframe.

# Nested command, rather hard to read, since we read from the inside out
head(select(gapminder,lifeExp))

## # A tibble: 6 x 1
##   lifeExp
##     <dbl>
## 1    28.8
## 2    30.3
## 3    32.0
## 4    34.0
## 5    36.1
## 6    38.4

# using the pipe operator: Start with gapminder, and then
gapminder %>% 
  
  # select the column (or variable) lifeExp, and then 
  select(lifeExp) %>% 
  
  # use the head() function to return the first few rows of the dataset 
  head()

## # A tibble: 6 x 1
##   lifeExp
##     <dbl>
## 1    28.8
## 2    30.3
## 3    32.0
## 4    34.0
## 5    36.1
## 6    38.4

Key functions in `dplyr`

There are 6 important verbs that you’ll typically use when working with data:

Extract columns/variables with select()
Extract rows/cases with filter()
Arrange/sort rows with arrange()
Make new columns/variables with mutate()
Make group summaries with group_by %>% summarise()

`function()`	Action performed
`select()`	Selects a subset of columns (or variables) from the data frame
`filter()`	Subsets observations based on their values
`arrange()`	Changes the order of observations based on their values
`mutate()`	Creates new columns (or variables)
`group_by()`	Changes the unit of analysis from the complete dataset to individual groups of columns
`summarise()`	Collapses the data frame to a smaller number of rows which summarise the larger data

Every dplyr verb follows the same pattern. The first argument is always a data frame, and the function always returns a data frame:

VERB(DATA_TO_TRANSFORM, STUFF_IT_DOES)

Pick columns with `select()`

If we want to select ], or drop, specific columns from a tibble, we use the select() verb. For instance, if we wanted to keep only the lifeExp and year columns:

gapminder %>% select(lifeExp, year)


## # A tibble: 1,704 x 2
##    lifeExp  year
##      <dbl> <int>
##  1    28.8  1952
##  2    30.3  1957
##  3    32.0  1962
##  4    34.0  1967
##  5    36.1  1972
##  6    38.4  1977
##  7    39.9  1982
##  8    40.8  1987
##  9    41.7  1992
## 10    41.8  1997
## # ... with 1,694 more rows

You can remove specific columns by prefacing the column names with a minus sign -. SO to drop -lifeExp from our tibble, we would use:

gapminder %>% select(-lifeExp)


## # A tibble: 1,704 x 5
##    country     continent  year      pop gdpPercap
##    <fct>       <fct>     <int>    <int>     <dbl>
##  1 Afghanistan Asia       1952  8425333      779.
##  2 Afghanistan Asia       1957  9240934      821.
##  3 Afghanistan Asia       1962 10267083      853.
##  4 Afghanistan Asia       1967 11537966      836.
##  5 Afghanistan Asia       1972 13079460      740.
##  6 Afghanistan Asia       1977 14880372      786.
##  7 Afghanistan Asia       1982 12881816      978.
##  8 Afghanistan Asia       1987 13867957      852.
##  9 Afghanistan Asia       1992 16317921      649.
## 10 Afghanistan Asia       1997 22227415      635.
## # ... with 1,694 more rows

You can also rename columns using select(), using the syntax select(new_name = old_name).

gapminder %>% select(year, country, life_expectancy = lifeExp)


## # A tibble: 1,704 x 3
##     year country     life_expectancy
##    <int> <fct>                 <dbl>
##  1  1952 Afghanistan            28.8
##  2  1957 Afghanistan            30.3
##  3  1962 Afghanistan            32.0
##  4  1967 Afghanistan            34.0
##  5  1972 Afghanistan            36.1
##  6  1977 Afghanistan            38.4
##  7  1982 Afghanistan            39.9
##  8  1987 Afghanistan            40.8
##  9  1992 Afghanistan            41.7
## 10  1997 Afghanistan            41.8
## # ... with 1,694 more rows

Alternatively, there’s a special rename() verb with the same syntax, i.e., rename(new_name = old_name) that will rename a column to a new name, while keeping all the other columns:

gapminder %>% rename(life_expectancy = lifeExp)


## # A tibble: 1,704 x 6
##    country     continent  year life_expectancy      pop gdpPercap
##    <fct>       <fct>     <int>           <dbl>    <int>     <dbl>
##  1 Afghanistan Asia       1952            28.8  8425333      779.
##  2 Afghanistan Asia       1957            30.3  9240934      821.
##  3 Afghanistan Asia       1962            32.0 10267083      853.
##  4 Afghanistan Asia       1967            34.0 11537966      836.
##  5 Afghanistan Asia       1972            36.1 13079460      740.
##  6 Afghanistan Asia       1977            38.4 14880372      786.
##  7 Afghanistan Asia       1982            39.9 12881816      978.
##  8 Afghanistan Asia       1987            40.8 13867957      852.
##  9 Afghanistan Asia       1992            41.7 16317921      649.
## 10 Afghanistan Asia       1997            41.8 22227415      635.
## # ... with 1,694 more rows

Pick rows with `filter()`

The filter() function takes two arguments: a tibble to transform, and a set of tests. It will return each row for which the test is TRUE.

This code, for instance, will look at the gapminder dataset and return all rows where country is equal to “Jordan”.

filter(gapminder, country == "Jordan")


## # A tibble: 12 x 6
##    country continent  year lifeExp     pop gdpPercap
##    <fct>   <fct>     <int>   <dbl>   <int>     <dbl>
##  1 Jordan  Asia       1952    43.2  607914     1547.
##  2 Jordan  Asia       1957    45.7  746559     1886.
##  3 Jordan  Asia       1962    48.1  933559     2348.
##  4 Jordan  Asia       1967    51.6 1255058     2742.
##  5 Jordan  Asia       1972    56.5 1613551     2111.
##  6 Jordan  Asia       1977    61.1 1937652     2852.
##  7 Jordan  Asia       1982    63.7 2347031     4161.
##  8 Jordan  Asia       1987    65.9 2820042     4449.
##  9 Jordan  Asia       1992    68.0 3867409     3432.
## 10 Jordan  Asia       1997    69.8 4526235     3645.
## 11 Jordan  Asia       2002    71.3 5307470     3845.
## 12 Jordan  Asia       2007    72.5 6053193     4519.

Notice that there are two equal signs (==). Please note that when testing for equality, we use a double equal sign, (==). If you had used a single equal sign, that would be the assignment operator, i.e., you set an argument (like data = gapminder); when you use two equal signs, you are running a logical a test.

Test	Meaning
`x < y`	Less than
`x > y`	Greater than
`x == y`	Equal to
`x <= y`	Less than or equal to
`x >= y`	Greater than or equal to
`x != y`	Not equal to
`x %in% y`	In (group membership)
`is.na(x)`	Is missing
`!is.na(x)`	Is not missing

Your turn: Use filter() and logical tests to show:
1. The data for China
2. All data for countries in Africa
3. All cases (rows) where life expectancy is greater than 80

You can also use multiple conditions, and these will extract rows that meet every test. By default, if you separate the tests with a comma, R will consider this an “and” test and find rows that are both Jordan and greater than 2000.

filter(gapminder, country == "Jordan", year > 2000)


## # A tibble: 2 x 6
##   country continent  year lifeExp     pop gdpPercap
##   <fct>   <fct>     <int>   <dbl>   <int>     <dbl>
## 1 Jordan  Asia       2002    71.3 5307470     3845.
## 2 Jordan  Asia       2007    72.5 6053193     4519.

If you have any programming experience, you can also use the common operators for “and” with “&”, “or” with “|”, and “not” with “!”:

Operator	Meaning
`a & b`	and
`a \| b`	or
`!a`	not

Your turn: Use filter() and logical tests to show:
1. India before 1970
2. Countries where life expectancy in 2007 is below 60
3. Countries where life expectancy in 2007 is below 60 and are not in Africa

Beware of some common mistakes! You can’t collapse multiple tests into one. Instead, use two separate tests:

# This won't work!
filter(gapminder, 1960 < year < 1980)

# This will work
filter(gapminder, 1960 < year, year < 1980)

Also, you can avoid stringing together lots of tests by using the %in% operator, which checks to see if a value is in a list of values.

# This works, but is tedious-- what if you wanted to pick a dozen countries?
filter(gapminder, 
       country == "Mexico" |  country == "United States" | country == "Canada" )

# This is more concise and easier to add other countries later
filter(gapminder, 
       country %in% c("Mexico", "United States", "Canada" ))

Sort data with `arrange()`

The arrange() verb sorts data. By default it sorts in ascending order, from minimum to maximum value:

gapminder %>% arrange(lifeExp)


## # A tibble: 1,704 x 6
##    country      continent  year lifeExp     pop gdpPercap
##    <fct>        <fct>     <int>   <dbl>   <int>     <dbl>
##  1 Rwanda       Africa     1992    23.6 7290203      737.
##  2 Afghanistan  Asia       1952    28.8 8425333      779.
##  3 Gambia       Africa     1952    30    284320      485.
##  4 Angola       Africa     1952    30.0 4232095     3521.
##  5 Sierra Leone Africa     1952    30.3 2143249      880.
##  6 Afghanistan  Asia       1957    30.3 9240934      821.
##  7 Cambodia     Asia       1977    31.2 6978607      525.
##  8 Mozambique   Africa     1952    31.3 6446316      469.
##  9 Sierra Leone Africa     1957    31.6 2295678     1004.
## 10 Burkina Faso Africa     1952    32.0 4469979      543.
## # ... with 1,694 more rows

You can sort in descending order (max to min) by using the desc() for the column/variable you want sorted:

gapminder %>% arrange(desc(lifeExp))


## # A tibble: 1,704 x 6
##    country          continent  year lifeExp       pop gdpPercap
##    <fct>            <fct>     <int>   <dbl>     <int>     <dbl>
##  1 Japan            Asia       2007    82.6 127467972    31656.
##  2 Hong Kong, China Asia       2007    82.2   6980412    39725.
##  3 Japan            Asia       2002    82   127065841    28605.
##  4 Iceland          Europe     2007    81.8    301931    36181.
##  5 Switzerland      Europe     2007    81.7   7554661    37506.
##  6 Hong Kong, China Asia       2002    81.5   6762476    30209.
##  7 Australia        Oceania    2007    81.2  20434176    34435.
##  8 Spain            Europe     2007    80.9  40448191    28821.
##  9 Sweden           Europe     2007    80.9   9031088    33860.
## 10 Israel           Asia       2007    80.7   6426679    25523.
## # ... with 1,694 more rows

You can sort by multiple columns by specifying them in a comma separated list. For example, we can sort by continent first and then sort by lifeExp (life expectancy) in descending order within each continent:

gapminder %>% 
  arrange(continent, desc(lifeExp))


## # A tibble: 1,704 x 6
##    country   continent  year lifeExp      pop gdpPercap
##    <fct>     <fct>     <int>   <dbl>    <int>     <dbl>
##  1 Reunion   Africa     2007    76.4   798094     7670.
##  2 Reunion   Africa     2002    75.7   743981     6316.
##  3 Reunion   Africa     1997    74.8   684810     6072.
##  4 Libya     Africa     2007    74.0  6036914    12057.
##  5 Tunisia   Africa     2007    73.9 10276158     7093.
##  6 Reunion   Africa     1992    73.6   622191     6101.
##  7 Tunisia   Africa     2002    73.0  9770575     5723.
##  8 Mauritius Africa     2007    72.8  1250882    10957.
##  9 Libya     Africa     2002    72.7  5368585     9535.
## 10 Algeria   Africa     2007    72.3 33333216     6223.
## # ... with 1,694 more rows

Add new columns with `mutate()`

You create new columns, or variables, with the mutate() function. You can create a single new column of gdp in the gapminder tibble as follows:

mutate(gapminder, gdp = gdpPercap * pop)


## # A tibble: 1,704 x 7
##    country     continent  year lifeExp      pop gdpPercap          gdp
##    <fct>       <fct>     <int>   <dbl>    <int>     <dbl>        <dbl>
##  1 Afghanistan Asia       1952    28.8  8425333      779.  6567086330.
##  2 Afghanistan Asia       1957    30.3  9240934      821.  7585448670.
##  3 Afghanistan Asia       1962    32.0 10267083      853.  8758855797.
##  4 Afghanistan Asia       1967    34.0 11537966      836.  9648014150.
##  5 Afghanistan Asia       1972    36.1 13079460      740.  9678553274.
##  6 Afghanistan Asia       1977    38.4 14880372      786. 11697659231.
##  7 Afghanistan Asia       1982    39.9 12881816      978. 12598563401.
##  8 Afghanistan Asia       1987    40.8 13867957      852. 11820990309.
##  9 Afghanistan Asia       1992    41.7 16317921      649. 10595901589.
## 10 Afghanistan Asia       1997    41.8 22227415      635. 14121995875.
## # ... with 1,694 more rows

And you can create multiple columns by including a comma-separated list of new columns to create:

mutate(gapminder, gdp = gdpPercap * pop,
                  pop_mill = round(pop / 1000000))


## # A tibble: 1,704 x 8
##    country     continent  year lifeExp      pop gdpPercap          gdp pop_mill
##    <fct>       <fct>     <int>   <dbl>    <int>     <dbl>        <dbl>    <dbl>
##  1 Afghanistan Asia       1952    28.8  8425333      779.  6567086330.        8
##  2 Afghanistan Asia       1957    30.3  9240934      821.  7585448670.        9
##  3 Afghanistan Asia       1962    32.0 10267083      853.  8758855797.       10
##  4 Afghanistan Asia       1967    34.0 11537966      836.  9648014150.       12
##  5 Afghanistan Asia       1972    36.1 13079460      740.  9678553274.       13
##  6 Afghanistan Asia       1977    38.4 14880372      786. 11697659231.       15
##  7 Afghanistan Asia       1982    39.9 12881816      978. 12598563401.       13
##  8 Afghanistan Asia       1987    40.8 13867957      852. 11820990309.       14
##  9 Afghanistan Asia       1992    41.7 16317921      649. 10595901589.       16
## 10 Afghanistan Asia       1997    41.8 22227415      635. 14121995875.       22
## # ... with 1,694 more rows

You can also run logical, conditional tests within mutate() using the ifelse() function. This works like the =IF function in Excel and it takes three arguments:

a logical test,
what happens if the test is true, and
what happens if the test is false:

ifelse(TEST, VALUE_IF_TRUE, VALUE_IF_FALSE)

We can create a new column that is a binary (TRUE/FALSE) indicator for whether year is after 1960:

mutate(gapminder, after_1960 = ifelse(year > 1960, TRUE, FALSE))


## # A tibble: 1,704 x 7
##    country     continent  year lifeExp      pop gdpPercap after_1960
##    <fct>       <fct>     <int>   <dbl>    <int>     <dbl> <lgl>     
##  1 Afghanistan Asia       1952    28.8  8425333      779. FALSE     
##  2 Afghanistan Asia       1957    30.3  9240934      821. FALSE     
##  3 Afghanistan Asia       1962    32.0 10267083      853. TRUE      
##  4 Afghanistan Asia       1967    34.0 11537966      836. TRUE      
##  5 Afghanistan Asia       1972    36.1 13079460      740. TRUE      
##  6 Afghanistan Asia       1977    38.4 14880372      786. TRUE      
##  7 Afghanistan Asia       1982    39.9 12881816      978. TRUE      
##  8 Afghanistan Asia       1987    40.8 13867957      852. TRUE      
##  9 Afghanistan Asia       1992    41.7 16317921      649. TRUE      
## 10 Afghanistan Asia       1997    41.8 22227415      635. TRUE      
## # ... with 1,694 more rows

We can also use text labels instead of TRUE and FALSE:

mutate(gapminder, 
       after_1960 = ifelse(year > 1960, "After 1960", "Before 1960"))


## # A tibble: 1,704 x 7
##    country     continent  year lifeExp      pop gdpPercap after_1960 
##    <fct>       <fct>     <int>   <dbl>    <int>     <dbl> <chr>      
##  1 Afghanistan Asia       1952    28.8  8425333      779. Before 1960
##  2 Afghanistan Asia       1957    30.3  9240934      821. Before 1960
##  3 Afghanistan Asia       1962    32.0 10267083      853. After 1960 
##  4 Afghanistan Asia       1967    34.0 11537966      836. After 1960 
##  5 Afghanistan Asia       1972    36.1 13079460      740. After 1960 
##  6 Afghanistan Asia       1977    38.4 14880372      786. After 1960 
##  7 Afghanistan Asia       1982    39.9 12881816      978. After 1960 
##  8 Afghanistan Asia       1987    40.8 13867957      852. After 1960 
##  9 Afghanistan Asia       1992    41.7 16317921      649. After 1960 
## 10 Afghanistan Asia       1997    41.8 22227415      635. After 1960 
## # ... with 1,694 more rows

Your turn: Use mutate() to:
1. Add an africa column that is TRUE if the country is on the African continent
2. Add a column log_GDP for the logarithm of GDP per capita, using log(gdpPercap)
3. Add an africa_asia column that says “Africa or Asia” if the country is in Africa or Asia, and “Not Africa or Asia” if it’s not

Combine multiple verbs with pipes (`%>%`)

What if you want to include only rows from 2002 and make a new column with the logged GDP per capita? Doing this requires both filter() and mutate(), so we need to find a way to use both at once.

One solution is to use intermediate data frames for each step:

gapminder_2002_filtered <- filter(gapminder, year == 2002)

gapminder_2002_logged <- mutate(gapminder_2002_filtered, log_gdpPercap = log(gdpPercap))

That works fine, but your environment panel will start getting full of lots of intermediate data frames.

Another solution is to nest the functions inside each other. Remember that all dplyr functions return data frames, so you can feed the results of one into another:

filter(mutate(gapminder, log_gdpPercap = log(gdpPercap)), 
       year == 2002)

That works too, but it gets really complicated once you have even more functions, and it’s hard to keep track of which function’s arguments go where. I’d avoid doing this entirely.

One really nice solution is to use the pipe operator, or %>%. The pipe takes an object on the left and passes it as the first argument of the function on the right.

# gapminder will automatically get placed in the _____ spot
gapminder %>% filter(_____, country == "Jordan")

These two lines of code do the same thing:

filter(gapminder, country == "Jordan")

gapminder %>% filter(country == "Jordan")

Using pipes, you always start with a data frame, pass it to one verb to do one thing, then pass the output of that verb (a dataframe) to the next verb that will do something else, and so on. When reading any code with a %>%, it’s easiest to read the %>% as “and then”. This would read:

Take the gapminder dataset and then filter it so that it only has rows from 2002 and then add a new column (mutate) with the logged GDP per capita

gapminder %>% 
  filter(year == 2002) %>% 
  mutate(log_gdpPercap = log(gdpPercap))

Summarise data by groups with `group_by() %>% summarise()`

The summarise() verb takes an entire frame and collapses all of the rows in a single number as it calculates summary information about it. For instance, the following code will start with the entire gapminder data, calculate average life expectancy, and return just a single value, namely avarage life expectnacy among all countries and all years :

gapminder %>% summarize(mean_life = mean(lifeExp))


## # A tibble: 1 x 1
##   mean_life
##       <dbl>
## 1      59.5

You can also make multiple summary variables, just like mutate(), and it will return a column for each:

gapminder %>% summarize(mean_life = mean(lifeExp),
                        sd_life = sd(lifeExp),
                        min_life = min(lifeExp),
                        max_life = max(lifeExp)
                        )


## # A tibble: 1 x 4
##   mean_life sd_life min_life max_life
##       <dbl>   <dbl>    <dbl>    <dbl>
## 1      59.5    12.9     23.6     82.6

Your turn: Use summarise() to calculate:
1. The first (minimum) year in the gapminder dataset
2. The last (maximum) year in the dataset
3. The number of rows in the dataset (use the dplyr cheatsheet)
4. The number of distinct countries in the dataset (use the dplyr cheatsheet)
5. Use filter() and summarise() to calculate the median, minimum, and maximum life expectancy on the African continent in 2007

Again, remember that summarise() on its own summarises the entire dataset, so you only get numbers in a single row. These values can be what you want, e.g., averages, standard deviations, and min/max values for the entire dataset. If you group your data into separate subgroups with group_by(), you can use summarise() to calculate summary statistics for each group.

The group_by() function puts rows into groups based on values in a column. If you run:

gapminder %>% group_by(continent)


## # A tibble: 1,704 x 6
## # Groups:   continent [5]
##    country     continent  year lifeExp      pop gdpPercap
##    <fct>       <fct>     <int>   <dbl>    <int>     <dbl>
##  1 Afghanistan Asia       1952    28.8  8425333      779.
##  2 Afghanistan Asia       1957    30.3  9240934      821.
##  3 Afghanistan Asia       1962    32.0 10267083      853.
##  4 Afghanistan Asia       1967    34.0 11537966      836.
##  5 Afghanistan Asia       1972    36.1 13079460      740.
##  6 Afghanistan Asia       1977    38.4 14880372      786.
##  7 Afghanistan Asia       1982    39.9 12881816      978.
##  8 Afghanistan Asia       1987    40.8 13867957      852.
##  9 Afghanistan Asia       1992    41.7 16317921      649.
## 10 Afghanistan Asia       1997    41.8 22227415      635.
## # ... with 1,694 more rows

…you won’t see anything different! R has put the dataset into separate invisible groups behind the scenes, but you haven’t done anything with those groups, so nothing has really happened. If you do things with those groups with summarise(), though, group_by() becomes much more useful.

For instance, this will take the gapminder data frame, group it by continent, and then summarize it by calculating the number of distinct countries in each group. It will return one row for each group, so there should be a row for each continent:

gapminder %>% 
  group_by(continent) %>% 
  summarize(n_countries = n_distinct(country))

## # A tibble: 5 x 2
##   continent n_countries
##   <fct>           <int>
## 1 Africa             52
## 2 Americas           25
## 3 Asia               33
## 4 Europe             30
## 5 Oceania             2

You can calculate multiple summary statistics, as before:

gapminder %>% 
  group_by(continent) %>% 
  summarize(n_countries = n_distinct(country),
            avg_life_exp = mean(lifeExp))

## # A tibble: 5 x 3
##   continent n_countries avg_life_exp
##   <fct>           <int>        <dbl>
## 1 Africa             52         48.9
## 2 Americas           25         64.7
## 3 Asia               33         60.1
## 4 Europe             30         71.9
## 5 Oceania             2         74.3

Your turn:
1. Calculate summary statistics for life expectancy for each continent. Calculate minimum, maximum, median, mean, and standard deviation, and total count (n)
2. Do the same, but for the year 2007 only

Finally, you can group by multiple columns and R will create subgroups for every combination of the groups and return the number of rows of combinations. For instance, we can calculate the average life expectancy by both year and continent and we’ll get 60 rows, since there are 5 continents and 12 years (5 × 12 = 60):

gapminder %>% 
  group_by(continent, year) %>% 
  summarize(avg_life_exp = mean(lifeExp))

## # A tibble: 60 x 3
## # Groups:   continent [5]
##    continent  year avg_life_exp
##    <fct>     <int>        <dbl>
##  1 Africa     1952         39.1
##  2 Africa     1957         41.3
##  3 Africa     1962         43.3
##  4 Africa     1967         45.3
##  5 Africa     1972         47.5
##  6 Africa     1977         49.6
##  7 Africa     1982         51.6
##  8 Africa     1987         53.3
##  9 Africa     1992         53.6
## 10 Africa     1997         53.6
## # ... with 50 more rows

A common mistake I have seen is that people use the summarise() function before any group_by(). Rememebr that if you summarise() first, you collapse the entire dataframe into a single row, so there is no group_by() that can be done on a single row of data!!

Further resources

dplyr and tidyr cheat sheet for examples.

Binary classification model

Tue, 28 Jul 2020 00:00:00 +0000

Data reshaping

Tue, 21 Jul 2020 00:00:00 +0000

Learning Objectives
1. Understand the concept of a wide and a long table format and for which purpose those formats are useful.
2. Understand what key-value pairs are.
3. Reshape a dataframe from long to wide format and back with the tidyr::pivot_longer() and tidyr::pivot_wider() commands.

Overview

It is often said that the vast majority of data analysis is spent on cleaning and preparing data. This is something that must be repeated many times over the course of analysis as new problems come to light or new data is collected.

Most people are used to analyze data in a spreadsheet or tabular format. For instance, if we wanted to study climate change, we can find data on the Combined Land-Surface Air and Sea-Surface Water Temperature Anomalies in the Northern Hemisphere at NASA’s Goddard Institute for Space Studies. The tabular data of temperature anomalies can be found here and part of that data set is shown below:

A lot of these tabular shape spreadsheets were designed for efficient data entry and not necessarily to undertake any kind of statistical analysis. The principles of tidy data provide a standard way to organise data values within a dataset. A standard makes initial data cleaning easier because you don’t need to start from scratch and reinvent the wheel every time.

Tidy data is a specific way of organising data in a consistent manner and structuring datasets to facilitate analysis with the tidyverse. The tidy data standard has been designed to facilitate exploratory data analysis; tidy datasets and tidy tools help make data analysis easier, allowing you to focus on the interesting domain problem, not on the logistics of cleaning data.

Before we proceed, a few definitions taken from Garret Grolemund and the vignette(“tidy-data”)

Variable: A quantity, quality, or property that you can measure.
Observation: A set of values that display the relationship between variables. To be an observation, values need to be measured under similar conditions, usually measured on the same observational unit at the same time.
Value: The state of a variable that you observe when you measure it.

There are three rules which make a dataset tidy:

Each variable must have its own column.
Each observation must have its own row.
Each value must have its own cell.

Figure 12.1 from R for Data Science

A tidy dataset is a long dataset, where each variable appears in one column, and each observation has its own row. The weather anomalies dataset is a wide dataset; the three variables are date (or year and month if you wanted to keep them separate), and delta (the actual temperature difference).

We will often need to reshape our datasets and should have a way to go:

from wide format to long (tidy) format using tidyr::gather() or tidyr::pivot_longer()
from long (tidy) to wide format using tidyr::spread() or tidyr::pivot_wider()

In a set of wonderful animations from Garrick Aden-Buie, this is the process of coverting from long format to wide and back

Let us review the basic tasks for tidying data using the R for Data Science gapminder subset.

table1

## # A tibble: 6 x 4
##   country      year  cases population
##   <chr>       <int>  <int>      <int>
## 1 Afghanistan  1999    745   19987071
## 2 Afghanistan  2000   2666   20595360
## 3 Brazil       1999  37737  172006362
## 4 Brazil       2000  80488  174504898
## 5 China        1999 212258 1272915272
## 6 China        2000 213766 1280428583

Note that in this data frame, each variable is in its own column (country, year, cases, and population), each observation is in its own row (i.e. each row is a different country-year pairing), and each value has its own cell.

`pivot_longer` or `gather` data

Gathering entails bringing a variable spread across multiple columns into a single column. For example, this version of table1 is not tidy because the year variable is in wide format, spread across multiple columns:

table4a

## # A tibble: 3 x 3
##   country     `1999` `2000`
## * <chr>        <int>  <int>
## 1 Afghanistan    745   2666
## 2 Brazil       37737  80488
## 3 China       212258 213766

The variables (columns) that a tidy dataframe would have would be country, year, and cases. We can use the pivot_longer or gather() function from the tidyr package to reshape the data frame and make this tidy. To do this we need three pieces of information:

The names of the columns that represent the values, not variables. Here, those are 1999 and 2000.
The key, or the name of the variable whose values form the column names. Here that is year.
The value, or the name of the variable whose values are spread over the cells. Here that is cases.

Notice that we create the names for key and value - they do not already exist in the data frame.

We implement this using the pivot_longer() or gather() function. pivot_longer() requires the newest version of tidyr

Once you have installed the newest version of tidyr, then you can use either pivot_longer() or gather()

table4a %>% 
  pivot_longer(cols=c(`1999`, `2000`), 
               names_to = "year", 
               values_to = "cases")

## # A tibble: 6 x 3
##   country     year   cases
##   <chr>       <chr>  <int>
## 1 Afghanistan 1999     745
## 2 Afghanistan 2000    2666
## 3 Brazil      1999   37737
## 4 Brazil      2000   80488
## 5 China       1999  212258
## 6 China       2000  213766

table4a %>% 
  gather(`1999`, `2000`, 
         key = year, 
         value = cases)

## # A tibble: 6 x 3
##   country     year   cases
##   <chr>       <chr>  <int>
## 1 Afghanistan 1999     745
## 2 Brazil      1999   37737
## 3 China       1999  212258
## 4 Afghanistan 2000    2666
## 5 Brazil      2000   80488
## 6 China       2000  213766

This operation would be called reshaping data wide to long.

`pivot_wider` or `spread` data

If we wanted to make a long table into a wide one, we use pivot_wider; spreading brings an observation spread across multiple rows into a single row. It is the reverse of gathering, or taking a wide dataset and making it long. For instance, take table2:

table2

## # A tibble: 12 x 4
##    country      year type            count
##    <chr>       <int> <chr>           <int>
##  1 Afghanistan  1999 cases             745
##  2 Afghanistan  1999 population   19987071
##  3 Afghanistan  2000 cases            2666
##  4 Afghanistan  2000 population   20595360
##  5 Brazil       1999 cases           37737
##  6 Brazil       1999 population  172006362
##  7 Brazil       2000 cases           80488
##  8 Brazil       2000 population  174504898
##  9 China        1999 cases          212258
## 10 China        1999 population 1272915272
## 11 China        2000 cases          213766
## 12 China        2000 population 1280428583

It violates the tidy data principle because each observation (unit of analysis is a country-year pairing) is split across multiple rows. To tidy the data frame, we need to know:

The key column, or the column that contains variable names. Here, it is type.
The value column, or the column that contains values for multiple variables. Here it is count.

Notice that unlike for gathering, when spreading the key and value columns are already defined in the data frame. We do not create the names ourselves, only identify them in the existing data frame.

table2 %>%
  pivot_wider(names_from = type, values_from = count)

## # A tibble: 6 x 4
##   country      year  cases population
##   <chr>       <int>  <int>      <int>
## 1 Afghanistan  1999    745   19987071
## 2 Afghanistan  2000   2666   20595360
## 3 Brazil       1999  37737  172006362
## 4 Brazil       2000  80488  174504898
## 5 China        1999 212258 1272915272
## 6 China        2000 213766 1280428583

table2 %>%
  spread(key = type, value = count)

## # A tibble: 6 x 4
##   country      year  cases population
##   <chr>       <int>  <int>      <int>
## 1 Afghanistan  1999    745   19987071
## 2 Afghanistan  2000   2666   20595360
## 3 Brazil       1999  37737  172006362
## 4 Brazil       2000  80488  174504898
## 5 China        1999 212258 1272915272
## 6 China        2000 213766 1280428583

This operation would be called reshaping data long to wide.

Separating

Separating splits multiple variables stored in a single column into multiple columns. For example in table3, the rate column contains both cases and population:

table3

## # A tibble: 6 x 3
##   country      year rate             
## * <chr>       <int> <chr>            
## 1 Afghanistan  1999 745/19987071     
## 2 Afghanistan  2000 2666/20595360    
## 3 Brazil       1999 37737/172006362  
## 4 Brazil       2000 80488/174504898  
## 5 China        1999 212258/1272915272
## 6 China        2000 213766/1280428583

This is a bad idea as you lose information. Tidy data principles require each column to contain a single variable. We can use separate() to split the column into two new columns:

table3 %>% 
  separate(rate, into = c("cases", "population"))

## # A tibble: 6 x 4
##   country      year cases  population
##   <chr>       <int> <chr>  <chr>     
## 1 Afghanistan  1999 745    19987071  
## 2 Afghanistan  2000 2666   20595360  
## 3 Brazil       1999 37737  172006362 
## 4 Brazil       2000 80488  174504898 
## 5 China        1999 212258 1272915272
## 6 China        2000 213766 1280428583

Uniting

Uniting is the inverse of separating - when a variable is stored in multiple columns, uniting brings the variable back into a single column. table5 splits the year variable into two columns:

table5

## # A tibble: 6 x 4
##   country     century year  rate             
## * <chr>       <chr>   <chr> <chr>            
## 1 Afghanistan 19      99    745/19987071     
## 2 Afghanistan 20      00    2666/20595360    
## 3 Brazil      19      99    37737/172006362  
## 4 Brazil      20      00    80488/174504898  
## 5 China       19      99    212258/1272915272
## 6 China       20      00    213766/1280428583

To bring them back, use the unite() function:

table5 %>% 
  unite(new, century, year)

## # A tibble: 6 x 3
##   country     new   rate             
##   <chr>       <chr> <chr>            
## 1 Afghanistan 19_99 745/19987071     
## 2 Afghanistan 20_00 2666/20595360    
## 3 Brazil      19_99 37737/172006362  
## 4 Brazil      20_00 80488/174504898  
## 5 China       19_99 212258/1272915272
## 6 China       20_00 213766/1280428583

# remove underscore
table5 %>% 
  unite(new, century, year, sep = "")

## # A tibble: 6 x 3
##   country     new   rate             
##   <chr>       <chr> <chr>            
## 1 Afghanistan 1999  745/19987071     
## 2 Afghanistan 2000  2666/20595360    
## 3 Brazil      1999  37737/172006362  
## 4 Brazil      2000  80488/174504898  
## 5 China       1999  212258/1272915272
## 6 China       2000  213766/1280428583

If we wanted to make gapminder a tabular, wide dataframe, we would use pivot_wider()

gapminder_life_exp_wide  <- gapminder %>% 
  select(country, continent, 
         lifeExp, year) %>% 
  pivot_wider(names_from = year, values_from = lifeExp) 


  gapminder_life_exp_wide

## # A tibble: 142 x 14
##    country continent `1952` `1957` `1962` `1967` `1972` `1977` `1982` `1987`
##    <fct>   <fct>      <dbl>  <dbl>  <dbl>  <dbl>  <dbl>  <dbl>  <dbl>  <dbl>
##  1 Afghan~ Asia        28.8   30.3   32.0   34.0   36.1   38.4   39.9   40.8
##  2 Albania Europe      55.2   59.3   64.8   66.2   67.7   68.9   70.4   72  
##  3 Algeria Africa      43.1   45.7   48.3   51.4   54.5   58.0   61.4   65.8
##  4 Angola  Africa      30.0   32.0   34     36.0   37.9   39.5   39.9   39.9
##  5 Argent~ Americas    62.5   64.4   65.1   65.6   67.1   68.5   69.9   70.8
##  6 Austra~ Oceania     69.1   70.3   70.9   71.1   71.9   73.5   74.7   76.3
##  7 Austria Europe      66.8   67.5   69.5   70.1   70.6   72.2   73.2   74.9
##  8 Bahrain Asia        50.9   53.8   56.9   59.9   63.3   65.6   69.1   70.8
##  9 Bangla~ Asia        37.5   39.3   41.2   43.5   45.3   46.9   50.0   52.8
## 10 Belgium Europe      68     69.2   70.2   70.9   71.4   72.8   73.9   75.4
## # ... with 132 more rows, and 4 more variables: `1992` <dbl>, `1997` <dbl>,
## #   `2002` <dbl>, `2007` <dbl>

Similarly, if we wanted to convert from the wide gapminder to the long one, we would use either gather or pivot_longer()

gapminder_life_exp_wide %>% 
  gather(key = "year", value = "lifeExp",
         -country, -continent) %>% 
  mutate(year = as.numeric(year))

## # A tibble: 1,704 x 4
##    country     continent  year lifeExp
##    <fct>       <fct>     <dbl>   <dbl>
##  1 Afghanistan Asia       1952    28.8
##  2 Albania     Europe     1952    55.2
##  3 Algeria     Africa     1952    43.1
##  4 Angola      Africa     1952    30.0
##  5 Argentina   Americas   1952    62.5
##  6 Australia   Oceania    1952    69.1
##  7 Austria     Europe     1952    66.8
##  8 Bahrain     Asia       1952    50.9
##  9 Bangladesh  Asia       1952    37.5
## 10 Belgium     Europe     1952    68  
## # ... with 1,694 more rows

gapminder_life_exp_wide %>% 
  pivot_longer(
    cols = c(-country, -continent), #keep country and continent
    names_to = "year", 
    values_to = "lifeExp",
         ) %>% 
  mutate(year = as.numeric(year))

## # A tibble: 1,704 x 4
##    country     continent  year lifeExp
##    <fct>       <fct>     <dbl>   <dbl>
##  1 Afghanistan Asia       1952    28.8
##  2 Afghanistan Asia       1957    30.3
##  3 Afghanistan Asia       1962    32.0
##  4 Afghanistan Asia       1967    34.0
##  5 Afghanistan Asia       1972    36.1
##  6 Afghanistan Asia       1977    38.4
##  7 Afghanistan Asia       1982    39.9
##  8 Afghanistan Asia       1987    40.8
##  9 Afghanistan Asia       1992    41.7
## 10 Afghanistan Asia       1997    41.8
## # ... with 1,694 more rows

RStudio primer on tidyr

Reshape Data

Recent versions of tidyr have renamed these core functions: gather() is now pivot_longer() and spread() is now pivot_wider(). The syntax for these pivot_*() functions is slightly different from what it was in gather() and spread(), so you can’t just replace the names. Even though, both gather() and spread() still work and won’t go away for a while, I think it’s worth learning the newer pivot_*() functions.

More resources

Pivoting data from columns to rows (and back!) in the tidyverse
Pivoting in tidyr
Hadley Wickham’s tidy data paper
Data Organization in Spreadsheets

Visualise Data

Tue, 21 Jul 2020 00:00:00 +0000

Learning Objectives
1. Produce scatter plots, boxplots, and time series plots using ggplot.
2. Set universal plot settings
3. Describe what faceting is and apply faceting in ggplot.
4. Modify the aesthetics of an existing ggplot plot (including axis labels and colour).
5. Build complex and customized plots from data in a data frame.

Overview

Above all else show the data.
–Edward Tufte, The Visual Display of Quantitative Information, 2001

“In brief… a statistical graphic is a mapping from data to aesthetic attributes (colour, shape, size) of geometric objects (points, lines, bars). The plot may also contain statistical transformations of the data and is drawn on a specific coordinates system”
– Hadley Wickham (ggplot2 creator)

It may seem verbose and unwieldy, but the idea of building a plot on a layer-by-layer basis is very powerful.

You begin a plot by defining the dataset we will use.
Then, we specify aesthetics, namely (x,y) coordinates, colour, size, etc.
Finally, we choose what geom (or geometric shape) we want to use to represent our data.

We can then add more layers, like legends, labels, facets. etc.

As we said, first we define the dataset we are using

library(gapminder) #load the package gapminder that contains the data

ggplot(data=gapminder)

We just get an empty canvas, as we haven’t done much with our dataset.

The next thing is to map aesthetics. In our case, we will map gdpPercap to the x-axis, and lifeExp to the y-axis.

ggplot(
  data = gapminder,
  mapping = aes(
    x = gdpPercap,
    y = lifeExp))

ggplot(
  data = gapminder,
  mapping = aes(
    x = gdpPercap,
    y = lifeExp)) +
  geom_point()

What if we wanted to colour the points by the continent each country is in? This is a change of the aesthetic properties, so we just add colour = continent.

ggplot(
  data = gapminder,
  mapping = aes(
    x = gdpPercap,
    y = lifeExp, 
    colour = continent)) +
  geom_point()

What if instead of a scatter plot we wanted to create a line plot? It would be the same code as before, but now the relevant geometry we should is geom_line insrtead of geom_point.

ggplot(
  data = gapminder,
  mapping = aes(
    x = gdpPercap,
    y = lifeExp, 
    colour = continent)) +
  geom_line()

However, this is not a particularly useful plot, so let us go back to our scatter plot.

ggplot(
  data = gapminder,
  mapping = aes(
    x = gdpPercap,
    y = lifeExp, 
    colour = continent,
    size = pop)) +
  geom_point()

ggplot(
  data = gapminder,
  mapping = aes(
    x = gdpPercap,
    y = lifeExp, 
    colour = continent,
    size = pop)) +
  geom_point()+
  scale_x_log10()

If we wanted to change the labels on the x-axis to dollars, we add labels = scales::dollar to the function scale_x_log10().

ggplot(
  data = gapminder,
  mapping = aes(
    x = gdpPercap,
    y = lifeExp, 
    colour = continent,
    size = pop)) +
  geom_point()+
  scale_x_log10(labels = scales::dollar)

Any graph should be properly labelled, and we can add labels by adding another layer: labs will add the relevant labels (title, subtitle, x- and y-axes, and a caption) as shown below.

ggplot(
  data = gapminder,
  mapping = aes(
    x = gdpPercap,
    y = lifeExp, 
    colour = continent,
    size = pop)) +
  geom_point() +
  scale_x_log10(labels = scales::dollar) +
  labs(title = "Life Expectancy vs GDP per capita",
       subtitle = "1952-2007", 
       x = "GDP per capita", 
       y = "Life Expectancy",
       caption = "Source: Gapminder"  
  )+
  NULL

Style advice: While you can have the entire code for a ggplot in one single line, please don’t!
First, it makes it very hard to read and understand.
Secondly, you build a ggplot in layers; by having each layer in a separate line, you can easily comment out a line (just add a hashtag # at the beginning of the line) and see what is the effect of removing that layer.
What about the final NULL? Well, it’s there to ensure that no matter how many lines you comment out, you have no orphan +s and your code will run fine.

Finally, we can change the default theme which is a plot on a grey background; for this graph, we have chosen theme_minimal().

ggplot(
  data = gapminder,
  mapping = aes(
    x = gdpPercap,
    y = lifeExp, 
    colour = continent,
    size = pop)) +
  geom_point() +
  scale_x_log10(labels = scales::dollar) +
  labs(title = "Life Expectancy vs GDP per capita",
       subtitle = "1952-2007", 
       x = "GDP per capita", 
       y = "Life Expectancy",
       caption = "Source: Gapminder"  
      ) +
  theme_minimal()+
  NULL

Your turn:
Try exprimenting with different themes.
1. Change theme_minimal() to theme_bw(). What’s the difference?
2. Now use theme_void() which is an even more minimal theme!

ggplot(
  data = gapminder,
  mapping = aes(
    x = gdpPercap,
    y = lifeExp, 
    colour = continent,
    size = pop, 
    )) +
  geom_point(alpha = 0.4) +
  scale_x_log10(labels = scales::dollar) +
  labs(title = "Life Expectancy vs GDP per capita",
       subtitle = "1952-2007", 
       x = "GDP per capita", 
       y = "Life Expectancy",
       caption = "Source: Gapminder"  
      ) +
  theme_minimal()

Layers

In essence, each geom layer specifies

A geom: the graphical object to be drawn (histogram, boxplot, density plot, etc.)
A stat: what “statistic” it is applied to
A position: how it is placed; identity, jitter, dodge, stack, fill

Unfortunately, due to an early design mistake I called these either stat_() or geom_(). A better decision would have been to call them layer_() functions: that’s a more accurate description because every layer involves a stat and a geom. – Hadley Wickham

Position adjustments are used, as the name says, to adjust the position of each geom. The following position adjustments and their defaults are shown below:

position_identity - default of most geoms– Doesn’t adjust position
position_jitter - default of geom_jitter. Adding random noise to a plot can sometimes make it easier to read. Jittering is particularly useful for small datasets with at least one discrete position.
position_dodge - default of geom_boxplot. Dodging preserves the vertical position of an geom while adjusting the horizontal position.
position_stack - default of geom_bar==geom_histogram and geom_area– it stacks bars on top of each other
position_fill - useful for geom_bar==geom_histogram and geom_area– stacks bars and standardises each stack to have constant height

Let us create a base plot of life expectancy, coloured by continent

life_exp_plot <- 
  ggplot(
    data = gapminder,
    mapping = aes(
      x = lifeExp,
      fill = continent)
  )

Nothing much happens, as we have just defined the base plot. Let us now plot a geom_histogram(), which uses position_fill as its default.

life_exp_plot + 
  geom_histogram()

life_exp_plot + 
  geom_histogram(
    position = "identity",
    alpha = 0.3
  )

life_exp_plot + 
  geom_density( alpha = 0.3)

If we again think what each layer specifies

A geom: density plot
A stat: density
A position: identity

What if we change the position and we use stack for the position layer?

life_exp_plot + 
  geom_histogram(
    position = "stack",
    alpha = 0.3
  )

life_exp_plot + 
  geom_histogram(alpha = 0.3)

Both plots are identical; in the second one, we didn’t specify what position should be, so ggplot used the default position for a histogram, which is position = stack.

Finallt, we can also use position = "fill" which stacks bars and standardises each stack to have constant height, or position = "dodge" (to separate each continent) for the position layer

life_exp_plot + 
  geom_histogram(
    position = "fill",
    alpha = 0.3
  )

life_exp_plot + 
  geom_histogram(
    position = "dodge",
    alpha = 0.5
  )

Facetting

facet_wrap() allows us to get the same graph, but looking at by changing another variable; in our case, we will look at the core scatterplot first by continent, and then by `year.

#define the core gapminder scatterplot of life expectancy vs GDP
# store it in an obect called `gapminder_scatterplot`
gapminder_scatterplot <-  
  ggplot(
    data = gapminder,
    mapping = aes(
      x = gdpPercap,
      y = lifeExp, 
      colour = continent, 
      alpha = 0.2)) +
  geom_point() +
  scale_x_log10(labels = scales::dollar) +
  labs(title = "Life Expectancy vs GDP per capita, 1952-2007", 
       x = "GDP per capita", 
       y = "Life Expectancy",
       caption = "Source: Gapminder"  
  ) +
  theme_minimal()


# We now add a new layer to our base plot: facet_wrap(~x), 
# where x is the variable you want to facet by

# first, facet the scatterplot by continent
gapminder_scatterplot +
  facet_wrap(~continent)

# then, facet the scatterplot by year
gapminder_scatterplot +
  facet_wrap(~year)

gapminder_scatterplot +
  facet_wrap(
    facets = vars(continent),
         nrow = 3) +
  theme(legend.position="none")

If you want to facet a plot and have its results appear in grid, we can use facet_grid(). You can define what the row and the columns in your grid should correspond to.

# use facet_grid(), where rows refer to continents
gapminder_scatterplot +
  facet_grid(vars(rows=continent)) +
  theme(legend.position="none")

# we can create a facet_grid where you can define *both* rows and columns
# in our scatterplot, we add a facet_grid() layer where columns = continents and rows =  year
gapminder_scatterplot+
  theme_minimal(8) + # just make the font size smaller
  facet_grid(
    cols = vars(continent), 
    rows = vars(year)
    ) +
  theme(legend.position="none")

Finally, if instead of a scatter plot we wanted to create a boxplot of life expectancy by continent, we use similar aesthetics, but the relevant geometry is geom_boxplot().

ggplot(
  data = gapminder,
  mapping = aes(
    x = continent,
    y = lifeExp, 
    fill = continent)) +
  geom_boxplot() +
  labs(title = "Life Expectancy among the continents, 1952-2007", 
       x = " ", # Empty, as the levels of the x-variable are the continets
       y = "Life Expectancy",
       caption = "Source: Gapminder"  
      ) +
  theme_minimal()

Tweaking graphics for publication quality

gapminder9707 <- gapminder %>% 
  group_by(continent, country) %>%
  filter(year %in% c(1997, 2002, 2007)) %>%
  summarise(avg_life = mean(lifeExp, na.rm = TRUE),
            avg_gdp = mean(gdpPercap, na.rm = TRUE),
            avg_population_millions = mean(pop/1000000, na.rm = TRUE)) %>% 
  ungroup()          
            
gapminder9707_plot <- ggplot(data = gapminder9707,
       mapping = aes(x = avg_gdp,
                     y = avg_life,
                     colour = continent,
                     size = avg_population_millions,
                     label = country)) +
  geom_point() +
  scale_x_log10(labels = scales::dollar) +
  theme_bw() +
  labs(title = "Life Expectancy vs GDP per capita, 1997-2007", 
       x = "Average GDP per capita", 
       y = "Average Life Expectancy",
       caption = "Source: Gapminder") +
  geom_text(nudge_y = -.8, size = 2.2, check_overlap = TRUE)+
  theme(legend.position="none") 


gapminder9707_plot

Making plots interactive using `plotly`

plotly::ggplotly(gapminder9707_plot)

Animated Graphs

Gapminder Animations - `transition_time()`

library(gganimate)

boxplot_animation <- ggplot(data = gapminder,
       mapping = aes(x = continent,
                     y = lifeExp,
                     fill = continent)) +
  geom_boxplot() +
  theme_bw() +
  theme(legend.position="none") +
  labs(title = "Year: {frame_time}", 
       x = "Continent", 
       y = "Life Expectancy") +  
  transition_time(year) +
  ease_aes("linear")


animate(boxplot_animation, height=600, width = 600)

If we want to animate the evolution of the relationship between life expectancy and GDP, similar to Hans Rosling’s 200 Countries, 200 Years, 4 Minutes, we can use the code below

animation1 <- ggplot(data = gapminder,
       mapping = aes(x = gdpPercap,
                     y = lifeExp,
                     colour = continent,
                     size = pop)) +
  geom_point(alpha = 0.5) +
  scale_x_log10(labels = scales::dollar) +
  theme_bw() +
  theme(legend.position="none") +
  labs(title = "Year: {frame_time}", 
       x = "GDP per capita", 
       y = "Life Expectancy") +    
  transition_time(year)+
  ease_aes("linear")

animate(animation1, height=600, width = 600)

Finally, instead of one scatter plot, if we wanted to facet our animation by continent, we just add the facet_wrap(~continent) line of code as shown below

faceted_animation <- ggplot(data = gapminder,
       mapping = aes(x = gdpPercap,
                     y = lifeExp,
                     colour = continent,
                     size = pop)) +
  geom_point(alpha = 0.5) +
  scale_x_log10(labels = scales::dollar) +
  theme_bw() +
  theme(legend.position="none") +
  facet_wrap(~continent) +
  labs(title = "Year: {frame_time}", 
       x = "GDP per capita", 
       y = "Life Expectancy") +    
  transition_time(year)+
  ease_aes("linear")

animate(faceted_animation, height=800, width = 800)

Why you should always plot your data

We have touched on the basics of ggplot visualisations, but in this section we wanted to discuss why one should always plot the data and not just rely on tables of summary statistics.

id	n	mean_x	mean_y	sd_x	sd_y	correlation
1	142	54.3	47.8	16.8	26.9	-0.064
2	142	54.3	47.8	16.8	26.9	-0.069
3	142	54.3	47.8	16.8	26.9	-0.068
4	142	54.3	47.8	16.8	26.9	-0.064
5	142	54.3	47.8	16.8	26.9	-0.060
6	142	54.3	47.8	16.8	26.9	-0.062
7	142	54.3	47.8	16.8	26.9	-0.069
8	142	54.3	47.8	16.8	26.9	-0.069
9	142	54.3	47.8	16.8	26.9	-0.069
10	142	54.3	47.8	16.8	26.9	-0.063
11	142	54.3	47.8	16.8	26.9	-0.069
12	142	54.3	47.8	16.8	26.9	-0.067
13	142	54.3	47.8	16.8	26.9	-0.066

Since our datasets contain values for X and Y, we can estimate 13 regression models and plot the values for each of the 13 intercepts and slope for X.

What data patterns can lie behind a correlation coefficient?

We will visualise 16 different datasets, all of which have a correlation of 0.50, and a sample of size n = 100.

RStudio’s primers for ggplot2

You can work through RStudio’s introductory primers for ggplot2; these are fairly short once you get used to the syntax of ggplot().

RStudios’s primers on visualising data

Further resources

Manipulate Data

Tue, 21 Jul 2020 00:00:00 +0000

Learning Objectives
1. Select certain variables (or columns) in a dataframe with the dplyr function select dplyr::select()
2. Select certain cases (or rows) in a dataframe according to filtering conditions with the dplyr function filter dplyr::filter()
3. Pass the output of one dplyr function to the input of another function with the ‘pipe’ operator %>%
4. Create new variables (columns) in a dataframe that are functions of existing columns with dplyr::mutate()
5. Use dplyr::group_by(), dplyr::summarise(), and dplyr::count() to split a dataframe into groups of observations, calculate summary statistics for each group, and also count the number of total observations in each group

The first argument is a data frame
Subsequent arguments describe what to do with the data frame
The result is a new data frame

The `pipe` operator, or `%>%`

# A list (or vector) of multiple values too:
my_first_list <- c(1, 2, 3, 5, 8, 13, 21, 34, 55, 89)
my_second_list <- c(1, 1, 2, 3, 5, 8, 13, 21, 34, 55)

# Define a function that takes X and adds 100
my_function <- function(x) {
  new_x <- x + 100
  return(new_x)
}

Functions work on single values and on lists (or vectors):

# call my_function with x=14 as an argument
my_function(14)

## [1] 114

# call my_function with x=my_first_list as an argument; this is a 
# vectorised operation, as it will add 100 to each value in my_first_list
my_function(my_first_list)

##  [1] 101 102 103 105 108 113 121 134 155 189

# call my_function with my_first_list+my_second_list as argument; this is a 
# vectorised operation, as it will first add my_first_list+my_second_list 
# and then add 100 to each value 
my_function(my_first_list+my_second_list)

##  [1] 102 103 105 108 113 121 134 155 189 244

We can nest functions inside each other and use mean(my_function(my_first_list))

mean(my_function(my_first_list))

## [1] 123

my_first_list %>% 
  my_function() %>% 
  mean()

## [1] 123

Suppose we wanted to look at the first few rows of life expectancy values, using the head() function, of the gapminder dataframe.

# Nested command, rather hard to read, since we read from the inside out
head(select(gapminder,lifeExp))

## # A tibble: 6 x 1
##   lifeExp
##     <dbl>
## 1    28.8
## 2    30.3
## 3    32.0
## 4    34.0
## 5    36.1
## 6    38.4

# using the pipe operator: Start with gapminder, and then
gapminder %>% 
  
  # select the column (or variable) lifeExp, and then 
  select(lifeExp) %>% 
  
  # use the head() function to return the first few rows of the dataset 
  head()

## # A tibble: 6 x 1
##   lifeExp
##     <dbl>
## 1    28.8
## 2    30.3
## 3    32.0
## 4    34.0
## 5    36.1
## 6    38.4

Key functions in `dplyr`

There are 6 important verbs that you’ll typically use when working with data:

Extract columns/variables with select()
Extract rows/cases with filter()
Arrange/sort rows with arrange()
Make new columns/variables with mutate()
Make group summaries with group_by %>% summarise()

`function()`	Action performed
`select()`	Selects a subset of columns (or variables) from the data frame
`filter()`	Subsets observations based on their values
`arrange()`	Changes the order of observations based on their values
`mutate()`	Creates new columns (or variables)
`group_by()`	Changes the unit of analysis from the complete dataset to individual groups of columns
`summarise()`	Collapses the data frame to a smaller number of rows which summarise the larger data

Every dplyr verb follows the same pattern. The first argument is always a data frame, and the function always returns a data frame:

VERB(DATA_TO_TRANSFORM, STUFF_IT_DOES)

Pick columns with `select()`

If we want to select ], or drop, specific columns from a tibble, we use the select() verb. For instance, if we wanted to keep only the lifeExp and year columns:

gapminder %>% select(lifeExp, year)


## # A tibble: 1,704 x 2
##    lifeExp  year
##      <dbl> <int>
##  1    28.8  1952
##  2    30.3  1957
##  3    32.0  1962
##  4    34.0  1967
##  5    36.1  1972
##  6    38.4  1977
##  7    39.9  1982
##  8    40.8  1987
##  9    41.7  1992
## 10    41.8  1997
## # ... with 1,694 more rows

You can remove specific columns by prefacing the column names with a minus sign -. SO to drop -lifeExp from our tibble, we would use:

gapminder %>% select(-lifeExp)


## # A tibble: 1,704 x 5
##    country     continent  year      pop gdpPercap
##    <fct>       <fct>     <int>    <int>     <dbl>
##  1 Afghanistan Asia       1952  8425333      779.
##  2 Afghanistan Asia       1957  9240934      821.
##  3 Afghanistan Asia       1962 10267083      853.
##  4 Afghanistan Asia       1967 11537966      836.
##  5 Afghanistan Asia       1972 13079460      740.
##  6 Afghanistan Asia       1977 14880372      786.
##  7 Afghanistan Asia       1982 12881816      978.
##  8 Afghanistan Asia       1987 13867957      852.
##  9 Afghanistan Asia       1992 16317921      649.
## 10 Afghanistan Asia       1997 22227415      635.
## # ... with 1,694 more rows

You can also rename columns using select(), using the syntax select(new_name = old_name).

gapminder %>% select(year, country, life_expectancy = lifeExp)


## # A tibble: 1,704 x 3
##     year country     life_expectancy
##    <int> <fct>                 <dbl>
##  1  1952 Afghanistan            28.8
##  2  1957 Afghanistan            30.3
##  3  1962 Afghanistan            32.0
##  4  1967 Afghanistan            34.0
##  5  1972 Afghanistan            36.1
##  6  1977 Afghanistan            38.4
##  7  1982 Afghanistan            39.9
##  8  1987 Afghanistan            40.8
##  9  1992 Afghanistan            41.7
## 10  1997 Afghanistan            41.8
## # ... with 1,694 more rows

Alternatively, there’s a special rename() verb with the same syntax, i.e., rename(new_name = old_name) that will rename a column to a new name, while keeping all the other columns:

gapminder %>% rename(life_expectancy = lifeExp)


## # A tibble: 1,704 x 6
##    country     continent  year life_expectancy      pop gdpPercap
##    <fct>       <fct>     <int>           <dbl>    <int>     <dbl>
##  1 Afghanistan Asia       1952            28.8  8425333      779.
##  2 Afghanistan Asia       1957            30.3  9240934      821.
##  3 Afghanistan Asia       1962            32.0 10267083      853.
##  4 Afghanistan Asia       1967            34.0 11537966      836.
##  5 Afghanistan Asia       1972            36.1 13079460      740.
##  6 Afghanistan Asia       1977            38.4 14880372      786.
##  7 Afghanistan Asia       1982            39.9 12881816      978.
##  8 Afghanistan Asia       1987            40.8 13867957      852.
##  9 Afghanistan Asia       1992            41.7 16317921      649.
## 10 Afghanistan Asia       1997            41.8 22227415      635.
## # ... with 1,694 more rows

Pick rows with `filter()`

The filter() function takes two arguments: a tibble to transform, and a set of tests. It will return each row for which the test is TRUE.

This code, for instance, will look at the gapminder dataset and return all rows where country is equal to “Jordan”.

filter(gapminder, country == "Jordan")


## # A tibble: 12 x 6
##    country continent  year lifeExp     pop gdpPercap
##    <fct>   <fct>     <int>   <dbl>   <int>     <dbl>
##  1 Jordan  Asia       1952    43.2  607914     1547.
##  2 Jordan  Asia       1957    45.7  746559     1886.
##  3 Jordan  Asia       1962    48.1  933559     2348.
##  4 Jordan  Asia       1967    51.6 1255058     2742.
##  5 Jordan  Asia       1972    56.5 1613551     2111.
##  6 Jordan  Asia       1977    61.1 1937652     2852.
##  7 Jordan  Asia       1982    63.7 2347031     4161.
##  8 Jordan  Asia       1987    65.9 2820042     4449.
##  9 Jordan  Asia       1992    68.0 3867409     3432.
## 10 Jordan  Asia       1997    69.8 4526235     3645.
## 11 Jordan  Asia       2002    71.3 5307470     3845.
## 12 Jordan  Asia       2007    72.5 6053193     4519.

Test	Meaning
`x < y`	Less than
`x > y`	Greater than
`x == y`	Equal to
`x <= y`	Less than or equal to
`x >= y`	Greater than or equal to
`x != y`	Not equal to
`x %in% y`	In (group membership)
`is.na(x)`	Is missing
`!is.na(x)`	Is not missing

Your turn: Use filter() and logical tests to show:
1. The data for China
2. All data for countries in Africa
3. All cases (rows) where life expectancy is greater than 80

filter(gapminder, country == "Jordan", year > 2000)


## # A tibble: 2 x 6
##   country continent  year lifeExp     pop gdpPercap
##   <fct>   <fct>     <int>   <dbl>   <int>     <dbl>
## 1 Jordan  Asia       2002    71.3 5307470     3845.
## 2 Jordan  Asia       2007    72.5 6053193     4519.

If you have any programming experience, you can also use the common operators for “and” with “&”, “or” with “|”, and “not” with “!”:

Operator	Meaning
`a & b`	and
`a \| b`	or
`!a`	not

Your turn: Use filter() and logical tests to show:
1. India before 1970
2. Countries where life expectancy in 2007 is below 60
3. Countries where life expectancy in 2007 is below 60 and are not in Africa

Beware of some common mistakes! You can’t collapse multiple tests into one. Instead, use two separate tests:

# This won't work!
filter(gapminder, 1960 < year < 1980)

# This will work
filter(gapminder, 1960 < year, year < 1980)

Also, you can avoid stringing together lots of tests by using the %in% operator, which checks to see if a value is in a list of values.

# This works, but is tedious-- what if you wanted to pick a dozen countries?
filter(gapminder, 
       country == "Mexico" |  country == "United States" | country == "Canada" )

# This is more concise and easier to add other countries later
filter(gapminder, 
       country %in% c("Mexico", "United States", "Canada" ))

Sort data with `arrange()`

The arrange() verb sorts data. By default it sorts in ascending order, from minimum to maximum value:

gapminder %>% arrange(lifeExp)


## # A tibble: 1,704 x 6
##    country      continent  year lifeExp     pop gdpPercap
##    <fct>        <fct>     <int>   <dbl>   <int>     <dbl>
##  1 Rwanda       Africa     1992    23.6 7290203      737.
##  2 Afghanistan  Asia       1952    28.8 8425333      779.
##  3 Gambia       Africa     1952    30    284320      485.
##  4 Angola       Africa     1952    30.0 4232095     3521.
##  5 Sierra Leone Africa     1952    30.3 2143249      880.
##  6 Afghanistan  Asia       1957    30.3 9240934      821.
##  7 Cambodia     Asia       1977    31.2 6978607      525.
##  8 Mozambique   Africa     1952    31.3 6446316      469.
##  9 Sierra Leone Africa     1957    31.6 2295678     1004.
## 10 Burkina Faso Africa     1952    32.0 4469979      543.
## # ... with 1,694 more rows

You can sort in descending order (max to min) by using the desc() for the column/variable you want sorted:

gapminder %>% arrange(desc(lifeExp))


## # A tibble: 1,704 x 6
##    country          continent  year lifeExp       pop gdpPercap
##    <fct>            <fct>     <int>   <dbl>     <int>     <dbl>
##  1 Japan            Asia       2007    82.6 127467972    31656.
##  2 Hong Kong, China Asia       2007    82.2   6980412    39725.
##  3 Japan            Asia       2002    82   127065841    28605.
##  4 Iceland          Europe     2007    81.8    301931    36181.
##  5 Switzerland      Europe     2007    81.7   7554661    37506.
##  6 Hong Kong, China Asia       2002    81.5   6762476    30209.
##  7 Australia        Oceania    2007    81.2  20434176    34435.
##  8 Spain            Europe     2007    80.9  40448191    28821.
##  9 Sweden           Europe     2007    80.9   9031088    33860.
## 10 Israel           Asia       2007    80.7   6426679    25523.
## # ... with 1,694 more rows

gapminder %>% 
  arrange(continent, desc(lifeExp))


## # A tibble: 1,704 x 6
##    country   continent  year lifeExp      pop gdpPercap
##    <fct>     <fct>     <int>   <dbl>    <int>     <dbl>
##  1 Reunion   Africa     2007    76.4   798094     7670.
##  2 Reunion   Africa     2002    75.7   743981     6316.
##  3 Reunion   Africa     1997    74.8   684810     6072.
##  4 Libya     Africa     2007    74.0  6036914    12057.
##  5 Tunisia   Africa     2007    73.9 10276158     7093.
##  6 Reunion   Africa     1992    73.6   622191     6101.
##  7 Tunisia   Africa     2002    73.0  9770575     5723.
##  8 Mauritius Africa     2007    72.8  1250882    10957.
##  9 Libya     Africa     2002    72.7  5368585     9535.
## 10 Algeria   Africa     2007    72.3 33333216     6223.
## # ... with 1,694 more rows

Add new columns with `mutate()`

You create new columns, or variables, with the mutate() function. You can create a single new column of gdp in the gapminder tibble as follows:

mutate(gapminder, gdp = gdpPercap * pop)


## # A tibble: 1,704 x 7
##    country     continent  year lifeExp      pop gdpPercap          gdp
##    <fct>       <fct>     <int>   <dbl>    <int>     <dbl>        <dbl>
##  1 Afghanistan Asia       1952    28.8  8425333      779.  6567086330.
##  2 Afghanistan Asia       1957    30.3  9240934      821.  7585448670.
##  3 Afghanistan Asia       1962    32.0 10267083      853.  8758855797.
##  4 Afghanistan Asia       1967    34.0 11537966      836.  9648014150.
##  5 Afghanistan Asia       1972    36.1 13079460      740.  9678553274.
##  6 Afghanistan Asia       1977    38.4 14880372      786. 11697659231.
##  7 Afghanistan Asia       1982    39.9 12881816      978. 12598563401.
##  8 Afghanistan Asia       1987    40.8 13867957      852. 11820990309.
##  9 Afghanistan Asia       1992    41.7 16317921      649. 10595901589.
## 10 Afghanistan Asia       1997    41.8 22227415      635. 14121995875.
## # ... with 1,694 more rows

And you can create multiple columns by including a comma-separated list of new columns to create:

mutate(gapminder, gdp = gdpPercap * pop,
                  pop_mill = round(pop / 1000000))


## # A tibble: 1,704 x 8
##    country     continent  year lifeExp      pop gdpPercap          gdp pop_mill
##    <fct>       <fct>     <int>   <dbl>    <int>     <dbl>        <dbl>    <dbl>
##  1 Afghanistan Asia       1952    28.8  8425333      779.  6567086330.        8
##  2 Afghanistan Asia       1957    30.3  9240934      821.  7585448670.        9
##  3 Afghanistan Asia       1962    32.0 10267083      853.  8758855797.       10
##  4 Afghanistan Asia       1967    34.0 11537966      836.  9648014150.       12
##  5 Afghanistan Asia       1972    36.1 13079460      740.  9678553274.       13
##  6 Afghanistan Asia       1977    38.4 14880372      786. 11697659231.       15
##  7 Afghanistan Asia       1982    39.9 12881816      978. 12598563401.       13
##  8 Afghanistan Asia       1987    40.8 13867957      852. 11820990309.       14
##  9 Afghanistan Asia       1992    41.7 16317921      649. 10595901589.       16
## 10 Afghanistan Asia       1997    41.8 22227415      635. 14121995875.       22
## # ... with 1,694 more rows

You can also run logical, conditional tests within mutate() using the ifelse() function. This works like the =IF function in Excel and it takes three arguments:

a logical test,
what happens if the test is true, and
what happens if the test is false:

ifelse(TEST, VALUE_IF_TRUE, VALUE_IF_FALSE)

We can create a new column that is a binary (TRUE/FALSE) indicator for whether year is after 1960:

mutate(gapminder, after_1960 = ifelse(year > 1960, TRUE, FALSE))


## # A tibble: 1,704 x 7
##    country     continent  year lifeExp      pop gdpPercap after_1960
##    <fct>       <fct>     <int>   <dbl>    <int>     <dbl> <lgl>     
##  1 Afghanistan Asia       1952    28.8  8425333      779. FALSE     
##  2 Afghanistan Asia       1957    30.3  9240934      821. FALSE     
##  3 Afghanistan Asia       1962    32.0 10267083      853. TRUE      
##  4 Afghanistan Asia       1967    34.0 11537966      836. TRUE      
##  5 Afghanistan Asia       1972    36.1 13079460      740. TRUE      
##  6 Afghanistan Asia       1977    38.4 14880372      786. TRUE      
##  7 Afghanistan Asia       1982    39.9 12881816      978. TRUE      
##  8 Afghanistan Asia       1987    40.8 13867957      852. TRUE      
##  9 Afghanistan Asia       1992    41.7 16317921      649. TRUE      
## 10 Afghanistan Asia       1997    41.8 22227415      635. TRUE      
## # ... with 1,694 more rows

We can also use text labels instead of TRUE and FALSE:

mutate(gapminder, 
       after_1960 = ifelse(year > 1960, "After 1960", "Before 1960"))


## # A tibble: 1,704 x 7
##    country     continent  year lifeExp      pop gdpPercap after_1960 
##    <fct>       <fct>     <int>   <dbl>    <int>     <dbl> <chr>      
##  1 Afghanistan Asia       1952    28.8  8425333      779. Before 1960
##  2 Afghanistan Asia       1957    30.3  9240934      821. Before 1960
##  3 Afghanistan Asia       1962    32.0 10267083      853. After 1960 
##  4 Afghanistan Asia       1967    34.0 11537966      836. After 1960 
##  5 Afghanistan Asia       1972    36.1 13079460      740. After 1960 
##  6 Afghanistan Asia       1977    38.4 14880372      786. After 1960 
##  7 Afghanistan Asia       1982    39.9 12881816      978. After 1960 
##  8 Afghanistan Asia       1987    40.8 13867957      852. After 1960 
##  9 Afghanistan Asia       1992    41.7 16317921      649. After 1960 
## 10 Afghanistan Asia       1997    41.8 22227415      635. After 1960 
## # ... with 1,694 more rows

Your turn: Use mutate() to:
1. Add an africa column that is TRUE if the country is on the African continent
2. Add a column log_GDP for the logarithm of GDP per capita, using log(gdpPercap)
3. Add an africa_asia column that says “Africa or Asia” if the country is in Africa or Asia, and “Not Africa or Asia” if it’s not

Combine multiple verbs with pipes (`%>%`)

One solution is to use intermediate data frames for each step:

gapminder_2002_filtered <- filter(gapminder, year == 2002)

gapminder_2002_logged <- mutate(gapminder_2002_filtered, log_gdpPercap = log(gdpPercap))

That works fine, but your environment panel will start getting full of lots of intermediate data frames.

Another solution is to nest the functions inside each other. Remember that all dplyr functions return data frames, so you can feed the results of one into another:

filter(mutate(gapminder, log_gdpPercap = log(gdpPercap)), 
       year == 2002)

That works too, but it gets really complicated once you have even more functions, and it’s hard to keep track of which function’s arguments go where. I’d avoid doing this entirely.

One really nice solution is to use the pipe operator, or %>%. The pipe takes an object on the left and passes it as the first argument of the function on the right.

# gapminder will automatically get placed in the _____ spot
gapminder %>% filter(_____, country == "Jordan")

These two lines of code do the same thing:

filter(gapminder, country == "Jordan")

gapminder %>% filter(country == "Jordan")

Take the gapminder dataset and then filter it so that it only has rows from 2002 and then add a new column (mutate) with the logged GDP per capita

gapminder %>% 
  filter(year == 2002) %>% 
  mutate(log_gdpPercap = log(gdpPercap))

Summarise data by groups with `group_by() %>% summarise()`

gapminder %>% summarize(mean_life = mean(lifeExp))


## # A tibble: 1 x 1
##   mean_life
##       <dbl>
## 1      59.5

You can also make multiple summary variables, just like mutate(), and it will return a column for each:

gapminder %>% summarize(mean_life = mean(lifeExp),
                        sd_life = sd(lifeExp),
                        min_life = min(lifeExp),
                        max_life = max(lifeExp)
                        )


## # A tibble: 1 x 4
##   mean_life sd_life min_life max_life
##       <dbl>   <dbl>    <dbl>    <dbl>
## 1      59.5    12.9     23.6     82.6

Your turn: Use summarise() to calculate:
1. The first (minimum) year in the gapminder dataset
2. The last (maximum) year in the dataset
3. The number of rows in the dataset (use the dplyr cheatsheet)
4. The number of distinct countries in the dataset (use the dplyr cheatsheet)
5. Use filter() and summarise() to calculate the median, minimum, and maximum life expectancy on the African continent in 2007

The group_by() function puts rows into groups based on values in a column. If you run:

gapminder %>% group_by(continent)


## # A tibble: 1,704 x 6
## # Groups:   continent [5]
##    country     continent  year lifeExp      pop gdpPercap
##    <fct>       <fct>     <int>   <dbl>    <int>     <dbl>
##  1 Afghanistan Asia       1952    28.8  8425333      779.
##  2 Afghanistan Asia       1957    30.3  9240934      821.
##  3 Afghanistan Asia       1962    32.0 10267083      853.
##  4 Afghanistan Asia       1967    34.0 11537966      836.
##  5 Afghanistan Asia       1972    36.1 13079460      740.
##  6 Afghanistan Asia       1977    38.4 14880372      786.
##  7 Afghanistan Asia       1982    39.9 12881816      978.
##  8 Afghanistan Asia       1987    40.8 13867957      852.
##  9 Afghanistan Asia       1992    41.7 16317921      649.
## 10 Afghanistan Asia       1997    41.8 22227415      635.
## # ... with 1,694 more rows

gapminder %>% 
  group_by(continent) %>% 
  summarize(n_countries = n_distinct(country))

## # A tibble: 5 x 2
##   continent n_countries
##   <fct>           <int>
## 1 Africa             52
## 2 Americas           25
## 3 Asia               33
## 4 Europe             30
## 5 Oceania             2

You can calculate multiple summary statistics, as before:

gapminder %>% 
  group_by(continent) %>% 
  summarize(n_countries = n_distinct(country),
            avg_life_exp = mean(lifeExp))

## # A tibble: 5 x 3
##   continent n_countries avg_life_exp
##   <fct>           <int>        <dbl>
## 1 Africa             52         48.9
## 2 Americas           25         64.7
## 3 Asia               33         60.1
## 4 Europe             30         71.9
## 5 Oceania             2         74.3

Your turn:
1. Calculate summary statistics for life expectancy for each continent. Calculate minimum, maximum, median, mean, and standard deviation, and total count (n)
2. Do the same, but for the year 2007 only

gapminder %>% 
  group_by(continent, year) %>% 
  summarize(avg_life_exp = mean(lifeExp))

## # A tibble: 60 x 3
## # Groups:   continent [5]
##    continent  year avg_life_exp
##    <fct>     <int>        <dbl>
##  1 Africa     1952         39.1
##  2 Africa     1957         41.3
##  3 Africa     1962         43.3
##  4 Africa     1967         45.3
##  5 Africa     1972         47.5
##  6 Africa     1977         49.6
##  7 Africa     1982         51.6
##  8 Africa     1987         53.3
##  9 Africa     1992         53.6
## 10 Africa     1997         53.6
## # ... with 50 more rows

A common mistake I have seen is that people use the summarise() function before any group_by(). Rememebr that if you summarise() first, you collapse the entire dataframe into a single row, so there is no group_by() that can be done on a single row of data!!

Further resources

dplyr and tidyr cheat sheet for examples.

Reshape Data

Tue, 21 Jul 2020 00:00:00 +0000

Learning Objectives
1. Understand the concept of a wide and a long table format and for which purpose those formats are useful.
2. Understand what key-value pairs are.
3. Reshape a dataframe from long to wide format and back with the tidyr::pivot_longer() and tidyr::pivot_wider() commands.

Overview

Before we proceed, a few definitions taken from Garret Grolemund and the vignette(“tidy-data”)

Variable: A quantity, quality, or property that you can measure.
Observation: A set of values that display the relationship between variables. To be an observation, values need to be measured under similar conditions, usually measured on the same observational unit at the same time.
Value: The state of a variable that you observe when you measure it.

There are three rules which make a dataset tidy:

Each variable must have its own column.
Each observation must have its own row.
Each value must have its own cell.

Figure 12.1 from R for Data Science

We will often need to reshape our datasets and should have a way to go:

from wide format to long (tidy) format using tidyr::gather() or tidyr::pivot_longer()
from long (tidy) to wide format using tidyr::spread() or tidyr::pivot_wider()

In a set of wonderful animations from Garrick Aden-Buie, this is the process of coverting from long format to wide and back

Let us review the basic tasks for tidying data using the R for Data Science gapminder subset.

table1

## # A tibble: 6 x 4
##   country      year  cases population
##   <chr>       <int>  <int>      <int>
## 1 Afghanistan  1999    745   19987071
## 2 Afghanistan  2000   2666   20595360
## 3 Brazil       1999  37737  172006362
## 4 Brazil       2000  80488  174504898
## 5 China        1999 212258 1272915272
## 6 China        2000 213766 1280428583

`pivot_longer` or `gather` data

table4a

## # A tibble: 3 x 3
##   country     `1999` `2000`
## * <chr>        <int>  <int>
## 1 Afghanistan    745   2666
## 2 Brazil       37737  80488
## 3 China       212258 213766

The names of the columns that represent the values, not variables. Here, those are 1999 and 2000.
The key, or the name of the variable whose values form the column names. Here that is year.
The value, or the name of the variable whose values are spread over the cells. Here that is cases.

Notice that we create the names for key and value - they do not already exist in the data frame.

We implement this using the pivot_longer() or gather() function. pivot_longer() requires the newest version of tidyr

Once you have installed the newest version of tidyr, then you can use either pivot_longer() or gather()

table4a %>% 
  pivot_longer(cols=c(`1999`, `2000`), 
               names_to = "year", 
               values_to = "cases")

## # A tibble: 6 x 3
##   country     year   cases
##   <chr>       <chr>  <int>
## 1 Afghanistan 1999     745
## 2 Afghanistan 2000    2666
## 3 Brazil      1999   37737
## 4 Brazil      2000   80488
## 5 China       1999  212258
## 6 China       2000  213766

table4a %>% 
  gather(`1999`, `2000`, 
         key = year, 
         value = cases)

## # A tibble: 6 x 3
##   country     year   cases
##   <chr>       <chr>  <int>
## 1 Afghanistan 1999     745
## 2 Brazil      1999   37737
## 3 China       1999  212258
## 4 Afghanistan 2000    2666
## 5 Brazil      2000   80488
## 6 China       2000  213766

This operation would be called reshaping data wide to long.

`pivot_wider` or `spread` data

table2

## # A tibble: 12 x 4
##    country      year type            count
##    <chr>       <int> <chr>           <int>
##  1 Afghanistan  1999 cases             745
##  2 Afghanistan  1999 population   19987071
##  3 Afghanistan  2000 cases            2666
##  4 Afghanistan  2000 population   20595360
##  5 Brazil       1999 cases           37737
##  6 Brazil       1999 population  172006362
##  7 Brazil       2000 cases           80488
##  8 Brazil       2000 population  174504898
##  9 China        1999 cases          212258
## 10 China        1999 population 1272915272
## 11 China        2000 cases          213766
## 12 China        2000 population 1280428583

It violates the tidy data principle because each observation (unit of analysis is a country-year pairing) is split across multiple rows. To tidy the data frame, we need to know:

The key column, or the column that contains variable names. Here, it is type.
The value column, or the column that contains values for multiple variables. Here it is count.

Notice that unlike for gathering, when spreading the key and value columns are already defined in the data frame. We do not create the names ourselves, only identify them in the existing data frame.

table2 %>%
  pivot_wider(names_from = type, values_from = count)

## # A tibble: 6 x 4
##   country      year  cases population
##   <chr>       <int>  <int>      <int>
## 1 Afghanistan  1999    745   19987071
## 2 Afghanistan  2000   2666   20595360
## 3 Brazil       1999  37737  172006362
## 4 Brazil       2000  80488  174504898
## 5 China        1999 212258 1272915272
## 6 China        2000 213766 1280428583

table2 %>%
  spread(key = type, value = count)

## # A tibble: 6 x 4
##   country      year  cases population
##   <chr>       <int>  <int>      <int>
## 1 Afghanistan  1999    745   19987071
## 2 Afghanistan  2000   2666   20595360
## 3 Brazil       1999  37737  172006362
## 4 Brazil       2000  80488  174504898
## 5 China        1999 212258 1272915272
## 6 China        2000 213766 1280428583

This operation would be called reshaping data long to wide.

Separating

Separating splits multiple variables stored in a single column into multiple columns. For example in table3, the rate column contains both cases and population:

table3

## # A tibble: 6 x 3
##   country      year rate             
## * <chr>       <int> <chr>            
## 1 Afghanistan  1999 745/19987071     
## 2 Afghanistan  2000 2666/20595360    
## 3 Brazil       1999 37737/172006362  
## 4 Brazil       2000 80488/174504898  
## 5 China        1999 212258/1272915272
## 6 China        2000 213766/1280428583

This is a bad idea as you lose information. Tidy data principles require each column to contain a single variable. We can use separate() to split the column into two new columns:

table3 %>% 
  separate(rate, into = c("cases", "population"))

## # A tibble: 6 x 4
##   country      year cases  population
##   <chr>       <int> <chr>  <chr>     
## 1 Afghanistan  1999 745    19987071  
## 2 Afghanistan  2000 2666   20595360  
## 3 Brazil       1999 37737  172006362 
## 4 Brazil       2000 80488  174504898 
## 5 China        1999 212258 1272915272
## 6 China        2000 213766 1280428583

Uniting

Uniting is the inverse of separating - when a variable is stored in multiple columns, uniting brings the variable back into a single column. table5 splits the year variable into two columns:

table5

## # A tibble: 6 x 4
##   country     century year  rate             
## * <chr>       <chr>   <chr> <chr>            
## 1 Afghanistan 19      99    745/19987071     
## 2 Afghanistan 20      00    2666/20595360    
## 3 Brazil      19      99    37737/172006362  
## 4 Brazil      20      00    80488/174504898  
## 5 China       19      99    212258/1272915272
## 6 China       20      00    213766/1280428583

To bring them back, use the unite() function:

table5 %>% 
  unite(new, century, year)

## # A tibble: 6 x 3
##   country     new   rate             
##   <chr>       <chr> <chr>            
## 1 Afghanistan 19_99 745/19987071     
## 2 Afghanistan 20_00 2666/20595360    
## 3 Brazil      19_99 37737/172006362  
## 4 Brazil      20_00 80488/174504898  
## 5 China       19_99 212258/1272915272
## 6 China       20_00 213766/1280428583

# remove underscore
table5 %>% 
  unite(new, century, year, sep = "")

## # A tibble: 6 x 3
##   country     new   rate             
##   <chr>       <chr> <chr>            
## 1 Afghanistan 1999  745/19987071     
## 2 Afghanistan 2000  2666/20595360    
## 3 Brazil      1999  37737/172006362  
## 4 Brazil      2000  80488/174504898  
## 5 China       1999  212258/1272915272
## 6 China       2000  213766/1280428583

If we wanted to make gapminder a tabular, wide dataframe, we would use pivot_wider()

gapminder_life_exp_wide  <- gapminder %>% 
  select(country, continent, 
         lifeExp, year) %>% 
  pivot_wider(names_from = year, values_from = lifeExp) 


  gapminder_life_exp_wide

## # A tibble: 142 x 14
##    country continent `1952` `1957` `1962` `1967` `1972` `1977` `1982` `1987`
##    <fct>   <fct>      <dbl>  <dbl>  <dbl>  <dbl>  <dbl>  <dbl>  <dbl>  <dbl>
##  1 Afghan~ Asia        28.8   30.3   32.0   34.0   36.1   38.4   39.9   40.8
##  2 Albania Europe      55.2   59.3   64.8   66.2   67.7   68.9   70.4   72  
##  3 Algeria Africa      43.1   45.7   48.3   51.4   54.5   58.0   61.4   65.8
##  4 Angola  Africa      30.0   32.0   34     36.0   37.9   39.5   39.9   39.9
##  5 Argent~ Americas    62.5   64.4   65.1   65.6   67.1   68.5   69.9   70.8
##  6 Austra~ Oceania     69.1   70.3   70.9   71.1   71.9   73.5   74.7   76.3
##  7 Austria Europe      66.8   67.5   69.5   70.1   70.6   72.2   73.2   74.9
##  8 Bahrain Asia        50.9   53.8   56.9   59.9   63.3   65.6   69.1   70.8
##  9 Bangla~ Asia        37.5   39.3   41.2   43.5   45.3   46.9   50.0   52.8
## 10 Belgium Europe      68     69.2   70.2   70.9   71.4   72.8   73.9   75.4
## # ... with 132 more rows, and 4 more variables: `1992` <dbl>, `1997` <dbl>,
## #   `2002` <dbl>, `2007` <dbl>

Similarly, if we wanted to convert from the wide gapminder to the long one, we would use either gather or pivot_longer()

gapminder_life_exp_wide %>% 
  gather(key = "year", value = "lifeExp",
         -country, -continent) %>% 
  mutate(year = as.numeric(year))

## # A tibble: 1,704 x 4
##    country     continent  year lifeExp
##    <fct>       <fct>     <dbl>   <dbl>
##  1 Afghanistan Asia       1952    28.8
##  2 Albania     Europe     1952    55.2
##  3 Algeria     Africa     1952    43.1
##  4 Angola      Africa     1952    30.0
##  5 Argentina   Americas   1952    62.5
##  6 Australia   Oceania    1952    69.1
##  7 Austria     Europe     1952    66.8
##  8 Bahrain     Asia       1952    50.9
##  9 Bangladesh  Asia       1952    37.5
## 10 Belgium     Europe     1952    68  
## # ... with 1,694 more rows

gapminder_life_exp_wide %>% 
  pivot_longer(
    cols = c(-country, -continent), #keep country and continent
    names_to = "year", 
    values_to = "lifeExp",
         ) %>% 
  mutate(year = as.numeric(year))

## # A tibble: 1,704 x 4
##    country     continent  year lifeExp
##    <fct>       <fct>     <dbl>   <dbl>
##  1 Afghanistan Asia       1952    28.8
##  2 Afghanistan Asia       1957    30.3
##  3 Afghanistan Asia       1962    32.0
##  4 Afghanistan Asia       1967    34.0
##  5 Afghanistan Asia       1972    36.1
##  6 Afghanistan Asia       1977    38.4
##  7 Afghanistan Asia       1982    39.9
##  8 Afghanistan Asia       1987    40.8
##  9 Afghanistan Asia       1992    41.7
## 10 Afghanistan Asia       1997    41.8
## # ... with 1,694 more rows

RStudio primer on tidyr

Reshape Data

Recent versions of tidyr have renamed these core functions: gather() is now pivot_longer() and spread() is now pivot_wider(). The syntax for these pivot_*() functions is slightly different from what it was in gather() and spread(), so you can’t just replace the names. Even though, both gather() and spread() still work and won’t go away for a while, I think it’s worth learning the newer pivot_*() functions.

More resources

Pivoting data from columns to rows (and back!) in the tidyverse
Pivoting in tidyr
Hadley Wickham’s tidy data paper
Data Organization in Spreadsheets

Side-by-side regression tables

Tue, 28 Jul 2020 00:00:00 +0000

huxtable::huxreg()

It’s often helpful to put the results from regression models in a side-by-side table so you can compare coefficients across different model specifications. If you’re unfamiliar with these kinds of tables, check out this helpful guide to how to read them.

We will use the huxreg() function in the huxtable package, the Palmer Penguins dataset, and a few regression models on trying to explain body_mass_g.

library(palmerpenguins)

model1 <- lm(body_mass_g ~ flipper_length_mm, data = penguins)
model2 <- lm(body_mass_g ~ flipper_length_mm + bill_length_mm, data = penguins)
model3 <- lm(body_mass_g ~ flipper_length_mm + bill_length_mm + species, data = penguins)
model4 <- lm(body_mass_g ~ flipper_length_mm + bill_length_mm + species + sex, data = penguins)

`huxtable::huxreg()`

Installing

huxtable is published on CRAN, so use the “Packages” panel in RStudio to install huxtable, or use:

install.packages("huxtable", dependencies = TRUE)

To knit to Word, you also need the flextable package, so install that too from the “Packages” panel, or use:

install.packages("flextable", dependencies = TRUE)

Using huxtable::huxreg() to knit to HTML and Word works very well.

Basic usage

As you go about builiding your linear models, you save them as object with different names, e.g., model1, model2, etc. Once you have the models you want to compate, pass them to huxreg():

library(huxtable)

huxreg(model1, model2, model3, model4)

	(1)	(2)	(3)	(4)
(Intercept)	-5780.831 ***	-5736.897 ***	-3904.387 ***	-759.064
	(305.815)	(307.959)	(529.257)	(541.377)
flipper_length_mm	49.686 ***	48.145 ***	27.429 ***	17.847 ***
	(1.518)	(2.011)	(3.176)	(2.902)
bill_length_mm		6.047	61.736 ***	21.633 **
		(5.180)	(7.126)	(7.148)
speciesChinstrap			-748.562 ***	-291.711 ***
			(81.534)	(81.502)
speciesGentoo			90.435	707.028 ***
			(88.647)	(94.359)
sexmale				465.395 ***
				(43.081)
N	342	342	342	333
R2	0.759	0.760	0.822	0.871
logLik	-2528.427	-2527.741	-2476.373	-2359.787
AIC	5062.855	5063.482	4964.745	4733.574
* p < 0.001; p < 0.01; * p < 0.05.

You can add column names, to make the output table more user friendly.

huxreg(list("A" = model1, "B" = model2, "C" = model3, "D" = model4))

	A	B	C	D
(Intercept)	-5780.831 ***	-5736.897 ***	-3904.387 ***	-759.064
	(305.815)	(307.959)	(529.257)	(541.377)
flipper_length_mm	49.686 ***	48.145 ***	27.429 ***	17.847 ***
	(1.518)	(2.011)	(3.176)	(2.902)
bill_length_mm		6.047	61.736 ***	21.633 **
		(5.180)	(7.126)	(7.148)
speciesChinstrap			-748.562 ***	-291.711 ***
			(81.534)	(81.502)
speciesGentoo			90.435	707.028 ***
			(88.647)	(94.359)
sexmale				465.395 ***
				(43.081)
N	342	342	342	333
R2	0.759	0.760	0.822	0.871
logLik	-2528.427	-2527.741	-2476.373	-2359.787
AIC	5062.855	5063.482	4964.745	4733.574
* p < 0.001; p < 0.01; * p < 0.05.

Statistics to display, bold significant variables, add captions

You can choose what statistics to display; for instance, if we wanted to show number of observatiosn, R^2 and adjusted R2, and the residual SE, we pass them to the statistics variable. We can also bold those variables that are significant at, say 0.05, and choose not to use stars to denote level of significance by using stars = NULL.

huxreg(model1, model2, model3, 
                 statistics = c('#observations' = 'nobs', 
                                'R squared' = 'r.squared', 
                                'Adj. R Squared' = 'adj.r.squared', 
                                'Residual SE' = 'sigma'), 
                 bold_signif = 0.05, 
                 stars = NULL
) %>% 
  set_caption('Comparison of models')

Comparison of models
	(1)	(2)	(3)
(Intercept)	-5780.831	-5736.897	-3904.387
	(305.815)	(307.959)	(529.257)
flipper_length_mm	49.686	48.145	27.429
	(1.518)	(2.011)	(3.176)
bill_length_mm		6.047	61.736
		(5.180)	(7.126)
speciesChinstrap			-748.562
			(81.534)
speciesGentoo			90.435
			(88.647)
#observations	342	342	342
R squared	0.759	0.760	0.822
Adj. R Squared	0.758	0.759	0.820
Residual SE	394.278	394.068	340.114

Warning 1: huxtable reformats all your tables

If your document creates any other tables (like with tidy()), huxtable automatically formats these tables in a fancy way. If you don’t want that, you can turn it off with this code—put it at the top of your document near where you load your libraries:

# Huxtable likes to automatically format *all* tables, which is annoying. 
# This turns that off.
options('huxtable.knit_print_df' = FALSE)

warning 2: Knitting to PDF is fragile

In order to knit to PDF, you need to install LaTeX, which you should have done when you installed tinytex. When using huxtable, before knitting to PDF for the first time on your computer, you need to run this in your console to install the LaTeX packages that R uses to knit huxtable tables to PDF:

huxtable::install_latex_dependencies()

If you’re using tinytex, you’ll also need to run this once on your computer:

tinytex::tlmgr_install("unicode-math")

Side-by-side regression tables

Tue, 28 Jul 2020 00:00:00 +0000

huxtable’s huxreg()

We will use the huxreg() function in the huxtable package, the Palmer Penguins dataset, and a few regression models on trying to explain body_mass_g.

library(palmerpenguins)

model1 <- lm(body_mass_g ~ flipper_length_mm, data = penguins)
model2 <- lm(body_mass_g ~ flipper_length_mm + bill_length_mm, data = penguins)
model3 <- lm(body_mass_g ~ flipper_length_mm + bill_length_mm + species, data = penguins)
model4 <- lm(body_mass_g ~ flipper_length_mm + bill_length_mm + species + sex, data = penguins)

huxtable’s `huxreg()`

Installing

huxtable is published on CRAN, so use the “Packages” panel in RStudio to install huxtable, or run this:

install.packages("huxtable", dependencies = TRUE)

To knit to Word, you also need the flextable package, and R doesn’t install that automatically for whatever reason, so install that too from the “Packages” panel, or run this too:

install.packages("flextable", dependencies = TRUE)

Basic usage

Feed huxreg() a bunch of models:

library(huxtable)

huxreg(model1, model2, model3, model4)

	(1)	(2)	(3)	(4)
(Intercept)	-5780.831 ***	-5736.897 ***	-3904.387 ***	-759.064
	(305.815)	(307.959)	(529.257)	(541.377)
flipper_length_mm	49.686 ***	48.145 ***	27.429 ***	17.847 ***
	(1.518)	(2.011)	(3.176)	(2.902)
bill_length_mm		6.047	61.736 ***	21.633 **
		(5.180)	(7.126)	(7.148)
speciesChinstrap			-748.562 ***	-291.711 ***
			(81.534)	(81.502)
speciesGentoo			90.435	707.028 ***
			(88.647)	(94.359)
sexmale				465.395 ***
				(43.081)
N	342	342	342	333
R2	0.759	0.760	0.822	0.871
logLik	-2528.427	-2527.741	-2476.373	-2359.787
AIC	5062.855	5063.482	4964.745	4733.574
* p < 0.001; p < 0.01; * p < 0.05.

Add column names:

huxreg(list("A" = model1, "B" = model2, "C" = model3, "D" = model4))

	A	B	C	D
(Intercept)	-5780.831 ***	-5736.897 ***	-3904.387 ***	-759.064
	(305.815)	(307.959)	(529.257)	(541.377)
flipper_length_mm	49.686 ***	48.145 ***	27.429 ***	17.847 ***
	(1.518)	(2.011)	(3.176)	(2.902)
bill_length_mm		6.047	61.736 ***	21.633 **
		(5.180)	(7.126)	(7.148)
speciesChinstrap			-748.562 ***	-291.711 ***
			(81.534)	(81.502)
speciesGentoo			90.435	707.028 ***
			(88.647)	(94.359)
sexmale				465.395 ***
				(43.081)
N	342	342	342	333
R2	0.759	0.760	0.822	0.871
logLik	-2528.427	-2527.741	-2476.373	-2359.787
AIC	5062.855	5063.482	4964.745	4733.574
* p < 0.001; p < 0.01; * p < 0.05.

Statistics to display, bold significant variables, add captions

huxreg(model1, model2, model3, 
                 statistics = c('#observations' = 'nobs', 
                                'R squared' = 'r.squared', 
                                'Adj. R Squared' = 'adj.r.squared', 
                                'Residual SE' = 'sigma'), 
                 bold_signif = 0.05, 
                 stars = NULL
) %>% 
  set_caption('Comparison of models')

Comparison of models
	(1)	(2)	(3)
(Intercept)	-5780.831	-5736.897	-3904.387
	(305.815)	(307.959)	(529.257)
flipper_length_mm	49.686	48.145	27.429
	(1.518)	(2.011)	(3.176)
bill_length_mm		6.047	61.736
		(5.180)	(7.126)
speciesChinstrap			-748.562
			(81.534)
speciesGentoo			90.435
			(88.647)
#observations	342	342	342
R squared	0.759	0.760	0.822
Adj. R Squared	0.758	0.759	0.820
Residual SE	394.278	394.068	340.114

Upside: HTML and Word

Knitting to HTML and Word works pretty flawlessly.

Downside 1: huxtable reformats all your tables

# Huxtable likes to automatically format *all* tables, which is annoying. 
# This turns that off.
options('huxtable.knit_print_df' = FALSE)

Downside 2: Knitting to PDF is fragile

In order to knit to PDF, you need to install LaTeX, which you did by installing tinytex. When using huxtable, before knitting to PDF for the first time on your computer, you need to run this in your console to install the LaTeX packages that R uses to knit huxtable tables to PDF:

huxtable::install_latex_dependencies()

If you’re using tinytex, you’ll also need to run this once on your computer:

tinytex::tlmgr_install("unicode-math")

Course materials | Data Analytics: Learning from Data

Homework 1

Pre-course assignment

Instructions

Finance Data

Finance data with the tidyquant package

Calculating financial returns

Summarising the data set

Minimum and maximum price of each stock by quarter

Sharpe Ratio

Investment Growth

Scatterplots of individual stocks returns versus S&P500 Index returns

Creating a portfolio of assets

Creating various portfolios by changing weights of assets

Data from the Federal Reserve Economic Data with tidyquant

Acknowledgments

Confidence Intervals

Confidence Intervals

Exploratory Data Analysis for Modelling

Exploratory Data Analysis for Modelling

EDA. Is Y skewed? Does it need to be transformed? Why?

GGally::ggpairs() to get scatter plot + correlation matrix

R Syntax, Vectors, missing data

R Syntax

Assignmnent Operator <-

R is case sensitive

Typos

Comments

R knows you’re not finished

Arithmetic Operations and Functions

Main Data types and Vectors

Your turn

Vectors

Manipulating vectors

Your turn

Factors

Your turn

Missing data, or NA

Your turn

Online Quiz: Variable Types - Vectors

R Syntax, Vectors, missing data

R Syntax

Assignmnent Operator <-

R is case sensitive

Typos

Comments

R knows you’re not finished

Arithmetic Operations and Functions

Main Data types and Vectors

Your turn

Vectors

Manipulating vectors

Your turn

Factors

Your turn

Missing data, or NA

Your turn

Online Quiz: Variable Types - Vectors

Visualise data

Introduction to ggplot2 by visualising numeric data.

Scatter plots and multiple panels using facet_wrap()

Animating changes

IMDB movie ratings: Scatterplots and relationships

IMDB movie ratings: Boxplots, violin plots

Multiple panels using facet_wrap() and facet_grid()

Visualise data

Introduction to ggplot2 by visualising numeric data.

Scatter plots and multiple panels using facet_wrap()

Animating changes

IMDB movie ratings: Scatterplots and relationships

IMDB movie ratings: Boxplots, violin plots

Multiple panels using facet_wrap() and facet_grid()

Installing R and RStudio

Installing R & RStudio

Install XCode if you have a Mac

Install R

Install RStudio IDE

Change character encoding to UTF-8, and UTF-8 only

Exiting R & RStudio

Updating R and RStudio

Finance data with the `tidyquant` package

Data from the Federal Reserve Economic Data with `tidyquant`

`GGally::ggpairs()` to get scatter plot + correlation matrix

Assignmnent Operator `<-`

Missing data, or `NA`

Assignmnent Operator `<-`

Missing data, or `NA`

Introduction to `ggplot2` by visualising numeric data.

Scatter plots and multiple panels using `facet_wrap()`

Multiple panels using `facet_wrap()` and `facet_grid()`

Introduction to `ggplot2` by visualising numeric data.

Scatter plots and multiple panels using `facet_wrap()`

Multiple panels using `facet_wrap()` and `facet_grid()`

Install `XCode` if you have a Mac

Change character encoding to `UTF-8`, and `UTF-8` only

Assignmnent Operator `<-`