project5.Rmd

---
title: "Chapter 2: Data manipulation using dplyr (part 1)"
description: |
  Learn how to manipulate your data with the dplyr package.
author:
  - name: Jewel Johnson 
date: 12-13-2021
output: 
  distill::distill_article:
    toc_depth: 3
preview: photos/dplyr.png
---
```{r setup, include=FALSE}
library(dplyr)
library(tidyr)
library(palmerpenguins)
library(rmarkdown)
library(knitr)
knitr::opts_chunk$set(echo = TRUE)
```

```{r, xaringanExtra-clipboard, echo=FALSE}
htmltools::tagList(
  xaringanExtra::use_clipboard(
    button_text = "<i class=\"fa fa-clone fa-2x\" style=\"color: #301e64\"></i>",
    success_text = "<i class=\"fa fa-check fa-2x\" style=\"color: #90BE6D\"></i>",
    error_text = "<i class=\"fa fa-times fa-2x\" style=\"color: #F94144\"></i>"
  ),
  rmarkdown::html_dependency_font_awesome()
)
```

---
# xaringanExtra package will help us to have inbuilt tabs insdie the article
---

```{r panelset, echo=FALSE}
xaringanExtra::use_panelset()
```

---
# loading javascipt file which helps in folding outputs similar to code folding
---
<script src="js/output_folding.js"></script>


---
#start editing from here
---

## Introduction to dplyr package

The `dplyr` package is a grammar of data manipulation just like how `ggplot2` is the grammar of data visualization. It helps us to apply a wide variety of functions such as;

1. Summarising the dataset
2. Applying selections and orderings as a function of a variable
3. Creating new variables as a function of existing variables

We will see in-depth how to manipulate our data like a boss!

## The pipe operator %>%

Perhaps the most amazing thing in making codes short and efficient is the pipe operator which is originally from the `magrittr` package which is made available for the `dplyr` package. The pipe operator helps you skip the intermediate steps of saving an object before you can use them in command. It does so by 'piping' together results from the first object to the function ahead of the pipe operator. The command `x %>% y %>% z` can be read as 'take the result of x and use it with function y and take that result and use it with function z'. This is the gist of what the pipe operator does. Allow me to demonstrate.

```{r, eval=FALSE}
library(ggplot2)
#dummy data
a <- c(sample(1:100, size = 50))
b <- c(sample(1:100, size = 50))
data <- as.data.frame(cbind(a,b))

#without %>%
data <- mutate(data, ab = a*b, twice_a = 2*a)
data_new <- filter(data, ab < 300, twice_a < 200)
ggplot(data_new, aes(ab, twice_a)) + geom_point()

#with %>%
data %>% mutate(ab = a*b, twice_a = 2*a) %>% 
  filter(ab < 300, twice_a < 200) %>%
  ggplot(aes(ab, twice_a)) + geom_point()
```

As you can see, with pipe operator `%>%`, we did not have to save any objects in the intermediate steps and also it improved the overall clarity of the code. I have used a few commands from the `dplyr` package in the example given above. So without further ado let us delve into the `dplyr` package. For this chapter, I will be using the penguin dataset from the popular `palmerpenguin` package as an example.

```{r, eval=FALSE}
#install palmerpenguins package
install.packages("palmerpenguins")
library(dplyr)
library(palmerpenguins)
```

## Grouping the data

### group_by()

The command `group_by()` allows us to group the data via existing variables. It allows for a 'split-apply-combine' way of getting output. First, it will split the data or group the data with the levels in the variable, then apply the function of our choice and then finally combine the results to give us a tabular output. On its own the command doesn't do anything, we use it in conjunction with other commands to get results based on the grouping we specify. The command `ungroup()` is used to ungroup the data.

## Summarising the data

### summarise()

The `summarise()` command allows you to get the summary statistics of a variable or a column in the dataset. The result is given as tabular data. Many types of summary statistics can be obtained using the `summarise()` function. Some of them are given below. To calculate average values it is necessary to drop `NA` values from the dataset. Use `drop_na()` command from the `tidyr` package. The comments denote what each summary statistic is.

```{r}
library(tidyr)
summary_data <- penguins %>% drop_na() %>%
  group_by(species) %>% # we are grouping/splitting the data according to species
  summarise(avg_mass = mean(body_mass_g), #mean mass
            median_mass = median(body_mass_g), #median mass
            max_mass = max(body_mass_g), #max value of mass, can also use min()
            standard_deviation_bill_length = sd(bill_length_mm), #standard deviation of bill_length
            sum_mass = sum(flipper_length_mm), #sum
            distinct_years = n_distinct(year), #distinct values in column year
            no_of_non_NAs = sum(!is.na(year)), #gives no of non NAs, 
            length_rows = n(), #length of the rows
            iqr_mass = IQR(body_mass_g), #inter quartile range of mass
            median_absolute_deviation_mass = mad(body_mass_g), #median absolute deviation of mass
            variance_mass = var(body_mass_g)) # variance
#viewing summary as a table
paged_table(summary_data)
```

The number of non `NA` values will be the same as that of `n()` result as we have used `drop_na()`command in the beginning.

The base function `summary()` in R also gives the whole summary statistics of a dataset

<div class="fold o">
```{r}
summary(penguins)
```
</div> 

## when to use group_by()

It can be confusing to decide when to use the `group_by()` function. In short, you should use it whenever you want any function to act separately on different groups present in the dataset. Here is a graphical representation of how the `summarise()` function is used to calculate the mean values of a dataset. When used with `group_by()` it calculates mean values for the respective groups in the data, but when `group_by()` is not used, it will calculate the mean value of the entire dataset irrespective of the different groups present and outputs a single column.

```{r echo=FALSE}
knitr::include_graphics('photos/group_by.png')
```

### count()

The `count()` command is used to count the number of rows of a variable. Has the same function as that of `n()`

```{r}
count <- penguins %>% group_by(species) %>%
  count(island)
#viewing count as a table
paged_table(count)
```

## Manipulating cases or observations

The following functions affect rows to give a subset of rows in a new table as output.

### filter()

Use `filter()` to filter rows corresponding to a given logical criteria


```{r, eval=FALSE}
penguins %>% filter(body_mass_g < 3000)
```

```{r, echo=FALSE}
paged_table(penguins %>% filter(body_mass_g < 3000))
```


### distinct()

Use `distinct()` to remove rows with duplicate or same values.

```{r, eval=FALSE}
penguins %>% group_by(species) %>% distinct(body_mass_g)
```

```{r, echo=FALSE}
paged_table(penguins %>% group_by(species) %>% distinct(body_mass_g))
```

### slice()

Use `slice()` to select rows by position. 

```{r, eval=FALSE}
penguins %>% slice(1:5) #slice from first row to fifth row
```

```{r, echo=FALSE}
paged_table(penguins %>% slice(1:5)) #slice from first row to fifth row
```

### slice_sample()

Use `slice_sample()` to randomly select rows from the dataset. Instead of `(n = )` you can also provide the proportion value (between 0 and 1) using `(prop = )`. For e.g. for a dataset with 10 rows, giving `(prop = 0.5)` will randomly sample 5 rows. Other related functions include;

* `preserve` : Values include `TRUE` to preserve grouping in a grouped dataset and `FALSE` to not preserve grouping while sampling.
* `weight_by` : Gives priority to a particular variable during sampling. An example is given below.
* `replace` : Values include `TRUE` if you want sampling with replacement which can result in duplicate values, `FALSE` if you want sampling without replacement.

::: {.l-page}
::: {.panelset}
::: {.panel}
[n = 4]{.panel-name}

```{r, eval=FALSE}
#samples 4 rows randomly
penguins %>% slice_sample(n = 4)
```

```{r, echo=FALSE}
#samples 4 rows randomly
paged_table(penguins %>% slice_sample(n = 4))
```

:::
::: {.panel}
[weight_by]{.panel-name}

```{r, eval=FALSE}
#sampling will favour rows with higher values of 'body_mass_g'
penguins %>% drop_na() %>% slice_sample(n = 4, weight_by = body_mass_g)
```

```{r, echo=FALSE}
#sampling will favour rows with higher values of 'body_mass_g'
paged_table(penguins %>% drop_na() %>% slice_sample(n = 4, weight_by = body_mass_g))
```

:::
:::
:::

### slice_min() and slice_max()

Use `slice_min()` to extract rows containing least values and use `slice_max()` to extract rows with greatest values. The function `with_ties = FALSE` is included to avoid tie values.

::: {.l-page}
::: {.panelset}
::: {.panel}
[slice_min()]{.panel-name}

```{r, eval=FALSE}
#first 4 rows containing least value in body mass
penguins %>% slice_min(body_mass_g, n = 4, with_ties = FALSE)
```

```{r, echo=FALSE}
paged_table(penguins %>% slice_min(body_mass_g, n = 4, with_ties = FALSE))
```

:::
::: {.panel}
[slice_max()]{.panel-name}

```{r, eval=FALSE}
#first 4 rows containing least value in body mass
penguins %>% slice_max(body_mass_g, n = 4, with_ties = FALSE)
```

```{r, echo=FALSE}
paged_table(penguins %>% slice_max(body_mass_g, n = 4,with_ties = FALSE))
```

:::
:::
:::

### slice_head() and slice_tail()

Use `slice_head()` to extract first set of rows and use `slice_tail()` to extract last set of rows.

::: {.l-page}
::: {.panelset}
::: {.panel}
[slice_head()]{.panel-name}

```{r, eval=FALSE}
#samples first 4 rows
penguins %>% slice_head(n = 4)
```

```{r, echo=FALSE}
paged_table(penguins %>% slice_head(n = 4))
```

:::
::: {.panel}
[slice_tail()]{.panel-name}

```{r, eval=FALSE}
#samples last 4 rows
penguins %>% slice_tail(n = 4)
```

```{r, echo=FALSE}
paged_table(penguins %>% slice_tail(n = 4))
```

:::
:::
:::

### arrange()

Use `arrange()` to arrange rows in a particular order.

```{r, eval=FALSE}
#arranging rows in descending order of bill length
#by default it arranges data by ascending order when no specifications are given
penguins %>% arrange(desc(bill_length_mm))
```


```{r, echo=FALSE}
#arranging rows in descending order of bill length
#by default it arranges data by ascending order when no specifications are given
paged_table(penguins %>% arrange(desc(bill_length_mm)))
```


### add_row()

Use `add_row()` to add rows to the dataset.

```{r, eval=FALSE}
Name <- c("a", "b")
Age <- c(12,13)
data.frame(Name, Age) %>% add_row(Name = "c", Age = 15)
```

```{r, echo=FALSE}
Name <- c("a", "b")
Age <- c(12,13)
paged_table(data.frame(Name, Age) %>% add_row(Name = "c", Age = 15))
```

## Manipulating variables or columns

The following functions affect columns to give a subset of columns in a new table as output.

### pull()

Use `pull()` to extract columns as a vector, by name or index. Only the first 10 results are shown for easy viewing.

```{r, R.options=list(max.print=10)}
penguins %>% pull(body_mass_g)
```

### select()

Use `select()` to extract columns as tables, by name or index.

```{r, eval=FALSE}
penguins %>% select(species, body_mass_g)
```


```{r, echo=FALSE}
paged_table(penguins %>% select(species, body_mass_g))
```


### relocate()

Use `relocate()` to move columns to new position. Results are not shown as these are trivial results.

```{r, eval=FALSE}
#relocates 'species' column to last position
penguins %>% relocate(species, .after = last_col())
```

```{r, echo=FALSE, eval=FALSE}
#relocates 'species' column to last position
paged_table(penguins %>% relocate(species, .after = last_col()))
```

```{r, eval=FALSE}
#relocates 'species' column before column 'year' and renames the column as 'penguins'
penguins %>% relocate(penguins = species, .before = year)
```

```{r, echo=FALSE, eval=FALSE}
#relocates 'species' column before column 'year' and renames the column as 'penguins'
paged_table(penguins %>% relocate(penguins = species, .before = year))
```

```{r, eval=FALSE}
#you can also relocate columns based on their class
#relocates all columns with 'character' class to last position
penguins %>% relocate(where(is.character), .after = last_col())
```

```{r, echo=FALSE, eval=FALSE}
#you can also relocate columns based on their class
#relocates all columns with 'character' class to last position
penguins %>% relocate(where(is.character), .after = last_col())
```

### rename()

Use `rename()` function to rename column names in the dataset.

<div class="fold o">
```{r}
#renames the column sex to gender
penguins %>% rename(gender = sex)
```
</div>

### mutate()

Use `mutate()` function to create new columns or variables.

```{r, eval=FALSE}
penguins %>% drop_na() %>% 
  group_by(species) %>%
  mutate(mean_mass = mean(body_mass_g))
```

```{r, echo=FALSE}
paged_table(penguins %>% drop_na() %>% 
  group_by(species) %>%
  mutate(mean_mass = mean(body_mass_g)))
```


### transmute()

Does the same function as `mutate()` but in the process will drop any other columns and give you a table with only the newly created columns.

```{r, eval=FALSE}
penguins %>% drop_na() %>% 
  group_by(species) %>%
  transmute(mean_mass = mean(body_mass_g))
```

```{r, echo=FALSE}
paged_table(penguins %>% drop_na() %>% 
  group_by(species) %>%
  transmute(mean_mass = mean(body_mass_g)))
```


### across()

Use `across()` to summarise or mutate columns in the same way. First example shows `across()` used with `summarise()` function.

```{r, eval=FALSE}
#summarise across columns body mass, bill length and bill depth
#and calculate the mean values
#since we are calculating mean values,
#NAs are dropped using 'drop_na() function from 'tidyr' package

penguins %>% drop_na() %>%
  group_by(species) %>%
  summarise(across(c(body_mass_g, bill_length_mm, bill_depth_mm), mean))
```

```{r, echo=FALSE}
paged_table(penguins %>% drop_na() %>%
  group_by(species) %>%
  summarise(across(c(body_mass_g, bill_length_mm, bill_depth_mm), mean)))
```

Second example showing `across()` used with `mutate()` function. We can efficiently create new columns using `mutate()` and `across()` together. Suppose we want to multiply all numerical values in a dataset with 2 and create new columns of those values. This can be done using the code below.

```{r, eval=FALSE}
# define the function
two_times <- function(x) {
  2*x
} 

# .name will rename the new columns with 'twice` prefix combined with existing col names
penguins %>% group_by(species) %>%
  mutate(across(where(is.numeric), two_times, .names = "two_times_{col}"))
```


```{r, echo=FALSE}
# define the function
two_times <- function(x) {
  2*x
} 

# .name will rename the new columns with 'twice` prefix combined with existing col names
paged_table(penguins %>% group_by(species) %>%
  mutate(across(where(is.numeric), two_times, .names = "two_times_{col}")))
```


The same code when used just with `mutate()` function will look like this

```{r, eval=FALSE}
# define the function
two_times <- function(x) {
  2*x
}

#using only 'mutate()' function
penguins %>% group_by(species) %>%
  mutate(twice_bill_lenght = two_times(bill_length_mm),
         twice_body_mass = two_times(body_mass_g),
         .....)
```

So in this code, I will have to manually type all the col names and apply the operation individually which is too much of a hassle. Now we can better appreciate how efficient it is in using `mutate()` and `across()` functions together.

### c_across()

The function `c_across()` is similar to the earlier mentioned `across()` function. But instead of doing a column-wise function, it applies function across columns in a row-wise manner. Now, most functions in R by default computes across columns, so to specify row-wise computation, we have to explicitly use the function `rowwise()` in conjunction with other functions. In the example below we will sum both bill and flipper lengths of the penguins in the `penguins` dataset and create a new column called 'sum_of_lengths'

```{r, eval=FALSE}
penguins %>% drop_na() %>%
  group_by(species) %>%
  rowwise() %>%
  transmute(sum_of_length = sum(c_across(c(bill_length_mm,flipper_length_mm))))
```

```{r, echo=FALSE}
paged_table(penguins %>% drop_na() %>%
  group_by(species) %>%
  rowwise() %>%
  transmute(sum_of_length = sum(c_across(c(bill_length_mm,flipper_length_mm)))))
```


## Summary

The `dplyr` package is the grammar of the data manipulation in R. It features well-made functions to help us summarise the data, group data by variables and manipulate columns and rows in the dataset. In this chapter, we learned in detail the different functions that help us manipulate data efficiently and have seen case examples also. In the next chapter, we will see the remaining set of functions in the `dplyr` package. See you!

<br>  

<a href="project6.html" class="btn button_round" style="float: right;">Next chapter:<br> 3. Data manipulation using dplyr (part 2)</a>
<a href="project4.html" class="btn button_round" style="float: left;">Previous chapter:<br> 1. Data tidying using tidyr</a>

<!-- adding share buttons on the right side of the page -->
<!-- AddToAny BEGIN -->
<div class="a2a_kit a2a_kit_size_32 a2a_floating_style a2a_vertical_style" style="right:0px; top:150px; data-a2a-url="https://jeweljohnsonj.github.io/jeweljohnson.github.io/" data-a2a-title="One-carat Blog">
<a class="a2a_button_twitter"></a>
<a class="a2a_button_whatsapp"></a>
<a class="a2a_button_telegram"></a>
<a class="a2a_button_google_gmail"></a>
<a class="a2a_button_pinterest"></a>
<a class="a2a_button_reddit"></a>
<a class="a2a_button_facebook"></a>
<a class="a2a_button_facebook_messenger"></a>
</div>
<script>
var a2a_config = a2a_config || {};
a2a_config.onclick = 1;
</script>
<script async src="https://static.addtoany.com/menu/page.js"></script>
<!-- AddToAny END -->

## References

1. Hadley Wickham, Romain François, Lionel Henry and Kirill Müller (2021). dplyr: A Grammar of Data Manipulation. R
  package version 1.0.7. https://CRAN.R-project.org/package=dplyr
  Here is the [link](https://github.com/rstudio/cheatsheets/blob/main/data-transformation.pdf) to the cheat sheet explaining each function in `dplyr`.
  
2. Horst AM, Hill AP, Gorman KB (2020). palmerpenguins: Palmer Archipelago (Antarctica) penguin data. R package version
  0.1.0. https://allisonhorst.github.io/palmerpenguins/
  
3. Hadley Wickham (2021). tidyr: Tidy Messy Data. R package version 1.1.4. https://CRAN.R-project.org/package=tidyr

## Last updated on {.appendix}

```{r, echo=FALSE}
Sys.time()
```