workshopBasic data wrangling with tidyverse
The following packages are used in this section: dplyr, tibble, and teachingtools. dplyr and tibble are part of the tidyverse family.
To make sure that the packages used in this section are installed and loaded, run the following code in your console.
xfun::pkg_attach2("teachingtools", "tidyverse")In the last section we saw how we could select specific parts of data frames using logic. However, this syntax can sometimes be a little awkward. To try get around this awkwardness a different dialect of R has been developed. This dialect is know as the tidyverse. This is the dialect I prefer and it’s the dialect that we teach undergrads at Sussex.
The package dplyr is part of the the tidyverse. dplyr is based around a collection of verbs for manipulating data tables. The primary verbs for sub-setting data tables are select(), filter(), and pull().
Selecting
Warning! There’s another select() function which is part of the MASS package. MASS::select() has nothing to do with dplyr::select() and if you load the MASS after loading tidyverse then none of your selecting will work as expected. That’s why should always load the tidyverse last or explicitly specify dplyr::select()
While you might not always load MASS explicitly it does automatically load with a lot of popular packages. For example, the multcomp package for doing multiple comparisons automatically loads MASS
The data table mtcars contains data about different cars—for example, their horse power (hp), their fuel efficiency (mpg) etc.
mtcars mpg cyl disp hp drat wt qsec vs am gear carb
Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4
Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4
Datsun 710 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1
Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1
Hornet Sportabout 18.7 8 360.0 175 3.15 3.440 17.02 0 0 3 2
Valiant 18.1 6 225.0 105 2.76 3.460 20.22 1 0 3 1
Duster 360 14.3 8 360.0 245 3.21 3.570 15.84 0 0 3 4
Merc 240D 24.4 4 146.7 62 3.69 3.190 20.00 1 0 4 2
Merc 230 22.8 4 140.8 95 3.92 3.150 22.90 1 0 4 2
Merc 280 19.2 6 167.6 123 3.92 3.440 18.30 1 0 4 4
Merc 280C 17.8 6 167.6 123 3.92 3.440 18.90 1 0 4 4
Merc 450SE 16.4 8 275.8 180 3.07 4.070 17.40 0 0 3 3
Merc 450SL 17.3 8 275.8 180 3.07 3.730 17.60 0 0 3 3
Merc 450SLC 15.2 8 275.8 180 3.07 3.780 18.00 0 0 3 3
Cadillac Fleetwood 10.4 8 472.0 205 2.93 5.250 17.98 0 0 3 4
Lincoln Continental 10.4 8 460.0 215 3.00 5.424 17.82 0 0 3 4
Chrysler Imperial 14.7 8 440.0 230 3.23 5.345 17.42 0 0 3 4
Fiat 128 32.4 4 78.7 66 4.08 2.200 19.47 1 1 4 1
Honda Civic 30.4 4 75.7 52 4.93 1.615 18.52 1 1 4 2
Toyota Corolla 33.9 4 71.1 65 4.22 1.835 19.90 1 1 4 1
Toyota Corona 21.5 4 120.1 97 3.70 2.465 20.01 1 0 3 1
Dodge Challenger 15.5 8 318.0 150 2.76 3.520 16.87 0 0 3 2
AMC Javelin 15.2 8 304.0 150 3.15 3.435 17.30 0 0 3 2
Camaro Z28 13.3 8 350.0 245 3.73 3.840 15.41 0 0 3 4
Pontiac Firebird 19.2 8 400.0 175 3.08 3.845 17.05 0 0 3 2
Fiat X1-9 27.3 4 79.0 66 4.08 1.935 18.90 1 1 4 1
Porsche 914-2 26.0 4 120.3 91 4.43 2.140 16.70 0 1 5 2
Lotus Europa 30.4 4 95.1 113 3.77 1.513 16.90 1 1 5 2
Ford Pantera L 15.8 8 351.0 264 4.22 3.170 14.50 0 1 5 4
Ferrari Dino 19.7 6 145.0 175 3.62 2.770 15.50 0 1 5 6
Maserati Bora 15.0 8 301.0 335 3.54 3.570 14.60 0 1 5 8
Volvo 142E 21.4 4 121.0 109 4.11 2.780 18.60 1 1 4 2We saw that we could subset particular columns in a data table using []. For example, if we just wanted the mpg column from mtcars data set we could use the following code.
mtcars["mpg"] mpg
Mazda RX4 21.0
Mazda RX4 Wag 21.0
Datsun 710 22.8
Hornet 4 Drive 21.4
Hornet Sportabout 18.7
Valiant 18.1
Duster 360 14.3
Merc 240D 24.4
Merc 230 22.8
Merc 280 19.2
Merc 280C 17.8
Merc 450SE 16.4
Merc 450SL 17.3
Merc 450SLC 15.2
Cadillac Fleetwood 10.4
Lincoln Continental 10.4
Chrysler Imperial 14.7
Fiat 128 32.4
Honda Civic 30.4
Toyota Corolla 33.9
Toyota Corona 21.5
Dodge Challenger 15.5
AMC Javelin 15.2
Camaro Z28 13.3
Pontiac Firebird 19.2
Fiat X1-9 27.3
Porsche 914-2 26.0
Lotus Europa 30.4
Ford Pantera L 15.8
Ferrari Dino 19.7
Maserati Bora 15.0
Volvo 142E 21.4To do the equivalent in the tidyverse we’d use the dplyr verb select(). This would give the following:
dplyr::select(
.data = mtcars, # our data table
mpg # the column to keep
) mpg
Mazda RX4 21.0
Mazda RX4 Wag 21.0
Datsun 710 22.8
Hornet 4 Drive 21.4
Hornet Sportabout 18.7
Valiant 18.1
Duster 360 14.3
Merc 240D 24.4
Merc 230 22.8
Merc 280 19.2
Merc 280C 17.8
Merc 450SE 16.4
Merc 450SL 17.3
Merc 450SLC 15.2
Cadillac Fleetwood 10.4
Lincoln Continental 10.4
Chrysler Imperial 14.7
Fiat 128 32.4
Honda Civic 30.4
Toyota Corolla 33.9
Toyota Corona 21.5
Dodge Challenger 15.5
AMC Javelin 15.2
Camaro Z28 13.3
Pontiac Firebird 19.2
Fiat X1-9 27.3
Porsche 914-2 26.0
Lotus Europa 30.4
Ford Pantera L 15.8
Ferrari Dino 19.7
Maserati Bora 15.0
Volvo 142E 21.4You’ll notice that when using select(), the column names are not placed in "". This is because in the tidyverse, columns of a data table are treated as variables, and variable names don’t take "".
We can also select multiple columns by just listing more names. We just use commas (,) to separate them. For example, if we want to keep mpg and cyl we do the following.
dplyr::select(
.data = mtcars, # our data table
mpg, cyl # the columns to keep
) mpg cyl
Mazda RX4 21.0 6
Mazda RX4 Wag 21.0 6
Datsun 710 22.8 4
Hornet 4 Drive 21.4 6
Hornet Sportabout 18.7 8
Valiant 18.1 6
Duster 360 14.3 8
Merc 240D 24.4 4
Merc 230 22.8 4
Merc 280 19.2 6
Merc 280C 17.8 6
Merc 450SE 16.4 8
Merc 450SL 17.3 8
Merc 450SLC 15.2 8
Cadillac Fleetwood 10.4 8
Lincoln Continental 10.4 8
Chrysler Imperial 14.7 8
Fiat 128 32.4 4
Honda Civic 30.4 4
Toyota Corolla 33.9 4
Toyota Corona 21.5 4
Dodge Challenger 15.5 8
AMC Javelin 15.2 8
Camaro Z28 13.3 8
Pontiac Firebird 19.2 8
Fiat X1-9 27.3 4
Porsche 914-2 26.0 4
Lotus Europa 30.4 4
Ford Pantera L 15.8 8
Ferrari Dino 19.7 6
Maserati Bora 15.0 8
Volvo 142E 21.4 4In addition to asking select() to keep specific columns, we can also specify that we want to drop specific columns (and keep the rest). To do this, we just put a minus - in front of the name of the column we want to drop
dplyr::select(
.data = mtcars, # our data table
-mpg, -cyl # columns to drop (keep the rest!)
) disp hp drat wt qsec vs am gear carb
Mazda RX4 160.0 110 3.90 2.620 16.46 0 1 4 4
Mazda RX4 Wag 160.0 110 3.90 2.875 17.02 0 1 4 4
Datsun 710 108.0 93 3.85 2.320 18.61 1 1 4 1
Hornet 4 Drive 258.0 110 3.08 3.215 19.44 1 0 3 1
Hornet Sportabout 360.0 175 3.15 3.440 17.02 0 0 3 2
Valiant 225.0 105 2.76 3.460 20.22 1 0 3 1
Duster 360 360.0 245 3.21 3.570 15.84 0 0 3 4
Merc 240D 146.7 62 3.69 3.190 20.00 1 0 4 2
Merc 230 140.8 95 3.92 3.150 22.90 1 0 4 2
Merc 280 167.6 123 3.92 3.440 18.30 1 0 4 4
Merc 280C 167.6 123 3.92 3.440 18.90 1 0 4 4
Merc 450SE 275.8 180 3.07 4.070 17.40 0 0 3 3
Merc 450SL 275.8 180 3.07 3.730 17.60 0 0 3 3
Merc 450SLC 275.8 180 3.07 3.780 18.00 0 0 3 3
Cadillac Fleetwood 472.0 205 2.93 5.250 17.98 0 0 3 4
Lincoln Continental 460.0 215 3.00 5.424 17.82 0 0 3 4
Chrysler Imperial 440.0 230 3.23 5.345 17.42 0 0 3 4
Fiat 128 78.7 66 4.08 2.200 19.47 1 1 4 1
Honda Civic 75.7 52 4.93 1.615 18.52 1 1 4 2
Toyota Corolla 71.1 65 4.22 1.835 19.90 1 1 4 1
Toyota Corona 120.1 97 3.70 2.465 20.01 1 0 3 1
Dodge Challenger 318.0 150 2.76 3.520 16.87 0 0 3 2
AMC Javelin 304.0 150 3.15 3.435 17.30 0 0 3 2
Camaro Z28 350.0 245 3.73 3.840 15.41 0 0 3 4
Pontiac Firebird 400.0 175 3.08 3.845 17.05 0 0 3 2
Fiat X1-9 79.0 66 4.08 1.935 18.90 1 1 4 1
Porsche 914-2 120.3 91 4.43 2.140 16.70 0 1 5 2
Lotus Europa 95.1 113 3.77 1.513 16.90 1 1 5 2
Ford Pantera L 351.0 264 4.22 3.170 14.50 0 1 5 4
Ferrari Dino 145.0 175 3.62 2.770 15.50 0 1 5 6
Maserati Bora 301.0 335 3.54 3.570 14.60 0 1 5 8
Volvo 142E 121.0 109 4.11 2.780 18.60 1 1 4 2dplyr also includes a few helpers that make it easier to select columns in more complex situations. Let’s take the data table two_scales. It has columns that correspond to some Woodcock Johnson sub-scales and some that correspond to WISC sub-scales.
two_scales# A tibble: 10 x 7
id WCJ_1 WCJ_2 WCJ_3 WISC_1 WISC_2 WISC_3
<chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 s001 4 4 4 4 4 4
2 s002 8 8 8 8 8 8
3 s003 6 6 6 6 6 6
4 s004 0 0 0 0 0 0
5 s005 2 2 2 2 2 2
6 s006 2 2 2 2 2 2
7 s007 6 6 6 6 6 6
8 s008 2 2 2 2 2 2
9 s009 8 8 8 8 8 8
10 s010 1 1 1 1 1 1If we wanted to keep all the WISC columns we could write out all the names.
dplyr::select(
.data = two_scales,
id, WISC_1, WISC_2, WISC_3
)# A tibble: 10 x 4
id WISC_1 WISC_2 WISC_3
<chr> <dbl> <dbl> <dbl>
1 s001 4 4 4
2 s002 8 8 8
3 s003 6 6 6
4 s004 0 0 0
5 s005 2 2 2
6 s006 2 2 2
7 s007 6 6 6
8 s008 2 2 2
9 s009 8 8 8
10 s010 1 1 1But we know that we just want to keep id and everything that starts with WISC_. dplyr includes a helper called starts_with() for just this occasion. Let’s see starts_with() in action.
dplyr::select(
.data = two_scales, # our data table
id, # keep the id column
starts_with(match = "WISC") # and everything that starts with WISC
)# A tibble: 10 x 4
id WISC_1 WISC_2 WISC_3
<chr> <dbl> <dbl> <dbl>
1 s001 4 4 4
2 s002 8 8 8
3 s003 6 6 6
4 s004 0 0 0
5 s005 2 2 2
6 s006 2 2 2
7 s007 6 6 6
8 s008 2 2 2
9 s009 8 8 8
10 s010 1 1 1In addition to starts_with(), there’s also ends_with(), and contains().
For example, if we just wanted to keep subscale 2 of each scale.
dplyr::select(
.data = two_scales, # our data table
id, # keep the id column
ends_with(match = "2") # and everything that ends with 2
)# A tibble: 10 x 3
id WCJ_2 WISC_2
<chr> <dbl> <dbl>
1 s001 4 4
2 s002 8 8
3 s003 6 6
4 s004 0 0
5 s005 2 2
6 s006 2 2
7 s007 6 6
8 s008 2 2
9 s009 8 8
10 s010 1 1All the helpers have an additional argument called ignore.case. Let’s see this argument at work in conjunction with contains(). Let’s say we’re getting data from a collaborator and they’re inconsistently labelling the WISC scores. Sometimes it’s WISC, sometimes Wis, other times WechslerIS etc. But at least they’re consistently using some string that includes is although it’s sometimes upper case and sometime lower case. We can use contains() and ignore.case to help us select the correct columns without having to check the data table and edit our code each time.
dplyr::select(
.data = two_scales, # our data table
id, # keep the id column
contains(match = "is", ignore.case = T) # and everything that contains is/IS
)# A tibble: 10 x 4
id WISC_1 WISC_2 WISC_3
<chr> <dbl> <dbl> <dbl>
1 s001 4 4 4
2 s002 8 8 8
3 s003 6 6 6
4 s004 0 0 0
5 s005 2 2 2
6 s006 2 2 2
7 s007 6 6 6
8 s008 2 2 2
9 s009 8 8 8
10 s010 1 1 1Finally, when we’re dealing with columns that have numbering like our columns we can use the num_range() helper to help us select columns that end with numbers in a specific range. We just have to specify the prefix we want to check and the number range we want to keep.
dplyr::select(
.data = two_scales,
id, num_range(prefix = "WISC_", range = 2:3)
)# A tibble: 10 x 3
id WISC_2 WISC_3
<chr> <dbl> <dbl>
1 s001 4 4
2 s002 8 8
3 s003 6 6
4 s004 0 0
5 s005 2 2
6 s006 2 2
7 s007 6 6
8 s008 2 2
9 s009 8 8
10 s010 1 1Filtering
filter(), as the name suggests is for filtering data tables (think how you would apply a data filter in Excel). To see it in action, let’s look at an example.
Let’s say that we want to take a subset of the mtcars data table so that we only keep the cars that have a fuel efficiency of above 20 mpg. We could do it with the code below.
mtcars[mtcars$mpg > 20, ] mpg cyl disp hp drat wt qsec vs am gear carb
Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4
Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4
Datsun 710 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1
Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1
Merc 240D 24.4 4 146.7 62 3.69 3.190 20.00 1 0 4 2
Merc 230 22.8 4 140.8 95 3.92 3.150 22.90 1 0 4 2
Fiat 128 32.4 4 78.7 66 4.08 2.200 19.47 1 1 4 1
Honda Civic 30.4 4 75.7 52 4.93 1.615 18.52 1 1 4 2
Toyota Corolla 33.9 4 71.1 65 4.22 1.835 19.90 1 1 4 1
Toyota Corona 21.5 4 120.1 97 3.70 2.465 20.01 1 0 3 1
Fiat X1-9 27.3 4 79.0 66 4.08 1.935 18.90 1 1 4 1
Porsche 914-2 26.0 4 120.3 91 4.43 2.140 16.70 0 1 5 2
Lotus Europa 30.4 4 95.1 113 3.77 1.513 16.90 1 1 5 2
Volvo 142E 21.4 4 121.0 109 4.11 2.780 18.60 1 1 4 2To do the equivalent in the tidyverse way using the dplyr verb filter() we would do the following
dplyr::filter(
.data = mtcars, # our data table
mpg > 20 # our logical
) mpg cyl disp hp drat wt qsec vs am gear carb
Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4
Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4
Datsun 710 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1
Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1
Merc 240D 24.4 4 146.7 62 3.69 3.190 20.00 1 0 4 2
Merc 230 22.8 4 140.8 95 3.92 3.150 22.90 1 0 4 2
Fiat 128 32.4 4 78.7 66 4.08 2.200 19.47 1 1 4 1
Honda Civic 30.4 4 75.7 52 4.93 1.615 18.52 1 1 4 2
Toyota Corolla 33.9 4 71.1 65 4.22 1.835 19.90 1 1 4 1
Toyota Corona 21.5 4 120.1 97 3.70 2.465 20.01 1 0 3 1
Fiat X1-9 27.3 4 79.0 66 4.08 1.935 18.90 1 1 4 1
Porsche 914-2 26.0 4 120.3 91 4.43 2.140 16.70 0 1 5 2
Lotus Europa 30.4 4 95.1 113 3.77 1.513 16.90 1 1 5 2
Volvo 142E 21.4 4 121.0 109 4.11 2.780 18.60 1 1 4 2We can use all the same logic with filter() that we’d usually use with subsetting, including subsetting by multiple criteria.
dplyr::filter(
.data = mtcars, # our data table
mpg > 15 & disp > 250 # our two logical criteria
) mpg cyl disp hp drat wt qsec vs am gear carb
Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1
Hornet Sportabout 18.7 8 360.0 175 3.15 3.440 17.02 0 0 3 2
Merc 450SE 16.4 8 275.8 180 3.07 4.070 17.40 0 0 3 3
Merc 450SL 17.3 8 275.8 180 3.07 3.730 17.60 0 0 3 3
Merc 450SLC 15.2 8 275.8 180 3.07 3.780 18.00 0 0 3 3
Dodge Challenger 15.5 8 318.0 150 2.76 3.520 16.87 0 0 3 2
AMC Javelin 15.2 8 304.0 150 3.15 3.435 17.30 0 0 3 2
Pontiac Firebird 19.2 8 400.0 175 3.08 3.845 17.05 0 0 3 2
Ford Pantera L 15.8 8 351.0 264 4.22 3.170 14.50 0 1 5 4Pulling
You’d have noticed that select() returns a data table. That is, it’s the equivalent to using [] for selecting a column. But what if we want to do the equivalent to $.
Just to refresh you on the difference. We’ll use some data about Star Wars characters
starwars# A tibble: 87 x 14
name height mass hair_color skin_color eye_color birth_year sex gender
<chr> <int> <dbl> <chr> <chr> <chr> <dbl> <chr> <chr>
1 Luke… 172 77 blond fair blue 19 male mascu…
2 C-3PO 167 75 <NA> gold yellow 112 none mascu…
3 R2-D2 96 32 <NA> white, bl… red 33 none mascu…
4 Dart… 202 136 none white yellow 41.9 male mascu…
5 Leia… 150 49 brown light brown 19 fema… femin…
6 Owen… 178 120 brown, gr… light blue 52 male mascu…
7 Beru… 165 75 brown light blue 47 fema… femin…
8 R5-D4 97 32 <NA> white, red red NA none mascu…
9 Bigg… 183 84 black light brown 24 male mascu…
10 Obi-… 182 77 auburn, w… fair blue-gray 57 male mascu…
# … with 77 more rows, and 5 more variables: homeworld <chr>, species <chr>,
# films <list>, vehicles <list>, starships <list>starwars["height"] # equivalent to select(.data = starwars, height)# A tibble: 87 x 1
height
<int>
1 172
2 167
3 96
4 202
5 150
6 178
7 165
8 97
9 183
10 182
# … with 77 more rowsstarwars$height # returns just the values [1] 172 167 96 202 150 178 165 97 183 182 188 180 228 180 173 175 170 180 66
[20] 170 183 200 190 177 175 180 150 NA 88 160 193 191 170 196 224 206 183 137
[39] 112 183 163 175 180 178 94 122 163 188 198 196 171 184 188 264 188 196 185
[58] 157 183 183 170 166 165 193 191 183 168 198 229 213 167 79 96 193 191 178
[77] 216 234 188 178 206 NA NA NA NA NA 165To do the equivalent of $ in tidyverse syntax uses the pull() function.
dplyr::pull(
.data = starwars, # our data table
var = height # the column to pull
) [1] 172 167 96 202 150 178 165 97 183 182 188 180 228 180 173 175 170 180 66
[20] 170 183 200 190 177 175 180 150 NA 88 160 193 191 170 196 224 206 183 137
[39] 112 183 163 175 180 178 94 122 163 188 198 196 171 184 188 264 188 196 185
[58] 157 183 183 170 166 165 193 191 183 168 198 229 213 167 79 96 193 191 178
[77] 216 234 188 178 206 NA NA NA NA NA 165So just like $ the pull() function returns a vector. One of the neat things about pull(), however, is that instead of just returning a regular vector you can ask for a named vector to be returned.
named_vect <- dplyr::pull(
.data = starwars, # our data table
var = height, # the column to pull
name = name # the name of the column to get the names from
)
named_vect[["R2-D2"]] # subset our named vector by name[1] 96