×

The following packages are used in this section: dplyr, tibble, and teachingtools. dplyr and tibble are part of the tidyverse family.

To make sure that the packages used in this section are installed and loaded, run the following code in your console.

xfun::pkg_attach2("teachingtools", "tidyverse")

In the last section we saw how we could select specific parts of data frames using logic. However, this syntax can sometimes be a little awkward. To try get around this awkwardness a different dialect of R has been developed. This dialect is know as the tidyverse. This is the dialect I prefer and it’s the dialect that we teach undergrads at Sussex.

The package dplyr is part of the the tidyverse. dplyr is based around a collection of verbs for manipulating data tables. The primary verbs for sub-setting data tables are select(), filter(), and pull().

# Selecting

×

Warning! There’s another select() function which is part of the MASS package. MASS::select() has nothing to do with dplyr::select() and if you load the MASS after loading tidyverse then none of your selecting will work as expected. That’s why should always load the tidyverse last or explicitly specify dplyr::select()

While you might not always load MASS explicitly it does automatically load with a lot of popular packages. For example, the multcomp package for doing multiple comparisons automatically loads MASS

The data table mtcars contains data about different cars—for example, their horse power (hp), their fuel efficiency (mpg) etc.

mtcars
                     mpg cyl  disp  hp drat    wt  qsec vs am gear carb
Mazda RX4           21.0   6 160.0 110 3.90 2.620 16.46  0  1    4    4
Mazda RX4 Wag       21.0   6 160.0 110 3.90 2.875 17.02  0  1    4    4
Datsun 710          22.8   4 108.0  93 3.85 2.320 18.61  1  1    4    1
Hornet 4 Drive      21.4   6 258.0 110 3.08 3.215 19.44  1  0    3    1
Hornet Sportabout   18.7   8 360.0 175 3.15 3.440 17.02  0  0    3    2
Valiant             18.1   6 225.0 105 2.76 3.460 20.22  1  0    3    1
Duster 360          14.3   8 360.0 245 3.21 3.570 15.84  0  0    3    4
Merc 240D           24.4   4 146.7  62 3.69 3.190 20.00  1  0    4    2
Merc 230            22.8   4 140.8  95 3.92 3.150 22.90  1  0    4    2
Merc 280            19.2   6 167.6 123 3.92 3.440 18.30  1  0    4    4
Merc 280C           17.8   6 167.6 123 3.92 3.440 18.90  1  0    4    4
Merc 450SE          16.4   8 275.8 180 3.07 4.070 17.40  0  0    3    3
Merc 450SL          17.3   8 275.8 180 3.07 3.730 17.60  0  0    3    3
Merc 450SLC         15.2   8 275.8 180 3.07 3.780 18.00  0  0    3    3
Cadillac Fleetwood  10.4   8 472.0 205 2.93 5.250 17.98  0  0    3    4
Lincoln Continental 10.4   8 460.0 215 3.00 5.424 17.82  0  0    3    4
Chrysler Imperial   14.7   8 440.0 230 3.23 5.345 17.42  0  0    3    4
Fiat 128            32.4   4  78.7  66 4.08 2.200 19.47  1  1    4    1
Honda Civic         30.4   4  75.7  52 4.93 1.615 18.52  1  1    4    2
Toyota Corolla      33.9   4  71.1  65 4.22 1.835 19.90  1  1    4    1
Toyota Corona       21.5   4 120.1  97 3.70 2.465 20.01  1  0    3    1
Dodge Challenger    15.5   8 318.0 150 2.76 3.520 16.87  0  0    3    2
AMC Javelin         15.2   8 304.0 150 3.15 3.435 17.30  0  0    3    2
Camaro Z28          13.3   8 350.0 245 3.73 3.840 15.41  0  0    3    4
Pontiac Firebird    19.2   8 400.0 175 3.08 3.845 17.05  0  0    3    2
Fiat X1-9           27.3   4  79.0  66 4.08 1.935 18.90  1  1    4    1
Porsche 914-2       26.0   4 120.3  91 4.43 2.140 16.70  0  1    5    2
Lotus Europa        30.4   4  95.1 113 3.77 1.513 16.90  1  1    5    2
Ford Pantera L      15.8   8 351.0 264 4.22 3.170 14.50  0  1    5    4
Ferrari Dino        19.7   6 145.0 175 3.62 2.770 15.50  0  1    5    6
Maserati Bora       15.0   8 301.0 335 3.54 3.570 14.60  0  1    5    8
Volvo 142E          21.4   4 121.0 109 4.11 2.780 18.60  1  1    4    2

We saw that we could subset particular columns in a data table using []. For example, if we just wanted the mpg column from mtcars data set we could use the following code.

mtcars["mpg"]
                     mpg
Mazda RX4           21.0
Mazda RX4 Wag       21.0
Datsun 710          22.8
Hornet 4 Drive      21.4
Valiant             18.1
Duster 360          14.3
Merc 240D           24.4
Merc 230            22.8
Merc 280            19.2
Merc 280C           17.8
Merc 450SE          16.4
Merc 450SL          17.3
Merc 450SLC         15.2
Lincoln Continental 10.4
Chrysler Imperial   14.7
Fiat 128            32.4
Honda Civic         30.4
Toyota Corolla      33.9
Toyota Corona       21.5
Dodge Challenger    15.5
AMC Javelin         15.2
Camaro Z28          13.3
Pontiac Firebird    19.2
Fiat X1-9           27.3
Porsche 914-2       26.0
Lotus Europa        30.4
Ford Pantera L      15.8
Ferrari Dino        19.7
Maserati Bora       15.0
Volvo 142E          21.4

To do the equivalent in the tidyverse we’d use the dplyr verb select(). This would give the following:

dplyr::select(
.data = mtcars, # our data table
mpg # the column to keep
)
                     mpg
Mazda RX4           21.0
Mazda RX4 Wag       21.0
Datsun 710          22.8
Hornet 4 Drive      21.4
Valiant             18.1
Duster 360          14.3
Merc 240D           24.4
Merc 230            22.8
Merc 280            19.2
Merc 280C           17.8
Merc 450SE          16.4
Merc 450SL          17.3
Merc 450SLC         15.2
Lincoln Continental 10.4
Chrysler Imperial   14.7
Fiat 128            32.4
Honda Civic         30.4
Toyota Corolla      33.9
Toyota Corona       21.5
Dodge Challenger    15.5
AMC Javelin         15.2
Camaro Z28          13.3
Pontiac Firebird    19.2
Fiat X1-9           27.3
Porsche 914-2       26.0
Lotus Europa        30.4
Ford Pantera L      15.8
Ferrari Dino        19.7
Maserati Bora       15.0
Volvo 142E          21.4

You’ll notice that when using select(), the column names are not placed in "". This is because in the tidyverse, columns of a data table are treated as variables, and variable names don’t take "".

We can also select multiple columns by just listing more names. We just use commas (,) to separate them. For example, if we want to keep mpg and cyl we do the following.

dplyr::select(
.data = mtcars, # our data table
mpg, cyl # the columns to keep
)
                     mpg cyl
Mazda RX4           21.0   6
Mazda RX4 Wag       21.0   6
Datsun 710          22.8   4
Hornet 4 Drive      21.4   6
Valiant             18.1   6
Duster 360          14.3   8
Merc 240D           24.4   4
Merc 230            22.8   4
Merc 280            19.2   6
Merc 280C           17.8   6
Merc 450SE          16.4   8
Merc 450SL          17.3   8
Merc 450SLC         15.2   8
Lincoln Continental 10.4   8
Chrysler Imperial   14.7   8
Fiat 128            32.4   4
Honda Civic         30.4   4
Toyota Corolla      33.9   4
Toyota Corona       21.5   4
Dodge Challenger    15.5   8
AMC Javelin         15.2   8
Camaro Z28          13.3   8
Pontiac Firebird    19.2   8
Fiat X1-9           27.3   4
Porsche 914-2       26.0   4
Lotus Europa        30.4   4
Ford Pantera L      15.8   8
Ferrari Dino        19.7   6
Maserati Bora       15.0   8
Volvo 142E          21.4   4

In addition to asking select() to keep specific columns, we can also specify that we want to drop specific columns (and keep the rest). To do this, we just put a minus - in front of the name of the column we want to drop

dplyr::select(
.data = mtcars, # our data table
-mpg, -cyl # columns to drop (keep the rest!)
)
                     disp  hp drat    wt  qsec vs am gear carb
Mazda RX4           160.0 110 3.90 2.620 16.46  0  1    4    4
Mazda RX4 Wag       160.0 110 3.90 2.875 17.02  0  1    4    4
Datsun 710          108.0  93 3.85 2.320 18.61  1  1    4    1
Hornet 4 Drive      258.0 110 3.08 3.215 19.44  1  0    3    1
Hornet Sportabout   360.0 175 3.15 3.440 17.02  0  0    3    2
Valiant             225.0 105 2.76 3.460 20.22  1  0    3    1
Duster 360          360.0 245 3.21 3.570 15.84  0  0    3    4
Merc 240D           146.7  62 3.69 3.190 20.00  1  0    4    2
Merc 230            140.8  95 3.92 3.150 22.90  1  0    4    2
Merc 280            167.6 123 3.92 3.440 18.30  1  0    4    4
Merc 280C           167.6 123 3.92 3.440 18.90  1  0    4    4
Merc 450SE          275.8 180 3.07 4.070 17.40  0  0    3    3
Merc 450SL          275.8 180 3.07 3.730 17.60  0  0    3    3
Merc 450SLC         275.8 180 3.07 3.780 18.00  0  0    3    3
Cadillac Fleetwood  472.0 205 2.93 5.250 17.98  0  0    3    4
Lincoln Continental 460.0 215 3.00 5.424 17.82  0  0    3    4
Chrysler Imperial   440.0 230 3.23 5.345 17.42  0  0    3    4
Fiat 128             78.7  66 4.08 2.200 19.47  1  1    4    1
Honda Civic          75.7  52 4.93 1.615 18.52  1  1    4    2
Toyota Corolla       71.1  65 4.22 1.835 19.90  1  1    4    1
Toyota Corona       120.1  97 3.70 2.465 20.01  1  0    3    1
Dodge Challenger    318.0 150 2.76 3.520 16.87  0  0    3    2
AMC Javelin         304.0 150 3.15 3.435 17.30  0  0    3    2
Camaro Z28          350.0 245 3.73 3.840 15.41  0  0    3    4
Pontiac Firebird    400.0 175 3.08 3.845 17.05  0  0    3    2
Fiat X1-9            79.0  66 4.08 1.935 18.90  1  1    4    1
Porsche 914-2       120.3  91 4.43 2.140 16.70  0  1    5    2
Lotus Europa         95.1 113 3.77 1.513 16.90  1  1    5    2
Ford Pantera L      351.0 264 4.22 3.170 14.50  0  1    5    4
Ferrari Dino        145.0 175 3.62 2.770 15.50  0  1    5    6
Maserati Bora       301.0 335 3.54 3.570 14.60  0  1    5    8
Volvo 142E          121.0 109 4.11 2.780 18.60  1  1    4    2

dplyr also includes a few helpers that make it easier to select columns in more complex situations. Let’s take the data table two_scales. It has columns that correspond to some Woodcock Johnson sub-scales and some that correspond to WISC sub-scales.

two_scales
# A tibble: 10 x 7
id    WCJ_1 WCJ_2 WCJ_3 WISC_1 WISC_2 WISC_3
<chr> <dbl> <dbl> <dbl>  <dbl>  <dbl>  <dbl>
1 s001      4     4     4      4      4      4
2 s002      8     8     8      8      8      8
3 s003      6     6     6      6      6      6
4 s004      0     0     0      0      0      0
5 s005      2     2     2      2      2      2
6 s006      2     2     2      2      2      2
7 s007      6     6     6      6      6      6
8 s008      2     2     2      2      2      2
9 s009      8     8     8      8      8      8
10 s010      1     1     1      1      1      1

If we wanted to keep all the WISC columns we could write out all the names.

dplyr::select(
.data = two_scales,
id, WISC_1, WISC_2, WISC_3
)
# A tibble: 10 x 4
id    WISC_1 WISC_2 WISC_3
<chr>  <dbl>  <dbl>  <dbl>
1 s001       4      4      4
2 s002       8      8      8
3 s003       6      6      6
4 s004       0      0      0
5 s005       2      2      2
6 s006       2      2      2
7 s007       6      6      6
8 s008       2      2      2
9 s009       8      8      8
10 s010       1      1      1

But we know that we just want to keep id and everything that starts with WISC_. dplyr includes a helper called starts_with() for just this occasion. Let’s see starts_with() in action.

dplyr::select(
.data = two_scales, # our data table
id, # keep the id column
starts_with(match = "WISC") # and everything that starts with WISC
)
# A tibble: 10 x 4
id    WISC_1 WISC_2 WISC_3
<chr>  <dbl>  <dbl>  <dbl>
1 s001       4      4      4
2 s002       8      8      8
3 s003       6      6      6
4 s004       0      0      0
5 s005       2      2      2
6 s006       2      2      2
7 s007       6      6      6
8 s008       2      2      2
9 s009       8      8      8
10 s010       1      1      1

In addition to starts_with(), there’s also ends_with(), and contains().

For example, if we just wanted to keep subscale 2 of each scale.

dplyr::select(
.data = two_scales, # our data table
id, # keep the id column
ends_with(match = "2") # and everything that ends with 2
)
# A tibble: 10 x 3
id    WCJ_2 WISC_2
<chr> <dbl>  <dbl>
1 s001      4      4
2 s002      8      8
3 s003      6      6
4 s004      0      0
5 s005      2      2
6 s006      2      2
7 s007      6      6
8 s008      2      2
9 s009      8      8
10 s010      1      1

All the helpers have an additional argument called ignore.case. Let’s see this argument at work in conjunction with contains(). Let’s say we’re getting data from a collaborator and they’re inconsistently labelling the WISC scores. Sometimes it’s WISC, sometimes Wis, other times WechslerIS etc. But at least they’re consistently using some string that includes is although it’s sometimes upper case and sometime lower case. We can use contains() and ignore.case to help us select the correct columns without having to check the data table and edit our code each time.

dplyr::select(
.data = two_scales, # our data table
id, # keep the id column
contains(match = "is", ignore.case = T) # and everything that contains is/IS
)
# A tibble: 10 x 4
id    WISC_1 WISC_2 WISC_3
<chr>  <dbl>  <dbl>  <dbl>
1 s001       4      4      4
2 s002       8      8      8
3 s003       6      6      6
4 s004       0      0      0
5 s005       2      2      2
6 s006       2      2      2
7 s007       6      6      6
8 s008       2      2      2
9 s009       8      8      8
10 s010       1      1      1

Finally, when we’re dealing with columns that have numbering like our columns we can use the num_range() helper to help us select columns that end with numbers in a specific range. We just have to specify the prefix we want to check and the number range we want to keep.

dplyr::select(
.data = two_scales,
id, num_range(prefix = "WISC_", range = 2:3)
)
# A tibble: 10 x 3
id    WISC_2 WISC_3
<chr>  <dbl>  <dbl>
1 s001       4      4
2 s002       8      8
3 s003       6      6
4 s004       0      0
5 s005       2      2
6 s006       2      2
7 s007       6      6
8 s008       2      2
9 s009       8      8
10 s010       1      1

# Filtering

filter(), as the name suggests is for filtering data tables (think how you would apply a data filter in Excel). To see it in action, let’s look at an example.

Let’s say that we want to take a subset of the mtcars data table so that we only keep the cars that have a fuel efficiency of above 20 mpg. We could do it with the code below.

mtcars[mtcars$mpg > 20, ]  mpg cyl disp hp drat wt qsec vs am gear carb Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4 Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4 Datsun 710 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1 Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1 Merc 240D 24.4 4 146.7 62 3.69 3.190 20.00 1 0 4 2 Merc 230 22.8 4 140.8 95 3.92 3.150 22.90 1 0 4 2 Fiat 128 32.4 4 78.7 66 4.08 2.200 19.47 1 1 4 1 Honda Civic 30.4 4 75.7 52 4.93 1.615 18.52 1 1 4 2 Toyota Corolla 33.9 4 71.1 65 4.22 1.835 19.90 1 1 4 1 Toyota Corona 21.5 4 120.1 97 3.70 2.465 20.01 1 0 3 1 Fiat X1-9 27.3 4 79.0 66 4.08 1.935 18.90 1 1 4 1 Porsche 914-2 26.0 4 120.3 91 4.43 2.140 16.70 0 1 5 2 Lotus Europa 30.4 4 95.1 113 3.77 1.513 16.90 1 1 5 2 Volvo 142E 21.4 4 121.0 109 4.11 2.780 18.60 1 1 4 2 To do the equivalent in the tidyverse way using the dplyr verb filter() we would do the following dplyr::filter( .data = mtcars, # our data table mpg > 20 # our logical )  mpg cyl disp hp drat wt qsec vs am gear carb Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4 Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4 Datsun 710 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1 Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1 Merc 240D 24.4 4 146.7 62 3.69 3.190 20.00 1 0 4 2 Merc 230 22.8 4 140.8 95 3.92 3.150 22.90 1 0 4 2 Fiat 128 32.4 4 78.7 66 4.08 2.200 19.47 1 1 4 1 Honda Civic 30.4 4 75.7 52 4.93 1.615 18.52 1 1 4 2 Toyota Corolla 33.9 4 71.1 65 4.22 1.835 19.90 1 1 4 1 Toyota Corona 21.5 4 120.1 97 3.70 2.465 20.01 1 0 3 1 Fiat X1-9 27.3 4 79.0 66 4.08 1.935 18.90 1 1 4 1 Porsche 914-2 26.0 4 120.3 91 4.43 2.140 16.70 0 1 5 2 Lotus Europa 30.4 4 95.1 113 3.77 1.513 16.90 1 1 5 2 Volvo 142E 21.4 4 121.0 109 4.11 2.780 18.60 1 1 4 2 We can use all the same logic with filter() that we’d usually use with subsetting, including subsetting by multiple criteria. dplyr::filter( .data = mtcars, # our data table mpg > 15 & disp > 250 # our two logical criteria )  mpg cyl disp hp drat wt qsec vs am gear carb Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1 Hornet Sportabout 18.7 8 360.0 175 3.15 3.440 17.02 0 0 3 2 Merc 450SE 16.4 8 275.8 180 3.07 4.070 17.40 0 0 3 3 Merc 450SL 17.3 8 275.8 180 3.07 3.730 17.60 0 0 3 3 Merc 450SLC 15.2 8 275.8 180 3.07 3.780 18.00 0 0 3 3 Dodge Challenger 15.5 8 318.0 150 2.76 3.520 16.87 0 0 3 2 AMC Javelin 15.2 8 304.0 150 3.15 3.435 17.30 0 0 3 2 Pontiac Firebird 19.2 8 400.0 175 3.08 3.845 17.05 0 0 3 2 Ford Pantera L 15.8 8 351.0 264 4.22 3.170 14.50 0 1 5 4 # Pulling You’d have noticed that select() returns a data table. That is, it’s the equivalent to using [] for selecting a column. But what if we want to do the equivalent to $.

Just to refresh you on the difference. We’ll use some data about Star Wars characters

starwars
# A tibble: 87 x 14
name  height  mass hair_color skin_color eye_color birth_year sex   gender
<chr>  <int> <dbl> <chr>      <chr>      <chr>          <dbl> <chr> <chr>
1 Luke…    172    77 blond      fair       blue            19   male  mascu…
2 C-3PO    167    75 <NA>       gold       yellow         112   none  mascu…
3 R2-D2     96    32 <NA>       white, bl… red             33   none  mascu…
4 Dart…    202   136 none       white      yellow          41.9 male  mascu…
5 Leia…    150    49 brown      light      brown           19   fema… femin…
6 Owen…    178   120 brown, gr… light      blue            52   male  mascu…
7 Beru…    165    75 brown      light      blue            47   fema… femin…
8 R5-D4     97    32 <NA>       white, red red             NA   none  mascu…
9 Bigg…    183    84 black      light      brown           24   male  mascu…
10 Obi-…    182    77 auburn, w… fair       blue-gray       57   male  mascu…
# … with 77 more rows, and 5 more variables: homeworld <chr>, species <chr>,
#   films <list>, vehicles <list>, starships <list>
starwars["height"] # equivalent to select(.data = starwars, height)
# A tibble: 87 x 1
height
<int>
1    172
2    167
3     96
4    202
5    150
6    178
7    165
8     97
9    183
10    182
# … with 77 more rows
starwars$height # returns just the values  [1] 172 167 96 202 150 178 165 97 183 182 188 180 228 180 173 175 170 180 66 [20] 170 183 200 190 177 175 180 150 NA 88 160 193 191 170 196 224 206 183 137 [39] 112 183 163 175 180 178 94 122 163 188 198 196 171 184 188 264 188 196 185 [58] 157 183 183 170 166 165 193 191 183 168 198 229 213 167 79 96 193 191 178 [77] 216 234 188 178 206 NA NA NA NA NA 165 To do the equivalent of $ in tidyverse syntax uses the pull() function.

dplyr::pull(
.data = starwars, # our data table
var = height # the column to pull
)
 [1] 172 167  96 202 150 178 165  97 183 182 188 180 228 180 173 175 170 180  66
[20] 170 183 200 190 177 175 180 150  NA  88 160 193 191 170 196 224 206 183 137
[39] 112 183 163 175 180 178  94 122 163 188 198 196 171 184 188 264 188 196 185
[58] 157 183 183 170 166 165 193 191 183 168 198 229 213 167  79  96 193 191 178
[77] 216 234 188 178 206  NA  NA  NA  NA  NA 165

So just like \$ the pull() function returns a vector. One of the neat things about pull(), however, is that instead of just returning a regular vector you can ask for a named vector to be returned.

named_vect <- dplyr::pull(
.data = starwars, # our data table
var = height, # the column to pull
name = name # the name of the column to get the names from
)
named_vect[["R2-D2"]] # subset our named vector by name
[1] 96