How to merge multiple data frames using base R

The merge function in R allows you to combine two data frames, much like the join function that is used in SQL to combine data tables. Merge, however, does not allow for more than two data frames to be joined at once, requiring several lines of code to join multiple data frames.

This post explains the methodology behind merging multiple data frames in one line of code using base R. We will be using the Reduce function, part of Funprog in base R v3.4.3. Funprog contains a suite of higher order functions which provide simple alternatives to laborious, long winded coding solutions.

The merge function

As described, merge is essentially the “join” of the R world. Whilst this post is not about the fine workings of merge, I will give a brief introduction.

Merge takes two data frames, x and y, and combines them based on one or more shared columns. Rows are combined where the data of these shared columns are equal, meaning we can combine columns from different data frames that refer to the same piece of data. For instance, take the following two data frames:

height <- data.frame("Character" = c("Luke", "Han", "Leia"),
                "height" = c("1.75m", "1.85m", "1.5m"))
gender <- data.frame("Character" = c("Luke", "Han", "Leia"),
                "gender" = c("m", "m", "f"))

It is clear that the two data frames are referring to the same characters, however it may be more useful to us if the two were combined into a single data frame. This is where merge comes in. Merge takes the following structure:

merge(x = height, y = gender, by = “Character”)

Here, we are looking to combine the height and gender data frames where the character columns are equal. To continue the SQL analogy, x is the left-hand table, y is the right-hand table, and merge is the LEFT JOIN operation. The “by” component is our “ON” clause. For example:

FROM height
LEFT JOIN gender
ON height.column1 = gender.column1

Running this merge function gives us the following output:


This is the result we were expecting, but what if we introduce a third data frame?

eyeColour <- data.frame("Character" = c("Luke","Han","Leia"),
                "eye_colour" = c("Brown", "Blue", "Brown"))

Sadly, merge does not allow us to simply add our eyeColour data frame as a third input (we only have x and y parameters available). That’s where Reduce comes in.

The Reduce function

Reduce takes a function and sequentially applies it to a given list of inputs, in our case a list of data frames. For example, imagine we have a function f which accepts two arguments, and a list of objects (a, b, c). Then Reduce(x, list(a, b, c)) would perform the following action:

f(a, f(b, c))

where the function x is first applied to data frames b and c, and is then applied to data frame a and the output of the first application of x. This allows us to avoid running and saving x(b, c), like this:

output_1 <- x(b, c)
output_2 <- x(a, output_1)

Applying Reduce to merge

In merge we have an example of a function that performs an action on two inputs. Reduce takes two parameters; f which stands for function and x which represents a vector. Reduce will sequentially apply the function f to the list x.

In our example, the function that we want to apply is merge, and the vector which we want to apply it to is a list of our data frames. First off, let’s try the following:

Reduce(merge, list(height, gender, eyeColour))

Perfect! But what if we wanted to specify the parameters within our merge function call? Well, we could define our own function which merges two data frames with specified parameters:

Reduce(function(x,y) merge(x = x, y = y, by = "Character"), 
       list(height, gender, eyeColour))

Here, we have specified our f as a custom function, which takes two parameters and applies the merge function to them. Within this custom function, we have specified our by parameter, which may be necessary for longer or more complex uses of Reduce.

Further reading

The function that we passed to Reduce is known in the world of functional programming as a lambda function, or an anonymous function; a single use function that is not named and saved. Functional programming is a principle around which R is built, and can provide many smart and elegant ways to achieve things that would otherwise require large amounts of coding. We may explore more of the functional programming features of R in future blog posts, however for now the following link provides a nice overview of the most used techniques:

by Jon Willis