Onboarding DataFrames.jl
Introduction
Working with data frames is one of the basic needs of any data scientist. In the Julia ecosystem DataFrames.jl is a package providing support for these operations. It was designed to be efficient and flexible.
Sometimes, however, novice users can be overwhelmed by the syntax due to its flexibility. Therefore data scientists often find it useful to use the packages that make it easier to do transformations of data frames.
Interestingly, these packages use metaprogramming, which might sound to novices as something scary, while in reality it is the opposite. Metaprogramming is used to make them easier to use.
Today I want do do a quick review of the main metaprogramming packages that are available in the ecosystem. I will not go into the details functionality and syntax of the packages, but rather just present them briefly and give my personal (opinionated) view of their status.
This post is written under Julia 1.10.1, DataFrames.jl 1.6.1, Chain.jl 0.5.0, DataFramesMeta.jl 0.15.2, DataFrameMacros.jl 0.4.1, and TidyData.jl 0.15.1.
A basic example
Let us start with a basic example of DataFrames.jl syntax, which we will later rewrite using metaprogramming:
julia> using Statistics
julia> using DataFrames
julia> df = DataFrame(id=[1, 2, 1, 2], v=1:4)
4×2 DataFrame
Row │ id v
│ Int64 Int64
─────┼──────────────
1 │ 1 1
2 │ 2 2
3 │ 1 3
4 │ 2 4
julia> transform(groupby(df, :id), :v => (x -> x .- mean(x)) => :v100)
4×3 DataFrame
Row │ id v v100
│ Int64 Int64 Float64
─────┼───────────────────────
1 │ 1 1 -1.0
2 │ 2 2 -1.0
3 │ 1 3 1.0
4 │ 2 4 1.0
The syntax looks complex and might be scary. Let us see if we can make it simpler.
Chain.jl
The first functionality we might want to use is to put the operations in a pipe. This is achieved with the Chain.jl package:
julia> using Chain
julia> @chain df begin
groupby(:id)
transform(:v => (x -> x .- mean(x)) => :v100)
end
4×3 DataFrame
Row │ id v v100
│ Int64 Int64 Float64
─────┼───────────────────────
1 │ 1 1 -1.0
2 │ 2 2 -1.0
3 │ 1 3 1.0
4 │ 2 4 1.0
We have achieved the benefit of a better visual separation of operations. In my opinion Chain.jl can be considered as a currently mostly accepted approach to piping operations in Julia (there are alternatives in the ecosystem but as far as I can tell they have lower adoption level).
DataFramesMeta.jl
Still the transform(:v => (x -> x .- mean(x)) => :v100) part looks verbose. Let us start by showing
how it can be made simpler using DataFramesMeta.jl:
julia> using DataFramesMeta
julia> @chain df begin
groupby(:id)
@transform(:v100 = :v .- mean(:v))
end
4×3 DataFrame
Row │ id v v100
│ Int64 Int64 Float64
─────┼───────────────────────
1 │ 1 1 -1.0
2 │ 2 2 -1.0
3 │ 1 3 1.0
4 │ 2 4 1.0
In my opinion the code is now really easy to read.
Here is the status of DataFramesMeta.jl:
- It is actively maintained.
- Its syntax is close to DataFrames.jl.
- It uses
:to signal that some name is a column of a data frame.
DataFrameMacros.jl
The DataFrameMacros.jl is another package that is closely tied to DataFrames.jl. Let us see how we can use it. Note that you need to restart the Julia session before running the code as the macro names are overlapping with DataFramesMeta.jl:
julia> using DataFrameMacros
julia> @chain df begin
groupby(:id)
@transform(:v100 = @bycol :v .- mean(:v))
end
4×3 DataFrame
Row │ id v v100
│ Int64 Int64 Float64
─────┼───────────────────────
1 │ 1 1 -1.0
2 │ 2 2 -1.0
3 │ 1 3 1.0
4 │ 2 4 1.0
Note the difference with the @bycol expression. It is needed because in DataFrameMacros.jl @transform by default vectorizes operations.
This is often more convenient for users, but sometimes (like in this case), one wants to suppress vectorization.
What is the status of DataFrameMacros.jl?
- It is maintained but less actively developed than DataFramesMeta.jl.
- Its syntax is close to DataFrames.jl, but several macros, for user convenience, vectorize operations by default (as opposed to Base Julia).
- It uses
:to signal that some text is a column of a data frame.
TidierData.jl
Now let us see the TidierData.jl package that is designed to follow dplyr from R:
julia> using TidierData
julia> @chain df begin
@group_by(id)
@mutate(v100 = v - mean(v))
@ungroup
end
4×3 DataFrame
Row │ id v v100
│ Int64 Int64 Float64
─────┼───────────────────────
1 │ 1 1 -1.0
2 │ 1 3 1.0
3 │ 2 2 -1.0
4 │ 2 4 1.0
If you know dplyr you should be at home with this syntax.
What is the status of TidierData.jl:
- It is actively maintained.
- It tries to guess as much as possible; the package automatically decides which functions should be vectorized (in our example
-was vectorized butmeanwas not). - You do not need a
:prefix in column names, the package uses scoping similar to R to resolve variable names.
As you can see, the R-style syntax is designed for maximum convenience, at the expense of control (a lot of “magic” happens behind the scenes; admittedly most of the time this magic is what novice users would want).
Conclusions
Here is a recap of what we have discussed:
- Meta-packages are here to make life easier for users. There is no need to be afraid of them.
- For piping I recommend using Chain.jl.
- Use plain DataFrames.jl if you are a die-hard Julia user and want all your code to be valid Julia syntax (I prefer it when writing production stuff).
- Use DataFramesMeta.jl if you want an experience most consistent with Base Julia (this is my personal preference for interactive sessions, but it requires most knowledge of Julia).
- DataFrameMacros.jl is an in-between package, it adds some more convenience (e.g. vectorization by default), but does not push it to the extreme
(it also has a super convenient
{}notation which you might find useful; I decided to skip it to keep the post simple to follow). - TidyData.jl goes for maximum convenience. It follows R-style and tries to guess what you most likely wanted to do. Users with
dplyrshould be able to start using it immediately.