Have a look at this, this and this posts for real usage scenarios. No intermediate results are materialised, and the join+aggregate is performed all in one go. We don't have to group/hash twice (one for aggregation and other for joining).Īnd more importantly, the operation what we wanted to perform is clear by looking at j in (2).Ĭheck this post for a detailed explanation of by =. We don't have to allocate memory for the intermediate result. We can either:Īggregate DT1 to get sum(z), 2) perform a join and 3) multiply (or) data.table wayĭT1 dplyr equivalentĭF1 %>% group_by(x, y) %>% summarise(z = sum(z)) %>%ĭo it all in one go (using by =. So those who prefer dplyr's syntax can use it with data.tables.īut it will still lack many features that data.table provides, including (sub)-assignment by reference. When implemented, this should settle the referential transparency issue altogether while providing the user with both possibilties.Īlso, once shallow() is exported dplyr's data.table interface should avoid almost all copies. We take care of everything internally to ensure that while also ensuring to copy columns you modify only when it is absolutely necessary. For example, if it is desirable to not modify the input data.table within a function, one can then do: foo 2L, newcol := 2L] # no need to copy (internally), as this column exists only in shallow copied DTĭT # have to copy (like base R / dplyr does always) otherwise original DT willīy not using shallow(), the old functionality is retained: bar 2L, x := 3L] # old behaviour, update column x in original DT.īy creating a shallow copy using shallow(), we understand that you don't want to modify the original object. Therefore we are working towards exporting shallow() function in data.table that will provide the user with both possibilities. But this is an incredibly useful feature: see this and this posts for interesting cases. Updating a data.table object by reference, especially within a function may not be always desirable. The dplyr equivalent would be (note that the result needs to be re-assigned): # copies the entire 'y' columnĪns % mutate(y = replace(y, which(x >= 1L), NA))Ī concern for this is referential transparency. # sub-assign by reference, updates 'y' in-placeīut dplyr will never update by reference. Note that Hadley's comment talks about speed (that dplyr is plentiful fast for him), whereas the major concern here is memory.ĭata.table interface at the moment allows one to modify/update columns by reference (note that we don't need to re-assign the result back to a variable). Operations involving filter() or slice() in dplyr can be memory inefficient (on both ames and data.tables). Grouping operations involving a subset of rows - i.e., DT type operations.īenchmark other operations such as update and joins.Īlso benchmark memory footprint for each operation in addition to runtime. On benchmarks, it would be great to cover these remaining aspects as well: See also updated benchmarks, which include Spark and pydatatable as well. Quite a few benchmarks (though mostly on grouping operations) have been added to the question already showing data.table gets faster than dplyr as the number of groups and/or rows to group by increase, including benchmarks by Matt on grouping from 10 million to 2 billion rows (100GB in RAM) on 100 - 10 million groups and varying grouping columns, which also compares pandas. By keeping related operations together, it allows to easily optimise operations for speed and more importantly memory usage, and also provide some powerful features, all while maintaining the consistency in syntax. To keep i, j and by together is by design. The data.table syntax is consistent in its form - DT. Note: unless explicitly mentioned otherwise, by referring to dplyr, we refer to dplyr's ame interface whose internals are in C++ using Rcpp. My intent is to cover each one of these as clearly as possible from data.table perspective. We need to cover at least these aspects to provide a comprehensive answer/comparison (in no particular order of importance): Speed, Memory usage, Syntax and Features.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |