Search code examples
rdataframedata-manipulationdcast

Manipulating large dataset with dcast


Apologies if this is a repeat question but I could not find the specific answer I am looking for. I have a dataframe with counts of different species caught on a given trip. A simplified example with 5 trips and 4 species is below:

trip = c(1,1,1,2,2,3,3,3,3,4,5,5)
species = c("a","b","c","b","d","a","b","c","d","c","c","d")
count = c(5,7,3,1,8,10,1,4,3,1,2,10)

dat = cbind.data.frame(trip, species, count)
dat

> dat
   trip species count
1     1       a     5
2     1       b     7
3     1       c     3
4     2       b     1
5     2       d     8
6     3       a    10
7     3       b     1
8     3       c     4
9     3       d     3
10    4       c     1
11    5       c     2
12    5       d    10

I am only interested in the counts of species b for each trip. So I want to manipulate this data frame so I end up with one that looks like this:

trip2 = c(1,2,3,4,5)
species2 = c("b","b","b","b","b")
count2 = c(7,1,1,0,0)

dat2 = cbind.data.frame(trip2, species2, count2)
dat2

> dat2
  trip2 species2 count2
1     1        b      7
2     2        b      1
3     3        b      1
4     4        b      0
5     5        b      0

I want to keep all trips, including trips where species b was not observed. So I can't just subset the data by species b. I know I can cast the data so species are the columns and then just remove the columns for the other species like so:

library(dplyr)
library(reshape2)
test = dcast(dat, trip ~ species, value.var = "count", fun.aggregate = sum)
test

> test
  trip  a b c  d
1    1  5 7 3  0
2    2  0 1 0  8
3    3 10 1 4  3
4    4  0 0 1  0
5    5  0 0 2 10

However, my real dataset has several hundred species caught on thousands of trips, and if I try to cast that many species to columns R chokes. There are way too many columns. Is there a way to specify in dcast that I only want to cast species b? Or is there another way to do this that doesn't require casting the data? Thank you.


Solution

  • Here is a data.table approach which I suspect will be very fast for you:

    library(data.table)
    setDT(dat)
    result <- dat[,.(species = "b", count = sum(.SD[species == "b",count])),by = trip]
    result
       trip species count
    1:    1       b     7
    2:    2       b     1
    3:    3       b     1
    4:    4       b     0
    5:    5       b     0