I have built a logistic regression model using the glm function from the stats package. I now would like to predict the outcome of this model on a large number of values, stored in a "ffdf" object (see ff package), however I do not find how to proceed:
How can I create a subset of my ffdf object, in order to keep only the variables (i.e. columns) to be used in my prediction? - needed to specify as an input in the predict function
How should I proceed next? Which function should be used between predict(), predict.glm(), predict.bigglm() (Maybe biglm package is helpful)?
Thank you in advance for your views on this!
Best regards
UPDATE
Thank you for your feedback BondedDust.
Let me be more precise, it is indeed a coding question, aiming at performing logistic regression based on an ffdf object (learning dataset), and predict the outcome of the model for another ffdf object (test dataset).
(1/3) Learning data set: ffdf object (created with ff package).
` class(train.random.sample)` >
[1] "ffdf"
below is the structure of the ffdf object in case of needs:
`str(train.random.sample) ` >
List of 3
$ virtual: 'data.frame': 27 obs. of 7 variables:
.. $ VirtualVmode : chr "integer" "integer" "integer" "integer" ...
.. $ AsIs : logi FALSE FALSE FALSE FALSE FALSE FALSE ...
.. $ VirtualIsMatrix : logi FALSE FALSE FALSE FALSE FALSE FALSE ...
.. $ PhysicalIsMatrix : logi FALSE FALSE FALSE FALSE FALSE FALSE ...
.. $ PhysicalElementNo: int 1 2 3 4 5 6 7 8 9 10 ...
.. $ PhysicalFirstCol : int 1 1 1 1 1 1 1 1 1 1 ...
.. $ PhysicalLastCol : int 1 1 1 1 1 1 1 1 1 1 ...
.. - attr(*, "Dim")= int 500000 27
.. - attr(*, "Dimorder")= int 1 2
$ physical: List of 27
.. $ id : list()
…
.. .. - attr(*, "class") = chr [1:2] "ff_vector" "ff"
.. $ click : list()
…
.. .. - attr(*, "class") = chr [1:2] "ff_vector" "ff"
.. $ hour : list()
…
.. .. - attr(*, "class") = chr [1:2] "ff_vector" "ff"
.. $ C1 : list()
…
.. .. - attr(*, "class") = chr [1:2] "ff_vector" "ff"
.. $ banner_pos : list()
.. ..- attr(*, "physical")=Class 'ff_pointer' <externalptr>
.. .. ..- attr(*, "vmode")= chr "integer"
.. .. ..- attr(*, "maxlength")= int 500000
.. .. ..- attr(*, "pattern")= chr "ffdf"
.. .. ..- attr(*, "filename")= chr "anonymized.ff"
.. .. ..- attr(*, "pagesize")= int 65536
.. .. ..- attr(*, "finalizer")= chr "delete"
.. .. ..- attr(*, "finonexit")= logi TRUE
.. .. ..- attr(*, "readonly")= logi FALSE
.. .. ..- attr(*, "caching")= chr "mmnoflush"
.. ..- attr(*, "virtual")= list()
.. .. ..- attr(*, "Length")= int 500000
.. .. ..- attr(*, "Symmetric")= logi FALSE
.. .. - attr(*, "class") = chr [1:2] "ff_vector" "ff"
.. $ site_id : list()
…
.. .. - attr(*, "class") = chr [1:2] "ff_vector" "ff"
.. $ site_domain : list()
…
.. .. - attr(*, "class") = chr [1:2] "ff_vector" "ff"
.. $ site_category : list()
…
.. .. - attr(*, "class") = chr [1:2] "ff_vector" "ff"
.. $ app_id : list()
…
.. .. - attr(*, "class") = chr [1:2] "ff_vector" "ff"
.. $ app_domain : list()
…
.. .. - attr(*, "class") = chr [1:2] "ff_vector" "ff"
.. $ app_category : list()
…
.. .. - attr(*, "class") = chr [1:2] "ff_vector" "ff"
.. $ device_id : list()
…
.. .. - attr(*, "class") = chr [1:2] "ff_vector" "ff"
.. $ device_ip : list()
….
.. .. - attr(*, "class") = chr [1:2] "ff_vector" "ff"
.. $ device_os : list()
…
.. .. - attr(*, "class") = chr [1:2] "ff_vector" "ff"
.. $ device_make : list()
…
.. .. - attr(*, "class") = chr [1:2] "ff_vector" "ff"
.. $ device_model : list()
…
.. .. - attr(*, "class") = chr [1:2] "ff_vector" "ff"
.. $ device_type : list()
…
.. .. - attr(*, "class") = chr [1:2] "ff_vector" "ff"
.. $ device_conn_type : list()
…
.. .. - attr(*, "class") = chr [1:2] "ff_vector" "ff"
.. $ device_geo_country: list()
…
.. .. - attr(*, "class") = chr [1:2] "ff_vector" "ff"
.. $ C17 : list()
…
.. .. - attr(*, "class") = chr [1:2] "ff_vector" "ff"
$ row.names: NULL
- attributes: List of 2
.. $ names: chr [1:3] "virtual" "physical" "row.names"
.. $ class: chr "ffdf"
(2/3) Logistic regression based on learning dataset:
Objective is to learn/ predict ‘click’ outcome based on ‘baser_pos’ input
`logreg1 <- glm(click ~ banner_pos, data = train.random.sample, family = "binomial")
summary(logreg1)` >
Call:
glm(formula = click ~ banner_pos, family = "binomial", data = train.random.sample)
Deviance Residuals:
Min 1Q Median 3Q Max
-1.0555 -0.6495 -0.5951 -0.5951 1.9071
Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -1.641416 0.004702 -349.12 <2e-16 xxx
banner_pos 0.192534 0.007595 25.35 <2e-16 xxx
---
Signif. codes: 0 ‘xxx’ 0.001 ‘xx’ 0.01 ‘x’ 0.05 ‘.’ 0.1 ‘ ’ 1
(Dispersion parameter for binomial family taken to be 1)
Null deviance: 458848 on 499999 degrees of freedom
Residual deviance: 458215 on 499998 degrees of freedom
AIC: 458219
Number of Fisher Scoring iterations: 4
`class(logreg1)`>
[1] "glm" "lm"
(3/3)Test dataset: ffdf object (created with ff package).
`class(df.test)` >
[1] "ffdf"
Test dataset structure is identical to training dataset, with ~4.8m rows
`str(df.test)`>
List of 3
$ virtual: 'data.frame': 26 obs. of 7 variables:
.. $ VirtualVmode : chr "integer" "integer" "integer" "integer" ...
.. $ AsIs : logi FALSE FALSE FALSE FALSE FALSE FALSE ...
.. $ VirtualIsMatrix : logi FALSE FALSE FALSE FALSE FALSE FALSE ...
.. $ PhysicalIsMatrix : logi FALSE FALSE FALSE FALSE FALSE FALSE ...
.. $ PhysicalElementNo: int 1 2 3 4 5 6 7 8 9 10 ...
.. $ PhysicalFirstCol : int 1 1 1 1 1 1 1 1 1 1 ...
.. $ PhysicalLastCol : int 1 1 1 1 1 1 1 1 1 1 ...
.. - attr(*, "Dim")= int 4769401 26
.. - attr(*, "Dimorder")= int 1 2
$ physical: List of 26
…
I could not succeed in predicting click outcome. I first tried to create a dataframe or ffdf object containing the banner_pos variable:
`modeldata <- df.test[["banner_pos"]]`
Then I tried to predict the outcome:
`predict.glm(object = logreg1, newdata = modeldata, type = "response")`
Error in as.data.frame.default(data) :
cannot coerce class "c("ff_vector", "ff")" to a data.frame
Is there something wrong in my code? Should I use other function leveraging other packages such as biglm?
Many thanks in advance for your views on that issue,
Best regards
Something similar as this will score your ffdf alongside your glm.
require(ff)
df.test$score <- ff(as.numeric(NA), length = nrow(df.test))
chunks <- chunk(df.test)
for(chunkrangeindex in chunks){
df.test$score[chunkrangeindex] <- predict(object = logreg1, newdata = df.test[chunkrangeindex, ], type = "response")
}