Half of the time, without pattern, I get rows omitted from the output even though I increase max.print and the global options in R studio. Details below:
I am running an lmer model on large datasets (more than 100,000 observations of 14 variables) with packages: lme4, lmerTest, tidyverse, lsmeans
I have several datasets that I run separately, but the datasets are not different in design from each other and they all originated from the same mother dataset. I just removed certain rows in each dataset to answer different questions. Variables in each dataset include hour, location, date, depth, temperature and transformed depth to meet assumptions.
For some reason after I run the lsmeans() function to get pairwise comparisons, half of the time there are rows omitted: [reached getOption("max.print") -- omitted 110 rows]
example code:
ft.model = lmer(dep.variable ~ fixed.factor + (1|random.factor), data = dataset)
anova(ft.model)
plot(ft.model, main = "residuals/fitted ft.model")
lsmeans(ft.model, pairwise~fixed.factor, adjust = "Tukey", max.print = 9999)
So it looks like this with some of my variables:
ft.hrdepth = lmer(depth ~ hr + (1|FishID), data = all)
anova(ft.hrdepth)
plot(ft.hrdepth, main = "residuals/fitted ft.hrdepth")
lsmeans(ft.hrdepth, pairwise~hr, adjust = "Tukey", max.print = 9999)
I have tried adding the max.print argument and increasing to 10000, 10000000, 9999, 99999999, Inf, and other combos. I have also gone into RStudio settings and changed the limit of lines displayed to similar numbers: Tools > Global Options > Code > Display > Limit length of lines displayed in console is: 1000000
I have also made lsmeans into an object and used the print() function to print the output or just the contrasts, but still no luck:
pws = lsmeans(ft.model, pairwise~fixed.factor, adjust = "Tukey", max.print = 9999)
print(pws)
pwss = pws[[contrasts]]
print(pwss)
I have tried repeatedly with the same exact dataset and different datasets, and see absolutely no pattern as to why it omits rows sometimes and doesn't omit rows sometimes.
I have no fix to view the lsmeans output in the console or at the bottom of the R Markdown file I have. (Note I also get the same result when I use an R script file instead of R mardown file = only 1/2 the time do I get all the rows). Please help! The only way I can guarantee seeing all the rows is by exporting the lsmeans contrasts output to a csv file, which is extremely inefficient and not helpful since I also ideally knit all my results to a single document:
pws = lsmeans(ft.model, pairwise~fixed.factor, adjust = "Tukey", max.print = 9999)
pwss = pws[[contrasts]]
write.csv(pwss,file = "pwss.csv")
FYI, I use RStudio Version 1.1.419 and R i386 3.4.3 My computer is an HP windows 10 Pro 64-bit operating system
example dataset:
structure(list(X = c(1L, 15L, 20L, 26L, 35L, 44L, 55L, 66L, 77L, 88L, 99L, 1111L, 44444L, 77777L, 8888L, 999L, 2222L, 4444L, 55555L, 66666L, 777L, 88888L, 99999L, 11111L, 57890L, 23456L, 9675L, 129873L, 22222L, 333L, 5555L, 123434L, 99944L, 88833L, 77744L, 66655L, 55544L, 44433L, 33322L, 22211L, 134534L, 111111L, 121111L, 131111L, 141111L, 151111L, 161111L, 123111L, 33444L, 5566L, 6677L, 9988L, 33888L, 87878L, 148483L, 139847L, 34231L, 34200L), OrigDate = structure(c(1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 12L, 16L, 6L, 1L, 2L, 3L, 13L, 15L, 1L, 18L, 19L, 8L, 14L, 10L, 7L, 23L, 9L, 1L, 4L, 22L, 19L, 18L, 16L, 15L, 13L, 12L, 11L, 9L, 24L, 20L, 21L, 23L, 25L, 27L, 28L, 22L, 11L, 4L, 5L, 7L, 11L, 17L, 26L, 25L, 11L, 11L), .Label = c("10/01/2012", "10/02/2012", "10/03/2012", "10/04/2012", "10/05/2012", "10/06/2012", "10/07/2012", "10/08/2012", "10/14/2012", "10/15/2012", "10/21/2012", "10/27/2012", "11/02/2012", "11/03/2012", "11/08/2012", "11/15/2012", "11/19/2012", "11/20/2012", "11/25/2012", "11/29/2012", "12/03/2012", "12/04/2012", "12/08/2012", "12/10/2012", "12/13/2012", "12/17/2012", "12/18/2012", "12/23/2012" ), class = "factor"), OrigTime = structure(c(1L, 5L, 6L, 8L, 9L, 11L, 13L, 18L, 22L, 25L, 27L, 32L, 47L, 3L, 2L, 52L, 33L, 40L, 39L, 21L, 4L, 45L, 48L, 7L, 28L, 14L, 16L, 55L, 23L, 44L, 49L, 19L, 30L, 24L, 41L, 15L, 29L, 42L, 20L, 10L, 17L, 26L, 31L, 38L, 56L, 36L, 50L, 43L, 57L, 51L, 46L, 35L, 12L, 37L, 54L, 34L, 58L, 53L), .Label = c("00:03:39", "00:10:58", "00:31:07", "00:41:09", "01:24:09", "01:48:18", "02:10:49", "02:17:49", "03:33:44", "04:41:53", "04:47:35", "05:13:35", "05:52:04", "06:04:16", "06:27:49", "06:45:23", "06:57:49", "07:03:07", "07:03:26", "07:19:39", "07:25:02", "08:12:43", "08:37:03", "08:44:48", "09:27:02", "09:30:58", "10:41:49", "10:52:47", "11:09:06", "11:10:01", "11:12:46", "11:15:04", "11:34:46", "11:59:42", "12:02:58", "12:36:34", "12:39:42", "13:20:47", "13:45:05", "13:50:05", "13:51:44", "14:04:15", "14:47:51", "14:51:57", "15:23:14", "15:37:53", "15:49:09", "16:08:00", "16:10:44", "16:37:25", "17:34:17", "18:17:16", "19:34:03", "20:23:36", "20:40:22", "21:20:51", "21:27:49", "22:44:33" ), class = "factor"), OrigDateTime = structure(c(1L, 3L, 4L, 9L, 10L, 11L, 12L, 13L, 14L, 15L, 5L, 6L, 26L, 35L, 31L, 8L, 19L, 27L, 39L, 47L, 2L, 40L, 43L, 34L, 45L, 18L, 33L, 58L, 17L, 7L, 28L, 56L, 42L, 41L, 36L, 46L, 38L, 25L, 24L, 16L, 48L, 44L, 54L, 57L, 50L, 52L, 53L, 55L, 21L, 29L, 30L, 32L, 23L, 37L, 51L, 49L, 22L, 20L), .Label = c("10/1/2012 0:03:39", "10/1/2012 0:41:09", "10/1/2012 1:24:09", "10/1/2012 1:48:18", "10/1/2012 10:41:49", "10/1/2012 11:15:04", "10/1/2012 14:51:57", "10/1/2012 18:17:16", "10/1/2012 2:17:49", "10/1/2012 3:33:44", "10/1/2012 4:47:35", "10/1/2012 5:52:04", "10/1/2012 7:03:07", "10/1/2012 8:12:43", "10/1/2012 9:27:02", "10/14/2012 4:41:53", "10/14/2012 8:37:03", "10/15/2012 6:04:16", "10/2/2012 11:34:46", "10/21/2012 19:34:03", "10/21/2012 21:27:49", "10/21/2012 22:44:33", "10/21/2012 5:13:35", "10/21/2012 7:19:39", "10/27/2012 14:04:15", "10/27/2012 15:49:09", "10/3/2012 13:50:05", "10/4/2012 16:10:44", "10/4/2012 17:34:17", "10/5/2012 15:37:53", "10/6/2012 0:10:58", "10/7/2012 12:02:58", "10/7/2012 6:45:23", "10/8/2012 2:10:49", "11/15/2012 0:31:07", "11/15/2012 13:51:44", "11/19/2012 12:39:42", "11/2/2012 11:09:06", "11/2/2012 13:45:05", "11/20/2012 15:23:14", "11/20/2012 8:44:48", "11/25/2012 11:10:01", "11/25/2012 16:08:00", "11/29/2012 9:30:58", "11/3/2012 10:52:47", "11/8/2012 6:27:49", "11/8/2012 7:25:02", "12/10/2012 6:57:49", "12/13/2012 11:59:42", "12/13/2012 21:20:51", "12/17/2012 20:23:36", "12/18/2012 12:36:34", "12/23/2012 16:37:25", "12/3/2012 11:12:46", "12/4/2012 14:47:51", "12/4/2012 7:03:26", "12/8/2012 13:20:47", "12/8/2012 20:40:22"), class = "factor"), TimeSerial = c(41183.0025347222, 41183.0584375, 41183.0752083333, 41183.0957060185, 41183.1484259259, 41183.1997106482, 41183.2444907407, 41183.2938310185, 41183.3421643519, 41183.3937731481, 41183.4457060185, 41183.4687962963, 41209.6591319444, 41228.0216087963, 41188.0076157407, 41183.7619907407, 41184.4824768519, 41185.5764467593, 41215.572974537, 41221.3090509259, 41183.0285763889, 41233.6411342593, 41238.6722222222, 41190.0908449074, 41216.4533217593, 41197.252962963, 41189.2815162037, 41251.8613657407, 41196.3590625, 41183.6194097222, 41186.6741203704, 41247.2940509259, 41238.4652893519, 41233.3644444444, 41228.5775925926, 41221.2693171296, 41215.4646527778, 41209.5862847222, 41203.3053125, 41196.1957523148, 41253.290150463, 41242.3965046296, 41246.4671990741, 41251.556099537, 41256.8894791667, 41261.5253935185, 41266.6926504629, 41247.6165625, 41203.8943171296, 41186.7321412037, 41187.6513078704, 41189.5020601851, 41203.2177662037, 41232.5275694444, 41260.8497222222, 41256.4997916667, 41203.9476041667, 41203.8153125), DateNum = c(41183L, 41183L, 41183L, 41183L, 41183L, 41183L, 41183L, 41183L, 41183L, 41183L, 41183L, 41183L, 41209L, 41228L, 41188L, 41183L, 41184L, 41185L, 41215L, 41221L, 41183L, 41233L, 41238L, 41190L, 41216L, 41197L, 41189L, 41251L, 41196L, 41183L, 41186L, 41247L, 41238L, 41233L, 41228L, 41221L, 41215L, 41209L, 41203L, 41196L, 41253L, 41242L, 41246L, 41251L, 41256L, 41261L, 41266L, 41247L, 41203L, 41186L, 41187L, 41189L, 41203L, 41232L, 41260L, 41256L, 41203L, 41203L), TimeNum = c(0.0025347222, 0.0584375, 0.0752083333, 0.0957060185, 0.1484259259, 0.1997106482, 0.2444907407, 0.2938310185, 0.3421643519, 0.3937731481, 0.4457060185, 0.4687962963, 0.6591319444, 0.0216087963, 0.0076157407, 0.7619907407, 0.4824768519, 0.5764467593, 0.572974537, 0.3090509259, 0.0285763889, 0.6411342593, 0.6722222222, 0.0908449074, 0.4533217593, 0.252962963, 0.2815162037, 0.8613657407, 0.3590625, 0.6194097222, 0.6741203704, 0.2940509259, 0.4652893519, 0.3644444444, 0.5775925926, 0.2693171296, 0.4646527778, 0.5862847222, 0.3053125, 0.1957523148, 0.290150463, 0.3965046296, 0.4671990741, 0.556099537, 0.8894791667, 0.5253935185, 0.692650463, 0.6165625, 0.8943171296, 0.7321412037, 0.6513078704, 0.5020601852, 0.2177662037, 0.5275694444, 0.8497222222, 0.4997916667, 0.9476041667, 0.8153125 ), depth = c(19.34, 19.34, 19.12, 18.46, 19.34, 19.34, 19.34, 19.12, 18.9, 19.12, 18.02, 20.88, 21.1, 20.44, 20.22, 20.44, 17.36, 18.9, 20, 17.8, 18.9, 17.36, 19.34, 16.92, 20.22, 20.22, 20, 9.89, 19.56, 18.68, 20, 11.9, 20.22, 19.56, 21.1, 16.7, 20, 21.1, 19.56, 16.92, 20.22, 18.24, 17.36, 20.66, 10.3, 20.88, 19.34, 21.32, 16.26, 17.58, 18.46, 20, 18.46, 17.58, 9.45, 10.8, 18.68, 18.46), FishID = c(5593L, 5593L, 5593L, 5593L, 5593L, 5593L, 5593L, 5593L, 5593L, 5593L, 5593L, 7510L, 7513L, 7508L, 7510L, 7509L, 7501L, 7508L, 7511L, 5593L, 7508L, 7501L, 5593L, 7502L, 7513L, 7501L, 7501L, 5593L, 7508L, 6747L, 7501L, 7514L, 5593L, 7501L, 7505L, 5593L, 7511L, 7513L, 5593L, 7508L, 6747L, 7510L, 7512L, 7509L, 7508L, 7501L, 7496L, 7510L, 5593L, 7501L, 7501L, 7503L, 7502L, 7514L, 5593L, 5593L, 7508L, 7508L), Location = structure(c(1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L), .Label = "PM", class = "factor"), Temp = c(28.7, 28.7, 28.7, 28.7, 28.7, 28.7, 28.7, 28.6, 28.6, 28.6, 28.6, 28.6, 27, 24.1, 28, 28.6, 28.6, 28.2, 25.3, 25.8, 28.7, 22.6, 22.7, 27.5, 25.5, 27.6, 27.7, 22.5, 27.8, 28.6, 27.9, 21.6, 22.6, 22.6, 23.7, 26, 25.7, 27, 27.9, 27.8, 22.7, 22, 21.7, 22.4, 20.8, 21.4, 21.8, 21.6, 27.8, 28, 27.8, 27.7, 27.9, 22.6, 21.6, 20.7, 27.8, 27.7), hr = structure(c(1L, 2L, 2L, 3L, 4L, 5L, 6L, 8L, 9L, 10L, 11L, 12L, 16L, 1L, 1L, 19L, 12L, 14L, 14L, 8L, 1L, 16L, 17L, 3L, 11L, 7L, 7L, 21L, 9L, 15L, 17L, 8L, 12L, 9L, 14L, 7L, 12L, 15L, 8L, 5L, 7L, 10L, 12L, 14L, 22L, 13L, 17L, 15L, 22L, 18L, 16L, 13L, 6L, 13L, 21L, 12L, 23L, 20L), .Label = c("01:00:00", "02:00:00", "03:00:00", "04:00:00", "05:00:00", "06:00:00", "07:00:00", "08:00:00", "09:00:00", "10:00:00", "11:00:00", "12:00:00", "13:00:00", "14:00:00", "15:00:00", "16:00:00", "17:00:00", "18:00:00", "19:00:00", "20:00:00", "21:00:00", "22:00:00", "23:00:00"), class = "factor"), sqrtdepth = c(4.3977266854592, 4.3977266854592, 4.37264222181509, 4.29651021178817, 4.3977266854592, 4.3977266854592, 4.3977266854592, 4.37264222181509, 4.34741302385683, 4.37264222181509, 4.24499705535823, 4.56946386351834, 4.59347363114234, 4.52106182218293, 4.49666543118342, 4.52106182218293, 4.16653333119993, 4.34741302385683, 4.47213595499958, 4.2190046219458, 4.34741302385683, 4.16653333119993, 4.3977266854592, 4.11339276024063, 4.49666543118342, 4.49666543118342, 4.47213595499958, 3.14483703870328, 4.4226688774992, 4.32203655699486, 4.47213595499958, 3.44963766213207, 4.49666543118342, 4.4226688774992, 4.59347363114234, 4.08656334834051, 4.47213595499958, 4.59347363114234, 4.4226688774992, 4.11339276024063, 4.49666543118342, 4.27083130081252, 4.16653333119993, 4.5453272709454, 3.20936130717624, 4.56946386351834, 4.3977266854592, 4.61735855224608, 4.03236903073119, 4.19285105864733, 4.29651021178817, 4.47213595499958, 4.29651021178817, 4.19285105864733, 3.07408522978788, 3.286335345031, 4.32203655699486, 4.29651021178817), sqdepth = c(374.0356, 374.0356, 365.5744, 340.7716, 374.0356, 374.0356, 374.0356, 365.5744, 357.21, 365.5744, 324.7204, 435.9744, 445.21, 417.7936, 408.8484, 417.7936, 301.3696, 357.21, 400, 316.84, 357.21, 301.3696, 374.0356, 286.2864, 408.8484, 408.8484, 400, 97.8121, 382.5936, 348.9424, 400, 141.61, 408.8484, 382.5936, 445.21, 278.89, 400, 445.21, 382.5936, 286.2864, 408.8484, 332.6976, 301.3696, 426.8356, 106.09, 435.9744, 374.0356, 454.5424, 264.3876, 309.0564, 340.7716, 400, 340.7716, 309.0564, 89.3025, 116.64, 348.9424, 340.7716), cudepth = c(7233.848504, 7233.848504, 6989.782528, 6290.643736, 7233.848504, 7233.848504, 7233.848504, 6989.782528, 6751.269, 6989.782528, 5851.461608, 9103.145472, 9393.931, 8539.701184, 8266.914648, 8539.701184, 5231.776256, 6751.269, 8000, 5639.752, 6751.269, 5231.776256, 7233.848504, 4843.965888, 8266.914648, 8266.914648, 8000, 967.361669, 7483.530816, 6518.244032, 8000, 1685.159, 8266.914648, 7483.530816, 9393.931, 4657.463, 8000, 9393.931, 7483.530816, 4843.965888, 8266.914648, 6068.404224, 5231.776256, 8818.423496, 1092.727, 9103.145472, 7233.848504, 9690.843968, 4298.942376, 5433.211512, 6290.643736, 8000, 6290.643736, 5433.211512, 843.908625, 1259.712, 6518.244032, 6290.643736)), class = "data.frame", row.names = c(NA, -58L))
I would love any input!
You should specify the max.print option with
options('max.print' = 100000) # or whatever value you want
getOption('max.print')
[1] 100000
When you do lsmeans(stuff, max.print = 10000), you're actually passing the argument (not option) of max.print to other functions in the emmeans package. That's what is meant by the ... argument for lsmeans (see ?lsmeans::lsmeans). Whatever max.print is passed to in this case doesn't change the value in getOption('max.print'). You could just as well do lsmeans(stuff, some_random_arg = 'whatever i want') and not have any error returned, but is completely meaningless.
You can make the change carry over from session to session by modifying your Rprofile. This question and this question might help getting you going in the right direction so you don't have to keep changing the max.print option everytime you start up a new R session. There may be an alternate way of doing it through RStudio, but I'm not familiar enough with the platform to help with that.