I used both quanteda and stm packages. The first one helps me to preprocess data, and I did topic modeling with the second packages.
When I try to use findthoughts function, I find following errors:
Error in if (!is.null(texts) && length(texts) != nrow(theta)) stop("Number of
provided texts and number of documents modeled do not match") :
missing value where TRUE/FALSE needed
I think this is because I removed the empty lines from my original text by using the following command
text <- rs[complete.cases(data), ]
and using sparsity=0.99, which also remove some less used words.
So the original text and new text does not match. However, I do not know how I can reach the new text file after dfm function?
In the reproducible example (not my own data), if we assume there are missing values in the text, can you help me to get the text after dfm function.
library(stm)
library(quanteda)
data <- corpus(gadarian, text_field = 'open.ended.response')
docvars(data)$text <- texts(data)
data <- dfm(data, stem = TRUE, remove = stopwords('english'),
remove_punct = TRUE) %>% dfm_trim(min_count = 2)
out <- convert(data, to = 'stm')
gadarian_3 <- stm(documents = out$documents,
vocab = out$vocab,
data = out$meta,
prevalence = ~ treatment + s(pid_rep),
K = 10, verbose = FALSE)
outputFit <- gadarian_3$runout[[1]]
thoughts1<-findThoughts(gadarian_3, texts=textdata , n=10, topics=1)$docs[[1]]
The texts are saved in the converted STM input object, here, the object called out. You added the original text as a document variable called text so this is accessible through out$meta$text.
str(out)
# List of 3
# $ documents:List of 341
# ..$ 1 : int [1:2, 1:11] 72 1 73 1 108 1 216 2 223 1 ...
# ..$ 2 : int [1:2, 1:7] 57 1 101 1 190 1 223 1 229 1 ...
# ..$ 3 : int [1:2, 1:16] 144 1 148 1 150 1 156 1 183 1 ...
# ..$ 4 : int [1:2, 1:27] 26 1 60 1 69 1 105 2 150 3 ...
# .. [list output truncated]
# $ vocab : chr [1:482] "#1" "1" "2" "3" ...
# $ meta :'data.frame': 341 obs. of 4 variables:
# ..$ MetaID : num [1:341] 0 0 0 0 0 0 0 0 0 0 ...
# ..$ treatment: num [1:341] 1 1 0 0 1 1 1 1 0 1 ...
# ..$ pid_rep : num [1:341] 1 1 0.333 0.5 0.667 ...
# ..$ text : chr [1:341] "problems caused by the influx of ..." [TRUNCATED]
So this will work:
thoughts1 <- findThoughts(gadarian_3, texts = out$meta$text,
n = 10, topics = 1)$docs[[1]]
head(thoughts1)
# [1] "as an arizona resident who lives 18 miles from the mexican-us border, and who has also spoken to some of these illegals while hiking in the huachuca mtns., i know these people, mostly, come here out of sheer desperation. sure, some are the same lazy, fat, undereducated jerks that lurk around our own mid-level businesses. but most simply are people who want what we all do: a comfortable life with as little thinking and suffering as possible, while reproducing at will. they have told me, babies in arms,that if they remain at home, they have no future but an early death. that they, maybe, should reduce their birth rate and/or not have children at all, if they cannot support them, simply will never occur to citizens of a catholic country, living a day's walk from a rich country that can be easily milked for what they consider a fortune in life support. there is no answer to this, so long as 95% of mexico's wealth is controlled by 5% of its people, and the only riches the others have lie in their children."
# [2] "people moving from one place to another, mostly for a better economic future."
# [3] "the construction of the fence along the border. the deaths of people smuggled into the us in unventilated trucks. people starving or freezing to death in the desert"
# [4] "i think of, first off, where i grew up. southern california is full of immigrants from much of south & central america."
# [5] "we need to protect our borders more. not enough agents covering too much distance."
# [6] "need better border build a wall like china did"