I am trying to implement one of the solutions to the question about How to align two GloVe models in text2vec?. I don't understand what are the proper values for input at GlobalVectors$new(..., init = list(w_i, w_j). How do I ensure the values for w_i and w_j are correct?
Here's a minimal reproducible example. First, prepare some corpora to compare, taken from the quanteda tutorial. I am using dfm_match(all_words) to try and ensure all words are present in each set, but this doesn't seem to have the desired effect.
library(quanteda)
# from https://quanteda.io/articles/pkgdown/replication/text2vec.html
# get a list of all words in all documents
all_words <-
data_corpus_inaugural %>%
tokens(remove_punct = TRUE,
remove_symbols = TRUE,
remove_numbers = TRUE) %>%
types()
# should expect this mean features in each set
length(all_words)
# these are our three sets that we want to compare, we want to project the
# change in a few key words on a fixed background of other words
corpus_1 <- data_corpus_inaugural[1:19]
corpus_2 <- data_corpus_inaugural[20:39]
corpus_3 <- data_corpus_inaugural[40:58]
my_tokens1 <- texts(corpus_1) %>%
char_tolower() %>%
tokens(remove_punct = TRUE,
remove_symbols = TRUE,
remove_numbers = TRUE)
my_tokens2 <- texts(corpus_2) %>%
char_tolower() %>%
tokens(remove_punct = TRUE,
remove_symbols = TRUE,
remove_numbers = TRUE)
my_tokens3 <- texts(corpus_3) %>%
char_tolower() %>%
tokens(remove_punct = TRUE,
remove_symbols = TRUE,
remove_numbers = TRUE)
my_feats1 <-
dfm(my_tokens1, verbose = TRUE) %>%
dfm_trim(min_termfreq = 5) %>%
dfm_match(all_words) %>%
featnames()
my_feats2 <-
dfm(my_tokens2, verbose = TRUE) %>%
dfm_trim(min_termfreq = 5) %>%
dfm_match(all_words) %>%
featnames()
my_feats3 <-
dfm(my_tokens3, verbose = TRUE) %>%
dfm_trim(min_termfreq = 5) %>%
dfm_match(all_words) %>%
featnames()
# leave the pads so that non-adjacent words will not become adjacent
my_toks1_2 <- tokens_select(my_tokens1, my_feats1, padding = TRUE)
my_toks2_2 <- tokens_select(my_tokens2, my_feats2, padding = TRUE)
my_toks3_2 <- tokens_select(my_tokens3, my_feats3, padding = TRUE)
# Construct the feature co-occurrence matrix
my_fcm1 <- fcm(my_toks1_2, context = "window", tri = TRUE)
my_fcm2 <- fcm(my_toks2_2, context = "window", tri = TRUE)
my_fcm3 <- fcm(my_toks3_2, context = "window", tri = TRUE)
Somewhere in the above steps I believe I need to ensure that the fcm for each set has all the words of all sets to get the matrix dimensions the same, but I'm not sure how to accomplish that.
Now fit the word embedding model for the first set:
library("text2vec")
glove1 <- GlobalVectors$new(rank = 50,
x_max = 10)
my_main1 <- glove1$fit_transform(my_fcm1,
n_iter = 10,
convergence_tol = 0.01,
n_threads = 8)
my_context1 <- glove1$components
word_vectors1 <- my_main1 + t(my_context1)
And here is where I get stuck, I want to initialise the second model with the first, so that the coordinate system will be comparable between the first and second models. I read that w_i and w_j are main and context words, and b_i and b_j are biases. I've found output for those in my first model object, but I get an error:
glove2 <- GlobalVectors$new(rank = 50,
x_max = 10,
init = list(w_i = glove1$.__enclos_env__$private$w_i,
b_i = glove1$.__enclos_env__$private$b_i,
w_j = glove1$.__enclos_env__$private$w_j,
b_j = glove1$.__enclos_env__$private$b_j))
my_main2 <- glove2$fit_transform(my_fcm2,
n_iter = 10,
convergence_tol = 0.01,
n_threads = 8)
The error is Error in glove2$fit_transform(my_fcm2, n_iter = 10, convergence_tol = 0.01, :
init values provided in the constructor don't match expected dimensions from the input matrix
Assuming I have identified w_i, etc., correctly in the first model, how can I get ensure they are the correct size?
Here's my session info:
sessionInfo()
R version 3.6.0 (2019-04-26)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS 10.15.2
Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] text2vec_0.6 quanteda_2.0.0
loaded via a namespace (and not attached):
[1] Rcpp_1.0.4 pillar_1.4.3 compiler_3.6.0 tools_3.6.0 stopwords_1.0
[6] digest_0.6.25 packrat_0.5.0 lifecycle_0.2.0 tibble_3.0.0 gtable_0.3.0
[11] lattice_0.20-40 pkgconfig_2.0.3 rlang_0.4.5 Matrix_1.2-18 fastmatch_1.1-0
[16] cli_2.0.2 rstudioapi_0.11 mlapi_0.1.0 parallel_3.6.0 RhpcBLASctl_0.20-17
[21] dplyr_0.8.5 vctrs_0.2.4 grid_3.6.0 tidyselect_1.0.0.9000 glue_1.3.2
[26] data.table_1.12.8 R6_2.4.1 fansi_0.4.1 lgr_0.3.4 ggplot2_3.3.0
[31] purrr_0.3.3 magrittr_1.5 scales_1.1.0 ellipsis_0.3.0 assertthat_0.2.1
[36] float_0.2-3 rsparse_0.4.0 colorspace_1.4-1 stringi_1.4.6 RcppParallel_5.0.0
[41] munsell_0.5.0 crayon_1.3.4.9000
Here is a working example. See ?rsparse::GloVe documentation for details.
library(rsparse)
data("movielens100k")
x = crossprod(sign(movielens100k))
model = GloVe$new(rank = 10, x_max = 5)
w_i = model$fit_transform(x = x, n_iter = 5, n_threads = 1)
w_j = model$components
init = list(w_i = t(w_i), model$bias_i, w_j = w_j, b_j = model$bias_j)
model2 = GloVe$new(rank = 10, x_max = 10, init = init)
w_i2 = model2$fit_transform(x)