Loess curve fitting Plotly in R starting in third observation

Question

I'm following this tutorial on Scatterplot with LOESS Smoother but I want to be able to apply second derivate to the LOESS smoothed line to check where it reaches maximum so I can tell how many clusters are optimal, as if it was the elbow for k-means.

perplexi <- structure(list(Perplexity = c(NA, NA, 660, 596, 552, 480, 464, 
                      415, 399, 370, 349, 340, 327, 314, 288), Clusters = c(1, 2, 3, 
                      4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15)), class = "data.frame", row.names = c(NA, 
                      -15L))

library(plotly)

p <- plot_ly(perplexi[3:15,],
             x = ~Clusters,
             color = I("black")) %>% 
  add_markers(y = ~Perplexity) %>% 
  add_lines(y = ~fitted(loess(Perplexity ~ Clusters)),
                         line = list(color = 'lightblue'),
                         name = "Loess Smoother",
                         showlegend = F) %>% 
  layout(xaxis = list(title = 'Clusters'),
         yaxis = list(title = 'Perplexity')) %>% 
  add_trace(y = ~Perplexity,
            name = 'Perplexity',
            mode = 'markers',
            showlegend = F)

p

d1 <- diff(perplex); k <- which.max(abs(diff(d1) / diff(perplex[-1])))

Could somebody please point out what to do next? I want k to be for the smoothed line instead of the actual numbers so I know how many topics to perform.

Answer 1

One approach would be to fit the loess outside of plotly and then take the derivative.

loess.result <-loess.smooth(perplexi$Clusters, y=perplexi$Perplexity, evaluation = 20)
slopes <- diff(loess.result$x)/diff(loess.result$y)

plot_ly(perplexi[3:15,],
             x = ~Clusters,
             color = I("black")) %>% 
  add_markers(y = ~Perplexity) %>% 
  add_lines(y = ~fitted(loess(Perplexity ~ Clusters)),
                         line = list(color = 'lightblue'),
                         name = "Loess Smoother") %>% 
  layout(xaxis = list(title = 'Clusters'),
         yaxis = list(title = 'Perplexity')) %>% 
  add_trace(y = ~Perplexity,
            name = 'Perplexity',
            mode = 'markers',
            showlegend = F) %>%
  add_trace(x = loess.result$x[-1], y = slopes * -10000, mode = "line", name = "Loess First Derivative")