How reliable is the Elbow curve in finding K in K-Means?
So I was trying to use the Elbow curve to find the value of optimum 'K' (number of clusters) in K-Means clustering.
The clustering was done for the average vectors (using Word2Vec) of a text column in my dataset (1467 rows). But looking at my text data, I can clearly find more than 3 groups the data can be grouped into.
I read the reasoning is to have a small value of k while keeping the Sum of Squared Errors (SSE) low. Can somebody tell me how reliable the Elbow Curve is? Also if there's something I'm missing.
Attaching the Elbow curve for reference. I also tried plotting it up to 70 clusters, exploratory.
.
The "elbow" is not even well defined so how can it be reliable?
You can "normalize" the values by the expected dropoff from splitting the data into k clusters and it will become a bit more readable. For example, the Calinski and Harabasz (1974) variance ratio criterion. It is essentially a rescaled version that makes much more sense.
- Implementation of Okapi BM25 in python
- How do I parse a string to a float or int?
- What is the best design pattern when creating an interface for functions that can return a variety of responses?
- How can I sandbox Python in pure Python?
- Currency formatting in Python
- How boxen plot is different from box plot?
- Sorting names by their high scores
- How do I serve both HTML and JSON content with Flask?
- Changing lowercase characters to uppercase and vice-versa in Python
- WARNING Not Found: /favicon.ico in Django
- SQL Alchemy - How to delete a model instance?
- What does the python operator =- do?
- How can I unit test the jinja2 template logic?
- python pandas dataframe select rows with ternary operator
- Selenium using Python - Geckodriver executable needs to be in PATH
- Gunicorn do not reload worker
- How to add timestamp to each request in uvicorn logs?
- How to extract video titles with the Youtube API(Python)
- Cartesian product of dict of lists in Python
- Round to 5 (or other number) in Python
- Why do I get a "ModuleNotFoundError" in VS Code despite the fact that I already installed the module?
- Python: Suppressing errors from going to commandline?
- Problems using MySQL with AWS Lambda in Python
- Is there any benefit of using a dictionary comprehension if an equivalent dict(zip(a, b)) is possible?
- SQLAlchemy - Get query results in same order as IN clause
- Python3 __pycache__ generating even if PYTHONDONTWRITEBYTECODE=1
- Matplotlib.animation: how to remove white margin
- bash script maybe syntax error not sure trying a until loop
- How to typehint dataclass field in function signature
- pdfium library is not working when compiling my code in python