scalaapache-sparkapache-spark-mllibword2vec
Why does word2vec only take one task for mapPartitionsWithIndex at Word2Vec.scala:323
I am running word2vec in spark and when it comes to fit(), only one task is observed in UI as in image:
.
As per the configuration, num-executors = 1000, executor-cores = 2. And the RDD coalesces to 2000 partitions. It takes quite a long time for mapPartitionsWithIndex. Can it be distributed to multiple executors or tasks?
Solution
setNumPartitions(numPartitions: Int) solves my problem. I did not check the default value.
Sets number of partitions (default: 1).
- Check a list contains an instance of each possible case class
- Append the "_commit_timestamp" Column to the Latest Data Version When Reading from a DeltaTable
- How to read the json file in spark using scala?
- How to add self-signed generated certificate to trusted certificates from inside a Java keystore?
- filter only not empty arrays dataframe spark
- How can I get all names of the arrays on Dataframe
- create a Spark DataFrame from a nested array of struct element?
- spark UI - Understand metrics memory used
- Tapir custom http code in successful response
- Remove list elements in a dataframe in scala
- Search a dataframe from a list and add column to say found or not
- Not able to Explode and select in the same expression in spark scala
- Why am I getting "No SLF4J Providers found" in my Scala + AKKA Jar?
- How to convert a java Map to immutable Scala map via java code?
- How to correctly read a CSV file while escaping delimiter comma placed within square brackets using Apache Spark and Scala?
- How to remove an item from a list in Scala having only its index?
- Multiple image operations in a single process with gm4java
- In Scala, how would I update specific items in a Seq using a filter?
- Scala actors as single-threaded queues
- What is the difference between Abstract Data Types and Algebraic Data Types
- How to cancel a future action if another future did failed?
- Manifest and abstract type resolution
- How to I convert a HttpEntity into a Protobuf object in Scala using Akka-http?
- How to keep reference to a created instance of Actor class with Akka?
- Scala: An extension method was tried, but could not be fully constructed (same extension name on multiple classes)
- Doobie - lifting arbitrary effect into ConnectionIO
- Why does Scala not have the concept of checked and unchecked exceptions?
- Trying to create a classic actor from inside of a akka typed application
- How to get a random element from a Set in Scala
- Convert Iterable[Any] to string in scala