How to fix error in R regarding spakly model

Can some help me with error im getting in R sparkly

kmeans_model <- iris_tbl %>%
   select(Petal_Width, Petal_Length) %>%
   ml_kmeans(centers = 3)

Error: java.lang.IllegalArgumentException: Field "features" does not exist. Available fields: Petal_Width, Petal_Length

    at org.apache.spark.sql.types.StructType$$anonfun$apply$1.apply(StructType.scala:274)

    at org.apache.spark.sql.types.StructType$$anonfun$apply$1.apply(StructType.scala:274)

    at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)

    at scala.collection.AbstractMap.getOrElse(Map.scala:59)

    at org.apache.spark.sql.types.StructType.apply(StructType.scala:273)

    at org.apache.spark.ml.util.SchemaUtils$.checkColumnTypes(SchemaUtils.scala:58)

    at org.apache.spark.ml.util.SchemaUtils$.validateVectorCompatibleColumn(SchemaUtils.scala:119)

    at org.apache.spark.ml.clustering.KMeansParams$class.validateAndTransformSchema(KMeans.scala:96)

    at org.apache.spark.ml.clustering.KMeans.validateAndTransformSchema(KMeans.scala:285)

    at org.apache.spark.ml.clustering.KMeans.transformSchema(KMeans.scala:382)

    at org.apache.spark.ml.PipelineStage.transformSchema(Pipeline.scala:74)

    at org.apache.spark.ml.clustering.KMeans$$anonfun$fit$1.apply(KMeans.scala:341)

    at org.apache.spark.ml.clustering.KMeans$$anonfun$fit$1.apply(KMeans.scala:340)

    at org.apache.spark.ml.util.Instrumentation$$anonfun$11.apply(Instrumentation.scala:183)

    at scala.util.Try$.apply(Try.scala:192)

    at org.apache.spark.ml.util.Instrumentation$.instrumented(Instrumentation.scala:183)

    at org.apache.spark.ml.clustering.KMeans.fit(KMeans.scala:340)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

    at java.lang.reflect.Method.invoke(Unknown Source)

    at sparklyr.Invoke.invoke(invoke.scala:139)

    at sparklyr.StreamHandler.handleMethodCall(stream.scala:123)

    at sparklyr.StreamHandler.read(stream.scala:66)

    at sparklyr.BackendHandler.channelRead0(handler.scala:51)

    at sparklyr.BackendHandler.channelRead0(handler.scala:4)

    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)

    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)

    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)

    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)

    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)

    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)

    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)

    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)

    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)

    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)

    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)

    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)

    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)

    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)

    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)

    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)

    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)

    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:138)

    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)

    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)

    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)

    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)

    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)

    at java.lang.Thread.run(Unknown Source)

Warning: Some components of ... were not used: centers

I already tried to use other function, but it doesn't wotk for 3 clusters and picking up only 2

kmeans_model <- iris_tbl %>% 
ml_kmeans(formula= ~ Petal_Width + Petal_Length, centers = 3)

#Warning: Some components of ... were not used: centers

print(kmeans_model)
#K-means clustering with 2 clusters
#
#Cluster centers:
#  Petal_Width Petal_Length
#1   1.6818182     4.925253
#2   0.2627451     1.492157
#
#Within Set Sum of Squared Errors =  86.39022>

Solution

The first line of your error is very straight forward:

Field "features" does not exist.

If you check the documentation for ?ml_kmeans you would see that you need to either specify a formula (your second attempt) or the features_col. Now a qucik note, in Spark features for a model are expected to be vectorized within one column of a data.frame

Your second error/warning message is also straight forward:

Warning: Some components of ... were not used: centers

centers is not a parameter in ml_kmeans. What you want to use is k

kmeans_model <- iris_tbl %>% 
       ml_kmeans(formula= ~ Petal_Width + Petal_Length, k = 3)

kmeans_model
# K-means clustering with 3 clusters
# 
# Cluster centers:
#   Petal_Width Petal_Length
# 1    1.359259     4.292593
# 2    0.246000     1.462000
# 3    2.047826     5.626087
# 
# Within Set Sum of Squared Errors =  31.41289

To run without a formula you need to use ft_vector_assembler

kmeans_model <- iris_tbl %>% 
  ft_vector_assembler(input_cols=c("Sepal_Width","Petal_Length"), output_col="features") %>%
  ml_kmeans(k = 3)