Unable to collect spark dataset as list or map with No applicable constructor error

Question

I have a class

@Getter
@Setter
@AllArgsConstructor
@NoArgsConstructor
public class Store {

    private Double probability;

    private String store;

}

and json file

{"probability":"0.26","store":"abc/s3"}
{"probability":"0.57","store":"abc/s1"}

I try to read it as a dataset and convert it to map. Reading as dataset is successful and able to operate on it using spark sql commands and also able to view the dataset using show(), etc

Dataset<Store> ds = ss.read().json(path).as(Encoders.bean(Store.class));
Map<String, Double> storeMap = ds.collectAsList().stream()
                .collect(Collectors.toMap(Store::getStore, Store::getProbability));

But converting to map fails with error. This error is on the command ds.collectAsList() itself

No applicable constructor/method found for actual parameters 

"org.apache.spark.unsafe.types.UTF8String"; 
candidates are: 
"public static java.lang.Double java.lang.Double.valueOf(java.lang.String) throws java.lang.NumberFormatException",
"public static java.lang.Double java.lang.Double.valueOf(double)"

What am i doing wrong ?

Answer 1

problem is not in your code but in your JSON data . It should be

{"probability":0.26,"store":"abc/s3"}
{"probability":0.57,"store":"abc/s1"}

Or change private Double probability; to private String probability;