Search code examples
javaapache-sparkamazon-s3dependencies

Fail to resolving dependencies of org.apache.hadoop:hadoop-aws:2.7.6

When I type following command to let my java spark to read the csv file from amazon s3

spark-submit --class myApp --packages org.apache.hadoop:hadoop-aws:2.7.6 --master local ./target/spark_pratice.jar

Error Message 


                ::::::::::::::::::::::::::::::::::::::::::::::

                ::              FAILED DOWNLOADS            ::

                :: ^ see resolution messages for details  ^ ::

                ::::::::::::::::::::::::::::::::::::::::::::::

                :: javax.activation#activation;1.1!activation.jar

                :: asm#asm;3.2!asm.jar

                :: org.apache.avro#avro;1.7.4!avro.jar

                :: com.thoughtworks.paranamer#paranamer;2.3!paranamer.jar

                :: org.xerial.snappy#snappy-java;1.0.4.1!snappy-java.jar(bundle)

                :: io.netty#netty;3.6.2.Final!netty.jar(bundle)

                ::::::::::::::::::::::::::::::::::::::::::::::
:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [
    download failed: javax.activation#activation;1.1!activation.jar, 
    download failed: asm#asm;3.2!asm.jar, 
    download failed: org.apache.avro#avro;1.7.4!avro.jar, 
    download failed: com.thoughtworks.paranamer#paranamer;2.3!paranamer.jar, 
    download failed: org.xerial.snappy#snappy-java;1.0.4.1!snappy-java.jar(bundle), 
    download failed: io.netty#netty;3.6.2.Final!netty.jar(bundle)
]

I don't know how to resolve this problem. Should I update anything ? maven ? or something else ?

Solution

  • Finally I try to download the missing jar files from maven repository maven repository

    And I put them into properly repository of .m2/repository