bluemix spark as a service - how to run spark-submit.sh with cygwin?

Question

The custom spark-submit.sh script provided on bluemix states the following restriction:

Restriction: Running the spark-submit.sh script is supported on Linux and Mac OS X.

Source: https://console.ng.bluemix.net/docs/services/AnalyticsforApacheSpark/index-gentopic3.html#genTopProcId4

Presumably the script will also work with cygwin as long as the correct binaries are available, e.g. bash, sed, grep, curl, hash?

Answer 1

I don't see why it should not work in Cygwin but there may be some things that may fail which you will know when you test different scenarios.

I tested this with cygwin and very simple program hellopy.py that prints out sparkcontext version and it works fine.

I get all the files stdout, sterr, spark-submit_xxx.log. So curl works.

Here is what i installed when i install cygwin(bash, sed, grep, curl, hash and OPENSSL, libdev). Get all the higher versions.(Curl 7.471. libcurl 7.48, OpenSSL 1.0.2 libssh)

This is what i see when i do curl -V $ curl -V curl 7.47.1 (i686-pc-cygwin) libcurl/7.48.0 OpenSSL/1.0.2h zlib/1.2.8 libidn/1.29 libpsl/0.13.0 (+libidn/1.29) libssh2/1.7.0 nghttp2/1.7.1 Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp Features: Debug IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets Metalink PSL

Note:- When you search for the above libraries in cygwin installation page, just check other supporting libraries like perl and other stuff(whatever comes in suggestion).

I hope it helps. Consider above as temporary solution until it is officially tested.

Thanks, Charles.