Search code examples
pythonapache-sparkpysparkglob

Extracting from S3 a file path with star (*) sign in Spark python

One of my files has the following path:

s3://bucket/folder/file[1]*[2]

Now, when I'm trying to read it, using sc.textFile command, it cannot read it because of the star sign (*).

Is there a way to read it properly?

(I tried * , but that didn't work)

Any help would be greatly appreciated.

Solution

  • I don't know about the packages you're using, but with boto3 I use urlparse.quote_plus.

    https://docs.python.org/2/library/urllib.html#urllib.quote_plus