pythonparquetazure-data-lakeazure-databricksazure-data-lake-gen2
Azure Databricks - Write Parquet file to Curated Zone
While writing parquet file back to DataLake Gen2 is creating additional files.
Example:
%python
rawfile = "wasbs://[email protected]/xxxx/2019-09-30/account.parquet"
curatedfile = "wasbs://[email protected]/xxxx-Curated/2019-09-30/account.parquet"
dfraw = spark.read.parquet(rawfile)
dfraw.write.parquet(curatedfile, mode = "overwrite")
display(dfraw)
File name supplied (account.parquet) is accounted to created folder rather creating file with that name.
How can these additional files be ignored and file written with name supplied.
Solution
When a user writes a file in a job, DBIO will perform the following actions for you.
- Tag files written with the unique transaction id.
- Write files directly to their final location.
- Mark the transaction as committed when the jobs commits.
It's not possible to do it directly to change the file name in Spark's save.
Spark uses Hadoop File Format, which requires data to be partitioned - that's why you have part- files. You can easily change filename after processing just like in the SO thread.
You may refer similar SO thread, which addressed similar issue.
Hope this helps.
- Python method chaining in functional programming style
- flask-jwt-extended: Fake Authorization Header during testing (pytest)
- For loop through the list unless empty?
- Polars make all groups the same size
- Is there a way to specify a default base-template for all templates in django?
- How to tackle time limit exceeded error in leetcode
- Is pd.get_dummies() updated in newer versions of Pandas making it default to Booleans (True/False) instead of (0/1)?
- What's the function like sum() but for multiplication? product()?
- How to type hint a dynamically-created dataclass
- Issue with pulling the data with EIA API with Python
- 403 Forbidden Error when scraping a site, user-agents already used and updated. Any ideas?
- Fullstack web-hosting services
- How to handle an AnalysisException on Spark SQL?
- Python requests is slow and takes very long to complete HTTP or HTTPS request
- Is there a way to modify an element in a Numpy array based on the value of other elements?
- Tkinter grid manager height/width nonconsistent
- Sql Alchemy Insert Statement failing to insert, but no error
- How can I create a Polars struct while eval-ing a list?
- Excel using win32com and python
- django: on pypy, psyco, unladen swallow or cpython, which one is the fastest?
- How convert a list into multiple columns and a dataframe?
- Name not defined in type annotation
- Static type checkers and Language Servers not recognizing attributes of objects that are subclasses
- How do I get multiple OID values in PySNMP?
- how to log a file in Django
- Is there a simple and efficient way to evaluate Elementary Symmetric Polynomials in Python?
- Iterating over two lists one after another
- Python -i flag for production
- What is PyCompilerFlags in Python C API?
- How to make Python check whether a variable is a number or letter