apache-sparkpysparkapache-spark-sqlazure-databricksdelta-lake
How to Retain DataType After Pyspark Merge with Databricks
I have a Databricks Dataframe with the following Schema:
You will notice the Data Type for the CreatedOn field is a DateType()
However, after doing a merge and loading the data with the variable lakeDataDrop
lakeDataDrop = spark.read.format("delta").load(saveloc)
the Data Type changes to TimestampType()
Can someone explain why the Data Type changes to TimestampType()?
I would like the Data Type to remain as a DateType()
Solution
It looks like in your dataset that you're using for merge the corresponding column has the timestamp type, so if you have schema evolution enabled, then the type is promoted from date to timestamp. Solution would be to cast the column in the "updates" dataframe that you're using in merge to date before merge.
- difference between spark.kubernetes.driver.request.cores, spark.kubernetes.driver.limit.cores and spark.driver.cores
- Pyspark Streaming data to Elastic search index from Kafka topic , running in Jupyter notebook, causing failure
- Spark Send DataFrame as body of HTTP Post request
- How to handle an AnalysisException on Spark SQL?
- How convert a list into multiple columns and a dataframe?
- PySpark Window functions: Aggregation differs if WindowSpec has sorting
- Using rangeBetween considering months rather than days in PySpark
- Pyspark replace strings in Spark dataframe column
- Spark read from MongoDB and filter by objectId indexed field
- How to specify file size using repartition() in spark
- BloomFilter mergeInPlace() producing unexpected behavior
- Spark reading from mutiple SQL databases in parallel
- Spark partition size greater than the executor memory
- Last day of quarter
- Java spark Map is empty
- corrupted record from json file in pyspark due to False as entry
- Chain several WHEN conditions in a scalable way in PySpark
- How to extract all elements from array of structs?
- How to get the schema definition from a dataframe in PySpark?
- Get index of item in array column in a Spark dataframe
- How to find difference between two sequential array items in spark sql
- How to bootstrap installation of Python modules on Amazon EMR?
- How to find installation directory of Apache Spark package in Homebrew?
- checksum error while writing data to delta table. Is there a way to fix this issue?
- Insert column at specified position
- Count particular characters within a column using Spark Dataframe API
- Stream-Static Join: How to refresh (unpersist/persist) static Dataframe periodically
- Can you construct pyspark.pandas.DataFrame from pyspark.sql.dataframe.DataFrame?
- Feed the result of one query to another in the same Spark Structured Streaming app
- AssertDataFrameEqual doesn't throw error with None dataframe in Pyspark