Does stage kill from Spark UI causes a reprocessing of the data?
In Spark UI there is this capability of killing an actively running stage:
When a stage is killed with this button, the tasks that are associated with this stage will be reprocessed? Or they'll be skipped?
I noticed that when a stage is killed, the associated job is killed as well, which makes me think that no reprocessing happens, but I'd like to know if there is an official doc or something which states this clearly (or if someone played around with this feature and knows how it works).
In active Stages, it’s possible to kill the active Stage with the kill link - as you show.
The App is killed as well normally, so no re-processing of Stage occurs. Neither can that killed Stage be skipped for current path of processing; that would be not so logical.
If you have an App with more independent paths of processing, then the other paths of processing continue.
Note that only in failed stages, a failure reason is shown. You can still see the output of History Server, of course.
That is to say, confirming your observations.
- Spark dataframe write method writing many small files
- How to calculate the size of dataframe in bytes in Spark?
- Spark Row object instantiated differently from overloaded prototypes?
- How to copy and convert parquet files to csv
- How to tune spark executor number, cores and executor memory?
- Spark/pyspark on same version but "py4j.Py4JException: Constructor org.apache.spark.api.python.PythonFunction does not exist"
- How to disable all logging info to the spark console from .net application
- How to resolve the following AWS Glue error while writing to Redshift using Spark: "ORA-01722: invalid number"?
- how to avoid row number in read_sql output
- Spark getItem shortcut
- Spark: fill spec value between flag values
- How to read Parquet file from S3 without spark? Java
- spark dataframe sum of column based on condition
- Check whether boolean column contains only True values
- reading csv file with header and tail into apache arrow
- Pyspark - how to initialize common DataFrameReader options separately?
- Open, High, Low, Close, Volume in PySpark using tick data
- Dataframe.write() produces csv file on single node jobs cluster, but not on 2+1 nodes cluster
- Internals of worker/executor usage during coalesce/repartition
- How to run Multi threaded jobs in apache spark using scala or python?
- Including null values in an Apache Spark Join
- Why is my PySpark row_number column messed up when applying a schema?
- Split a datafarme column based on another column - Column is not iterable
- Issue with Multiple Spark Structured Streaming Jobs Consuming Same Kafka Topic
- Pass system property to spark-submit and read file from classpath or custom path
- Pandas cannot read parquet files created in PySpark
- conditional split based on list of column
- Explode JSON array into rows
- How to use pyspark regex to correctly break data with pipe delimited with literal pipe inside?
- Meaning of Exchange in Spark Stage