apache-sparkjupyteramazon-emrlivy
Increasing Spark application timeout in Jupyter/Livy
I'm using a shared EMR cluster with Jupyterhub installed. If my cluster is under heavy load, I get an error 
How do I increase the timeout for a spark application from 60 seconds to something greater like 900 seconds (15 mins)?
Solution
I've found the correct file to adjust the timeout.
/etc/jupyter/conf/config.json
"livy_session_startup_timeout_seconds": 900
Now the timeout is set to 900 seconds vs 60 before.
- Spark sending LIMIT to SQL Server on display function
- Start of the week on Monday in Spark
- Spark dataframe not adding columns with null values
- Azure Data Factory trigger Databricks cluster termination
- Spark-delta not working when upgrade to spark 3.5.0 and delta 3.1.0
- Spark Streaming - Refresh Static Data
- How to use Jts in spark in java
- Casting RDD to a different type (from float64 to double)
- symbolic reference class is not accessible: class sun.util.calendar.ZoneInfo, from interface spark.sql.catalyst.util.SparkDateTimeUtils
- Removing NULL items from PySpark arrays
- What is the fastest way to calculate a daily balance with compound interest in Pandas or Spark?
- Spark dataframe write method writing many small files
- How to calculate the size of dataframe in bytes in Spark?
- Spark Row object instantiated differently from overloaded prototypes?
- How to copy and convert parquet files to csv
- How to tune spark executor number, cores and executor memory?
- Spark/pyspark on same version but "py4j.Py4JException: Constructor org.apache.spark.api.python.PythonFunction does not exist"
- How to disable all logging info to the spark console from .net application
- How to resolve the following AWS Glue error while writing to Redshift using Spark: "ORA-01722: invalid number"?
- how to avoid row number in read_sql output
- Spark getItem shortcut
- Spark: fill spec value between flag values
- How to read Parquet file from S3 without spark? Java
- spark dataframe sum of column based on condition
- Check whether boolean column contains only True values
- reading csv file with header and tail into apache arrow
- Pyspark - how to initialize common DataFrameReader options separately?
- Open, High, Low, Close, Volume in PySpark using tick data
- Dataframe.write() produces csv file on single node jobs cluster, but not on 2+1 nodes cluster
- Internals of worker/executor usage during coalesce/repartition