apache-sparkhadoopapache-zeppelin
How can I tell which application I'm running in Spark?
I use Zeppelin to run Spark queries and we have the Spark application UI.
But, we only have one of the UI views for each of the running applications.
https://spark-history-server.[myURL].com/history/application_112378123821_98123
I have Zeppelin running on Hadoop. I'd like to find out what my application ID should be.
Is there a way that I can find this out from within a Zeppelin notebook?
Solution
Use sc.applicationId to get the application id for the current job.
Using Spark Session:
spark.sparkContext.applicationId
- Logging using Logback on Spark StandAlone
- How to properly checkpoint a dataframe in PySpark
- How to construct Dataframe from a Excel (xls,xlsx) file in Scala Spark?
- Is there any preference on the order of select and filter in spark?
- Spark: What is the difference between repartition and repartitionByRange?
- Updating values in apache parquet file
- Difference between ReduceByKey and CombineByKey in Spark
- Task not serializable exception while running apache spark job
- Apache Spark with Spring boot - failed to start exception Factory method 'javaSparkContext' threw exception with message: javax/servlet/Servlet
- which is the best way to convert json into a dataframe?
- How to read a file stored in adls gen 2 using pandas?
- How to define partitioning of DataFrame?
- Spark Shell: spark.executor.extraJavaOptions is not allowed to set Spark options
- Read CSV with "§" as delimiter using Databricks autoloader
- What are the benefits of Apache Beam over Spark/Flink for batch processing?
- How do I update a Spark setting in SparkR?
- How to handle accented letter in Pyspark
- difference between spark.kubernetes.driver.request.cores, spark.kubernetes.driver.limit.cores and spark.driver.cores
- Pyspark Streaming data to Elastic search index from Kafka topic , running in Jupyter notebook, causing failure
- Spark Send DataFrame as body of HTTP Post request
- How to handle an AnalysisException on Spark SQL?
- How convert a list into multiple columns and a dataframe?
- PySpark Window functions: Aggregation differs if WindowSpec has sorting
- Using rangeBetween considering months rather than days in PySpark
- Pyspark replace strings in Spark dataframe column
- Spark read from MongoDB and filter by objectId indexed field
- How to specify file size using repartition() in spark
- BloomFilter mergeInPlace() producing unexpected behavior
- Spark reading from mutiple SQL databases in parallel
- Spark partition size greater than the executor memory