How to query a persisted dataframe in spark job (A) from another spark job (B)
There are two spark streaming jobs running on different containers - let's call them teacher job and student job. Both are reading from two different kafka topics. When a student message comes into the student spark job, I need to 'query' the teacher job's persisted data to retrieve the teacher associated with that student (in this example, the student only has one teacher but a teacher can have many students). How can I persist a key value pair(or a teacher data frame) in the teacher job and then retrieve/lookup that teacher in the student job so I can process that student knowing it's teacher? Can I use persist() in one job and unpersist() in another?
From the evidence it appears that Spark Structured Streaming with Kafka Integration utilizing Stream - Stream Join is the way to go.
- How to properly checkpoint a dataframe in PySpark
- How to construct Dataframe from a Excel (xls,xlsx) file in Scala Spark?
- Is there any preference on the order of select and filter in spark?
- Spark: What is the difference between repartition and repartitionByRange?
- Updating values in apache parquet file
- Difference between ReduceByKey and CombineByKey in Spark
- Task not serializable exception while running apache spark job
- Apache Spark with Spring boot - failed to start exception Factory method 'javaSparkContext' threw exception with message: javax/servlet/Servlet
- which is the best way to convert json into a dataframe?
- How to read a file stored in adls gen 2 using pandas?
- How to define partitioning of DataFrame?
- Spark Shell: spark.executor.extraJavaOptions is not allowed to set Spark options
- Read CSV with "§" as delimiter using Databricks autoloader
- What are the benefits of Apache Beam over Spark/Flink for batch processing?
- How do I update a Spark setting in SparkR?
- How to handle accented letter in Pyspark
- difference between spark.kubernetes.driver.request.cores, spark.kubernetes.driver.limit.cores and spark.driver.cores
- Pyspark Streaming data to Elastic search index from Kafka topic , running in Jupyter notebook, causing failure
- Spark Send DataFrame as body of HTTP Post request
- How to handle an AnalysisException on Spark SQL?
- How convert a list into multiple columns and a dataframe?
- PySpark Window functions: Aggregation differs if WindowSpec has sorting
- Using rangeBetween considering months rather than days in PySpark
- Pyspark replace strings in Spark dataframe column
- Spark read from MongoDB and filter by objectId indexed field
- How to specify file size using repartition() in spark
- BloomFilter mergeInPlace() producing unexpected behavior
- Spark reading from mutiple SQL databases in parallel
- Spark partition size greater than the executor memory
- Last day of quarter