Search code examples
apache-sparkibm-clouddata-science-experience

Spark history server is not showing 'complete' applications

I am trying to performance tune a slow running DSX job.

I have navigated to the spark history server from the underlying spark service on Bluemix (as per this question).

I have executed a cell containing some basic spark code:

In [1]:
x = sc.parallelize(range(1, 1000000))
x.collect()

Out[1]:
[1,
 2,
 3,
 4,
 5,
 ...

I have then refreshed the Job History Server page in the browser, however, the spark history server is not showing any complete applications:

enter image description here

How can I find the 'complete' applications?

Update

The spark service I'm referring to is IBM's managed spark service on Bluemix so I don't have any control over the configuration.

Update 2

It looks as though the dates are getting corrupted which is why I'm not seeing completed jobs:

enter image description here

Solution

  • I have taken this up with the spark service engineering team - it is a known issue.