Does anyone know how to access the old driver log files from the databricks platform (User interface) from a specific cluster? I'm only able to see 4 files generated today.
I have the impression that the oldest logs are deleted on a regular basis. Am I wrong?
If the cluster is shut down, you can see old logs up to 30 days. The running cluster would show only driver logs since the latest restart. You can enable cluster log delivery to have driver/executor/init script logs delivered to DBFS/S3 location. You can follow the below doc to configure it.
https://docs.databricks.com/clusters/configure.html#cluster-log-delivery-1