Search code examples
apache-sparkmachine-learningpysparkapache-spark-mlmachine-learning-model

Is it possible to save a trained Spark ML model or crossvalidator to postgree database?

Is it possible to save a trained Spark ML model or crossvalidator to postgree database ? I could find ways to save it to disk or HDFS but can we save it in RDBMS(e.g. Postgree) table ?

Solution

  • In general it is not. ML models use specific structure with Parquet files storing model, metadata and optionally data. It is possible to create wrappers that could store the same data in the relational database, but there is no such functionality in Spark at this moment.

    For some subset of models you can:

    • Serialize model using Java serialization.
    • Store saved model as a Large Object.

    This is limited to non-distributed models.