Importing a Model with Scikit Learn on Vertex

Question

I'm trying to import a model from my local but everytime I get the same error from gcp logs. The framework is scikit-learn AttributeError: Can't get attribute 'preprocess_text' on <module 'model_server' from '/usr/app/model_server.py'> The code snippet with this problem is

complaints_clf_pipeline = Pipeline(
    [
        ("preprocess", text.TfidfVectorizer(preprocessor=utils.preprocess_text, ngram_range=(1, 2))),
        ("clf", naive_bayes.MultinomialNB(alpha=0.3)),
    ]
)

this

preprocess_text 

comes from the cell above, but I keep receiving this issue with model_server which is not present on my code.

Can someone help?

I tried to refactor the code but got the same error, tried to undo this pipeline structure but then I got another error while trying to consult the model by API.

Answer 1

GCP is trying to load the model, but it can't find the preprocess_text function because it's not included in the serialized model.

Save the scikit-learn pipeline, functions like preprocess_text are not automatically saved with the model. To ensure that GCP knows where to find this function, you can either:

Define preprocess_text inside the same script where you're loading the model, or Package utils as part of your deployment (including it in your GCP deployment files) so that the preprocess_text function is available in the same environment.

import pickle
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB

class CustomTextClassifier:
    def __init__(self):
        self.pipeline = Pipeline(
            [
                ("preprocess", TfidfVectorizer(preprocessor=self.preprocess_text, ngram_range=(1, 2))),
                ("clf", MultinomialNB(alpha=0.3)),
            ]
        )

    def preprocess_text(self, text):
        
        return text.lower() 

    def train(self, X, y):
        self.pipeline.fit(X, y)

    def predict(self, X):
        return self.pipeline.predict(X)


model = CustomTextClassifier()
# train model with your data...
with open('model.pkl', 'wb') as f:
    pickle.dump(model, f)