Search code examples
pythonapache-sparktensorflowpysparkdata-science-experience

TensorFrames not working with Tensorflow on Data Science Experience


This is a follow up from this question. I've imported the following jars into my notebook:

pixiedust.installPackage("http://central.maven.org/maven2/com/typesafe/scala-logging/scala-logging-slf4j_2.10/2.1.2/scala-logging-slf4j_2.10-2.1.2.jar") pixiedust.installPackage("http://central.maven.org/maven2/com/typesafe/scala-logging/scala-logging-api_2.10/2.1.2/scala-logging-api_2.10-2.1.2.jar")

But when I do an extremely basic command using tensorframes, I get the following error:

import tensorframes as tfs
from pyspark.sql import Row
data = [Row(x=[float(x), float(2 * x)],
            key=str(x % 2),
            z = float(x+1)) for x in range(1, 6)]
df = spark.createDataFrame(data)
tfs.print_schema(df)


.... NoClassDefFoundError: org.tensorflow.DataType ....

I can't update Tensorflow any more than it is, and even the latest version using python 3.5 is giving me an error. How should I be implementing tensorframes in IBM's Data Science Experience?


Solution

  • I found a jar that will get passed the above code, but I can't do any computations using TensorFrames. I suppose I can ask another follow up question:

    pixiedust.installPackage("http://central.maven.org/maven2/org/tensorflow/libtensorflow/1.1.0/libtensorflow-1.1.0.jar")