Search code examples
pyspark

creating a new column using the cosine function

I have a table. I need to create a new column, which will be the result of taking the cosine from the original one.

Column A
10
20

I tried the following option

import numpy as np import pyspark.sql.functions as F

df = (df .withColumn('new', np.cos(F.col('Column A))))

At the same time , I received the following error:

"loop of ufunc does not support argument 0 of type Column which has no callable sin method"

Solution

  • Try with cos function instead of numpy function.

    To get use numpy function you need to convert dataframe to pandas dataframe using toPandas() function.

    Example:

    from pyspark.sql.functions import cos
df.withColumn('new', cos(col("A"))).show()