Search code examples
pythonapache-sparkpysparkjupyter-notebookspark-koalas

What does this mean ? WARNING:root:'PYARROW_IGNORE_TIMEZONE' environment variable was not set

I am working in Python on a Jupyter Notebook, and I got this warning:

WARNING:root:'PYARROW_IGNORE_TIMEZONE' environment variable was not set.

I tried to remove it, but I couldn't. I tried to set PYARROW_IGNORE_TIMEZONE to 1, as I saw on some forums but it didn't work.

Here is my code :

PYARROW_IGNORE_TIMEZONE=1
import databricks.koalas as ks
import pyspark
from pyspark.sql import SparkSession, functions
from pyspark.sql.types import *
import datetime

What's wrong with it ?

I am using spark and koalas.

Solution

  • If you want to set environment variable, you should use os. Otherwise you're just setting the variable in Python, but it doesn't get exported to the environment.

    import os
os.environ["PYARROW_IGNORE_TIMEZONE"] = "1"