pythonpython-3.xpandasdataframepyspark
AttributeError: 'DataFrame' object has no attribute 'iteritems'
I am using pandas to read csv on my machine then I create a pyspark dataframe from pandas dataframe.
df = spark.createDataFrame(pandas_df)
I updated my pandas from version 1.3.0 to 2.0
Now, I am getting this error:
AttributeError: 'DataFrame' object has no attribute 'iteritems'
Solution
Found an answer on github: https://github.com/YosefLab/Compass/issues/92
It is an issue going on.
iteritems is removed from pandas 2.0
For now I need to downgrade pandas back to version 1.5.3
Edit:
Other workarounds may be
Use the latest Spark (3.4.1)
https://spark.apache.org/downloads.html
For pandas >=2.0
You can also assign DataFrame.items to DataFrame.iteritems
import pandas as pd
pd.DataFrame.iteritems = pd.DataFrame.items
- How to web scrap Economic Calendar data from TradingView and load into Dataframe?
- How do I force a Mako function to require named arguments?
- How to scrape a csv table contained within a JavaScript button using Python?
- range() with x Base (not base 10)
- Accessing power plans with WMI
- FIFO class in python library?
- Selenium in Python: "NoSuchElementException: Message: no such element: Unable to locate element"
- How to decrypt password from JavaScript CryptoJS.AES.encrypt(password, passphrase) in Python
- Convert UTC to local time zone in 'Views'
- Python: Unable to Render Tex in Matplotlib
- How to translate this Math Formula in Haskell or Python? (Was translated in PHP)
- spaCy incorrectly recognizing finger as verb
- How to calculate rolling / moving average using python + NumPy / SciPy?
- MoviePy TextClip set color to RGB value
- Aggregate columns that fall within range
- Installing Python on iSH
- Adjusting the spacing between the edge of the plot and the x-axis
- How to implement CRTP functionality in python?
- Open a new window when clicking on a mesh in Open3D
- selenium.common.exceptions.InvalidArgumentException: Message: invalid argument error invoking get() with urls read from text file with Selenium Python
- Python String to Int Or None
- Is there a way to do some pre-process/post-process every time I call a function in python
- Get the title of a window of another program using the process name
- Sum of the [maximums of all subarrays multiplied by their lengths], in linear time
- Struggling to upload data onto MongoDB
- Django: TemplateSyntaxError: Could not parse the remainder
- How to fix: "Unknow command: manage.py" in a Django project in PyCharm (newbie)
- Create a python file using cmd in windows
- How to guess the numerical Solution for Mathieu's Equation
- Leetcode 417 BFS Time Limit Exceeded