pythonamazon-web-servicesamazon-emrmrjob
mrjob in emr is running only 1 MRStep out of 3 MRSteps and cluster is shutting down
The error looks something like this :- Terminating cluster: j-SDOP2KOKWYZM
botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the AddJobFlowSteps operation: A job flow that is shutting down, terminated, or finished may not be modified.
Solution
Following the error. It is clear that the cluster is terminating after step 1.
This issue is because the botocore package is deprecated.
A solution to this could be:
- Start a persistent cluster
- Use that cluster ID to run the mr job in emr
Commands:
mrjob create-cluster
Make sure you have configured the cluster-info in mrjob.config file. The above command lets you create a persistent cluster.
python3 MovieSimilarities.py -r emr --cluster-id "your-cluster-id"
--items=ml-100k/u.item ml-100k/u.data > sims2t.txt
Now specify the cluster id to run your cluster.
- Python code to retrieve system info in format
- How to use threading.Lock in async function while object can be accessed from multiple thread
- How does defining what a function will return work
- How to improve pandas DF processing time on different combinations of calculated data
- How split (chunk) a list of customers along month/year?
- Sagemaker batch transformer with my own pre-trained model
- Given an array of N integers, return the max sum of 3 NON-adjacent elements
- Error "table ... already exits" when using to_sql(...,if_exists='append') with Pandas and SQLite
- Saving tkinter canvas with black background and white line objects using PIL results in fully white image
- change grid interval in astropy curved axis
- Qt/PySide6: What is the best way to implement an infinite data fetching loop with a QThread?
- Number of installations statistics for PyPI packages?
- how can I do a maximum likelihood regression using scipy.optimize.minimize
- Intersection of two lists including duplicates?
- Python RegEx: match words in a string only if they are not preceded by a specific character
- Python Async Thread-safe Semaphore
- Is there an elegant way to use struct and namedtuple instead of this?
- What's convention for naming a class or method as "class" in Python?
- Telegram bot to send auto message every n hours with python-telegram-bot
- How to modify the default font in Tkinter?
- Pandas set_levels on MultiIndex: Level values must be unique
- python use of corrcoeff to achieve matlab's corr function
- rolling_sum where the window_size is in another column
- Python: How to check if a nested list is essentially empty?
- Read python output from another python script
- Trouble figuring out input arguments to SciPy RegularGridInterpolator for 2D interpolation
- Converting html to text with Python
- condition formatting rule formala want to pass text value using variable in python openpyxl
- Process inside daemon immediately exits when using FIFO
- ipywidgets interactive with variable containers of widgets