Best way to write to CSV and append to list simulatenously in Python

Question

Background:
I have the following code the uses Selenium to find a list of links from this Simply Recipe Index URL and stores them in a linklist list. The code then iterates through a linklist and for every link, it will download recipe text, before storing the data in a recipe_list

from bs4 import BeautifulSoup
import requests
from splinter import Browser
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import selenium
import time
import csv

#set up chromedriver for WINDOWS
driver=webdriver.Chrome('chromedriver.exe')
url = "https://www.simplyrecipes.com/index/"
driver.get(url) 
response=requests.get(url)
soup=BeautifulSoup(response.text,'html.parser')

#set up chromedriver for MAC
driver=webdriver.Chrome("/Users/williamforsyth/Documents/uc_davis/Homework_Repos/group-project-2/Kathryn/chromedriver")
url = "https://www.simplyrecipes.com/index/"
driver.get(url) 
response=requests.get(url)
soup=BeautifulSoup(response.text,'html.parser')

linklist=[]
links=soup.find_all('a')
for link in links:
    linklist.append(link)
linklist_text=[]
for i in range(164,1068):
    linklist_text.append(linklist[i].text)

recipe_list=[]
for link in linklist_text:
    time.sleep(0.3)
    target=driver.find_element_by_partial_link_text(link)
    target.click()
    time.sleep(0.1)
    cards = driver.find_elements_by_class_name("grd-title-link")
    for i in range(0,len(cards)):
        try:
            newcards = driver.find_elements_by_class_name("grd-title-link")
            time.sleep(0.3)
            newcards[i].click()
            time.sleep(0.3)
            recipe=driver.find_element_by_id("sr-recipe-callout")
            recipe_list.append(recipe.text)
            driver.back()
            time.sleep(0.3)
        except:
            continue
    driver.get(url)

Issue:
This code wasn't written with a feature I'd now like to implement, in mind, which is that I'd like to append the recipe.text from every loop to a csv file. This is the code that I'd like to try and append to a CSV as well as what the code does now, which is append to the recipe_list

    recipe_list.append(recipe.text)

Is there a simple way to implement this, without having to rework the entire code?

Answer 1

you want to "..append the recipe.text from every loop to a csv file"

If you just want to dump it to file each iteration of your TRY loop, then you could import a reader and replace your "recipe_list.append(recipe.text)" with something like:

import csv

with open('recipe_output.csv', 'w', newline='') as csvfile:
    writer = csv.writer(csvfile, delimiter=',')
    writer.writerow(recipe.text)

Set the "with open()" and "writer =" lines somewhere before the loop begins, and use this "writer.writerow" line to replace your current "recipe_list.append(recipe.text)" line. This will give you a file dump for every loop into a CSV file, instead of dumping the lot from your appended list at the end