So I am scraping a site for urls and storing these urls in a db I am looking for the lightest way to compare the new scraped urls with the urls stored to only add the ones that arent currently stored in the db. I am working in php with a mysql db.
Give the URL a UNIQUE
index. In your query :
INSERT IGNORE INTO table (field1,field2) VALUES ('value1','value2')
This will skip the url when they already exist in the database.