salesforce api call returns too many responses within for loop in Python

Question

I am making a salesforce bulk API call to fetch the data. I am using a simple_salesforce library. I want to fetch the data where my id equals a specific value, I also need to return a few responses as I have a list of ids. My data looks like the following:

ids_dict = [{'ID-1010': 'abc'}, {'ID-1020': 'def'}]

Here is code:

for key, value in ids_dict.items():
    desired_opp = value
    sql = sf.bulk.OpportunityLineItem.query("SELECT Name, Price FROM OpportunityLineItem where Opportunity_ID__c = '%s'" % desired_opp)
    sql_response = []
    sql_response.append(sql)

What being returned is a list with multiple responses with def ids. Where I need only two responses for respectful ids.

Answer 1

with my experience i have found it best to use sf.query(query=SOQL) and use sf.query_more(next_URL, True) to get the rest of the records

Since query only returns 2000 records, you need need to use .query_more() to get more records

From the simple-salesforce docs

SOQL queries are done via:

sf.query("SELECT Id, Email FROM Contact WHERE LastName = 'Jones'")

If, due to an especially large result, Salesforce adds a nextRecordsUrl to your query result, such as "nextRecordsUrl" : "/services/data/v26.0/query/01gD0000002HU6KIAW-2000", you can pull the additional results with either the ID or the full URL (if using the full URL, you must pass ‘True’ as your second argument)

sf.query_more("01gD0000002HU6KIAW-2000")
sf.query_more("/services/data/v26.0/query/01gD0000002HU6KIAW-2000", True)

Also maybe you want to change your sql statement to use "in" instead of "=" to select multiple values it can equal to at once. That way you only make one API call to salesforce until you go to request more records. (removes wasted calls on multiple less than 2000 record searches)

Here is an example of using this

 data = [] # list to hold all the records

 SOQL = "SELECT Name, Price FROM OpportunityLineItem where Opportunity_ID__c in ('abc', 'def')"

 results = sf.query(query=SOQL) # api call

 ## loop through the results and add the records
 for rec in results['records']:
     rec.pop('attributes', None) # remove extra data
     data.append(rec) # add the record to the list


 ## check the 'done' attrubite in the response to see if there are more records
 ## While 'done' == False (more records to fetch) get the next page of records
 while(results['done'] == False):
     ## attribute 'nextRecordsUrl' holds the url to the next page of records
     results = sf.query_more(results['nextRecordsUrl', True])

     ## repeat the loop of adding the records
     for rec in results['records']:
         rec.pop('attributes', None)
         data.append(rec)

Looping through the records and using the data

 ## loop through the records and get their attribute values
 for rec in data:
     # the attribute name will always be the same as the salesforce api name for that value
     print(rec['Name'])
     print(rec['Price'])