When it comes to creating/updating custom objects, can I use both dedupeFields or lookupField when pushing the data to Marketo?
What is the difference between the two?
I'm not sure what do you mean under lookupField, as there is no such input field described in the API documentation of the Sync Custom Objects endpoint. (That is the endpoint to create or update custom objects.)
In the other hand, you do not need such a standalone lookup field, as with the input array you provide the objects you want to create or update, with all their important values. Have a look at the sample payload in the docs.
When input is used together with the optional dedupeBy and action fields, you have full control over which object you want to create or update.
Also, the endpoint expects the name of the dedupe field under dedupeBy key, as opposed to dedupeFields. So the name is singular; you can provide a single field name use, and it does what you can expect: if the value in the field for a given record is not unique, an error will be returned for the individual record.