Search code examples
shopware6shopware6-api

Synchronizing millions of products

My team and I are trying update a huge number (millions) of products through integration with ERP. We want to use the sync api.

https://forum.shopware.com/t/sync-api-upsert-mit-productnumber-als-unique-key/68556 Explains what we want to do. Our ERP system is not aware of the product id (UUID) from shopware and only knows a product SKU. This leaves us with having to do a product lookup in shopware for each product number, to get product id and then update the product data.

Is there a workaround so we can upsert by product number or other great ideas on how we can speed up things?

Kind regards

One idea is to generate our own product UUID based on MD5 hash of the product number. This way we will always know the uuid without having to do lookup in database.

Solution

  • There are several approaches to solve this. This are some which i use in my daily business.

    API-only

    If you do can only work with the sync-API, then i would recommend to build a sep. storage only for product synchronization. This would contain the following information:

    1. SKU
    2. Shopware-UUID
    3. hash

    Every synchronized product will have an existing UUID. If there is no UUID, you know that the product has not been synchronized to Shopware. The hash itself can be used internally to check for changes in the data. If there is a change, that specific entity will be updated then. In that way a simple and fast lookup would be possible without the API.

    Direct access to the db

    This is not my recommendation if you do not have a team of developers. Direct access to the database will provide you the fastest ways to access that information. Actually we do this for merchants with lots of product data but you will loose some benefits like the automatically queued events for updating entities in the catalog, prices, categories, etc. when using the sync api. So i only recommend it for read access which is what you want.

    About the UUID

    The approach you described is exactly what UUID v5 is. Shopware uses UUID v4 internally, which is just a random 128 bit sequence while v5 utilizes sha1 to generate that. So this is a good practice.