Search code examples
gomicroservices

How to design a multi-file processor using Golang?


I'm trying to figure out how to design a service that can handle multiples files format, using micro-services for example:

I have a customer using a file format A, another using format B and other using format C.

After processing a specific format for each customer, I need to transform this format into an common format, and insert into database.

The first thing I tried is to design one service per customer but all of them need to know the base format and if an update is needed in this base format, I need to update all of there services.

I'm trying to decouple the service processor and have a single place to translate to base format.

If my customer services know about the base format, if this format changes, I need to update all of them. If I have a service that translate the format, this service needs to know all customer formats.

How to design this solution?


Solution

  • Plan internal data format to suit purpose of your application first, it should be convenient for next operations. All the application should work with internal data format and only specific input-output adapters should know about specific customers formats. Then write adapters from custom data formats A, B, C, ... to convert to basic internal format. These adapters should be as thin as possible, they should have minimal logics - just convert data and nothing more. This way it will be easier to maintain them. You are right - if you change internal format you have to review and possibly change all adapters. That's why they should be very thin. Think which defaults can you have? I recommend to start with few customer formats to debug all concepts. Wheather make the solution as a set of microservices of as modules in a single monilith - has no much sense in this way.