Design Pattern for Data Consolidation Layer (ETL)
I have to design a software using asp .net core which collects data from various datasources (s. picture below). E.g. DataSource1 and DataSource2 are including product data like attributes. DataSource 3 is including the assets of those products. I thought first of all I will collect the data from each datasource and persists them in own datasource with the defined entity below. I have the advantage later at translating or tranforming the data to use one abstract entity.
My question which pattern should be good for this system? Repository, Pipeline,...?? Could you show me some pseudo code?
What about DI if I use interfaces but should have multiplied instances of datasources?
A pattern (or a set of patterns) should be applied to solve a specific problem/complexity.
I think the pattern the pattern you need here is Facade.
The problem that it will solve is that it will hide the complexity of the 'three data sources' for your client.
Within the Facade you would merge the data into a reasonable entity.
Additionally, you could make use of the Proxy pattern, which could give you the 'cache' functionality for the 'merged' entities, which could solve the second complexity you describe.
I am not sure I understand the idea of persisting these items into a fourth datastore, that might be an overkill - but in any case, that can also be achieved with the proxy class - it's just that the cache would be more permanent - if your domain 'allows' it.
As for Repository (pattern) - well, I believe that it's very likely that any reasonable solution that you apply that will hide the details of your data access will end up to be a an implementation of a Repository.
I wouldn't be too strict about naming the patterns and sticking to sample code in books or articles. Patterns are high level guidelines that can be adjusted to needs.
- Sort program not working, not sure why
- Fast & accurate atan/arctan approximation algorithm
- What's the difference between strtok_r and strtok_s in C?
- How memory address for pointer to arrays is same as an element in 2D array?
- Which is the best way to suppress "unused variable" warning
- How to use ellipsis in c's case statement?
- How can I exclude non-numeric keys? CS50 Caesar Pset2
- Fast ceiling of an integer division in C / C++
- Is there an invalid pthread_t id?
- How to Implement Universal Setter/Getter Functions for Interrupt-Driven Variables in Embedded C?
- How does SIMD (avx) processing work? for example, if I want 10 32 bit floats how do i fit in a 256 bit avx vector?
- FDCAN problems on STM32G4
- How does the call macro enable mutual recursion between functions f and g in this Hanoi Tower implementation?
- Running test on Rocket core CPU - global variable initialized to 0 is unsuccessful, output wrong value instead
- Interacting with C arrays without knowing the size
- Combination of two strings
- Avoiding strcpy overflow destination warning
- carriage return by fgets
- How to use special characters in C?
- Why does 1.0/100.0 == 0.1/10.0 give True?
- Is it correct to compare pointers in C?
- Force free() to return malloc memory back to OS
- How can I print to standard error in C with 'printf'?
- What is the standard behavior of fread in C on Windows?
- How is strtok removing lines it shouldn't have access to?
- Using array as smart point in C
- Assigning string to malloced 2d char array not working as intended
- How to refactor repetition inside a Makefile?
- Why does an empty preprocessor command still evaluate to something?
- How to implement variable sized array within C struct