TPL Dataflow pipeline design basics
I try to create well-designed TPL dataflow pipeline with optimal using of system resources. My project is a HTML parser that adds parsed values into SQL Server DB. I already have all methods of my future pipeline, and now my question is what is the optimal way to place them in Dataflow blocks, and how much blocks i should use? Some of methods are CPU-bound, and some of them - I/O-bound(loading from Internet, SQL Server DB queries). For now I think that placing each I/O operation in separate block is the right way like on this scheme:
What are the basic rules of designing pipelines in that case?
One way to choose how to divide the blocks is to decide which parts you want to scale independently of the others. A good starting point is to divide the CPU-bound portions from the I/O-bound portions. I'd consider combining the last two blocks, since they are both I/O-bound (presumably to the same database).
- How can I overload a method with @QueryParam in Jersey/Spring?
- Increment the 4-digit, 0-padded substring at the end of a string
- VSCode ipynb notebook showing the plots several times over
- How to parse a dynamic image path in Rails 8 inside Stimulus controller (Popshaft Asset Pipeline)?
- Is it possible to use raw SQL within a Spring Repository
- 68000 Assembly - Finding the Minimum in an Array
- Getting type of a property of a TypeScript class using keyof operator
- ESLint error when trying to lint Angular templates
- Access delegated Gmail account to fetch labeled emails
- Pandas ValueError : Must have equal len keys and value when setting with an iterable
- Couldn't start project on Android: It took too long to start the Android emulator: Pixel_2_API_30
- Rcpp Rf_warningcall compiler warnings
- Modify the name of factor variables in lm function(summary function)
- Generate random float64 numbers in specific range using Golang
- Error: "attribute value must be constant". Can I constract a constant from an enum at compile time?
- update with rowlock in MSSQL server
- Integrating tailwind v4 alpha with vite postcss config
- Any generic CSS libraries for web development?
- Sort defaultdict by key?
- Is it possible to update a PowerPoint slide with new data (in C#)?
- Reading SQL query using Polars and cx_Oracle gives error
- Which Stripe webhooks confirm Subscription and Product payments?
- How to make a "sticky" CircleView that stays attached to the edges of another views in SwiftUI?
- Blazor 9 MapStaticAssets with nuget
- AWS EC2 ECS - How many tasks should I place on a single instance?
- How memory address for pointer to arrays is same as an element in 2D array?
- How do I change Project language level on Android Studio
- Recursively render tree in Rails
- Error Classpath is empty. Please build the project first e.g. by running apache kafka
- Extreme value analysis and quantile estimation using log Pearson type 3 (Pearson III) distribution - R vs Python