I'm using spring boot and angular to develop an app that tracks orders from log files (up to 1gb) I'm willing for your suggestions about the best solutions to read, processing and validating files in spring ,I have only thought about spring batch but is there any other robust solutions
Okey after good researches i have discouvred various solutions i think that this link is worth it : https://www.java-success.com/processing-large-files-efficiently-java-part-1/
Q3. What are the different data sizes, and what technologies can be used to process them? A3. In general, data sizes can be classified as shown below.
1) Small size data is < 10 GB in multiple files. It fits in a single machine’s memory when you process them by streaming to conserve memory. Java’s file processing APIs, Apache commons File APIs, Spring batch framework or Java EE 7 batch processing framework can be used.
2) Medium size data is 10 GB to 1 TB in multiple files. Fits in a single machine’s disk space. Process them by splitting or streaming as you won’t be able read all the contents into memory. Spring batch framework or Java EE 7 batch processing framework can be used.
3) Big data is > 1 TB in multiple files. Stored across multiple machines and processed in distributed fashion. E.g. run a map reduce or a Spark job.