i'm currently developing and e-commerce backend on NestJS and looking for the most correct way to implement ElasticSearch into the app, the thing is, i dont really get how ElasticSearch would know about the records in my PostgreSQL database which I handled all those CRUD operations through ORM tool like Prisma and never ever synced with it with ElasticSearch,didnt even create index on backend for the records that I wanted ElasticSearch to reach out to search for.So, i guess the only way that ElasticSearch to get the most up-to-date records from the DB is probably to use Logstash in order for it to pick the records from Postgre so that it can send them all out to ElasticSearch...I'm just now bouncing between the both options back and forth, either i will create index for each product before creating records on backend, meaning any change on database, which is PostgreSQL, wont communicate with ElasticSearch, thus ElasticSearch wont be able to perform effective searching, ending up not retrieving the updated data from there, or i will connect PostgreSQL to Logstash as a data source, which looks best practise. Another thing hanging over my head is, how big companies handle that and how to implement such advanced search engine into NestJS project, either with ELK or not, i dont need workarounds, since this is going to be a relatively advanced e-commerce app and I have to apply only best practises.
any recommendations, tips are highly appreciated
As per your use case, these are the quick tips you can follow -
Index creation - As you have an E-Commerce website, assuming your data won't be a time series data. In that case, you can create simple indexes where you can add all records. As per your search requirement, you need to check whether you want to denormalize data from the Postgres table and index as a single record in the ES index OR you can create different indexes for different search requirements.
Data Migration - In such a use case, You can use Elasticsearch to store searchable data. Logstash is the best tool that will help to sync data from Postgres to Elasticsearch.
Apart from logstash, People use kafka to stream data directly to the Elasticsearch (This is only recommended if Kafka is already in place in your system).
You can also write another ingestion for the Elasticsearch where your data will be inserted at the same time in PG and ES.
All operations will be in realtime sync.
But the Logstash is pretty much known and heavily used for such syncing. So ELK will help to sync your data.
NestJS - You can use the Elasticsearch package to query synced data from Elasticsearch. You can give a try with this.