Search code examples
postgresqlelasticsearchsynchronization

The best way to syncronize postgres and elasticsearch

I have a huge amount of data in postgres and have some indexes in elasticsearch that store the same data, just in a different format and not complete. When developing my own program that uses debezium I faced a number of problems, especially with synchronization and thought that maybe there are some ready-made solutions for my problem.

The main requirements for the solution:

  1. unnoticeable load on the database or its absence
  2. small delay in synchronization
  3. the ability to send requests to third-party API in order to get the missing data
  4. the ability make multiple changes in elasticsearch for one change in postgres.

I would be very grateful if you can recommend something

Solution

    1. You can try Elastic PostgreSQL connector. You can use for one time sync or you can schedule recurring sync.

    2. You can use logstash with JDBC input plugin, it will pull data from postgreSQL and push into Elasticsearch. You can do one time sync or schedule recurring sync.