I have a number of Elasticsearch indices which are updated on a regular basis, and I'd like to have these updates reflected in a Neo4j graph database, with a minimal delay between the indexing of the item in Elasticsearch and its being updated into Neo4j.
(Also it's important to have administrative work kept low: I'd just want to update a cypher statement to define the mapping between the index documents and the graph, for example, so that once I've created an index, it's limited work to have its data added to Neo4j).
Is it possible to do this using Elastic search watchers and triggers? Or by some other means?
I'm aware of the APOC library's Elasticsearch functions in Neo4j, but I'd rather push from Elasticsearch on update than pull from Neo4j. GraphAware also has an extension, but it seems aimed mostly at the Neo4j to ES direction.
ElasticSearch triggers can post a HTTP request and Neo4j has a HTTP Api, so that seems doable: https://neo4j.com/docs/http-api/current/
If I were to do this in production I would probably develop a microservice that listens to the webhooks and stores the data into Neo4j. This way you can introduce retry policy as well as log errors if nothing else.
After doing a bit of research you could configure the ElasticSearch to push updates to Kafka with tools such as https://github.com/malike/elasticsearch-kafka-watch, and then ingest the data with the Neo4j Kafka Sink.