How to join indices in elasticsearch and kibana
I know I am thinking of this in SQL terms so I know there is going to be a shift in thinking.
Basically I am ingesting webhooks from GitHub about actions and I want to pull some data together for a visualization.
I want to be able to join these two indices together such that I can pull the path data from the workflow_run index that corresponds to the entries in the workflow_job index so that I can show the file associated with the job entries. Ultimately I am going to group them together and list the top jobs that fail.
A sample of the indices is here:
What is the correct way to visualize this data in Kibana?
I know I am thinking of this in SQL terms so I know there is going to be a shift in thinking.
TL;DR: That is correct. If you want to use Kibana you need to denormalize everything. If it is not in a single index with homogeneous records you are not going to be a happy camper.
There is pretty much no support for many-to-many relationships here. Neither Kibana nor Elasticsearch can handle them. There is a limited support for one-to-many relationships in Elasticserch. If you could make it one-to-many relationship (for example, you could create a single workflow record) then Elasticsearch could handle one-to-many relationships using join field or nested records. There are several queries. Please note that all records in this case have to reside in the same index in case of join fields or even the same record in case of nested fields.
Unfortunately neither nested fields nor [join fields] (https://github.com/elastic/kibana/issues/3730) are properly supported by Kibana at the moment. There are some features like querying nested fields available, but overall most of your queries and aggregation will be either not possible or will break unexpectedly.
- How to view the different values in a geopoint to a Data table in Kibana Elasticsearch Maps
- Not able to start elastic search service
- Can I filter an array in elastic?
- Pyspark Streaming data to Elastic search index from Kafka topic , running in Jupyter notebook, causing failure
- How to move shards around in a cluster
- OData services with Elastic search
- How to query elasticsearch from spring with LocalDate
- Elasticsearch query to get values of multiple attributes
- I want to sort my elastic search aggregation data with most recent login for each user by there name
- bucket_script inside filter aggregation throws error
- Elasticsearch delete_by_query version conflict
- Case insensitive exact match in ElasticSearch
- How do I enable remote access/request in Elasticsearch 2.0?
- Fluent-bit - Splitting json log into structured fields in Elasticsearch
- Elasticsearch Devtools Queries (Regexp/Wildcard) - Not Working
- Elastic Search Analyzer at search time not working
- Elasticsearch 7.2.0: master not discovered or elected yet, an election requires at least X nodes
- Create TermsQuery with List<String> using elasticsearch java api client
- Elasticsearch: Use loop in Painless script
- How can I check nulls in Logstash pipeline in filter plugin?
- How to input a JSON file (array form) in Logstash?
- OpenTelemetry Export to Elastic Search without Elastic APM
- Can't connect to Elasticsearch with Node.Js on Kubernetes (self signed certificate in certificate chain)
- trying to pass params to elasticsearch getting null_pointer_exception
- 'Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes" error when indexing a list of dictionaries
- Regexp filter in Opensearch (or Elasticsearch)
- Shards and replicas in Elasticsearch
- Search document with empty array field, on ElasticSearch
- how to rename an index in a cluster?
- Merge two indices value by a common field in elasticsearch