I'm curious what people have used as the @Id in their Java entity for Logstash data being pulled with Spring-Data-Elasticsearch? The log data funneled into Logstash does not have an obvious candidate. The closest thing might be timestamp but it's not guaranteed to be unique. Also apparently Spring does not support ZonedDateTime as an @Id.
I'm using Spring Boot 2.0.0.RELEASE. The default logback logstash encoder writes nicely to the file system. I have Filebeat send it out to Logstash then to Elasticsearch (5.5.x).
I'm writing a Java application to do various things with the logs. I'm utilizing Spring Data Elasticsearch (3.0.5.Release).
An example of a record in Elasticsearch as written per above:
{
"_index":"logstash-2017.09.08",
"_type":"log",
"_id":"AV5j2z-qpNewjNB_ukIA",
"_version":1,
"_score":1,
"_source":{
"offset":18441,
"level":"ERROR",
"input_type":"log",
"source":"/var/log/somecompany/configuration_service/json/configuration_service_json_events.json",
"message":" *************************** APPLICATION FAILED TO START *************************** Description: The Tomcat connector configured to listen on port 8888 failed to start. The port may already be in use or the connector may be misconfigured. Action: Verify the connector's configuration, identify and stop any process that's listening on port 8888, or configure this application to listen on another port. ",
"type":"log",
"tags":[
"beats",
"beats_input_codec_plain_applied"
],
"@timestamp":"2017-09-08T23:37:41.639Z",
"application_name":"configuration-service",
"thread_name":"main",
"level_value":40000,
"@version":1,
"beat":{
"hostname":"f3758598e319",
"name":"f3758598e319",
"version":"5.5.2"
},
"host":"f3758598e319",
"logger_name":"org.springframework.boot.diagnostics.LoggingFailureAnalysisReporter",
"application_version":"3.14.0-SNAPSHOT"
}
}
I discovered my answer shortly after posting.
If I declare a String id and annotate it as @Id, then the framework properly pulls back the internal autogenerated _id field from E.S.
@Id
private String id;
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}