Elasticsearch returns zero hits when using termQuery in QueryBuilders

Question

I am building a Java app that searches through data from Elasticsearch (Data comes in from kafka to logstash and then elasticsearch in json format). When I use QueryBuilders.queryStringQuery(reqId) I get all results back no problem but when I use QueryBuilders.termQuery("routingRequestID", reqId); I get 0 hits even though the reqId is present in ES data.

    RestHighLevelClient client = new RestHighLevelClient(
            RestClient.builder(new HttpHost("127.0.0.1", 9200, "http")));

    @GetMapping("/q/{reqId}")
    public String searchByReqId(@PathVariable("reqId") final String reqId) throws IOException {
        String[] indexes = {"devglan-log-test"};

        QueryBuilder queryBuilder = QueryBuilders.termQuery("routingRequestID", reqId);
        // QueryBuilder queryBuilder = QueryBuilders.queryStringQuery(reqId);

        SearchSourceBuilder searchSource = SearchSourceBuilder.searchSource().query(queryBuilder).from(0).size(1000);
        System.out.println(searchSource.query());

        SearchRequest searchRequest = new SearchRequest(indexes, searchSource);
        System.out.println(searchRequest.source().toString());

        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        System.out.println(searchResponse.toString());
        SearchHits hits = searchResponse.getHits();
        SearchHit[] searchHits = hits.getHits();
        for (SearchHit hit : searchHits) {
            System.out.println(hit.toString());
        }

        return "success";
    }

{
   took: 633,
   timed_out: false,
   _shards: {
      total: 1,
      successful: 1,
      skipped: 0,
      failed: 0
   },
   hits: {
      total: {
         value: 1,
         relation: "eq"
      },
      max_score: 1.6739764,
      hits: [
      {
         _index: "devglan-log-test",
         _type: "_doc",
         _id: "k4qAPXEBCzyTR4XVXPb2",
         _score: 1.6739764,
         _source: {
            @version: "1",
            message: "
                      {"requestorRole":"role3", "requestorGivenName":"doe", "requestorSurName":"male", 
                       "requestorOrganizationName":"dob", "reqd":"address", 
                       "requestorC":"city", "routingRequestID":"7778787898778879"}",
            @timestamp: "2020-04-03T00:45:53.917Z"
        }
      }
    ]
  }
}

Query generated by searchSource.query():

{
  "term" : {
    "routingRequestID" : {
      "value" : "2421",
      "boost" : 1.0
    }
  }
}

Query generated in searchRequest.source().toString():

{"from":0,"size":1000,"query":{"term":{"routingRequestID":{"value":"2421","boost":1.0}}}}

Results:

{"took":0,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":0,"relation":"eq"},"max_score":null,"hits":[]}}

All help is truly appreciated and please don't skip the post if you know how to help. *highfive emoji*

Answer 1

So the problem was that all info was in one field. I solved the issue by changing logstash configurations and then using matchQuery. Here is what you need to add to your logstash config file if you are using kafka and json format:

input {
   kafka {
      bootstrap_servers => "kafka ip"
      topics => ["your kafka topics"]
   }
}
filter {
      json {
        source => "message"
      }
      mutate {
         remove_field => ["message"]
      }
    }

by the way I am using elasticsearch 7.4, latest logstash and latest kafka v. Best of luck and thanks to everyone who tried to help! I appreciate it! Here is the link for elasticsearch logstash plugin that will guide you through different options: https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html