I try to get csv data from kibana server by making get request.
After this server send me response with data which i write the csv file and save somewhere.
Every time csv file with single line "pending" inside.
Kibana logs shows Status code 503
But if i put url in browser i can get csv file with correct data.
Probably i need to wait more for response from kibana.
From my perspective the problem is next: server can send me the response with single line "pending" which means that it needs more time for preparing right response.
I tried to increase client time but it doesn't work as well
client := http.Client{
Timeout: 10 * time.Second,
}
The Idea with go routines and channels/wait group was next: they force Get request to wait for getting right data instead of "pending" and 503 status code
import (
"encoding/json"
"errors"
"fmt"
"io/ioutil"
"log"
"net/http"
"sync"
)
type KibanaReportResponse struct {
Path string `json:"path"`
}
var urlKibanaBase = "http://localhost:5601"
var urlKibanaPost = urlKibanaBase + "/api/reporting/generate/csv_searchsource?"
var urlParameters = "jobParams=%28browserTimezone%3AEurope%2FBerlin%2Ccolumns%3A%21%28%29%2CobjectType%3Asearch%2CsearchSource%3A%28fields%3A%21%28%28field%3A%27%2A%27%2Cinclude_unmapped%3Atrue%29%29%2Cindex%3Aec074c00-1f62-11ec-8056-8d208a1f6e77%2Cparent%3A%28filter%3A%21%28%29%2Cindex%3Aec074c00-1f62-11ec-8056-8d208a1f6e77%2Cquery%3A%28language%3Akuery%2Cquery%3A%27%27%29%29%2Csort%3A%21%28%28_score%3Adesc%29%29%2CtrackTotalHits%3A%21t%2Cversion%3A%21t%29%2Ctitle%3A%27Discover%20search%20%5B2021-09-27T09%3A19%3A44.977%2B02%3A00%5D%27%29"
var urlWithParam = urlKibanaPost + urlParameters
func main() {
var wg sync.WaitGroup
wg.Add(1)
pathCsvFile := getCsvPathFromKibana(urlWithParam)
go getCsvFile(urlKibanaBase, pathCsvFile, &wg)
defer wg.Wait()
}
func getCsvPathFromKibana(urlKib string) string {
resKibana := KibanaReportResponse{}
client := &http.Client{}
if req, err := http.NewRequest("POST", urlKib, nil); err != nil {
log.Println("Given a method, URL, andoptional body are wrong", err)
} else {
req.Header.Add("kbn-xsrf", "true")
if res, err := client.Do(req); err != nil {
log.Println("Probably problems depends on client policy or HTTP connection", err)
} else {
if err := json.NewDecoder(res.Body).Decode(&resKibana); err != nil {
log.Println("Problem by Json decoding \n", err)
} else {
return resKibana.Path
}
}
}
return resKibana.Path
}
func getCsvFile(urlKibanaBase string, pathCsvFile string, wg *sync.WaitGroup) error {
defer wg.Done()
res, err := http.Get(urlKibanaBase + pathCsvFile)
if err != nil {
return err
}
defer res.Body.Close()
switch res.StatusCode {
case 200:
dataBody, err := ioutil.ReadAll(res.Body)
if err != nil {
return err
}
err = ioutil.WriteFile("data.csv", dataBody, 0666)
if err != nil {
return err
}
return nil
case 503:
fmt.Println("probably 503/pending")
return errors.New("probably 503/pending")
}
return nil
}
curl request
curl -v localhost:5601/api/reporting/jobs/download/ku5k3rxz00xs7fac46c0k12u
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 5601 (#0)
> GET /api/reporting/jobs/download/ku5k3rxz00xs7fac46c0k12u HTTP/1.1
> Host: localhost:5601
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< kbn-csv-contains-formulas: true
< kbn-max-size-reached: false
< content-disposition: inline; filename="Discover search [2021-09-27T09:19:44.977+02:00].csv"
< content-type: text/csv; charset=utf-8
< x-content-type-options: nosniff
< referrer-policy: no-referrer-when-downgrade
< kbn-name: 3ae0cbefece4
< kbn-license-sig: 58d9fcb437ac7d3ac54d538e6d5ff9a039fde738ed3a940fa530e6d8b6ef6740
< cache-control: private, no-cache, no-store, must-revalidate
< content-length: 9013976
< vary: accept-encoding
< accept-ranges: bytes
< Date: Wed, 29 Sep 2021 14:06:19 GMT
< Connection: keep-alive
< Keep-Alive: timeout=120
<
{ [13865 bytes data]
100 8802k 100 8802k 0 0 57.6M 0 --:--:-- --:--:-- --:--:-- 57.6M
* Connection #0 to host localhost left intact
Screenshot that shows that i can get data by typing url path to the browser
Update:
I try to call my function recursively. But it is bad solution even if it's working now.
case 503:
getCsvFile(urlKibanaBase, pathCsvFile)
Kibana Server needs some time to prepare response. I've added a bit of code for signaling the channel to wait some time and after waiting repeat request to client
if res.StatusCode != 200 {
time.Sleep(30 * time.Second)
}