Bulk insert into elasticsearch using Elasticsearch.net or PlainElastic.net

Question

I've being messing about with both Elasticsearch.net (http://nest.azurewebsites.net/) and PlainElastic.Net(https://github.com/Yegoroff/PlainElastic.Net) and have been able to insert single documents into elasticsearch. I am now trying to figure out how to perform a bulk insert. I know each of these two .net libraries have documentation around this, but the data I wish to insert is stored in a dictionary where the key is the ID of the document and value is the document and I cannot figure out how to do it.

Here is some code I have (using Elasticsearch.net) :

var conn = new Uri("http://localhost:9200");
var config = new ConnectionConfiguration(conn);
var client = new ElasticsearchClient(config);

var myJson = @"{""Col1"" : ""Hello World"", ""col2"" : ""asdfasdf"" }";
var myjson2 = @"{""Col2"" : ""Hello World Again"", ""col2"" : ""zxcvzxcv"" }";

Dictionary<string, string> jsonCollection = new Dictionary<string, string>();

jsonCollection.Add("1", myJson);
jsonCollection.Add("2", myjson2);

Answer 1

I use PlainElastic.Net and this is how RAW data should look like

POST /_bulk
{ "index" :{ "_index": "myIndex", "_type": "myType", "_id": 1  }}
{ "id": 1, "name": "My category \"ONE\" "}
{ "index" :{ "_index": "myIndex", "_type": "myType", "_id": 2  }}
{ "id": 2, "name": "My second category \t "}
{ "index" :{ "_index": "myIndex", "_type": "myType", "_id": 3 }}
{ "id": 3, "name": "My third category \r\n "}

keep in mind that new line is at the end of every row (yes even after last line)

so vb.net should look like this:

Dim bulkData As String = "{ ""index"": { ""_index"": ""myIndex"", ""_type"": ""myType"", ""_id"": 1  }}" & vbNewLine & _
                         "{ ""id"": 1, ""name"": ""My category""}" & vbNewLine & _
                         "{ ""index"": { ""_index"": ""myIndex"", ""_type"": ""myType"", ""_id"": 2  }}" & vbNewLine & _
                         "{ ""id"": 2, ""name"": ""My second category""} " & vbNewLine & _
                         "{ ""index"": { ""_index"": ""myIndex"", ""_type"": ""myType"", ""_id"": 3  }}" & vbNewLine & _
                         "{ ""id"": 3, ""name"": ""My third category""} " & vbNewLine

Dim ESConn as New ElasticConnection("localhost", 9200)
Dim response As String = ESConn.Post("/_bulk", bulkData)

c# version I didn't test but you'll get the idea

string bulkData = @"{ ""index"": { ""_index"": ""myIndex"", ""_type"": ""myType"", ""_id"": 1}}
{ ""id"": 1, ""name"": ""My category""}
{ ""index"": { ""_index"": ""myIndex"", ""_type"": ""myType"", ""_id"": 2}}
{ ""id"": 2, ""name"": ""My second category""}
{ ""index"": { ""_index"": ""myIndex"", ""_type"": ""myType"", ""_id"": 3}}
{ ""id"": 3, ""name"": ""My third category""} \n";

ElasticConnection ESConn = New ElasticConnection("localhost", 9200);
string response = ESConn.Post("/_bulk", bulkData);

You can create JSON manually or with Newtonsoft.Json