sync mongo data to elastic using logstash

Question

I want to sync my mongodb data(local mongodb) to elastic search(local elastic) using logstash-plugin of mongodb

I have install logstash plugin using

bin/logstash-plugin install logstash-input-mongodb .

Then i created a mongodata.conf file in /usr/share/logstash directory. When I execute the conf file then it shows --> Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties

My config file is:

input{ 
 mongodb{
   uri => "mongodb://localhost:27017/reporterDB"
   placeholder_db_dir => "/opt/logstash-mongodb/"
   placeholder_db_name => "logstash_sqlite.db"
   collection => "iam_ms_test"
   batch_size => 5000
  }
}
filter{ 

 }
output {
    stdout { codec => rubydebug }
    elasticsearch {
            action => "index"
            hosts => "localhost:9200"
            user => elastic
    password => changeme
            index => "mongo_log"
            document_type => "document_type"
            document_id => "%{id}"     
    }
  }

I am getting below lines in logstash-plain.log file

[2019-11-01T15:41:00,869][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-11-01T15:41:00,871][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>6, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>750, :thread=>"#<Thread:0x351f7fd1@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:245 run>"}
[2019-11-01T15:41:01,068][INFO ][logstash.inputs.mongodb  ] Registering MongoDB input
[2019-11-01T15:41:01,116][ERROR][logstash.pipeline        ] Error registering plugin {:pipeline_id=>"main", :plugin=>"<LogStash::Inputs::MongoDB uri=>\"mongodb://localhost:27017/anchorReports\", placeholder_db_dir=>\"/opt/logstash-mongodb/\", placeholder_db_name=>\"logstash_sqlite.db\", collection=>\"hi_p5m\", batch_size=>5000, id=>\"ec7682e8c6c5676deca84d5072c5f7865120a107ffce81ce21caa878c6e4ed09\", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>\"plain_441f95b8-cc8a-4b9e-a45f-657ed2011e2b\", enable_metric=>true, charset=>\"UTF-8\">, since_table=>\"logstash_since\", since_column=>\"_id\", since_type=>\"id\", parse_method=>\"flatten\", isodate=>false, retry_delay=>3, generateId=>false, unpack_mongo_id=>false, message=>\"Default message...\", interval=>1>", :error=>"Java::JavaSql::SQLException: path to '/opt/logstash-mongodb/logstash_sqlite.db': '/opt/logstash-mongodb' does not exist", :thread=>"#<Thread:0x351f7fd1@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:245 run>"}
[2019-11-01T15:41:01,869][ERROR][logstash.pipeline        ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Sequel::DatabaseConnectionError: Java::JavaSql::SQLException: path to '/opt/logstash-mongodb/logstash_sqlite.db': '/opt/logstash-mongodb' does not exist>, :backtrace=>["org.sqlite.core.CoreConnection.open(org/sqlite/core/CoreConnection.java:190)", "org.sqlite.core.CoreConnection.<init>(org/sqlite/core/CoreConnection.java:74)", "org.sqlite.jdbc3.JDBC3Connection.<init>(org/sqlite/jdbc3/JDBC3Connection.java:24)", "org.sqlite.jdbc4.JDBC4Connection.<init>(org/sqlite/jdbc4/JDBC4Connection.java:23)", "org.sqlite.SQLiteConnection.<init>

"(org/sqlite/SQLiteConnection.java:45)", 

"org.sqlite.JDBC.createConnection(org/sqlite/JDBC.java:114)", 

"org.sqlite.JDBC.connect(org/sqlite/JDBC.java:88)"

I want the records on my elastic search under `index("mongo_log"). I also want to know the uses of placeholder_db_dir and placeholder_db_name and whats should be these values when we are using mongodb as the input database.

Answer 1

Problem solved! actually the directory opt/logstash was not created . So I manually create the logstash folder under opt. After that i gave Write permission to that directory , so that when we execute the command for logstash then it can create file inside this folder.