Collaborating on a new filebeat module

Question

I am building a new filebeat module for a custom application log and I wish to collaborate on it with a colleague of mine. I understood that a clone is a local copy for me only, without a chance for making my changes visible to my colleague without having them first merged to the https://github.com/elastic/beats/ repo, so I opted to fork the beats repository instead of cloning it and then cloned that fork.

I set up my dev environment by following the instructions. Running make under filebeat gives no output so I assume everything is a-ok.

I generate the module, fileset, and set up my ingest pipeline for the fileset. My pipeline.json looks like this:

{
  "description": "Pipeline for parsing CA Service Desk Manager stdlogs",
  "processors": [
    {
        "grok": {
            "field": "message",
            "patterns": [
                "%{TIMESTAMP:casdm.stdlog.timestamp} %{HOSTNAME:casdm.stdlog.hostname} %{PROCESS:casdm.stdlog.process.name} %{PID:casdm.stdlog.process.id} %{LOGLEVEL:casdm.stdlog.level} %{FILENAME:casdm.stdlog.file.name} %{POSINT:casd.stdlog.file.line} %{DATA:casdm.stdlog.message}"
            ],
            "pattern_definitions": {
                "TIMESTAMP": "%{MONTHNUM2}/%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}",
                "PROCESS": "%{USERNAME}",
                "PID": "%{POSINT}",
                "LOGLEVEL": "(FATAL|EXIT|SIGNIFICANT|SEVERE_ERROR|ERROR|WARNING|INFORMATION|MILESTONE|TRACE|VERBOSE)",
                "FILENAME": "(?:[A-Za-z0-9_. -]+)"
            }
        }
    }
  ],
  "on_failure" : [{
    "set" : {
      "field" : "error.message",
      "value" : "{{ _ingest.on_failure_message }}"
    }
  }]
}

It's not all the way there but being new to filebeat module creation I am excluding all the special cases of the log format for now.

I am then instructed to generate the fields based on the pipeline configuration. Everything looks good:

~/go/src/github.com/jvalkonen/beats/filebeat $ make create-fields MODULE=casdm FILESET=stdlog
Fields.yml generated for casdm/stdlog

However, no fields.yml (regardless of the case) is found under the module/casdm/stdlog/_meta/ and the module level module/casdm/_meta/fields.yml doesn't contain any of my fields set in the module/casdm/stdlog/ingest/pipeline.json. This already indicates there is something wrong, but I don't know what as the output indicates all is well. If I then run the make update to generate documentation and configuration I get an error, which doesn't seem to indicate issues with the pipeline.json but the development setup itself and possibly the fact that I forked the project instead of just cloning it:

~/go/src/github.com/jvalkonen/beats/filebeat $ make update
mage update
Error: failed to find github.com/elastic/beats/dev-tools/mage in the project's vendor
failed to find github.com/elastic/beats/dev-tools/mage in the project's vendor
make: *** [update] Error 1

Something along the build pipeline is referring to the elastic github path instead of my forked repo. I could go search and replace all of those references but am I trying to do this completely backwards and is there an easier way to achieve what I'm trying to do?

My questions are:

1. On a general level, can one collaborate on a cloned repo or does it need to be forked first?
2. Being rather new to git and GitHub, is there an easier way of collaborating on a GitHub repo without becoming a contributor or messing up the source repo?
3. Are my assumptions about the source of my make errors even close? If yes, what's the proper solution to it?

Answer 1

Turns out my issue was a combination of being a git and go noob and either failing to read some instructions on how to setup the dev environment properly, or those instructions not being there.

In any case, this is what I needed to do (as instructed on this Elastic community thread:

1. Install correct version of Go (had that already)
2. Clone the official beats repo (this is where I went wrong, I forked it and cloned my fork)
3. Fork the official beats repo on github
4. Set my fork as the origin for the cloned repo and add the official beats repo as upstream
5. Setup path variables and install mage (I had the environment set up but I don't remember seeing the installation command for mage, so I guess this was one fault on my process)

After this I no longer get errors and so far all the commands I've needed to work have worked, including building filebeat from scratch.