I am attempting to write a Grafana backend plugin in C++. I am doing this in order to integrate with a custom tool (called "lighthouse") that has a C++ API. I created a C++ gRPC application that implements the Grafana plugin protocol as this documentation suggests.
My question is this: how are the connection settings, the gRPC server port for instance, communicated between the main Grafana process and the backend plugin. I was expecting to be able to specify the connection string or port number in the plugin.json configuration file but it doesn't appear that there is any such field.
I installed my plugin and attempted to load it. It didn't work, of course, but gave me this log output:
logger=plugin.loader t=2023-01-11T11:58:25.984652862Z level=debug msg="Loading plugin" path=/var/lib/grafana/plugins/lighthouse-datasource/plugin.json
logger=plugin.loader t=2023-01-11T11:58:25.984878566Z level=debug msg="Plugin is unsigned" id=lighthouse-datasource
logger=plugin.signature.validator t=2023-01-11T11:58:25.984904895Z level=warn msg="Permitting unsigned plugin. This is not recommended" pluginID=lighthouse-datasource pluginDir=/var/lib/grafana/plugins/lighthouse-datasource
logger=plugin.loader t=2023-01-11T11:58:25.984925224Z level=info msg="Plugin registered" pluginID=lighthouse-datasource
logger=plugin.lighthouse-datasource t=2023-01-11T11:58:25.984963175Z level=debug msg="starting plugin" path=/var/lib/grafana/plugins/lighthouse-datasource/lighthouse_backend_linux_amd64 args=[/var/lib/grafana/plugins/lighthouse-datasource/lighthouse_backend_linux_amd64]
logger=plugin.lighthouse-datasource t=2023-01-11T11:58:25.985195211Z level=debug msg="plugin started" path=/var/lib/grafana/plugins/lighthouse-datasource/lighthouse_backend_linux_amd64 pid=7065
logger=plugin.lighthouse-datasource t=2023-01-11T11:58:25.98523206Z level=debug msg="waiting for RPC address" path=/var/lib/grafana/plugins/lighthouse-datasource/lighthouse_backend_linux_amd64
logger=plugin.loader t=2023-01-11T11:59:31.176451438Z level=error msg="Could not start plugin" pluginId=lighthouse-datasource err="timeout while waiting for plugin to start"
logger=plugin.lighthouse-datasource t=2023-01-11T11:59:31.177010088Z level=debug msg="plugin process exited" path=/var/lib/grafana/plugins/lighthouse-datasource/lighthouse_backend_linux_amd64 pid=7065 error="signal: killed"
The message msg="waiting for RPC address" suggests that the Grafana process is waiting for something, perhaps the plugin itself, to provide the gRPC address. If this interpretation is correct, how is the plugin supposed to provide this information?
I was eventually able to figure this out by looking for the origin of the "waiting for RPC address" message. As it turns out that message comes from code in the HashiCorp go-plugin. As described there the plugin implements the serve portion while the plugin user (Grafana in this case) implements the client portion.
After scanning through that code I was able to piece together how things fit together. Grafana spawns the plugin executable as a subprocess and then attaches pipes to capture its stdout and stderr. Grafana expects a special pipe-delimited value to be output to stdout that includes the connection information. For my simple case the value output looks something like this:
1|2|unix|/tmp/some_path_for_uds|grpc
Grafana logs the plugin's stderr output which is useful for debugging.
One thing I got wrong above is that if you're running Unix/Linux then the plugin architecture expects gRPC to use a Unix domain socket. This makes sense but didn't occur to me earlier. So in my case the address+port I was expecting turned out just be the UDS filesystem path as shown above.
Here is the final crude version of my C++ plugin entry point:
#include <cstdint>
#include <fstream>
#include <iomanip>
#include <iostream>
#include <memory>
#include <sstream>
#include <string>
#include <grpcpp/grpcpp.h>
#include "DiagnosticsServiceImpl.h"
#include "Logger/Logger.h"
void RunServer(const std::string& udsAddress)
{
const std::string addressURI("unix://" + udsAddress);
::unlink(udsAddress.c_str());
DiagnosticsServiceImpl service;
grpc::EnableDefaultHealthCheckService(true);
grpc::reflection::InitProtoReflectionServerBuilderPlugin();
grpc::ServerBuilder builder;
// Listen on the given address without any authentication mechanism.
builder.AddListeningPort(addressURI, grpc::InsecureServerCredentials());
// Register "service" as the instance through which we'll communicate with
// clients. In this case it corresponds to an *synchronous* service.
builder.RegisterService(&service);
// Finally assemble the server.
std::unique_ptr<grpc::Server> server(builder.BuildAndStart());
std::cerr << "Server listening on " << udsAddress;
// Build start handshake output
std::ostringstream strm;
strm << "1|2|unix|" << udsAddress << "|grpc\n";
const std::string startOutput(strm.str());
// Now output it
std::cout << startOutput << std::endl;
// Wait for the server to shutdown. Note that some other thread must be
// responsible for shutting down the server for this call to ever return.
server->Wait();
}
int main(int argc, char** argv)
{
const std::string udsAddress("/tmp/lighthouse_backend_plugin_uds");
std::cerr << "Grafana Lighthouse Backend Plugin starting...";
RunServer(udsAddress);
std::cerr << "Grafana Lighthouse Backend Plugin terminating...";
return 0;
}
This took me several days to figure out so hopefully it will save someone some time in the future.