Fiware-cygnus documentation mentions that it is based on Apache Flume. However, it is not clear whether I can use native Flume sinks to persist events arriving from Orion Context Broker. Is this something I can easily do, with little (or ideally zero) coding? If not -- would be good to know why (and whether this can be supported going forward). Thanks!
You can use native Flume sinks by simply configuring them. Nothing has been changed in Cygnus in terms of configuration management, thus you can configure a Orion-like sink or a native one.
Nevertheless, there are differences between Orion-like and native Flume sinks.
The first one is the Orion-like sinks store the relevant data with certain structure, and the Flume native sinks will store the notified raw data. I mean, if you receive a Json-based notification such as:
{
"subscriptionId" : "51c0ac9ed714fb3b37d7d5a8",
"originator" : "localhost",
"contextResponses" : [
{
"contextElement" : {
"attributes" : [
{
"name" : "speed",
"type" : "float",
"value" : "112.9",
"metadatas": []
},
{
"name" : "oil_level",
"type" : "float",
"value" : "74.6",
"metadatas": []
}
],
"type" : "car",
"isPattern" : "false",
"id" : "car1"
},
"statusCode" : {
"code" : "200",
"reasonPhrase" : "OK"
}
]
}
OrionHDFSSink will store something like:
{"recvTimeTs":"1429535775","recvTime":"2015-04-20T12:13:22.41.124Z","fiware-servicePath":"4wheels","entityId":"car1","entityType":"car","attrName":"speed","attrType":"float","attrValue":"112.9","attrMd":[]}
But a native HDFS sink (or any other one) will persist the entire notified json.
The second main difference if the handling of the notified fiware-service and fiware-servicePath. Cygnus's sinks are able to deal with these values in order to map the notified data into specific data structures (folders, databases, tables, resources, queues...). This is very important for multi-tenancy purposes.
Third, Cygnus adds sinks for storages not covered by native Flume, such as CKAN, STH, MongoDB, MySQL or DynamoDB.
There are many other differences: