Since it was announced the access to Cosmos' WebHDFS in FIWARE Lab is protected with OAuth2, Cygnus (version 0.8.1) is not able to persist Orion's context data in Cosmos HDFS:
15/07/09 08:45:21 INFO handlers.OrionRestHandler: Starting transaction (1436424291-649-0000000000)
15/07/09 08:45:21 INFO handlers.OrionRestHandler: Received data ({ "subscriptionId" : "51c0ac9ed714fb3b37d7d5a8", "originator" : "localhost", "contextResponses" : [ { "contextElement" : { "attributes" : [ { "name" : "temperature", "type" : "centigrade", "value" : "26.5" } ], "type" : "Room", "isPattern" : "false", "id" : "Room1" }, "statusCode" : { "code" : "200", "reasonPhrase" : "OK" } } ]})
15/07/09 08:45:21 INFO handlers.OrionRestHandler: Event put in the channel (id=109760451, ttl=10)
15/07/09 08:45:21 INFO sinks.OrionSink: Event got from the channel (id=109760451, headers={timestamp=1436424321536, content-type=application/json, transactionId=1436424291-649-0000000000, fiware-service=def_serv, fiware-servicepath=def_serv_path, ttl=10, destination=room1_room}, bodyLength=460)
15/07/09 08:45:22 INFO sinks.OrionHDFSSink: [hdfs-sink] Persisting data at OrionHDFSSink. HDFS file (def_serv/def_serv_path/room1_room/room1_room.txt), Data ({"recvTime":"2015-07-09T06:45:21.536Z","temperature":"26.5", "temperature_md":[]})
15/07/09 08:45:22 ERROR sinks.OrionSink: Persistence error (The /user/frb/def_serv/def_serv_path/room1_room directory could not be created in HDFS. HttpFS response: 503 Service unavailable)
15/07/09 08:45:22 INFO sinks.OrionSink: An event was put again in the channel (id=109760451, ttl=9)
15/07/09 08:45:22 INFO sinks.OrionSink: Finishing transaction (1436424291-649-0000000000)
I figure out an OAuth2 token has to be configured somewhere, but how?
IMPORTANT: Before implementing any of the commands below, backup your Cygnus configurations since they will be still valid with minor changes.
Sadly, available Cygnus versions have no support for OAuth2. Until now. You will have to upgrade to the latest 0.8.2, which allows to configure a token through this OrionHDFSSink parameter:
<your_agent_nam>.sinks.<your_hdfs_sink_name>.oauth2_token = <token>
Upgrading to Cygnus 0.8.2 is as easy as installing it from the FIWARE repo:
$ (sudo) yum clean all # just to clean the yum cache
$ (sudo) yum list cygnus # this will show you 0.82 is available
$ (sudo) sudo rpm -e -vv --allmatches --nodeps --noscripts --notriggers cygnus # this is needed if you have installed a version < 0.8.0
$ (sudo) yum install cygnus # this installs 0.8.2
Remember the FIWARE repo is setup by editing /etc/yum.repos.d/fiware.repo (most probably you already have it):
[Fiware]
name=FIWARE repository
baseurl=http://repositories.testbed.fi-ware.eu/repo/rpm/x86_64/
gpgcheck=0
enabled=1
In the example below, we are moving from a 0.7.0 version to latest 0.8.2:
$ sudo yum clean all
Loaded plugins: fastestmirror, security
Cleaning repos: ...
Cleaning up Everything
Cleaning up list of fastest mirrors
$ sudo yum list cygnus
Loaded plugins: fastestmirror, security
Determining fastest mirrors
* base: sunsite.rediris.es
* epel: ftp.cica.es
* extras: sunsite.rediris.es
* updates: sunsite.rediris.es
...
Installed Packages
cygnus.x86_64 0.7.0-g4cddd26 @Fiware
Available Packages
cygnus.x86_64 0.8.2-0.g7353160 Fiware
$ sudo rpm -e -vv --allmatches --nodeps --noscripts --notriggers cygnus
D: loading keyring from pubkeys in /var/lib/rpm/pubkeys/*.key
D: couldn't find any keys in /var/lib/rpm/pubkeys/*.key
D: loading keyring from rpmdb
...
D: closed db index /var/lib/rpm/Name
D: closed db index /var/lib/rpm/Packages
D: closed db environment /var/lib/rpm
$ sudo yum install cygnus
Loaded plugins: fastestmirror, security
Loading mirror speeds from cached hostfile
* base: sunsite.rediris.es
* epel: ftp.cica.es
* extras: sunsite.rediris.es
* updates: sunsite.rediris.es
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package cygnus.x86_64 0:0.7.0-g4cddd26 will be updated
---> Package cygnus.x86_64 0:0.8.2-0.g7353160 will be an update
--> Finished Dependency Resolution
Dependencies Resolved
==============================================================================================================================================================
Package Arch Version Repository Size
==============================================================================================================================================================
Updating:
cygnus x86_64 0.8.2-0.g7353160 Fiware 94 M
Transaction Summary
==============================================================================================================================================================
Upgrade 1 Package(s)
Total download size: 94 M
Is this ok [y/N]:
Downloading Packages:
cygnus-0.8.2-0.g7353160.x86_64.rpm | 94 MB 00:03
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Updating : cygnus-0.8.2-0.g7353160.x86_64 1/2
[INFO] Creating cygnus user
[INFO] Creating log directory
Done
Cleanup : cygnus-0.7.0-g4cddd26.x86_64 2/2
[INFO] Uninstall the cygnus
Stopping Cygnus chatrooms... [ OK ]
[INFO] Deleting links
rm: no se puede borrar «/etc/cygnus/flume.conf»: No existe el fichero o el directorio
[INFO] Removing application log files
[INFO] Deleting the cygnus folder
[INFO] Deleting the cygnus user
Done
Updated:
cygnus.x86_64 0:0.8.2-0.g7353160
Complete!
In order to check you have the right version, you can do several things. For instance, using the rpm command:
$ rpm -qa | grep cygnus
cygnus-0.8.2-0.g7353160.x86_64
Or using the yum list:
$ (sudo) yum list cygnus
Loaded plugins: fastestmirror, security
Loading mirror speeds from cached hostfile
* base: sunsite.rediris.es
* epel: ftp.cica.es
* extras: sunsite.rediris.es
* updates: sunsite.rediris.es
Installed Packages
cygnus.x86_64 0.8.2-0.g7353160 @Fiware
Or simply check the oauth2_token parameter is there ;)
$ cat /usr/cygnus/conf/agent.conf.template |grep oauth2_token
cygnusagent.sinks.hdfs-sink.oauth2_token = xxxxxxxx
IMPORTANT: Backup your Cygnus configurations since they will be still valid with minor changes:
es.tid.fiware.fiwareconnectors.cygnus.etc but com.telefonica.iot.cygus.etc.matching_table.conf (CSV-like content) now is named as grouping_rules.conf (JSON-like content with the same semantic than the CSV-like version).