Search code examples
fiwarefiware-orion

Migrating previously collected datasets to FIWARE backend

Having at hand, the task of migrating previously collected environmental datasets (weather, airquality, noise etc) from sensors deployed in different locations, and stored in several tables of MySQL database, to my instance of fiware Orion CB, and thus persisted to fiware backend.

The challenges are many:

  • the data isn't stored in fiware standards, so must be transformed according to the fiware data models.
  • not all tables are a good candidates of being transformed to an Entity.
  • some Entities need have field values from several tables as attributes. For instance, defining AirQualityObserved Entity-type would have attributes from these tables: airquality, co, co2, no2 and deployment. So mapping these attributes to a particular Entity-type is a challenge.

As this is a one-time upload (not live data), I am thinking of two possibilities to go about it.

  1. Add an LwM2M client, to keep sending data to an IoTAgent and eventually passed to Orion CB until the last record.
  2. Create a Python script that "pretends" to be a contextProvider to the Orion instance, sending data (say every 5sec) until the last record.

I have not come across a case in my literature search that addresses such a situation. Is there any recommendations from FIWARE Foundation for situations similar to this?

How would you suggest about data fields --> Entity's attributes mapping that actually need be combined from several tables?

Solution

  • IOTA usage makes sense when you have live data (I mean, a real device sending information to the FIWARE platform). However, you say this is a one-time upload, so the Python script option seems better this case.

    (A little terminological comment here: your script will take the role of context producer. A context provider is a different actor, related with registrations and query/update forwarding. See this piece of documentation for additional detail).

    With regards to the data fields to Entity's attributes mapping I don't have any particular suggestion. This is just a matter of analyzing the data model (i.e. entity attributes) and find how to set that information from your data in the tables.