Search code examples
jsongomapsmgo

Inserting JSON or a map from map[string]interface{} to MongoDB collection sets ints and floats as strings

I know the title seems generic and a duplicate, but i've tried many of the options from previous questions, and I can't use a struct here

My system is using the messaging service NATS to sends maps between a subscriber and a publisher. The subscriber takes the received map, and inserts it as a document in a MongoDB collection

The problem I have is that floats and ints are inserted as strings!

In my code, the recipe is a configuration file that sets the datatypes of the columns received in the map. Think of it as a series of keys like this:

  • String column: "string",
  • Int column: "int"

Here's the code that creates the map with the right datatypes

mapWithCorrectDataTypes := make(map[string]interface{})

    for columnNameFromDataTypesInRecipe, datatypeForColumnInRecipe := range dataTypesFromRecipeForColumns {

    for natsMessageColumn, natsMessageColumnValue := range mapFromNATSMessage {

            //If the column in the NATS message is found in the recipe, format the data as dictated in the recipe
                    if natsMessageColumn == columnNameFromDataTypesInRecipe {

                        if datatypeForColumnInRecipe.(string) == "string" {

                            natsMessageColumnValue = natsMessageColumnValue.(string)
                            mapWithCorrectDataTypes[columnNameFromDataTypesInRecipe] = natsMessageColumnValue
                        }

                        if datatypeForColumnInRecipe.(string) == "int" {

                            convertedInt, err := strconv.Atoi(mapFromNATSMessage[columnNameFromDataTypesInRecipe].(string))
                            if err != nil {
                                fmt.Println("ERROR -->", err)
                            }
                            mapWithCorrectDataTypes[columnNameFromDataTypesInRecipe] = convertedInt
                        }

                        if datatypeForColumnInRecipe.(string) == "float64" {

                            convertedFloat, err := strconv.ParseFloat(mapFromNATSMessage[columnNameFromDataTypesInRecipe].(string), 64)

                            if err != nil {
                                fmt.Println("ERROR -->", err)
                            }
                            mapWithCorrectDataTypes[columnNameFromDataTypesInRecipe] = convertedFloat

                            fmt.Println("TYPE -->", reflect.TypeOf(mapWithCorrectDataTypes[columnNameFromDataTypesInRecipe]))
                        }
                    } else {
                        //If column not found in the recipe, format as a string
                        mapWithCorrectDataTypes[natsMessageColumn] = natsMessageColumnValue.(string)
                    }
                }
            }

From the last line I put in a print statement for float64s to check that the datatype for this key in the map is correct, and it passes this test!

My question is this: If the data types are correctly being set in the map, why when the map is inserted as a document in MongoDB are the floats and ints set as strings?!

What I have tried so far:

Marshalling and unmarshalling the map as an interface, then inserting the record:

        jsonVersionOfMap, err := json.Marshal(mapWithCorrectDataTypes)

        if err != nil {
            fmt.Println("ERROR -->", err)
        }

        var interfaceForJSON interface{}

        json.Unmarshal(jsonVersionOfMap, &interfaceForJSON)

        fmt.Println("JSON -->", interfaceForJSON)
        err = mongoConnection.Insert(interfaceForJSON)

        if err != nil {
            fmt.Println("Error inserting MongoDB documents", err)
        }

What am I missing here?

See the result with the incorrectly formatted data:

enter image description here

Solution

  • this may not be a fix. But i've resolved the issue i've been having. I'm using a publisher and a subscriber via NATS. Previously I was creating a map with all the data, then sending that out as a message, then the subscriber takes the map from the message, and processing the datatypes (on the subscriber side)

    To fix the problem that I was experiencing, I instead formatted the maps' values on the publisher side. So I instead moved my code that checks for the datatype over to the NATS publisher, and not the code that processes the incoming message

    I understand this isn't an ideal solution, but if you're using NATS and find you're having the same issue. Try this