json.Unmarshal and "push" to array of structs

Question

I have a task to read records from AWS MSK (in AWS lambda) and prepare some kind of payload to send it to Facebook. Records from AWS MSK are base64 encoded, but once I decode them, I get JSON string. Now I don't understand how json.Unmarshal(decodedParams) in to &jsonPayload.Data which is array of structs

type Payload struct {
    Data Data `json:"data,required"`
}

type Data []struct{
    Event             string         `json:"event_name,required"`
    EventTime         int            `json:"event_time,required"`
    EventSourceUrl    string         `json:"event_source_url,omitempty,required"`
    EventActionSource string         `json:"action_source,omitempty,required"`
    EventId           int            `json:"event_id,required"`
    UserData          UserDataType   `json:"user_data,required"`
    CustomData        CustomDataType `json:"custom_data,omitempty"`
}

type CustomDataType struct {
    SearchString     string      `json:"search_string,omitempty"`
    Value            json.Number `json:"value,omitempty"`
    Currency         string      `json:"currency,omitempty"`
}
type UserDataType struct {
    IpAddress string `json:"client_ip_address,omitempty,required"`
    UserAgent string `json:"client_user_agent,omitempty,required"`
}

// ProcessEvent function Using AWS Lambda computed event
func ProcessEvent(event events.KafkaEvent) {

    jsonPayload := Payload{}

    for _, mapper := range event.Records {

        for _, record := range mapper {

            // Base64 decode string from MSK Kafka
            decodedParams, err := base64.StdEncoding.DecodeString(record.Value)
            if err != nil {
                log.Fatal("Error decoding fb event params: ", err)
            }

            // json.Unmarshal and push to Data []structs???
            unmErr := json.Unmarshal(decodedParams, &jsonPayload.Data)
            if unmErr != nil {
                fmt.Println("UNMARSHAL ERROR")
                fmt.Println(unmErr)
            }
        }
    }
}

func main() {
    lambda.Start(ProcessEvent)
}

End result of payload should be similar to this one

{
   "data":[
      {
         "event_name":"Purchase",
         "event_time":1627975460,
         "action_source":"email",
         "user_data":{
            "em":[
               "7b17fb0bd173f625b58636fb796407c22b3d16fc78302d79f0fd30c2fc2fc068"
            ],
            "ph":[
               null
            ]
         },
         "custom_data":{
            "currency":"USD",
            "value":"142.52"
         }
      },
      {
         "event_name":"PageView",
         "event_time":1627975460,
         "action_source":"email"
      }
   ]
}

Currently I'm getting error

json: cannot unmarshal object into Go value of type main.Data

Since I'm fairly new with GO I wonder if I'm on the right path and how to push decoded json string in to Data []struct? If you need any additional informations, please let me know and I will provide. Thank you!

Answer 1

Every single iteration of event.Records is likely to overwrite whatever you have the, more of that try to imagine .Data as slice of []data

type data struct {
    Event             string         `json:"event_name,required"`
    EventTime         int            `json:"event_time,required"`
    EventSourceUrl    string         `json:"event_source_url,omitempty,required"`
    EventActionSource string         `json:"action_source,omitempty,required"`
    EventId           int            `json:"event_id,required"`
    UserData          UserDataType   `json:"user_data,required"`
    CustomData        CustomDataType `json:"custom_data,omitempty"`
}

type Data []data

So you will have code that unmarshal and push to Data slice small data struct, similar to this.

var d data
unmErr := json.Unmarshal(decodedParams, &d)
if unmErr != nil {
    fmt.Println("UNMARSHAL ERROR")
    fmt.Println(unmErr)
}

jsonPayload.Data = append(jsonPayload.Data, d)