Remove duplicates in nested objects of object

Question

Trying to parse some network descriptions from devices, hitting a problem where a description could be duplicate across types. This causes errors further down the road in code. What is the best choice with lodash or javascript to compare each nested object and remove duplicates until they don't exist in the arrays. The goal is that string 1 - 111 would be removed until it only exists in one object.

const data = {
    "hostname": {
        "typeA": [
            "string 1 - 111",
            "string 2 - 222",
            "string 3 - 333"
        ],
        "typeB": [
            "string 1 - 111",
            "string 4 - 444",
            "string 5 - 555"
        ],
        "typeC": [
            "string 1 - 111",
            "string 6 - 666"
        ]
    }
}

Answer 1

You simply need to iterate over each type array and filter based on a Set of previously seen values. Here mutating data in place using filter() to both populate the Set and to return the filtered array.

const data = {
  hostname: {
    typeA: ['string 1 - 111', 'string 2 - 222', 'string 3 - 333'],
    typeB: ['string 1 - 111', 'string 4 - 444', 'string 5 - 555'],
    typeC: ['string 1 - 111', 'string 6 - 666'],
  },
};

const seen = new Set();

for (const key of Object.keys(data.hostname)) {
  data.hostname[key] = data.hostname[key].filter((s) => {
    const _seen = seen.has(s);
    seen.add(s);

    return !_seen;
  });
}

console.log(data);