How can I deserialize JSON that has duplicate keys without losing any of the values?

Question

I am trying to deseralize some JSON that has duplicate keys and they might be in any number. The JSON looks like this:

...
"abilities": [
    {
        "ability_id": 5134,
        "ability_level": 3
    }, 
    {
        "ability_id": 5136,
        "ability_level": 3
    }
],
"abilities": [
    {
        "ability_id": 7710,
        "ability_level": 4
    }
]
...

original json

And my Rust code is:

#[derive(Deserialize, Debug)]
pub struct Ancient {
    score: usize,
    tower_state: usize,
    barracks_state: usize,
    picks: Option<Vec<HeroId>>,
    bans: Option<Vec<HeroId>>,
    players: Vec<PlayerDetailed>,
    abilities: Option<Vec<Ability>> // has many keys
}

original rust

The struct Ancient is part of another struct that in json.

Only the last abilities has many keys and that too in variable numbers, I have read this but am unable to change it into my liking (I want the end user to still have a struct). I have even tried #[serde(flatten)] but it just makes everything None. If possbile I would like to change it too:

#[derive(Deserialize, Debug)]
pub struct Ancient {
    score: usize,
    tower_state: usize,
    barracks_state: usize,
    picks: Option<Vec<HeroId>>,
    bans: Option<Vec<HeroId>>,
    players: Vec<PlayerDetailed>,
    abilities_list: Option<abilities<Option<Vec<Ability>>>> 
}

Answer 1

The JSON specification does not explicitly forbid duplicate keys, but most implementations will ignore all but one of them. With derived serde::Deserialize implementations any serde deserializer including serde_json will panic if there is a duplicate key.

If you can't change the json format, you can implement Deserialize manually for one of your types.

Let's simplify your structs and Json a little.

JSON:

{
    "name": "sn99",
    "abilities": [
        {
            "ability_id": 5134,
            "ability_level": 3
        }, 
        {
            "ability_id": 5136,
            "ability_level": 3
        }
    ],
    "abilities": [
        {
            "ability_id": 7710,
            "ability_level": 4
        }
    ]
}

And parse into structs like this:

use serde::Deserialize;

#[derive(Debug, Deserialize)]
struct Ability {
    ability_level: u32,
    ability_id: u32,
}

#[derive(Debug)]
struct Abilities(Vec<Ability>);

#[derive(Debug, Deserialize)]
struct Entity {
    name: String,
    #[serde(flatten)]
    abilities: Abilities,
}

Note that Abilities does not derive Deserialize; we'll implement it manually:

use serde::de::{self, MapAccess, Visitor, Error as _};

impl<'de> Deserialize<'de> for Abilities {
    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
    where
        D: de::Deserializer<'de>,
    {
        struct MyVisitor;

        impl<'d> Visitor<'d> for MyVisitor {
            type Value = Vec<Ability>;

            fn expecting(&self, f: &mut std::fmt::Formatter<'_>) -> Result<(), std::fmt::Error> {
                f.write_str("a map of abilities")
            }

            fn visit_map<M>(self, mut access: M) -> Result<Self::Value, M::Error>
            where
                M: MapAccess<'d>,
            {
                let mut abilities = Vec::new();
                while let Some((key, mut value)) = access.next_entry()? {
                    if key == "abilities" {
                        abilities.append(&mut value);
                    } else {
                        return Err(M::Error::unknown_field(key, &["abilities"]));
                    }
                }
                Ok(abilities)
            }
        }
        Ok(Abilities(deserializer.deserialize_map(MyVisitor)?))
    }
}

let entity: Entity = serde_json::from_str(input).unwrap();
println!("{:#?}", entity);

Produces:

Entity {
    name: "sn99",
    abilities: Abilities(
        [
            Ability {
                ability_level: 3,
                ability_id: 5134,
            },
            Ability {
                ability_level: 3,
                ability_id: 5136,
            },
            Ability {
                ability_level: 4,
                ability_id: 7710,
            },
        ],
    ),
}