i am struggling with conversion from JSON(deserialised from API-provicded data) to polars dataframe.
below are my structs and defined function:
// structs for option price query
#[derive(Serialize, Deserialize, Debug)]
pub struct OptionResponse {
pub tables: Vec<OptionTable>,
}
#[derive(Serialize, Deserialize, Debug)]
pub struct OptionTable {
pub table: OptionDataTable,
}
#[derive(Serialize, Deserialize, Debug)]
pub struct OptionDataTable {
pub volume: Vec<f64>,
pub open: Vec<f64>,
pub close: Vec<f64>,
pub high: Vec<f64>,
pub low: Vec<f64>,
}
pub async fn ifind_option_read(option_code: &str, access_token: &str, start_time_str: &str, end_time_str: &str) -> Result<(), Box<dyn std::error::Error>> {
let url = "https://quantapi.51ifind.com/api/v1/high_frequency";
// create the high-frequency prices request
let request_data = maplit::hashmap! {
"codes" => option_code,
"indicators" => "volume,open,close,high,low",
"starttime" => start_time_str,
"endtime" => end_time_str,
};
let request_json = serde_json::to_string(&request_data)?;
// send the HTTP POST request
let client = reqwest::Client::new();
let response = client
.post(url)
.header(reqwest::header::CONTENT_TYPE, "application/json")
.header("access_token", access_token)
.body(request_json)
.send()
.await?;
let response_json: OptionResponse = response.json().await?;
let response_body = &response_json.tables[0].table;
println!("{:?}", &response_body);
// Convert response_json to byte array
let json_bytes = serde_json::to_vec(&response_body)?;
// read the response into df, per method of JsonReader
let df = JsonReader::new(Cursor::new(json_bytes))
.with_batch_size(1024)
.infer_schema_len(None)
.finish().unwrap();
println!("{:?}", df);
Ok(())
}
the response_body returned looks very normal to me and looks like
OptionDataTable { volume: [0.0, 0.0, 0.0], open: [1263.0, 1263.0, 1263.0], close: [1263.0, 1263.0, 1263.0], high: [1263.0, 1263.0, 1263.0], low: [1263.0, 1263.0, 1263.0] }
neverthless the JsonReader cant read it as a df. this is confusing. is this a json format problem?
previously the deserialize function in
polars/polars-json/src/json/deserialize.rs
has a todo! macro causing this error.
pub fn deserialize(json: &BorrowedValue, data_type: DataType) -> Result<Box<dyn Array>, Error> {
match json {
BorrowedValue::Array(rows) => match data_type {
DataType::LargeList(inner) => Ok(_deserialize(rows, inner.data_type)),
_ => todo!("read an Array from a non-Array data type"),
},
_ => todo!("read an Array from a non-Array JSON"),
}
}
now it is replaced with Ok(_deserialize(&[json], data_type))
with commit #207e0b9 everything should be fine now.