I am reading some data from a sql database using odbc_api
in Rust that has a mixture of column types. However, I noticed that the fields in the table that have varchar(max)
fail to return back from the table. I am imagining it is related to the TextRowSet
string buffer size, but even if I change the value from 4 KiB to a massively higher limit, I still do not receive back the varchar columns. Everything builds without warning or error.
code:
use anyhow::Error;
use odbc_api::{buffers::TextRowSet, Cursor, Environment, ResultSetMetadata};
use std::io;
const BATCH_SIZE: usize = 100;
fn main() -> Result<(), Error>
{
// Write sql query to stdout
let out = io::stdout();
let mut writer = csv::Writer::from_writer(out);
// Establish environment
let environment = Environment::new()?;
let connection_string = "
Driver={ODBC Driver 17 for SQL Server};\
Server=my_server;\
db=my_db;\
trusted_connection=yes;
";
let conn = environment.connect_with_connection_string(connection_string)?;
// Set up query and execute
let qry = "select top 10 * from my_tbl";
match conn.execute(&qry, ())
{
Err(e) => println!("{}", e),
Ok(Some(mut cursor)) => {
// Write column names to stdout
let headline: Vec<String> = cursor.column_names()?.collect::<Result<_,_>>()?;
writer.write_record(headline)?;
// Use schema in cursor to initialize a text buffer large enough to hold the largest
// possible strings for each column to an upper limit of 4Kib.
let mut buffers = TextRowSet::for_cursor(BATCH_SIZE, &mut cursor, Some(4096))?;
// Bind the buffer to the cursor. It will now be filled with every call to fetch.
let mut row_set_cursor = cursor.bind_buffer(&mut buffers)?;
// Iterate over batches
while let Some(batch) = row_set_cursor.fetch()?
{
// Within a batch, iterate over every row
for row_index in 0..batch.num_rows()
{
//println!("{}", row_index);
let record = (0..batch.num_cols()).map(|col_index| {
batch.at(col_index, row_index)
.unwrap()
});
// Writes row as csv
writer.write_record(record)?;
}
}
},
Ok(None) => {
eprintln!("Query came back empty. No output created.");
}
}
Ok(())
}
odbc_api docs: https://docs.rs/odbc-api/latest/odbc_api/
**Hacky Solution EDIT: **
So by going to the source tables and modifying the fields that were varchar(max)
to varchar(255)
, I effectively solved the problem. Can anyone explain what happened here?
Microsoft SQL Server is reporting an element size of 0
for columns of type VARCHAR(max)
. Dutifully the text buffer allocates just enough memory for the terminating zero per field, while staying well under the specified upper bound. Since I am the author of odbc-api
I fixed it in version 0.50.2
which I released after reading your question.