Search code examples
sql-serverrustodbc

Issue writing varchar(max) Column Types in SQL Serve using Rust odbc_api


I am reading some data from a sql database using odbc_api in Rust that has a mixture of column types. However, I noticed that the fields in the table that have varchar(max) fail to return back from the table. I am imagining it is related to the TextRowSet string buffer size, but even if I change the value from 4 KiB to a massively higher limit, I still do not receive back the varchar columns. Everything builds without warning or error.

code:

use anyhow::Error;
use odbc_api::{buffers::TextRowSet, Cursor, Environment, ResultSetMetadata};
use std::io;

const BATCH_SIZE: usize = 100;
fn main() -> Result<(), Error> 
{   
    // Write sql query to stdout
    let out = io::stdout();
    let mut writer = csv::Writer::from_writer(out);

    // Establish environment
    let environment = Environment::new()?; 
    let connection_string = "
        Driver={ODBC Driver 17 for SQL Server};\
        Server=my_server;\
        db=my_db;\
        trusted_connection=yes;
    ";
    let conn = environment.connect_with_connection_string(connection_string)?;
    
    // Set up query and execute
    let qry = "select top 10 * from my_tbl";
    match conn.execute(&qry, ())
    {
        Err(e) => println!("{}", e),
        Ok(Some(mut cursor)) => {
            // Write column names to stdout
            let headline: Vec<String> = cursor.column_names()?.collect::<Result<_,_>>()?;
            writer.write_record(headline)?;
            
            // Use schema in cursor to initialize a text buffer large enough to hold the largest
            // possible strings for each column to an upper limit of 4Kib.
            let mut buffers = TextRowSet::for_cursor(BATCH_SIZE, &mut cursor, Some(4096))?;

            // Bind the buffer to the cursor. It will now be filled with every call to fetch.
            let mut row_set_cursor = cursor.bind_buffer(&mut buffers)?;

            // Iterate over batches
            while let Some(batch) = row_set_cursor.fetch()?
            {   
                // Within a batch, iterate over every row
                for row_index in 0..batch.num_rows()
                {
                    //println!("{}", row_index);
                    let record = (0..batch.num_cols()).map(|col_index| {
                        batch.at(col_index, row_index)
                             .unwrap()
                    });
                    // Writes row as csv
                    writer.write_record(record)?;
                }
            }
        },
        Ok(None) => {
            eprintln!("Query came back empty. No output created.");
        }
    }

    Ok(())
}

odbc_api docs: https://docs.rs/odbc-api/latest/odbc_api/

**Hacky Solution EDIT: **

So by going to the source tables and modifying the fields that were varchar(max) to varchar(255), I effectively solved the problem. Can anyone explain what happened here?


Solution

  • Microsoft SQL Server is reporting an element size of 0 for columns of type VARCHAR(max). Dutifully the text buffer allocates just enough memory for the terminating zero per field, while staying well under the specified upper bound. Since I am the author of odbc-api I fixed it in version 0.50.2 which I released after reading your question.