I'm building an app using salvo web framework in rust. The app will have multiple types of db connections (postgres no tls, postgres with tls, redis no tls, redis with tls). This will be user defined from a config file. Each of the connection types can have N number of instances. Each of those instances will have its own pool and can be referenced by its name.
Here is what I'm thinking of implementing
use core::panic;
use std::collections::HashMap;
use bb8_postgres::PostgresConnectionManager;
use serde_json::Value;
use tokio_postgres::NoTls;
use bb8::Pool;
pub struct AnyConnection {
name: String,
details: HashMap<String, Value>,
secure: bool
}
impl AnyConnection {
pub fn new(name: String, details: HashMap<String, Value>, secure: bool) -> Self {
return AnyConnection { name: name, details: details, secure: secure };
}
}
#[salvo::async_trait]
pub trait PostgresPool {
async fn make_unsecure(&self) -> Option<Pool<PostgresConnectionManager<NoTls>>>;
}
#[salvo::async_trait]
impl PostgresPool for AnyConnection {
async fn make_unsecure(&self) -> Option<Pool<PostgresConnectionManager<NoTls>
>> {
let manager = PostgresConnectionManager::new_from_stringlike(
&self.details.get("DATABASE_URL").unwrap().to_string(),
tokio_postgres::NoTls,
);
let conn = match manager {
Ok(m) => m,
Err(err) => panic!("Error opening a connection for: {} with error: {:?}", &self.name, err.to_string())
};
let pool = match Pool::builder().max_size(5).build(conn).await {
Ok(p) => p,
Err(err) => panic!("Error creating pool for: {} with error: {:?}", &self.name, err.to_string())
};
return Some(pool);
}
}
Here is how I'm initializing the pool in main.rs
let mut conns: HashMap<String, AnyConnection> = HashMap::new();
let mut details: HashMap<String, Value> = HashMap::new();
details.insert("DATABASE_URL".to_string(), Value::String("postgresql://localhost:5432/customers?user=postgres&password=postgres".to_string()));
let customers_db: AnyConnection = AnyConnection::new("customers_db".to_string(), details, false);
let pg = PostgresPool::make_unsecure(&customers_db).await;
Where do I cache/store the conns HashMap which keeps the reference of connection pool objects? Are the connections "safely" returns to the pool that is stored in HashMap ?
UPDATE:
Based on the answer, I re-wrote the handle to this
#[async_trait]
impl Handler for HandlerWithPool {
async fn handle(&self, req: &mut Request, _depot: &mut Depot, res: &mut Response, _ctrl: &mut FlowCtrl) {
let delay: Option<u64> = req.query::<u64>("delay");
// self.db_pools is available here
let db_pools = self.get_pools();
let pgpool = db_pools.get("pg").unwrap();
let pgconn = match pgpool {
AnyConn::Postgres(p) => {
println!("Pool State: {:?}", p.state()); // shows more then 1 connection
Some(p)
},
_ => None
};
let rows = pgconn.unwrap().get().await.unwrap().query("select * from users where id = $1", &[&"User-123"]).await.unwrap();
for row in rows {
//println!("Row {:?}", row);
}
let dur = time::Duration::from_millis(delay.unwrap() * 1000);
thread::sleep(dur);
// Now you can do watever you need to with db_pool
res.render("Hello World!");
}
To test out the concurrency of the connection pools, I added a delay query parameter. Using that, I create my first request with delay 10 seconds and second with delay 1 seconds. It seems that second request does not complete until the first request is completed.
So it seems that it is locking the connection pool somewhere which is blocking the second request from fetching connections until first request returns it.
@harmic to answer your other question, I do not need to create pools dynamically. All the pools will be set at server start.
Caveat Emptor: I have not used Salvo in anger, this answer is written based on their documentation.
Where do I cache/store the conns HashMap which keeps the reference of connection pool objects?
So you have a data structure containing some resources, which you create in your main function, and you need to access within your handler functions.
To do this, you can create a struct which will house the connection pool objects and implement Handler for it, as described in the Salvo documentation.
use std::collections::HashMap;
use salvo::prelude::*;
struct AnyConnection;
struct HandlerWithPool {
db_pools: HashMap<String, AnyConnection>,
}
#[async_trait]
impl Handler for HandlerWithPool {
async fn handle(&self, _req: &mut Request, _depot: &mut Depot, res: &mut Response, _ctrl: &mut FlowCtrl) {
// self.db_pools is available here
let db_pool = self.db_pools.get("my_pool").unwrap();
// Now you can do watever you need to with db_pool
res.write_body("Hello World").unwrap();
}
}
#[tokio::main]
async fn main() {
tracing_subscriber::fmt().init();
let db_pools = HashMap::new();
// Populate db_pools here
let hello_handler = HandlerWithPool { db_pools };
let router = Router::new().get(hello_handler);
let acceptor = TcpListener::new("127.0.0.1:5800").bind().await;
Server::new(acceptor).serve(router).await;
}
Your question does not make it clear what the expected usage pattern of the HashMap (which I have called db_pools) will be. I am assuming that you will read your config file on startup and then initialize it accordingly, with one or more bb8::Pool structs.
Note that inside the handler method you have immutable access to &self, which means you can't update the db_pools HashMap. If your intended usage is to create more pools dynamically inside the handler, you would need to wrap it in a Mutex.
Instead of the rather complex trait & struct heirarchy you have proposed, I'd go with an enum to store the different types of pool, like this:
enum AnyConnection {
Postgresql(bb8::Pool<PostgresConnectionManager>),
Redis(bb8::Pool<RedisConnectionManager>),
}
Are the connections "safely" returns to the pool that is stored in HashMap ?
Assuming you are using bb8::Pool instances to manage the connections, and you are talking about connections that you have obtained from the Pool::get method, then yes - you will get a reference to a database connection wrapped in a PooledConnection, and when that goes out of scope the connection is returned to the pool, regardless of what owns the pool.