Panache reactiveTransactional timeout with no stack trace

Question

Hi I have played a lot with the following code and has read https://github.com/quarkusio/quarkus/issues/21111

I think I am facing a very similar issue, where it will work the first 4 times and then it stops working and things are stuck and eventually showing.

2022-09-15 23:21:21,029 ERROR [io.sma.rea.mes.provider] (vert.x-eventloop-thread-16) SRMSG00201: Error caught while processing a message: io.vertx.core.impl.NoStackTraceThrowable: Timeout

I have seen such exact behaviours in multiple bug reports and discussion threads.

I am using quarkus-hibernate-reactive-panache + quarkus-smallrye-reactive-messaging with kafka (v2.12)

@Incoming("words-in")
@ReactiveTransactional
public Uni<Void> storeToDB(Message<String> message) {
    return storeMetamodels(message).onItemOrFailure().invoke((v, throwable) -> {
        if (throwable == null) {
            Log.info("Successfully stored");
            message.ack();
        } else {
            Log.error(throwable, throwable);
            message.nack(throwable);
        }
    });
}

private Uni<Void> storeMetamodels(Message<String> message) {
    List<EntityMetamodel> metamodels = Lists.newArrayList();
        for (String metamodelDsl : metamodelDsls.getMetamodelDsls()) {
            try {
                EntityMetamodel metamodel = new EntityMetamodel();
                metamodel.setJsonSchema("{}") 
                metamodels.add(metamodel);
            } catch (IOException e) {
                Log.error(e, e);
            }

        }

    return Panache.getSession().chain(session -> session.setBatchSize(10)
            .persistAll(metamodels.toArray((Object[]) new EntityMetamodel[metamodels.size()])));
}

NOTE This same code works if it is running on RestEasy Reactive but I need to move the actual processing and storing to DB away from rest easy as it will be a large process and I do not want it to be stuck on the Rest API waiting for a few minutes.

Hope some Panache or smallrye reactive messaging experts can shed some lights.

Answer 1

Could you try this approach, please?

@Inject
Mutiny.SessionFactory sf;

@Incoming("words-in")
public Uni<Void> storeToDB(Message<String> message) {
    return storeMetamodels(message).onItemOrFailure().invoke((v, throwable) -> {
        if (throwable == null) {
            Log.info("Successfully stored");
            message.ack();
        } else {
            Log.error(throwable, throwable);
            message.nack(throwable);
        }
    });
}

private Uni<Void> storeMetamodels(Message<String> message) {
    List<EntityMetamodel> metamodels = Lists.newArrayList();
        for (String metamodelDsl : metamodelDsls.getMetamodelDsls()) {
            try {
                EntityMetamodel metamodel = new EntityMetamodel();
                metamodel.setJsonSchema("{}") 
                metamodels.add(metamodel);
            } catch (IOException e) {
                Log.error(e, e);
            }

        }

    return sf
        .withTransaction( session -> session
            .setBatchSize(10)
            .persistAll(metamodels.toArray((Object[]) new EntityMetamodel[metamodels.size()]))
        );
}

I suspect you've hit a bug where the session doesn't get closed at the end of storeToDB. Because the session doesn't get closed when injected using Panache or dependency injection, the connection stays open and you hit the limit of connections that can stay open.

At the moment, using the session factory makes it easier to figure out when the session gets closed.