Reducing log noise for http requests, contextmanager or custom logging level?

Question

I have a logger that I use for most of my programs that I set at logging.INFO. This to me seems the most reasonable setting as the stuff I log isn't necessarily a warning, etc... Problem is, requests and urllib use logging.INFO for each one of their HTTP requests. This creates super long logs for some of my ETL processes, and it's not necessarily helpful in any way.

I was thinking about two solutions, the first being to create a context managing function that sets the script's overall level to logging.WARNING just during the ETL portion where there is no useful logging info. The other option is to create a custom logging level in-between logging.INFO and logging.WARNING. Which would be more re-usable and work more efficiently for the purpose?

Answer 1

When I configure my logger, I always add logging.getLogger('requests').propagate = False to avoid contaminating my logs with logging from the Requests library.

For example:

def init_logger(config_file=None, log_level=logging.INFO):
    # Configure the logging system
    if config_file:
        logging.config.fileConfig(config_file, disable_existing_loggers=False)
    else:
        logging.basicConfig(level=log_level)

    # Avoid output from the requests module
    logging.getLogger('requests').propagate = False


def main():
    init_logger(…)


if __name__ == '__main__':
    main()