Is there any way to change the log message format in scrapy?

Question

I would like to modify the scrapy log messages to contain user id at the beginning of it. for example, instead of this

2015-03-03 17:09:34+0530 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware

Is it possible to make it appear like

**user_id**:2015-03-03 17:09:34+0530 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware

Additionaly, is there a way to log only the user messages; i.e, Log only the messages I want logged which I will supply from the code?

Any help would be greatly appreciated. Thanks in advance!

Answer 1

Scrapy uses Twisted's logging systems which are build on top of logging standard library. I don't have answer for you, how to change date format of logs generated by scrapy itself.

For you second question I might have answer. I think you should completely silence scrapy logging by setting

LOG_ENABLED = False

in you settings.py. Then, configure logging module for yourself (on top of spider module)

logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(levelname)8s\t%(message)s', )

and then log using logging.log, logging.info, logging.debug etc.

I leave to you figure out desired logging format.