boto3 filter_log_events doesn't return cloudwatch logs

Question

I use python 3.7, boto3 1.9.196

there's multiple log streams in a single log group (more then 800)

group name : my-board

stream names : my-board-20191223-00, my-board-20191223-01 ... my-board-20191223-23

each stream names set with current datetime (YYYYMMDD-HH)

self._client.filter_log_events(
    logGroupName=log_group,
    startTime=start_timestamp,
    endTime=end_timestamp,
    filterPattern=filterPattern,
    limit=limit,
)

eg) startTime : 2019-11-07 00:00:00, endTime : 2019-11-08 23:59:59

it returns multiple events

'events': [ blabla... ]
'searchedLogStreams':[
{
    'logStreamName': 'my-board-20191101-17',
    'searchedCompletely': True
}, 
    blabla...
{
    'logStreamName': 'my-board-20191107-14',
    'searchedCompletely': True
}, {
    'logStreamName': 'my-board-20191107-21',
    'searchedCompletely': True
}, {
    'logStreamName': 'my-board-20191107-22',
    'searchedCompletely': True
}, {
    'logStreamName': 'my-board-20191107-23',
    'searchedCompletely': True
}, {
    'logStreamName': 'my-board-20191108-00',
    'searchedCompletely': False # FALSE
}]

eg2) startTime : 2019-12-23 00:00:00, endTime : 2019-12-24 23:59:59

but, it doesn't return any events!

'events': [], # EMPTY
'searchedLogStreams': [{
    'logStreamName': 'my-board-20191101-17',
    'searchedCompletely': True
},
    blabla...
{
    'logStreamName': 'my-board-20191102-17',
    'searchedCompletely': True
}, {
    'logStreamName': 'my-board-20191105-16',
    'searchedCompletely': True
}, {
    'logStreamName': 'my-board-20191112-13',
    'searchedCompletely': True
}, {
    'logStreamName': 'my-board-20191112-14',
    'searchedCompletely': True
}, {
    'logStreamName': 'my-board-20191112-19',
    'searchedCompletely': True
}, {
    'logStreamName': 'my-board-20191112-20',
    'searchedCompletely': True
}, {
    'logStreamName': 'my-board-20191112-21',
    'searchedCompletely': True
}, {
    'logStreamName': 'my-board-20191112-22',
    'searchedCompletely': True
}, {
    'logStreamName': 'my-board-20191112-23',
    'searchedCompletely': True # TRUE
}]

I think it's because there's too many log streams

why it doesn't work? and how to fix it?

Answer 1

If you get a next token in the response, keep doing There will be token that return nothing but there will be one with result you keep scrolling.

currentToken = response['nextToken']
response = client.filter_log_events(
        limit = 10000,
        filterPattern= filterPattern,
        startTime= start_timestamp,
        logGroupName= log_group,
        nextToken=currentToken)
print(response)

I was in same situation. My best guess is that the scrolling and the filtering are independent. If you remove the filter param (time, pattern in subsequence call you'll find it return events that does not match the initial call).