I am composing a Splunk query to summarize AWS ELB access logs. The purpose is to gather all the information needed to inform a load test that approximates production traffic.
Many of our URLs include path parameters. For example, a directory component may be a GUID, or a user's account ID, etc. In order to collapse all calls to a particular endpoint into a single string, I need to find and replace those path parameters with a string that represent the parameter. For example: /users/3h5l2h5fn1 becomes /users/{accountId}. This is easy enough, I just use a sequence of | rex field=path mode=sed commands.
The problem I'm encountering is that about the time I hit 30 such commands, the query starts failing. There's no clear indication why, the job just gets killed.
My question for the Stack Overflow experts is: Is there another way of doing a series of 50+ find-and-replace on the path field? One that won't break Splunk?
this sounds like a job for transforms.conf or maybe just to properly extract all those fields with props.conf.
If you're waiting until search time to mask data, you're still storing all of what you're trying to mask
Lastly, eval myfield=replace(my_field,"regex","literal string") is almost always faster, in my experience, than rex mode=sed
Doc.Splunk references for eval, rex: