I try to build script that execute some grep search in my logs and print the results. I try to use Envoy because is more easy than subprocess but when I execute grep command it gives me back an error of no such file o directory.
The dir structure is easy:
My test.py is easy:
import envoy
def test(value):
search = "grep 'cv="+str(value)+"' ./web_logs/log/log_*"
print(search) #check of the search string
r = envoy.run(search)
print(r.status_code, r.std_out, r.std_err)#check of the command
response = r.std_out
if __name__ == "__main__":
test(2)
The output is:
grep 'cv=2' ./web_logs/log/log_*
(2, '', 'grep: ./web_logs/log/log_*: No such file or directory\n')
If i run the same command:
grep 'cv=2' ./web_logs/log/log_*
I can find the occurrence of the string "cv=2" in the log files.
Where is the error?
Update after the answers The problem is in using of * that envoy cannot explode without use of glob module so I using the subprocess as it is and I try to study better the using of glob module to improve envoy.
The new code I used is:
import subprocess
def test(value):
search = "grep 'cv="+str(value)+"' ./web_logs/log/log_*"
print(search) #check of the search string
proc = subprocess.check_output(search, shell=True)
print proc.split('\n')
if __name__ == "__main__":
test(2)
@baptistemm is actually right in that since you're not running bash as part of your process the globbing is not working.
However what's happening is a bit deeper.
When you run a sub process it can be done by one of several system services (system calls).
Here's the correct way to do this:
import envoy
def test(value):
search = "/bin/sh -c \"grep 'cv="+str(value)+"' ./web_logs/log/log_*\""
print(search) #check of the search string
r = envoy.run(search)
print(r.status_code, r.std_out, r.std_err)#check of the command
response = r.std_out
if __name__ == "__main__":
test(2)
Running the command as a shell command will take care of globbing.
Whenever a sub process is executed, it eventually gets translated into an execve system call (or equivalent).
In C library there're helper functions such as system(3) and popen(3) which wrap around execve(2) to provide easier ways of executing processes. system launches a shell and passes its argument as is to -c option of the shell. popen does extra magic, kinda like what envoy is doing in python.
In envoy, the argument is parsed for | (see def expand_args(command):) in the envoy code. and then uses the equivalent of popen to execute the processes. envoy is essentially what the shell does with the | marker (splits things up across the | and then uses popen).
What envoy is NOT doing is interpreting * as the shell does, as in expanding it to match files using a glob function of some sort. Bash does. Thus my answer.
A fun exercise would be for you to contribute code to envoy :-) and make it do the globbing.