Search code examples
linuxbashshellcommand-linesh

Shell script - Filter data from log file which having actual process count value

I have one following log file (input-data-file.log) and i want to do some filter on input file using shell script (.sh) which give me output which having actual count >0 value (Exclude 0 (zero) count from process data) along with server name in output file.

I have below input file --> input-data-file.log :

01:36:04|[user@SERVER-1 bin]$ ./script.sh
Data loading lists...
HALTED_SOFTSTOP: 0
ASYNC_QUEUED: 0
ACTIVE: 7
HALTED: 0
HALTING: 0
WAITING: 0
WAITING_ON_IO: 3
INTERRUPTED_MAN: 0
INTERRUPTED_AUTO: 0
01:36:18|[user@SERVER-1 bin]$ ./script.sh -list
Data loading lists...
HALTED_SOFTSTOP: []
ASYNC_QUEUED: []
ACTIVE: [328465405, 328460561, 328465417, 328465409, 328465411]
HALTED: []
HALTING: []
WAITING: []
WAITING_ON_IO: [328465211, 328465267, 328466202]
INTERRUPTED_MAN: []
INTERRUPTED_AUTO: []
01:36:33|[user@SERVER-1 bin]$ exit
logout
Connection to SERVER-1 closed.

01:36:43|[user@SERVER-2 bin]$ ./script.sh
Data loading lists...
HALTED_SOFTSTOP: 0
ASYNC_QUEUED: 0
ACTIVE: 1
HALTED: 4
HALTING: 0
WAITING: 0
WAITING_ON_IO: 7
INTERRUPTED_MAN: 0
INTERRUPTED_AUTO: 0
01:36:51|[user@SERVER-2 bin]$ ./script.sh -list
Data loading lists...
HALTED_SOFTSTOP: []
ASYNC_QUEUED: []
ACTIVE: [1731005633, 1731007235]
HALTED: [1729578637, 1729078919, 1729411521, 1730174742]
HALTING: []
WAITING: []
WAITING_ON_IO: [1578123006, 1578122748, 1578122627, 1578121867, 1578122892, 1578121839, 1578122653, 1578122731, 1578122789, 1578122942, 1578121834, 1578121868, 1578121795, 1578122746, 1578122644]
INTERRUPTED_MAN: []
INTERRUPTED_AUTO: []
01:36:58|[user@SERVER-2 bin]$ exit
logout
Connection to SERVER-2 closed.

01:37:49|[user@SERVER-3 bin]$ ./script.sh
Data loading lists...
HALTED_SOFTSTOP: 0
ASYNC_QUEUED: 0
ACTIVE: 0
HALTED: 0
HALTING: 0
WAITING: 0
WAITING_ON_IO: 0
INTERRUPTED_MAN: 0
INTERRUPTED_AUTO: 0
01:37:57|[user@SERVER-3 bin]$ ./script.sh -list
Data loading lists...
HALTED_SOFTSTOP: []
ASYNC_QUEUED: []
ACTIVE: []
HALTED: []
HALTING: []
WAITING: []
WAITING_ON_IO: []
INTERRUPTED_MAN: []
INTERRUPTED_AUTO: []
01:38:05|[user@SERVER-3 bin]$ exit
logout
Connection to SERVER-3 closed.

My excepted output in new file as below (i.e. output-data-file.log):

01:36:04|[user@SERVER-1 bin]$ ./script.sh
Data loading lists...
ACTIVE: 7
WAITING_ON_IO: 3
01:36:18|[user@SERVER-1 bin]$ ./script.sh -list
Data loading lists...
ACTIVE: [328465405, 328460561, 328465417, 328465409, 328465411]
WAITING_ON_IO: [328465211, 328465267, 328466202]
01:36:33|[user@SERVER-1 bin]$ exit
logout
Connection to SERVER-1 closed.

01:36:43|[user@SERVER-2 bin]$ ./script.sh
Data loading lists...
ACTIVE: 1
HALTED: 4
WAITING_ON_IO: 7
01:36:51|[user@SERVER-2 bin]$ ./script.sh -list
Data loading lists...
ACTIVE: [1731005633, 1731007235]
HALTED: [1729578637, 1729078919, 1729411521, 1730174742]
WAITING_ON_IO: [1578123006, 1578122748, 1578122627, 1578121867, 1578122892, 1578121839, 1578122653, 1578122731, 1578122789, 1578122942, 1578121834, 1578121868, 1578121795, 1578122746, 1578122644]
01:36:58|[user@SERVER-2 bin]$ exit
logout
Connection to SERVER-2 closed.

Any help would be really appreciated? Thanks in well advance!

Solution

  • You can iterate you log file for each server by putting in shell script as:

    count=1
while read -r line;
do
    echo ${line} | sed -n ${count}p | grep -E '[1-9]'
    ((count++))
done < input-data-file.log

    apart from this you can accumlate whole data in an another file and than sort it as per your needs ex:

    cat input-data-file.log |grep -E '[1-9]' > output.txt

    It will give you the following output:

    ACTIVE: 7
WAITING_ON_IO: 3
01:36:18|[user@SERVER-1 bin]$ ./script.sh -list
ACTIVE: [328465405, 328460561, 328465417, 328465409, 328465411]
WAITING_ON_IO: [328465211, 328465267, 328466202]
01:36:33|[user@SERVER-1 bin]$ exit
Connection to SERVER-1 closed.
01:36:43|[user@SERVER-2 bin]$ ./script.sh
ACTIVE: 1
HALTED: 4
WAITING_ON_IO: 7
01:36:51|[user@SERVER-2 bin]$ ./script.sh -list
ACTIVE: [1731005633, 1731007235]
HALTED: [1729578637, 1729078919, 1729411521, 1730174742]
WAITING_ON_IO: [1578123006, 1578122748, 1578122627, 1578121867, 
1578122892, 1578121839, 1578122653, 1578122731, 1578122789, 1578122942, 
1578121834, 1578121868, 1578121795, 1578122746, 1578122644]
01:36:58|[user@SERVER-2 bin]$ exit
Connection to SERVER-2 closed.
01:37:49|[user@SERVER-3 bin]$ ./script.sh
01:37:57|[user@SERVER-3 bin]$ ./script.sh -list
01:38:05|[user@SERVER-3 bin]$ exit
Connection to SERVER-3 closed.