I'm working with a text file, which consists of many similar reports of the following structure:
['NetNGlyc-1.0 Server Output - DTU Health Tech\n',
' Asn-Xaa-Ser/Thr sequons in the sequence output below are highlighted in blue.\n',
' Asparagines predicted to be N-glycosylated are highlighted in red.\n',
"Output for 'Sequence'\n",
'Name: Sequence Length: 923\n',
'MERGLPLLCAVLALVLAPAGAFRNDKCGDTIKIESPGYLTSPGYPHSYHPSEKCEWLIQAPDPYQRIMINFNPHFDLEDR 80 \n',
'DCKYDYVEVFDGENENGHFRGKFCGKIAPPPVVSSGPFLFIKFVSDYETHGAGFSIRYEIFKRGPECSQNYTTPSGVIKS 160 \n',
'PGFPEKYPNSLECTYIVFVPKMSEIILEFESFDLEPDSNPPGGMFCRYDRLEIWDGFPDVGPHIGRYCGQKTPGRIRSSS 240 \n',
'GILSMVFYTDSAIAKEGFSANYSVLQSSVSEDFKCMEALGMESGEIHSDQITASSQYSTNWSAERSRLNYPENGWTPGED 320 \n',
'SYREWIQVDLGLLRFVTAVGTQGAISKETKKKYYVKTYKIDVSSNGEDWITIKEGNKPVLFQGNTNPTDVVVAVFPKPLI 400 \n',
'TRFVRIKPATWETGISMRFEVYGCKITDYPCSGMLGMVSGLISDSQITSSNQGDRNWMPENIRLVTSRSGWALPPAPHSY 480 \n',
'INEWLQIDLGEEKIVRGIIIQGGKHRENKVFMRKFKIGYSNNGSDWKMIMDDSKRKAKSFEGNNNYDTPELRTFPALSTR 560 \n',
'FIRIYPERATHGGLGLRMELLGCEVEAPTAGPTTPNGNLVDECDDDQANCHSGTGDDFQLTGGTTVLATEKPTVIDSTIQ 640 \n',
'SEFPTYGFNCEFGWGSHKTFCHWEHDNHVQLKWSVLTSKTGPIQDHTGDGNFIYSQADENQKGKVARLVSPVVYSQNSAH 720 \n',
'CMTFWYHMSGSHVGTLRVKLRYQKPEEYDQLVWMAIGHQGDHWKEGRVLLHKSLKLYQVIFEGEIGKGNLGGIAVDDISI 800 \n',
'NNHISQEDCAKPADLDKKNPEIKIDETGSTPGYEGEGEGDKNISRKPGNVLKTLDPILITIIAMSALGVLLGAVCGVVLY 880 \n',
'CACWHNGMSERNLSALENYNFELVDGVKLKKDKLNTQSTYSEA\n',
'................................................................................ 80\n',
'.....................................................................N.......... 160\n',
'................................................................................ 240\n',
'....................N........................................................... 320\n',
'.................................................................N.............. 400\n',
'................................................................................ 480\n',
'................................................................................ 560\n',
'................................................................................ 640\n',
'................................................................................ 720\n',
'................................................................................ 800\n',
'................................................................................ 880\n',
'........................................... 960\n',
'\n',
'(Threshold=0.5)\n',
'----------------------------------------------------------------------\n',
'SeqName Position Potential Jury N-Glyc\n',
' agreement result\n',
'----------------------------------------------------------------------\n',
'Sequence 150 NYTT 0.5361 (5/9) + \n',
'Sequence 261 NYSV 0.5599 (6/9) + \n',
'Sequence 300 NWSA 0.4157 (6/9) - \n',
'Sequence 386 NPTD 0.7736 (9/9) +++ WARNING: PRO-X1. \n',
'Sequence 522 NGSD 0.3918 (9/9) -- \n',
'Sequence 842 NISR 0.4662 (6/9) - \n',
'Sequence 892 NLSA 0.4099 (6/9) - \n',
'----------------------------------------------------------------------\n',
'\n',
'\n',
'Graphics in PostScript\n',
'\n',
'\n',
'Go back.\n']
The resulting file that I'm trying to get is a list of elements, where each element would be a string, containing only the info that I want to be left. The final list structure that I'm trying to get is something like that:
['Sequence 150 NYTT 0.5361 (5/9) + \n
Sequence 261 NYSV 0.5599 (6/9) + \n
Sequence 300 NWSA 0.4157 (6/9) - \n',
'Sequence 150 NYTT 0.5361 (5/9) + \n
Sequence 261 NYSV 0.5599 (6/9) + \n
Sequence 300 NWSA 0.4157 (6/9) - \n
Sequence 466 NYSV 0.6178 (7/9) + \n
Sequence 300 NWSA 0.4157 (6/9) - \n',
'Sequence 150 NYTT 0.5361 (5/9) + \n
Sequence 261 NYSV 0.5599 (6/9) + \n
Sequence 300 NWSA 0.4157 (6/9) - \n',
...]
I managed to separate the outputs with the following code:
import re
with open('/path_to_text_file/file.txt', 'r') as file:
test_output = file.readlines()
test_string = ''.join(map(str, test_output)) # convert the list into string
# here I decided to split the outputs by 'Go back' substring
# 1. first split by "\n\n" preceeding the 'Go back' substring
# 2. then by ".\n" following the 'Go back'
# 3. then by "\n" left
test_string_split = ' '.join(map(str, ' '.join(map(str, test_string.split('\n\n'))).split('.\n')))
# split into element by *'Go back'* substring
processed_test = ''.join(test_string_split).split('Go back')
Now what I have in my hands is a list of elements, where each element comprises a single output. But I haven't managed yet to strip this outputs of all unnecessary text preserving the structure of the list, where each element came from a single report. I tried the following logic:
res = [] # create a list for the final result
# split each output in the text file by '\n'
for output in processed_test:
output_split = ''.join(output).split('\n')
# then check each line of the output for the 'Sequence' substring
for string in output_split:
string_el = ''.join(string)
if re.match("Sequence.*", string_el):
res.append(string_el) # save matches to the resulting list
But what I get is a list of elements, where each element comprises a separate "Sequence"-line:
['Sequence 522 NGSD 0.3918 (9/9) -- ',
'Sequence 842 NISR 0.4662 (6/9) - ',
'Sequence 892 NLSA 0.4099 (6/9) - ',
'Sequence 63 NYTV 0.7796 (9/9) +++ ',
'Sequence 209 NITL 0.7032 (8/9) + ',
'Sequence 297 NVSI 0.6256 (8/9) + ',
'Sequence 365 NLSQ 0.6403 (7/9) + ',
'Sequence 522 NTSH 0.5207 (6/9) + ',
'Sequence 696 NCSI 0.6619 (9/9) ++ ',
...
...
...]
Is there a way of parsing a list inside the elements themselves so as to preserve the structure of the list? The idea is that I need to understand from which report comes the info on the sequences.
IIUC you wanat to do the following:
That can be done as follows:
Code
import ast
import os
def make_reports(file_path):
with open(file_path, 'r') as f:
stack = [[]] # start with 1st report empty
# Convert string into Python list
lines = ast.literal_eval(f.read())
for line in lines:
# Loop through all lines in list
if line.startswith('Sequence'):
# Append Sequence to current group
stack[-1].append(line)
elif line.startswith('Go back'):
stack.append([]) # Start new report
# Convert to a dataframe, with each Report enumeratd (i.e. 0, 1, 2, ...)
dfs = []
for i, seqs in enumerate(stack):
if seqs:
# TWo column dataframe: Sequence and Report number
dfs.append(pd.DataFrame({f'Sequences':seqs, 'Report':[i]*len(seqs)}))
result = pd.concat(dfs, ignore_index=True, sort=False)
# Write to results file (uses input file path and append -result to name)
result.to_csv(f'{os.path.splitext(file_path)[0]}-result.txt',
encoding='utf-8',
index=False)
return result
Usage
make_reports('test.txt')
Input File: test.txt
Obtained by replicating posted data two more times to obtain multiple reports
['NetNGlyc-1.0 Server Output - DTU Health Tech\n',
' Asn-Xaa-Ser/Thr sequons in the sequence output below are highlighted in blue.\n',
' Asparagines predicted to be N-glycosylated are highlighted in red.\n',
"Output for 'Sequence'\n",
'Name: Sequence Length: 923\n',
'MERGLPLLCAVLALVLAPAGAFRNDKCGDTIKIESPGYLTSPGYPHSYHPSEKCEWLIQAPDPYQRIMINFNPHFDLEDR 80 \n',
'DCKYDYVEVFDGENENGHFRGKFCGKIAPPPVVSSGPFLFIKFVSDYETHGAGFSIRYEIFKRGPECSQNYTTPSGVIKS 160 \n',
'PGFPEKYPNSLECTYIVFVPKMSEIILEFESFDLEPDSNPPGGMFCRYDRLEIWDGFPDVGPHIGRYCGQKTPGRIRSSS 240 \n',
'GILSMVFYTDSAIAKEGFSANYSVLQSSVSEDFKCMEALGMESGEIHSDQITASSQYSTNWSAERSRLNYPENGWTPGED 320 \n',
'SYREWIQVDLGLLRFVTAVGTQGAISKETKKKYYVKTYKIDVSSNGEDWITIKEGNKPVLFQGNTNPTDVVVAVFPKPLI 400 \n',
'TRFVRIKPATWETGISMRFEVYGCKITDYPCSGMLGMVSGLISDSQITSSNQGDRNWMPENIRLVTSRSGWALPPAPHSY 480 \n',
'INEWLQIDLGEEKIVRGIIIQGGKHRENKVFMRKFKIGYSNNGSDWKMIMDDSKRKAKSFEGNNNYDTPELRTFPALSTR 560 \n',
'FIRIYPERATHGGLGLRMELLGCEVEAPTAGPTTPNGNLVDECDDDQANCHSGTGDDFQLTGGTTVLATEKPTVIDSTIQ 640 \n',
'SEFPTYGFNCEFGWGSHKTFCHWEHDNHVQLKWSVLTSKTGPIQDHTGDGNFIYSQADENQKGKVARLVSPVVYSQNSAH 720 \n',
'CMTFWYHMSGSHVGTLRVKLRYQKPEEYDQLVWMAIGHQGDHWKEGRVLLHKSLKLYQVIFEGEIGKGNLGGIAVDDISI 800 \n',
'NNHISQEDCAKPADLDKKNPEIKIDETGSTPGYEGEGEGDKNISRKPGNVLKTLDPILITIIAMSALGVLLGAVCGVVLY 880 \n',
'CACWHNGMSERNLSALENYNFELVDGVKLKKDKLNTQSTYSEA\n',
'................................................................................ 80\n',
'.....................................................................N.......... 160\n',
'................................................................................ 240\n',
'....................N........................................................... 320\n',
'.................................................................N.............. 400\n',
'................................................................................ 480\n',
'................................................................................ 560\n',
'................................................................................ 640\n',
'................................................................................ 720\n',
'................................................................................ 800\n',
'................................................................................ 880\n',
'........................................... 960\n',
'\n',
'(Threshold=0.5)\n',
'----------------------------------------------------------------------\n',
'SeqName Position Potential Jury N-Glyc\n',
' agreement result\n',
'----------------------------------------------------------------------\n',
'Sequence 150 NYTT 0.5361 (5/9) + \n',
'Sequence 261 NYSV 0.5599 (6/9) + \n',
'Sequence 300 NWSA 0.4157 (6/9) - \n',
'Sequence 386 NPTD 0.7736 (9/9) +++ WARNING: PRO-X1. \n',
'Sequence 522 NGSD 0.3918 (9/9) -- \n',
'Sequence 842 NISR 0.4662 (6/9) - \n',
'Sequence 892 NLSA 0.4099 (6/9) - \n',
'----------------------------------------------------------------------\n',
'\n',
'\n',
'Graphics in PostScript\n',
'\n',
'\n',
'Go back.\n',
'NetNGlyc-1.0 Server Output - DTU Health Tech\n',
' Asn-Xaa-Ser/Thr sequons in the sequence output below are highlighted in blue.\n',
' Asparagines predicted to be N-glycosylated are highlighted in red.\n',
"Output for 'Sequence'\n",
'Name: Sequence Length: 923\n',
'MERGLPLLCAVLALVLAPAGAFRNDKCGDTIKIESPGYLTSPGYPHSYHPSEKCEWLIQAPDPYQRIMINFNPHFDLEDR 80 \n',
'DCKYDYVEVFDGENENGHFRGKFCGKIAPPPVVSSGPFLFIKFVSDYETHGAGFSIRYEIFKRGPECSQNYTTPSGVIKS 160 \n',
'PGFPEKYPNSLECTYIVFVPKMSEIILEFESFDLEPDSNPPGGMFCRYDRLEIWDGFPDVGPHIGRYCGQKTPGRIRSSS 240 \n',
'GILSMVFYTDSAIAKEGFSANYSVLQSSVSEDFKCMEALGMESGEIHSDQITASSQYSTNWSAERSRLNYPENGWTPGED 320 \n',
'SYREWIQVDLGLLRFVTAVGTQGAISKETKKKYYVKTYKIDVSSNGEDWITIKEGNKPVLFQGNTNPTDVVVAVFPKPLI 400 \n',
'TRFVRIKPATWETGISMRFEVYGCKITDYPCSGMLGMVSGLISDSQITSSNQGDRNWMPENIRLVTSRSGWALPPAPHSY 480 \n',
'INEWLQIDLGEEKIVRGIIIQGGKHRENKVFMRKFKIGYSNNGSDWKMIMDDSKRKAKSFEGNNNYDTPELRTFPALSTR 560 \n',
'FIRIYPERATHGGLGLRMELLGCEVEAPTAGPTTPNGNLVDECDDDQANCHSGTGDDFQLTGGTTVLATEKPTVIDSTIQ 640 \n',
'SEFPTYGFNCEFGWGSHKTFCHWEHDNHVQLKWSVLTSKTGPIQDHTGDGNFIYSQADENQKGKVARLVSPVVYSQNSAH 720 \n',
'CMTFWYHMSGSHVGTLRVKLRYQKPEEYDQLVWMAIGHQGDHWKEGRVLLHKSLKLYQVIFEGEIGKGNLGGIAVDDISI 800 \n',
'NNHISQEDCAKPADLDKKNPEIKIDETGSTPGYEGEGEGDKNISRKPGNVLKTLDPILITIIAMSALGVLLGAVCGVVLY 880 \n',
'CACWHNGMSERNLSALENYNFELVDGVKLKKDKLNTQSTYSEA\n',
'................................................................................ 80\n',
'.....................................................................N.......... 160\n',
'................................................................................ 240\n',
'....................N........................................................... 320\n',
'.................................................................N.............. 400\n',
'................................................................................ 480\n',
'................................................................................ 560\n',
'................................................................................ 640\n',
'................................................................................ 720\n',
'................................................................................ 800\n',
'................................................................................ 880\n',
'........................................... 960\n',
'\n',
'(Threshold=0.5)\n',
'----------------------------------------------------------------------\n',
'SeqName Position Potential Jury N-Glyc\n',
' agreement result\n',
'----------------------------------------------------------------------\n',
'Sequence 150 NYTT 0.5361 (5/9) + \n',
'Sequence 261 NYSV 0.5599 (6/9) + \n',
'Sequence 300 NWSA 0.4157 (6/9) - \n',
'Sequence 386 NPTD 0.7736 (9/9) +++ WARNING: PRO-X1. \n',
'Sequence 522 NGSD 0.3918 (9/9) -- \n',
'Sequence 842 NISR 0.4662 (6/9) - \n',
'Sequence 892 NLSA 0.4099 (6/9) - \n',
'----------------------------------------------------------------------\n',
'\n',
'\n',
'Graphics in PostScript\n',
'\n',
'\n',
'Go back.\n',
'NetNGlyc-1.0 Server Output - DTU Health Tech\n',
' Asn-Xaa-Ser/Thr sequons in the sequence output below are highlighted in blue.\n',
' Asparagines predicted to be N-glycosylated are highlighted in red.\n',
"Output for 'Sequence'\n",
'Name: Sequence Length: 923\n',
'MERGLPLLCAVLALVLAPAGAFRNDKCGDTIKIESPGYLTSPGYPHSYHPSEKCEWLIQAPDPYQRIMINFNPHFDLEDR 80 \n',
'DCKYDYVEVFDGENENGHFRGKFCGKIAPPPVVSSGPFLFIKFVSDYETHGAGFSIRYEIFKRGPECSQNYTTPSGVIKS 160 \n',
'PGFPEKYPNSLECTYIVFVPKMSEIILEFESFDLEPDSNPPGGMFCRYDRLEIWDGFPDVGPHIGRYCGQKTPGRIRSSS 240 \n',
'GILSMVFYTDSAIAKEGFSANYSVLQSSVSEDFKCMEALGMESGEIHSDQITASSQYSTNWSAERSRLNYPENGWTPGED 320 \n',
'SYREWIQVDLGLLRFVTAVGTQGAISKETKKKYYVKTYKIDVSSNGEDWITIKEGNKPVLFQGNTNPTDVVVAVFPKPLI 400 \n',
'TRFVRIKPATWETGISMRFEVYGCKITDYPCSGMLGMVSGLISDSQITSSNQGDRNWMPENIRLVTSRSGWALPPAPHSY 480 \n',
'INEWLQIDLGEEKIVRGIIIQGGKHRENKVFMRKFKIGYSNNGSDWKMIMDDSKRKAKSFEGNNNYDTPELRTFPALSTR 560 \n',
'FIRIYPERATHGGLGLRMELLGCEVEAPTAGPTTPNGNLVDECDDDQANCHSGTGDDFQLTGGTTVLATEKPTVIDSTIQ 640 \n',
'SEFPTYGFNCEFGWGSHKTFCHWEHDNHVQLKWSVLTSKTGPIQDHTGDGNFIYSQADENQKGKVARLVSPVVYSQNSAH 720 \n',
'CMTFWYHMSGSHVGTLRVKLRYQKPEEYDQLVWMAIGHQGDHWKEGRVLLHKSLKLYQVIFEGEIGKGNLGGIAVDDISI 800 \n',
'NNHISQEDCAKPADLDKKNPEIKIDETGSTPGYEGEGEGDKNISRKPGNVLKTLDPILITIIAMSALGVLLGAVCGVVLY 880 \n',
'CACWHNGMSERNLSALENYNFELVDGVKLKKDKLNTQSTYSEA\n',
'................................................................................ 80\n',
'.....................................................................N.......... 160\n',
'................................................................................ 240\n',
'....................N........................................................... 320\n',
'.................................................................N.............. 400\n',
'................................................................................ 480\n',
'................................................................................ 560\n',
'................................................................................ 640\n',
'................................................................................ 720\n',
'................................................................................ 800\n',
'................................................................................ 880\n',
'........................................... 960\n',
'\n',
'(Threshold=0.5)\n',
'----------------------------------------------------------------------\n',
'SeqName Position Potential Jury N-Glyc\n',
' agreement result\n',
'----------------------------------------------------------------------\n',
'Sequence 150 NYTT 0.5361 (5/9) + \n',
'Sequence 261 NYSV 0.5599 (6/9) + \n',
'Sequence 300 NWSA 0.4157 (6/9) - \n',
'Sequence 386 NPTD 0.7736 (9/9) +++ WARNING: PRO-X1. \n',
'Sequence 522 NGSD 0.3918 (9/9) -- \n',
'Sequence 842 NISR 0.4662 (6/9) - \n',
'Sequence 892 NLSA 0.4099 (6/9) - \n',
'----------------------------------------------------------------------\n',
'\n',
'\n',
'Graphics in PostScript\n',
'\n',
'\n',
'Go back.\n']
Output File: test-results.txt
Columns are Sequences, Report (for report index)
Sequences,Report
"Sequence 150 NYTT 0.5361 (5/9) +
",0
"Sequence 261 NYSV 0.5599 (6/9) +
",0
"Sequence 300 NWSA 0.4157 (6/9) -
",0
"Sequence 386 NPTD 0.7736 (9/9) +++ WARNING: PRO-X1.
",0
"Sequence 522 NGSD 0.3918 (9/9) --
",0
"Sequence 842 NISR 0.4662 (6/9) -
",0
"Sequence 892 NLSA 0.4099 (6/9) -
",0
"Sequence 150 NYTT 0.5361 (5/9) +
",1
"Sequence 261 NYSV 0.5599 (6/9) +
",1
"Sequence 300 NWSA 0.4157 (6/9) -
",1
"Sequence 386 NPTD 0.7736 (9/9) +++ WARNING: PRO-X1.
",1
"Sequence 522 NGSD 0.3918 (9/9) --
",1
"Sequence 842 NISR 0.4662 (6/9) -
",1
"Sequence 892 NLSA 0.4099 (6/9) -
",1
"Sequence 150 NYTT 0.5361 (5/9) +
",2
"Sequence 261 NYSV 0.5599 (6/9) +
",2
"Sequence 300 NWSA 0.4157 (6/9) -
",2
"Sequence 386 NPTD 0.7736 (9/9) +++ WARNING: PRO-X1.
",2
"Sequence 522 NGSD 0.3918 (9/9) --
",2
"Sequence 842 NISR 0.4662 (6/9) -
",2
"Sequence 892 NLSA 0.4099 (6/9) -
",2