Search code examples
pythonjsonlines

How to extract elements from a JSONL file with changing elements?


I want to extract "text" from the tokens in a JSONL file. If a label is present then I want to extract that as well. If it is not present then I want to insert "O" as a value for the label

{"text":"This is the first sentence.","_input_hash":2083129218,"_task_hash":-536378640,"spans":[],"meta":{"score":0.5,"pattern":65},"answer":"accept","tokens":[
{"text":"This","id":0},
{"text":"is","id":1},
{"text":"the","id":2},
{"text":"first","id":3},
{"text":"sentence","id":4},
{"text":".","id":5}]}
{"text":"This is coded in python.","_input_hash":2083129218,"_task_hash":-536378640,"spans":[],"meta":{"score":0.5,"pattern":65},"answer":"accept","tokens":[
{"text":"This","id":0},
{"text":"is","id":1},
{"text":"coded","id":2},
{"text":"in","id":3},
{"text":"python","label":"Programming"},
{"text":".","id":5}]}

The code that can be used to extract the text and id from the tokens is as follows if no label is present: (thanks to @DeveshKumarSingh in my previous question)

import jsonlines

#Open the file, iterate over the tokens and make the tuples
result = [(idx+1, i['text'], i['id']+1) for idx, obj in enumerate(jsonlines.open('file.txt')) for i in obj['tokens']]

print(result)

Expected output:

enter image description here


Solution

  • You can use dict.get to find the label when present, else replace it with a default value O that is i.get('label','O')

    import jsonlines
    
    #Open the file, iterate over the tokens and make the tuples
    result = [(idx+1, i['text'], i.get('label','O')) for idx, obj in enumerate(jsonlines.open('file.txt')) for i in obj['tokens']]
    
    print(result)
    

    The output will be

    [(1, 'This', 'O'),
     (1, 'is', 'O'), 
    (1, 'the', 'O'), 
    (1, 'first', 'O'), 
    (1, 'sentence', 'O'), 
    (1, '.', 'O'), 
    (2, 'This', 'O'), 
    (2, 'is', 'O'), 
    (2, 'coded', 'O'), 
    (2, 'in', 'O'), 
    (2, 'python', 'Programming'), 
    (2, '.', 'O')]