I have the output of a LexisNexis batch download of news articles in both html and txt format. The file itself contains the headers, metadata, and body of several different news articles that I need to systematically separate and save as independent txt files. The head of the txt version looks like:
> head(textz, 100)
[1] ""
[2] " 1 of 103 DOCUMENTS"
[3] ""
[4] ""
[5] " Foreign Affairs"
[6] ""
[7] " May 2013 - June 2013"
[8] ""
[9] "Why the U.S. Army Needs Armor Subtitle: The Case for a Balanced Force"
[10] ""
[11] "BYLINE: Chris McKinney, Mark Elfendahl, and H. R. McMaster Authors BIOS: CHRIS"
[12] "MCKINNEY is a Lieutenant Colonel in the U.S. Army and an adviser to the Saudi"
[13] "Arabian National Guard. MARK ELFENDAHL is a Colonel in the U.S. Army and a"
[14] "student at the Joint Advanced Warfighting School in Norfolk, Virginia. H. R."
[15] "MCMASTER is a Major General in the U.S. Army and Commander of the Maneuver"
[16] "Center of Excellence at Fort Benning, Georgia."
[17] ""
[18] "SECTION: Vol. 92 No. 4 PAGE: 129"
[19] ""
[20] "LENGTH: 2856 words"
[21] ""
[22] ""
[23] "Ever since World War II, the United States has depended on armored forces --"
[24] "forces equipped with tanks and other protected vehicles -- to wage its wars."
....
....
A snapshot of the html version looks like this:
<DOC NUMBER=103>
<DOCFULL> -->
<br><div class="c0">
<p class="c1"><span class="c2">103 of 103 DOCUMENTS</span></p>
</div>
<br><div class="c0">
<br><p class="c1"><span class="c2">The New York Times</span></p>
</div>
<br><div class="c3">
<p class="c1"><span class="c4">July</span>
<span class="c2"> 26, 2011 Tuesday</span>
<span class="c2">Â </span>
<span class="c2">Â <br>Late Edition - Final</span></p>
</div>
<br><div class="c5">
<p class="c6"><span class="c7">A Step Toward Trust With China</span></p>
</div>
<br><div class="c5">
<p class="c6"><span class="c8">BYLINE: </span><span class="c2">By MIKE MULLEN. </span></p>
<p class="c9"><span class="c2">Mike Mullen, a </span>
<span class="c4">Navy admiral,</span><span class="c2"> is the chairman of the Joint Chiefs of Staff.
</span></p>
</div>
<br><div class="c5">
<p class="c6"><span class="c8">SECTION: </span>
<span class="c2">Section A; Column 0; Editorial Desk; OP-ED CONTRIBUTOR; Pg. 23</span></p>
</div>
<br><div class="c5">
<p class="c6"><span class="c8">LENGTH: </span>
<span class="c2">794 words</span></p>
</div>
<br><div class="c5">
<p class="c9"><span class="c2">Washington</span></p>
<p class="c9"><span class="c2">THE military relationship between the United States and China is one of the world's most important. And yet, clouded by some misunderstanding and suspicion, it remains among the most challenging. There are issues on which we disagree and are tempted to confront each other. But there are crucial areas where our interests coincide, on which we must work together.
</span></p>
The unique documents are separated by the "[0-9] of [0-9] DOCUMENTS" lines in each, but between the grep family and strsplit I have been unable to find a way to split the txt (or html) file in R in a way that cleanly separates the component articles and allows me to save them as independent txt files. A thorough search of other questions threads was either unhelpful or required use of Python. Any advice would be great!
To split the txt version, assuming text is in doc_text, and write each to a sequentially named files .txt, file2.txt etc.
lapply for file writing adapted from @P Lapointe
texts <- unlist(strsplit(doc_text, "\\s+\\d+\\sof\\s\\d+\\sDOCUMENTS") )
texts <- texts[-1] # drop the first empty split
lapply (1:length(texts), function(i){ write(texts[i], paste0("file", i, ".txt"))})