Search code examples
jsonpython-3.xpython-2.7os.walk

Python Os.walk misses few files to process in the directory

Out of 10 files in the directory, only 8 files are processed and 2 files are not processed. But if I delete all the 8 files and try running with the missed 2 files it is working. Why Os.walk is missing files? Also is there a way to process all the files in the directory one after another without missing any. Note: The solution will be used for the folder that contains 100K JSON files.

for root, dirs, files in os.walk('D:/M'):
     for file in files:
        if file.endswith(".json"):
            Strfil=os.path.join(root,file)
            with open(Strfil, 'r') as json_file:
Solution

  • For file system related things it is better to use the pathlib module

    With pathlib you can do something like this. 

    from pathlib import Path

json_files = list(Path("D:/M").glob("**/*.json"))
for f in json_files:
    with open(f, 'r') as json_file: