I'm trying to figure out how to find lines containing duplicate data in a file. here I load a single line and break it down to original variables and then search the same data file for each variable separately, listing the lines that match to a new file for further processing.
@echo off
:: reset temp files
set "db=%temp%\lbdb"
del "%db%"
copy /y nul "%db%" >nul
set "dupes=%temp%\dupes"
del "%dupes%"
copy /y nul "%dupes%" >nul
:: rebuild database (full,dura,size,strict,id)
:: maybe add option to keep database if not too old?
setlocal enabledelayedexpansion
for /f "delims=" %%v in ('dir "d:\videos" /b /s /a-d 2^>nul') do @(
for /f "tokens=2 delims=," %%i in ('ffprobe -v quiet -show_entries "format=duration" -of csv "%%v"') do @(
@set "full=%%v" & @call reader >nul
echo %%v/%%i/%%~zv/!strict!/!id!
)
)>>"%db%"
:: for each line in database, read variables and call for processing
for /f "tokens=1,2,3,4,5 delims=/" %%a in ('sort "%db%"') do if exist "%%a" call :checks "%%a" "%%b" "%%c" "%%d" "%%e"
exit /b
:checks
:: full is unique, compare %dura%, %size%, %strict% and %id% separately
set "full=%~1" & set "dura=%~2" & set "size=%~3" & set "strict=%~4" & set "id=%~5"
:: find each type of match and list possibly same videos
:: how to exclude the current loaded line from results?
:: findstr /c:"/%dura%/" "%db%" >>"%dupes%"
:: findstr /c:"/%size%/" "%db%" >>"%dupes%"
:: findstr /c:"/%strict%/" "%db%" >>"%dupes%"
findstr /c:"/%id%" "%db%" >>"%dupes%"
exit /b
problem being that each "loaded" line (being used to do the searching) will obviously find itself in that file. if only a single line contains a match that should be ignored, and doubles or more should be collected so they can be processed. must I remove the whole line from the file when loading that line? I'm hoping for a speedier solution.
edit:
example %db% (looking good)
d:\videos\series\comedy\classic\shit my dad says\Shit My Dad Says S01E01 Pilot [Comedy] r10.0 720p x264 AC3 tt1612578.mkv/1272.772000/586337002/SHIT?MY?DAD?SAYS?S01E01/showShitMyDadSaysS01E01
d:\videos\series\comedy\classic\shit my dad says\Shit My Dad Says S01E02 Wi-Fight [Comedy] r6.6 720p x264 AC3 tt1612578.mkv/1274.400000/586061951/SHIT?MY?DAD?SAYS?S01E02/showShitMyDadSaysS01E02
d:\videos\series\comedy\classic\shit my dad says\Shit My Dad Says S01E03 The Truth About Dads and Moms [Comedy] r6.6 720p x264 AC3 tt1612578.mkv/1230.000000/587071468/SHIT?MY?DAD?SAYS?S01E03/showShitMyDadSaysS01E03
d:\videos\series\comedy\classic\shit my dad says\Shit My Dad Says S01E04 Code Ed [Comedy] r6.6 720p x264 AC3 tt1612578.mkv/1243.000000/587071468/SHIT?MY?DAD?SAYS?S01E04/showShitMyDadSaysS01E04
d:\videos\series\comedy\classic\shit my dad says\Shit My Dad Says S01E04.mkv/1243.000000/587071468/SHIT?MY?DAD?SAYS?S01E04/showShitMyDadSaysS01E04
what I want the resulting %dupes% to be:
d:\videos\series\comedy\classic\shit my dad says\Shit My Dad Says S01E04 Code Ed [Comedy] r6.6 720p x264 AC3 tt1612578.mkv/1243.000000/587071468/SHIT?MY?DAD?SAYS?S01E04/showShitMyDadSaysS01E04
d:\videos\series\comedy\classic\shit my dad says\Shit My Dad Says S01E04.mkv/1243.000000/587071468/SHIT?MY?DAD?SAYS?S01E04/showShitMyDadSaysS01E04
and finally, what I get because of the data finding itself in the %db%. note that I have commented in what search was used for the result. the comments do not actually appear in the file
:: searching showShitMyDadSaysS01E01
d:\videos\series\comedy\classic\shit my dad says\Shit My Dad Says S01E01 Pilot [Comedy] r10.0 720p x264 AC3 tt1612578.mkv/1272.772000/586337002/SHIT?MY?DAD?SAYS?S01E01/showShitMyDadSaysS01E01
:: searching showShitMyDadSaysS01E02
d:\videos\series\comedy\classic\shit my dad says\Shit My Dad Says S01E02 Wi-Fight [Comedy] r6.6 720p x264 AC3 tt1612578.mkv/1274.400000/586061951/SHIT?MY?DAD?SAYS?S01E02/showShitMyDadSaysS01E02
:: searching showShitMyDadSaysS01E03
d:\videos\series\comedy\classic\shit my dad says\Shit My Dad Says S01E03 The Truth About Dads and Moms [Comedy] r6.6 720p x264 AC3 tt1612578.mkv/1230.000000/587071468/SHIT?MY?DAD?SAYS?S01E03/showShitMyDadSaysS01E03
:: searching showShitMyDadSaysS01E04
d:\videos\series\comedy\classic\shit my dad says\Shit My Dad Says S01E04 Code Ed [Comedy] r6.6 720p x264 AC3 tt1612578.mkv/1243.000000/587071468/SHIT?MY?DAD?SAYS?S01E04/showShitMyDadSaysS01E04
d:\videos\series\comedy\classic\shit my dad says\Shit My Dad Says S01E04.mkv/1243.000000/587071468/SHIT?MY?DAD?SAYS?S01E04/showShitMyDadSaysS01E04
phew, that was hard to explain
I finally figured this out. one of those moments where the simple solution was too simple to notice. this goes through 27.000 files in a breeze, only taking 1h 5min where the old version I deleted took ~10 hours to complete. and best of all, this retains the relevant data so the weekly scan only takes ~20 minutes.
@echo off
echo %time:~0,2%:%time:~3,2% updating video information...
setLocal enableDelayedExpansion
for /f "delims=" %%v in ('dir "d:\videos" /b /s /a-d 2^>nul') do (
if not exist "c:%%~pv" md "c:%%~pv" && attrib +h "c:\videos && echo first time takes a while (~400 videos/minute), please be patient...
if not exist "c:%%~pnxv" for /f "tokens=2 delims=," %%i in ('ffprobe -v quiet -show_entries "format=duration" -of csv "%%v"') do (
set "full=%%v" & call reader >nul
echo %%v/%%i/%%~zv/!id!/!strict!>"c:%%~pnxv"
)
)
echo %time:~0,2%:%time:~3,2% gathering database...
del "%temp%\db" 2>nul
for /f "delims=" %%v in ('dir "c:\videos" /b /s /a-d 2^>nul') do (
rem if video is missing remove the data
if not exist "d:%%~pnxv" del "c:%%~pnxv"
rem is it too much to load the line here and check the fast properties to make sure it's valid?
if exist "c:%%~pnxv" type "c:%%~pnxv" >>"%temp%\db"
)
echo %time:~0,2%:%time:~3,2% comparing by identity and size+duration data...
del "%temp%\dupes" 2>nul
:: for each line in database, read variables and call for processing. sorting is useless, is it safe tor emove?
for /f "tokens=1,2,3,4,5 delims=/" %%a in ("%temp%\db") do if exist "%%a" call :compare "%%a" "%%b" "%%c" "%%d" "%%e"
:: if we have collected dupes list, execute the job for the dupe files
if exist "%temp%\dupes" (echo %time:~0,2%:%time:~3,2% moving aside...) else (echo %time:~0,2%:%time:~3,2% no duplicates found.)
if exist "%temp%\dupes" for /f "tokens=1,2,3,4,5 delims=/" %%a in ("%temp%\dupes") do if exist "%%a" call :execute "%%a"
if exist "%temp%\dupes" echo %time:~0,2%:%time:~3,2% finished.
exit /b
:compare
:: full is unique, compare %dura%, %size%, %strict% and %id%
set "full=%~1" & set "dura=%~2" & set "size=%~3" & set "id=%~4" & set "strict=%~5"
if "%dura%"=="" echo duration missing %1 %2 %3 %4 %5 & pause
if "%id%"=="" echo id missing %1 %2 %3 %4 %5 & pause
:: find all lines matching properties (incl. self line) w/temp storage
findstr /c:"%id%" "%temp%\db" >"%temp%\analysis"
:: check for both identical duration and size. %size% can be shortened for some flexibility
findstr /c:"/%dura%/%size%/" "%temp%\db" >>"%temp%\analysis"
:: collect all lines except the one containing the current file
findstr /v /c:"%full%" "%temp%\analysis" >>"%temp%\dupes"
exit /b
:execute
:: move to wherever user wants them
set "full=%~1"
call report %~n0 locate duplicates
exit /b