I have a client with thousands of audio/video files that they stream internally, all are segmented (.ts) and saved in an S3 bucket with appropriate metadata in an SQL database. Now they've asked me to create two "live" streams, one for audio, one for video that they can set and forget.
Not wanting to re-segment everything or concatenate all the files I'm trying to hack in a "live" m3u8 that slides through the already existing files (they're all encoded the exact same way).
What I've done is generate a "radio playlist" that is saved into a database at 40-second intervals (x3 .ts per m3u8), each tagged with a start and end time and appropriate EXT-X-MEDIA-SEQUENCE. Then I select between NOW() and push the file.
It works but sometimes the timing is right and it hits the same grouping for the first and last file and buffers out. I have complete control over the player (VideoJS) and server to get this working.
This is the code I have so far...any way I could make this work? I haven't tried playing with buffers on vJS yet (don't know how to...)
All the basic file information is stored in the database like this
INSERT INTO `contenido_audio_hls` (`id`, `audio_s`, `duration`) VALUES ('f2z7dcwc0l7rleig', '["10.000000","10.000000","10.000000","10.000000","10.000000","10.000000","10.000000","10.000000","10.000000","10.000000","10.000000","10.000000","10.000000","10.000000","10.000000","10.000000","10.000000","10.000000","10.000000","4.100000"]', 10);
When a playlist is generated, I pull out the required data
$radio = sql("SELECT lista_contenido.orden,lista_contenido.contenido,contenido_audio_hls.audio_s FROM lista_listas LEFT JOIN lista_contenido ON (lista_listas.id = lista_contenido.lista) LEFT JOIN contenido_audio_hls ON (lista_contenido.contenido = contenido_audio_hls.id) WHERE (lista_listas.tipo = 'radio') ORDER BY lista_contenido.orden ASC");
foreach($radio['data'] as $k=>$v) {
$arreglo = json_decode($v['audio_s'],TRUE);
foreach($arreglo as $kk=>$vv) {
$puro[] = array("extinf"=>'#EXTINF:'.$vv.',',"id"=>$v['contenido'],"segment"=>$kk);
}
}
I loop through them to create groups
$segundos = 0;
$grupo = 1;
$contador = 1;
foreach($puro as $k=>$v) {
if($segundos <= 30) {
$m3u8[$grupo][] = $puro[$k];
$contador++;
} else {
$m3u8[$grupo][] = $puro[$k];
$grupo = $grupo + $contador;
$segundos = 0;
}
$segundos = $segundos + 10;
}
Then put them into their own table
$largo = 0;
foreach($m3u8 as $k=>$v) {
$ini = sprintf('%02d:%02d:%02d',($largo/3600),($largo/60%60),$largo%60);
$localfin = $largo + 40;
$fin = sprintf('%02d:%02d:%02d',($localfin/3600),($localfin/60%60),$localfin%60);
$query = "INSERT INTO lista_m3u8 (ini,fin,tipo,sequence,data) VALUES('".$ini."','".$fin."','radio','".$k."','".json_encode($v)."')";
sql($query);
$largo = $largo + 40;
}
Which gives me this
INSERT INTO `lista_m3u8` (`ini`, `fin`, `tipo`, `sequence`, `data`) VALUES ('06:54:00', '06:54:40', 'radio', 580636, '[{"extinf":"#EXTINF:10.000000,","id":"f2z7de0quwgehw23","segment":14},{"extinf":"#EXTINF:10.000000,","id":"f2z7de0quwgehw23","segment":15},{"extinf":"#EXTINF:10.000000,","id":"f2z7de0quwgehw23","segment":16},{"extinf":"#EXTINF:10.000000,","id":"f2z7de0quwgehw23","segment":17}]');
INSERT INTO `lista_m3u8` (`ini`, `fin`, `tipo`, `sequence`, `data`) VALUES ('06:54:40', '06:55:20', 'radio', 582504, '[{"extinf":"#EXTINF:10.000000,","id":"f2z7de0quwgehw23","segment":18},{"extinf":"#EXTINF:10.000000,","id":"f2z7de0quwgehw23","segment":19},{"extinf":"#EXTINF:0.766667,","id":"f2z7de0quwgehw23","segment":20},{"extinf":"#EXTINF:10.000000,","id":"f2z7dft8c217xyp","segment":0}]');
INSERT INTO `lista_m3u8` (`ini`, `fin`, `tipo`, `sequence`, `data`) VALUES ('06:55:20', '06:56:00', 'radio', 584375, '[{"extinf":"#EXTINF:10.000000,","id":"f2z7dft8c217xyp","segment":1},{"extinf":"#EXTINF:10.000000,","id":"f2z7dft8c217xyp","segment":2},{"extinf":"#EXTINF:10.000000,","id":"f2z7dft8c217xyp","segment":3},{"extinf":"#EXTINF:10.000000,","id":"f2z7dft8c217xyp","segment":4}]');
INSERT INTO `lista_m3u8` (`ini`, `fin`, `tipo`, `sequence`, `data`) VALUES ('06:56:00', '06:56:40', 'radio', 586249, '[{"extinf":"#EXTINF:10.000000,","id":"f2z7dft8c217xyp","segment":5},{"extinf":"#EXTINF:10.000000,","id":"f2z7dft8c217xyp","segment":6},{"extinf":"#EXTINF:10.000000,","id":"f2z7dft8c217xyp","segment":7},{"extinf":"#EXTINF:10.000000,","id":"f2z7dft8c217xyp","segment":8}]');
Then the m3u8 is generated
$audio = sql("SELECT sequence, data FROM lista_m3u8 WHERE tipo = 'radio' AND ini <= DATE_FORMAT(NOW(),'%H:%i:%s') AND fin >= DATE_FORMAT(NOW(),'%H:%i:%s')");
$sale = '#EXTM3U'.PHP_EOL;
$sale .= '#EXT-X-VERSION:3'.PHP_EOL;
$sale .= '#EXT-X-MEDIA-SEQUENCE:'.$audio['data'][0]['sequence'].PHP_EOL;
$sale .= '#EXT-X-TARGETDURATION:10'.PHP_EOL;
$arreglo = json_decode($audio['data'][0]['data'],TRUE);
foreach($arreglo as $k=>$v) {
$sale .= $v['extinf'].PHP_EOL;
$sale .= S3URL("bucket-audio",$v['id']."/segment".sprintf('%05d',$v['segment']).".ts",(count($arreglo) * 25)).PHP_EOL;
}
header("Content-type: application/x-mpegURL");
echo $sale.PHP_EOL;
I believe I have solved it. I have just come back from leaving this playing for the past 16 hours and it's still going, my AWS logs confirm this.
I was originally approaching this the wrong way, trying to generate canned m3u8 files; what I really needed to do was know two things:
1.- What segment (regardless of original file) should be playing now?
2.- How many segments would have been played from the start of the "stream" (file 0, segment 0)?
The new method now takes the original playlist and creates a row for each segment, indicating its start time, duration, segment file and position in the stream. Then the m3u8 is generated with a few segments behind and a few after, calculating the correct EXT-X-MEDIA-SEQUENCE from the beginning of the stream. I also added EXT-X-DISCONTINUITY between the files so it doesn't hang up on receiving unexpected headers.
So now, I get the list of files/segments from my original table:
$ini = 0;
$conteo = 0;
$radio = sql("SELECT lista_contenido.orden,lista_contenido.contenido,contenido_audio_hls.audio_s FROM lista_listas LEFT JOIN lista_contenido ON (lista_listas.id = lista_contenido.lista) LEFT JOIN contenido_audio_hls ON (lista_contenido.contenido = contenido_audio_hls.id) WHERE (lista_listas.tipo = 'radio') ORDER BY lista_contenido.orden ASC");
foreach($radio['data'] as $k=>$v) {
$arreglo = json_decode($v['audio_s'],TRUE);
$seg = 0;
foreach($arreglo as $kk=>$vv) {
sql("INSERT INTO lista_m3u8 (tipo,orden,contenido,segmento,extinf,ini) VALUES('radio','".$conteo."','".$v['contenido']."','segment".sprintf('%05d',$seg).".ts','".$vv."','".sprintf('%02d:%02d:%02d',($ini/3600),($ini/60%60),$ini%60)."')");
$ini = $ini + ceil($vv * 1);
$seg++;
$conteo++;
}
}
Which gives me a table like so:
INSERT INTO `lista_m3u8` (`tipo`, `orden`, `contenido`, `segmento`, `extinf`, `ini`) VALUES ('radio', 20, 'f2z7ddw7r6bb7gfy', 'segment00018.ts', 10.000000000000, '00:03:11');
INSERT INTO `lista_m3u8` (`tipo`, `orden`, `contenido`, `segmento`, `extinf`, `ini`) VALUES ('radio', 21, 'f2z7ddw7r6bb7gfy', 'segment00019.ts', 10.000000000000, '00:03:21');
INSERT INTO `lista_m3u8` (`tipo`, `orden`, `contenido`, `segmento`, `extinf`, `ini`) VALUES ('radio', 22, 'f2z7ddw7r6bb7gfy', 'segment00020.ts', 6.066667079926, '00:03:31');
INSERT INTO `lista_m3u8` (`tipo`, `orden`, `contenido`, `segmento`, `extinf`, `ini`) VALUES ('radio', 23, 'f2z7df1bb66be7h3', 'segment00000.ts', 10.000000000000, '00:03:38');
INSERT INTO `lista_m3u8` (`tipo`, `orden`, `contenido`, `segmento`, `extinf`, `ini`) VALUES ('radio', 24, 'f2z7df1bb66be7h3', 'segment00001.ts', 10.000000000000, '00:03:48');
INSERT INTO `lista_m3u8` (`tipo`, `orden`, `contenido`, `segmento`, `extinf`, `ini`) VALUES ('radio', 25, 'f2z7df1bb66be7h3', 'segment00002.ts', 10.000000000000, '00:03:58');
INSERT INTO `lista_m3u8` (`tipo`, `orden`, `contenido`, `segmento`, `extinf`, `ini`) VALUES ('radio', 26, 'f2z7df1bb66be7h3', 'segment00003.ts', 10.000000000000, '00:04:08');
This creates thousands of db rows (~9000 for a 24-hour radio stream), but they're indexed by time, so SELECTing is instantaneous.
The final m3u8 script does this:
$actual = sql("SELECT orden, extinf, contenido, segmento, ini FROM lista_m3u8 WHERE tipo = 'radio' AND ini >= DATE_FORMAT(DATE_SUB(NOW(), INTERVAL 1 MINUTE),'%H:%i:%s') AND ini <= DATE_FORMAT(DATE_ADD(NOW(), INTERVAL 3 MINUTE),'%H:%i:%s') ORDER BY orden ASC");
$sale = '#EXTM3U'.PHP_EOL;
$sale .= '#EXT-X-VERSION:3'.PHP_EOL;
$sale .= '#EXT-X-MEDIA-SEQUENCE:'.($actual['data'][$actual['total']-1]['orden'] - $actual['total']).PHP_EOL;
$sale .= '#EXT-X-TARGETDURATION:10'.PHP_EOL;
$contenido = $actual['data'][0]['contenido'];
foreach($actual['data'] as $k=>$v) {
if($v['contenido'] != $contenido) { $sale .= "#EXT-X-DISCONTINUITY".PHP_EOL; }
$sale .= "#EXTINF:".$v['extinf'].",".PHP_EOL;
$sale .= S3URL("audio-bucket",$v['contenido']."/".$v['segmento'],180).PHP_EOL;
$contenido = $v['contenido'];
}
header("Content-type: application/x-mpegURL");
echo $sale.PHP_EOL;
Notice two things going on here, EXT-X-MEDIA-SEQUENCE is calculated by subtracting the current segment's position from the list as a whole and XT-X-DISCONTINUITY is put in between file changes.
I'm going to do some more testing to see if this works across browsers (I've only tested Chrome and IEG so far); but I believe it's a workable solution.