Search code examples
phpalgorithmdice

create array union for advanced taxt compare like Jaccard index


I'm trying to develop functionality to create an union on 2 arrays to compare large text. For example:

$myText1 = "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam quis eleifend sem. Maecenas at elit varius erat malesuada mollis. Nullam vulputate, velit vel posuere finibus, ex quam imperdiet lorem, eu gravida quam purus et augue. Nunc dictum nunc vehicula mattis mollis. Quisque pharetra lorem id ultrices feugiat. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Suspendisse sed tortor odio. Proin consequat sed purus quis congue. Donec faucibus nec magna ac sodales."
$myText2="Mauris eget metus non mi tristique varius. Ut suscipit ante id condimentum interdum. Pellentesque egestas, leo quis tincidunt ornare, nulla metus sagittis lorem, in interdum justo tellus ac lectus. Vivamus auctor bibendum eros vel cursus. Praesent semper mauris dolor, sit amet placerat orci vestibulum non. Aenean consequat ultrices massa, in congue urna condimentum nec. Pellentesque eu faucibus dolor. Ut eu accumsan nunc, vel egestas dolor. Sed convallis mi et orci interdum tincidunt."
$str1array = explode(' ', $myText1 );
$str2array = explode(' ', $myText2);
$dallas = array();
foreach($str1array as $str){
    $dallas[] = trim($str, " .,;\t\n\r\0\x0B");
}
$phoenix = array();
foreach($str2array as $str){
    $phoenix[] = trim($str, " .,;\t\n\r\0\x0B");
}
$inter = array_intersect($dallas, $phoenix);
$union = array_unique(array_merge($str1array, $str2array));
$indiceDeJaccard = count($inter)/count($union);
$coefficientDeDice = (2*count($inter))/(count($str1array)+count($str2array));

But the problem with array_unique is the suppression of all duplicate content in the original text and create false data for calculate Jaccard index.


Solution

  • Sorry to all but i have the answer to my question:

    $inter = array_intersect($dallas, $phoenix);
    $diff = array_diff($dallas, $phoenix);
    $union = array_merge($inter, $diff);