Perl: Using Storable and Thread::Queue correctly

Question

I'm trying to use Storable to help gather the data generated by multiple threads. Without the threading parts, the Storable bits of code work well. However, with the example code I provided, I get the following error:

thread failed to start: Invalid value for shared scalar at ./storable2.pl line 55.

Line 55 is $hash{"Simpsons"}{"Krusty"} = \%test2_hash;

@ikegami had advised me in an earlier post that the error is caused because my reference is ponting to an unshared variable. I tried sharing the variable with the code my %test2_hash : shared; (which is part of the example code) but I guess I'm using it incorrectly.

I also tried creating a new hash that was shared:

my %shared_hash;
%test2_hash = %{retrieve($tmp_filename)};
%shared_hash = share(%test2_hash);

However that results in the same error.

Here is my example code:

#!/usr/bin/perl
# storable.pl

use strict;
use warnings;
use Storable qw(store retrieve);
use File::Temp;
use threads;
use threads::shared;
use Thread::Queue;
use constant NUM_WORKERS => 10;
use Data::Dumper qw(Dumper);

my @out_array;

main();

sub main
{
    start_threads();
    foreach my $item (@out_array) {
        print "item: $item\n";
    }
}

sub start_threads
{
    my $queue = Thread::Queue->new();
    foreach (1..NUM_WORKERS) {
        async {
            while (my $job = $queue->dequeue()) {
                test1($job);
            }
        };
    }
    my @sentiments = ("Axe Murderer", "Mauler", "Babyface", "Dragon");
    $queue->enqueue(@sentiments);
    $queue->enqueue(undef) for 1..NUM_WORKERS;
    $_->join() for threads->list();

    my @return_array = @out_array;
    return @return_array;   
}

sub test1
{
    my $fighter = $_[0];
    my $tmp_filename : shared;
    my %hash : shared;
    my %test2_hash : shared;
    $tmp_filename = get_temp_filename();
    test2($fighter, $tmp_filename);
    %test2_hash = %{retrieve($tmp_filename)};

    $hash{"Simpsons"}{"Krusty"} = \%test2_hash;
    push @out_array, \%test2_hash;
    return;
}

sub test2
{
    my ($fighter, $tmp_filename) = @_;
    my %test2_hash;
    $test2_hash{"Zuffa"}{"LLC"} = $fighter;
    store(\%test2_hash, $tmp_filename);
}

sub get_temp_filename {
    my $fh = File::Temp->new(
        TEMPLATE => 'tempXXXXX',
        DIR      => 'tmpdir',
        SUFFIX   => '.tmp',
    );
    return $fh->filename;
}

What is the correct way to use Storable when implemented alongside Thread::Queue? Or is the problem purely related to incorrect Thread::Queue implementation?

Sorry @ikegami :( I really tried to experiment around with the tools I have, but I've hit a wall again.

UPDATE: I updated the code. Using strict and warnings I get the following output:

thread failed to start: Invalid value for shared scalar at ./storable2.pl line 52.

Line 52 is %test2_hash = %{retrieve($tmp_filename)};

Answer 1

Since you're only sharing the structure in order to transmit it, you could bypass the entire issue by serializing your data structure instead of sharing it.

You could use Storable's freeze and thaw to serialize and restore your data structure.

$queue->enqueue(freeze($_)) for @sentiments;

while (my $job = $queue->dequeue()) {
    test1(thaw($job));
}

Alternatively, you could use Thread::Queue::Any, a module like Thread::Queue which serializes queued items automatically.