My C++11 test show that sort(vector<string>) is even slower than C++03, any error?

Question

With right value reference and move semantics, C++11's swap/sort speed, should be equal or greater than C++03. So I designed a simple experiment to test this.

I compile and run it with -O2, with c++03 and c++11 standard.

$g++ test.cpp -O2 && ./a.out
10240000 end construction
sort 10240000 spent1.40035

$g++ test.cpp -O2 -std=c++11 && ./a.out
10240000 end construction
sort 10240000 spent2.25684

So it seems with C++11 enabled, program is slower. I'm on a very new mac and gcc environment:

$gcc -v
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 7.3.0 (clang-703.0.31)
Target: x86_64-apple-darwin15.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

Below is source code:

#include<string>
#include<algorithm>
#include<vector>
#include<cstdlib>
#include<cstdio>
#include<iostream>
#include<ctime>
using namespace std;
string randomString()
{
    const size_t scale=600;
    char ret[scale];
    for(size_t i=0;i<scale;++i)
    {
        double rand0to1=(double)rand()/RAND_MAX;
        ret[i]=(char)rand0to1*92+33;
    }
    return ret;
}
int main()
{
    srand(time(NULL));
    const size_t scale=10240000;
    vector<string> vs;
    vs.reserve(scale);
    for(size_t i=0;i<scale;++i)
    {
        vs.push_back(randomString());
    }
    cout<<vs.size()<<" end construction\n";
    clock_t begin=clock();
    sort(vs.begin(),vs.end());
    clock_t end=clock();
    double duration=(double)(end-begin)/CLOCKS_PER_SEC;
    cout<<"sort "<<scale<<" spent"<<duration<<"\n";
    return 0;
}

Any error with my program or understanding, how to explain my test result?

Really need your expertise on this!

Answer 1

Your test code has several issues.

1. The string you generate in ret is not null terminated so it will contain garbage from the stack which is likely to change with compiler settings. This the most likely cause of your strange results: the c++11 version sorting longer strings.

2. Your casts lead to strings which are all identical. Not an actual problem with measurements but probably not what you're interested in testing.

3. You should not use a truly random seed for benchmarking. You want to produce the same strings on every run to get reproducibility.

This fixed version of the code:

#include<string>
#include<algorithm>
#include<vector>
#include<cstdlib>
#include<cstdio>
#include<iostream>
#include<ctime>
using namespace std;
string randomString()
{
    const size_t scale=600;
    char ret[scale];
    for(size_t i=0;i<scale;++i)
    {
        double rand0to1=(double)rand()/RAND_MAX;
        ret[i]=(char)(rand0to1*92+33);
    }
    ret[scale-1] = 0;
    return ret;
}
int main()
{
    srand(1);
    const size_t scale=10240000;
    vector<string> vs;
    vs.reserve(scale);
    for(size_t i=0;i<scale;++i)
    {
        vs.push_back(randomString());
    }
    cout<<vs.size()<<" end construction\n";
    clock_t begin=clock();
    sort(vs.begin(),vs.end());
    clock_t end=clock();
    double duration=(double)(end-begin)/CLOCKS_PER_SEC;
    cout<<"sort "<<scale<<" spent "<<duration<<"\n";
    return 0;
}

produces what I believe you were expecting:

$ g++ -O2 -std=c++03 test.cpp && ./a.out
10240000 end construction
sort 10240000 spent 10.8765

$ g++ -O2 -std=c++11 test.cpp && ./a.out
10240000 end construction
sort 10240000 spent 8.72834

By the way, g++ from Xcode on a mac is actually clang. But the results are similar:

$ clang++ -O2 -std=c++03 test.cpp && ./a.out
10240000 end construction
sort 10240000 spent 10.9408

$ clang++ -O2 -std=c++11 test.cpp && ./a.out
10240000 end construction
sort 10240000 spent 8.33261

Tested with g++ 6.2.1 and clang 3.9.0. The -std=c++03 switch is important as without it, g++ compiles in a mode which gives the fast times.