How much memory(MB) can the vector variable occupy in enclave of Intel sgx?

Question

I want to immigrate PageRank algorithm in the sgx enclave. The algorithm uses vector to save the edge relationship and matrix.

vector<size_t> num_outgoing; // number of outgoing links per column
vector< vector<size_t> > rows; // the rowns of the hyperlink matrix
map<string, size_t> nodes_to_idx; // mapping from string node IDs to numeric
map<size_t, string> idx_to_nodes; // mapping from numeric node IDs to string
vector<double> pr; // the pagerank table

The application runs well when it stores less than 9000 edges. Once it increases to 10000 edges or more, the application crushes, and throws out an unhandled exceptionenter image description here . I also run the same code outside the enclave, it runs well when it stores 90000 edges.

By debugging, I found the application fails at the places below.

if (rows.size() <= max_dim) {
    max_dim = max_dim + 1;
    rows.resize(max_dim);
    if (num_outgoing.size() <= max_dim) {
        num_outgoing.resize(max_dim);
    }
}

However, the variable 'rows' cannot be resized larger once it owns 13896 element. I'm confused that 'rows' only occupies about 300kb and 'num_outgoing' only occupies about 100kb. It's far less than the allow size. There is 128MB space in total for the enclave application.

My enclave config file is listed as follows. I try to change the value of StackMaxSize, however, it seems useless.

<EnclaveConfiguration>
    <ProdID>0</ProdID>
    <ISVSVN>0</ISVSVN>
    <StackMaxSize>0x400000</StackMaxSize>
    <HeapMaxSize>0x100000</HeapMaxSize>
    <TCSNum>1</TCSNum>
    <TCSPolicy>1</TCSPolicy>
    <DisableDebug>0</DisableDebug>
    <MiscSelect>0</MiscSelect>
    <MiscMask>0xFFFFFFFF</MiscMask>
    <EnableKSS>0</EnableKSS>
    <ISVEXTPRODID_H>0</ISVEXTPRODID_H>
    <ISVEXTPRODID_L>0</ISVEXTPRODID_L>
    <ISVFAMILYID_H>0</ISVFAMILYID_H>
    <ISVFAMILYID_L>0</ISVFAMILYID_L>
</EnclaveConfiguration>

The format of the input edge is shown as below. The first number is the "from" node, the second number is the "to" node

1 0
2 0
3 1
4 3
5 4
6 0
7 1
8 0
9 1
10 0

I wonder how to config enclave to make it allow bigger vector variable? The problem exists on both win10 and ubuntu.

Answer 1

All seems like you are running out of memory. The memory limits (stack and heap) are set in the config file using StackMaxSize and HeapMaxSize (see Developer Reference for details). The size of the EPC (128MB or 256MB) has nothing to do with it. Here, you are not bounded by the EPC size but by the heap and stack.

Increasing the stack size does not change anything because dynamically allocated memory resides on heap. In turn, you should be looking at the max heap size. Currently, you have it set to 0x100000 (=1MB), which is quickly used up by your data. Try increasing it and see if you can accommodate larger vectors.

I think on Windows your enclave has to entirely fit into EPC but on Linux you can create enclaves with tens of GBs. Be aware that once your (Linux) enclave memory usage exceeds ~90 MB you will start noticing EPC paging and, with it, a huge performance degradation.