How to make content div full height when using padding on wrapper?

I know similar questions have been asked numerous times, but I've tried a number of answers and none seem to work for my situation. I've found a lot of solutions for sticky footers, but that's not exactly what I'm looking for, or maybe I haven't figured out how to use correctly for my situation:

I have fully fixed position navigation (header/sidebars/footer). The content flows on top of the header and footer.
My content wrapper layer is currently a 100% width/ 100% min-height layer above the navigation layer, and using simple padding on top/bottom and margins on left/right, I'm able to show the navigation elements.
Inside the wrapper layer I have my content div, which has my page background and a box-shadow. But I can't get the content div to expand to the full height of the wrapper, because the wrapper height is based on percentage. So everything works perfectly when I have content that fills or overflows the window, but when I don't, the content div is too small.

Sticky footer doesn't work in this situation because the sticky footer just covers the content (the content itself is still 100% height). I basically want the content to be min-height 100% - minus the 50px header and 50px footer.

Is there any css solution to this without using a javascript hack or calc()? I would be willing to have the header + footer not be fix positioned - but I want to keep the box-shadow on the content div (i.e. 50px from top and bottom of the page).

JSFiddle: https://jsfiddle.net/tyhenry/n9rpbj3m/5/

HTML:

<div id="nav">
    <div id="header">
        header
    </div>
    <div id="left-side">
        left sidebar
    </div>
    <div id="right-side">
        right sidebar
    </div>
    <div id="footer">
        footer
    </div>
</div>
<div id="page-layer">

    <div class="page">
        content<br>
        content<br>
        content<br>
    </div>
</div>

CSS:

* {
    box-sizing: border-box;
}

html, body{
    background-color: #eeffff;
    text-align: center;
    margin: 0;
    padding: 0;
    height: 100%;
}

#header {
    position: fixed;
    top:0;
    left:0;
    width: 100%;
    height: 50px;
    background-color: #ddd;
}

#left-side {
    background-color: #bbb;
    position: fixed;
    left: 0;
    top: 50px;
    width: 50px;
    height: 100%;
}

#right-side {
    background-color: #bbb;
    position: fixed;
    right: 0;
    top: 50px;
    width: 50px;
    height: 100%;
    overflow: hidden;
}

#footer {
    position: fixed;
    bottom: 0;
    left: 0;
    width: 100%;
    height: 50px;
    background-color: #aaa;
}

#page-layer{
    position:relative;
    padding: 50px 0 50px 0;
    margin: 0 50px 0 50px;
    min-height: 100%;
}

.page {
    background-color: #fff;
    box-shadow: 0px 1px 20px 0px rgba(0, 0, 0, 0.8);
    width:100%;
}

Solution

If you change the display property to flex it fills the whole (height between header and footer) and adjusts when you resize.

#page-layout {
  display: flex;
}