Google App Engine: Correctly configuring a static site with advanced routing

Question

I've spent several shameful hours trying to solve this to no avail...

Problem:

I have a static website I am developing that is 100% pre-processed via Grunt & Assemble (if you are familiar with Jekyll, it is essentially the same concept). It also has a simple static Blog component which houses category directories of various names. As such, I need the catch-all in my app.yaml to route them appropriately.

However, I also would also like to have a custom error page to show in place of the standard GAE page status.

It seems that you cannot accomplish accounting for both scenarios in app.yaml alone because you can only use the catch-all target once.

Here is the logic in my current app.yaml

- url: (.*)/
  static_files: dist\1/index.html
  upload: dist/index.html
  expiration: "15m"

- url: /(.*)
  static_files: dist/\1/index.html
  upload: dist/(.*)/index.html
  expiration: "15m"

This is perfect for my use case because it routes any path to an index file if it exists in the current directory. However, because it uses the catch-all, I cannot again use it for something like the following

- url: /(.*)
  static_files: custom_error.html

or depend on

error_handlers:
  - file: custom_error.html

because it only renders for paths with no matching url pattern...

Ideas:

My next thoughts were that I may be able to accomplish this with some advanced routing via an external Python script

- url: /.*
  script: main.app

but after trying a myriad of various setups I have yet to stumble onto a way to accomplish this.

One example of a trail of breadcrumbs I was on would be

import os
import webapp2
import jinja2

# vars
jinja_environment = jinja2.Environment(loader=jinja2.FileSystemLoader('jinja'), extensions=['jinja2.ext.autoescape'], autoescape=True)

class mainHandler(webapp2.RequestHandler):
    def get(self):
        if (an-index-file-exists-in-this-directory)
            # follow the same static file logic as the app.yaml
            # static_files: dist/\1/index.html
            # upload: dist/(.*)/index.html
        else:
            template = jinja_environment.get_template('404/index.html')
            context =  {
                'page_title': '404',
            }
            self.response.out.write(template.render(context))
            self.error(404)

app = webapp2.WSGIApplication([
    ('/.*', mainHandler)
], debug=False)

I'm not even sure if taking it into an external python file would help solve the issue or not, but this was my awkward stab at it.

Does anyone have any ideas on how you can achieve custom error pages when your catch-all pattern is being used for another important purpose?

Update: Solved

Ok I finally have this figured out, but because Stack Overflow thinks I'm not cool enough to answer my own question (low point threshold?), I've posted the solution here:

https://gist.github.com/dustintheweb/c5e6e4ee1a64d50d7f87

Good luck!

Answer 1

As @Anthuin's answer points out, you can't write (nor modify) those index.html files on disk (nor create new ones dynamically), so it makes little sense to try to read them from disk -- the "disk" available to GAE apps is read-only (and split between the part only available for static serving, and the part readable by the app code itself).

Rather, unless the index.html files are huge (unlikely, I suspect), I'd keep them in the GAE datastore for your app. A really simple model:

from google.appengine.ext import ndb

class Indx(ndb.Model):
    body = ndb.TextProperty()

assuming the path is no longer than 500 characters and the body up to a megabyte. Then, your MainHandler becomes pretty simple:

class MainHandler(webapp2.RequestHandler):

    def get(self):
        key = ndb.Key('Indx', self.request.path)
        ent = key.get()
        if ent:
            self.response.write(ent.body)
        else:
            # your existing 404-returning code goes here

The app.yaml routing /.* to this script, and your app = code, need no change.

Now, the only thing that remains is, how did you expect to write, or modify, these index.html files (now datastore entities)? I have no idea because as long as they were to be files your app couldn't possibly have written nor modified them. In any case, now that they're in the data store, the writing also becomes very simple:

def write_indx(path, body):
    ent = Indx(body=body, id=path)
    ent.put()

Note that there is no real "modify" -- to possibly "modify" some index.html's "body", you'll actually read the previous one, make a new body string, and rewrite the entity with the above write_indx.

Potential problems include: bodies of more than 1 MB; and, keys (paths) of more than 500 characters. The former can easily be worked around by using, as @Anhuin suggested, Google Cloud Storage instead of the GAE datastore; the latter may perhaps be a problem because even GCS object names have limits (differently from GCS object lengths) -- specifically, 1024 bytes once the name is converted into utf-8. Is either of these issues likely to be a problem for you? If so, let us know!