I am trying to implement a map tile engine on Google App Engine.
The map data is stored in the database, the datastore (big table). The problem is that 20 requests might come in at approximately the same time to draw 20 tiles based upon the same set of rows in the database.
So 20 requests come in, if I write the code to read from the database for each request, then I will be doing 20 read's, which are the same, from the database, one read for each tile image output. Since each read is the same query, it doesn't make sense to do the same query 20 times. In fact, this is very inefficient.
Can anyone suggest a better way to do this?
If I use the memcache, I need to put the data into memcache, but there are 20 requests coming in at the same time for the data, then if I do a nieve implementation then 20 processes will be writing to memcache, since they are all going at the same time in parallel.
I am programming in Google Go version 1 beta on Google App Engine, I refer to the Python doc's here since they are more complete.
References:
Google datastore http://code.google.com/appengine/docs/python/datastore/overview.html
Leaflet JS I am using for showing map tiles http://leaflet.cloudmade.com/
To clarify.
I generate the tile images from data in the database, that is, I query the database for the data (this is not the tile image), then I draw the data into a image and render the image as a JPEG. As GAE is efficient for drawing images on the server side http://blog.golang.org/2011/12/from-zero-to-go-launching-on-google.html
I don't know about how Google App Engine does it, but MySQL has a query cache so that if the same query gets asked twice in a row, then it uses the results from the first to answer the second. Google is smart about things, so hopefully they do that as well. (You might be able to figure out if they are by timing it.)
One thing you might need to make sure of is that the queries are exactly the same, not just returning the same results. For example, you don't want query1 to be SELECT lat, lng FROM mytable WHERE tileX=1 AND tileY=1 and query2 to be SELECT lat, lng FROM mytable WHERE tileX=1 AND tileY=2
I make tiles with gazillions of polygons, and when I did timing and optimization, I found to my surprise that it was faster to return ALL values and weed out the ones I didn't want in PHP than it was to stick in a WHERE clause to the SQL. I think that partly it was because the WHERE clause was different for every tile so the MySQL server couldn't cache effectively.