Search code examples
openstreetmapmbtilesvector-tiles

How can I merge osm files while retaining all node, way, and relation information?


Background

I am trying to make a reduced size vector mbtiles file for the world for a compact offline maps solution I am working on for Windows desktop. To do so I have found OsmAnd's reduced size World_basemap_2.obf to be exactly what I need, but in the mbtiles OSM format so I can generate an mbtiles. The obf was created using the following OSM files generated from shape files. They are available at http://builder.osmand.net/basemap/

The problem I have is that in order to make an mbtiles they need to be combined into one OSM file. osm2vectortiles' guide to generate an mbtiles found here http://osm2vectortiles.org/docs/own-vector-tiles/.

First I extract them to deal with the osm files.

Problem

When I use osmconvert to combine the files using the --complete-ways and --complex-ways flags, a lot of relations and ways are removed and the filesize is extremely different.

For ex:

osmconvert proc_line_tertiary_out.osm --complete-ways --complex-ways -o=tertiary.o5m

results in a file 133,691 kb in size

osmconvert tertiary.o5m --complete-ways --complex-ways -o=tertiary.csv

results in a CSV with a total of 13131405 nodes, ways, and relations

osmconvert tertiary.o5m proc_line_ferry_out.osm.o5m --complete-ways --complex-nodes -o=combined.o5m

merging tertiary.o5m with proc_line_ferr_out.osm.o5m (generated the same as shown above) and converting to CSV

results in a CSV with 1195239 total nodes, ways, and relations... which is significantly less and is very concerning. Combining two files should not make fewer nodes, ways, and relations than the original files.

Question

What am I doing wrong?

  • Is there an alternative way to combine osm files? (I've read that osmosis can't deal with negative nodes.)

  • Is there an alternative way to generate an mbtiles, possibly from multiple osm files?

Files trying to combine

  • points.osm.bz2
  • polygon_aeroway_military_tourism.osm.bz2
  • polygon_lake_water.osm.bz2
  • polygon_natural_landuse.osm.bz2
  • proc_line_admin_level_out.osm.bz2
  • proc_line_ferry_out.osm.bz2
  • proc_line_motorway_out.osm.bz2
  • proc_line_primary_out.osm.bz2
  • proc_line_railway_out.osm.bz2
  • proc_line_secondary_out.osm.bz2
  • proc_line_tertiary_out.osm.bz2
  • proc_line_trunk_out.osm.bz2
  • cities.osm.bz2
  • coastline.osm.bz2

OSMConvert

osmconvert explains that

Merging two or more Geographical Areas

Under certain conditions, OSM data files can be merged. If they hold objects (nodes, ways, relations) with the same id, they must have the same content. For example: If a way crosses the geographical border between two regional files, the way dataset must contain every node reference, even the references to nodes which do not lie within the borders of the regarding file. I.e., such a region must not have been cut out using the option --drop-brokenrefs.

Does that mean I can't do what I am trying to do? I want to keep all nodes, ways, and relations, even if the files don't have them in common.

Attempt

osmconvert polygon_aeroway_military_tourism.osm polygon_natural_landuse.osm -o=result.osm

Result

    <?xml version='1.0' encoding='UTF-8'?>
    <osm version="0.6" generator="osmconvert 0.7T">
        <node id="-10000000000001" lat="-1.0015443" lon="-80.5349586"/>
        <way id="-100000001" version="1">
            <nd ref="-10000000000001"/>
            <nd ref="-10000000000002"/>
            <nd ref="-10000000000003"/>
            <nd ref="-10000000000004"/>
            <nd ref="-10000000000005"/>
            <nd ref="-10000000000006"/>
            <nd ref="-10000000000007"/>
            <nd ref="-10000000000008"/>
            <nd ref="-10000000000009"/>
            <nd ref="-10000000000010"/>
            <nd ref="-10000000000011"/>
            <nd ref="-10000000000012"/>
            <nd ref="-10000000000013"/>
            <nd ref="-10000000000014"/>
            <nd ref="-10000000000015"/>
            <nd ref="-10000000000016"/>
            <nd ref="-10000000000017"/>
            <nd ref="-10000000000018"/>
            <nd ref="-10000000000019"/>
            <nd ref="-10000000000020"/>
            <nd ref="-10000000000021"/>
            <nd ref="-10000000000022"/>
            <nd ref="-10000000000023"/>
            <nd ref="-10000000000024"/>
            <nd ref="-10000000000025"/>
            <nd ref="-10000000000026"/>
            <nd ref="-10000000000027"/>
            <nd ref="-10000000000028"/>
            <nd ref="-10000000000029"/>
            <nd ref="-10000000000030"/>
            <nd ref="-10000000000031"/>
            <nd ref="-10000000000032"/>
            <nd ref="-10000000000033"/>
            <nd ref="-10000000000034"/>
            <nd ref="-10000000000035"/>
            <nd ref="-10000000000036"/>
            <nd ref="-10000000000037"/>
            <nd ref="-10000000000038"/>
            <nd ref="-10000000000039"/>
            <nd ref="-10000000000040"/>
            <nd ref="-10000000000041"/>
            <nd ref="-10000000000042"/>
            <nd ref="-10000000000043"/>
            <nd ref="-10000000000044"/>
            <nd ref="-10000000000045"/>
            <nd ref="-10000000000046"/>
            <nd ref="-10000000000047"/>
            <nd ref="-10000000000048"/>
            <nd ref="-10000000000049"/>
            <nd ref="-10000000000050"/>
            <nd ref="-10000000000051"/>
            <nd ref="-10000000000001"/>
        </way>
    </osm>

Solution

  • the options --complete-ways and --complex-ways are meant for usage when applying borders. If you are not clipping OSM data (options -b or -B) better do not use --complete-ways nor --complex-ways unless you are certain what they do with your data. At least I am not.

    The way of choice to merge OSM data using osmconvert is this:

    osmconvert input1.o5m input2.o5m input3.o5m -o=output.o5m
    

    There is some further information about osmconvert in OSM Wiki:

    https://wiki.openstreetmap.org/wiki/Osmconvert