Exporting OSM to 3D formats

December 29 2014

For an upcoming project, I needed a 3D model of all of the buildings in Manhattan. I knew that this 3d data existed in OpenStreetMaps and I trusted that it would be possible to convert it to my format of choice somehow, so first I concentrated on getting the data I needed.

I found that you could download OSM data by simply hitting the “export” button on openstreetmap.org…

…but unfortunately (and understandably), OSM doesn’t take too kindly to being asked to export 40 square miles of one of the most densely populated areas in the world. For small datasets (areas with few features), export from openstreetmap.org is probably fine. But in my case, the export tool just kind of silently failed. So I cried.

Area Extracts

I was determined not to have to download small areas and merge them into one bigger file, so I looked around a bit and found that the OSM wiki lists several Country and area extracts - meaning large areas merged into one big OSM file. Both geofabrik.de and mapzen have New York extracts updated daily and weekly (respectively).

I went with new-york_new-york.osm.pbf from Mapzen because it is limited to New York City whereas the geofabrik one is all of New York State.

HELPFUL TIP: To preview the data at each step along the way, go get QGIS. It will open your OSM files. It’s a bit of a pain to install the prerequisites (GDAL Framework and MatPlotLib), but it’ll be worth it to see what you are doing.

Progress! But not quite good enough. I only wanted Manhattan…

The island of Manhattan is awkward in that it is tall and skinny, slanting significantly to the East. Even very specific NYC “area extract” from Mapzen is still a big rectangular bounding box including tons of New Jersey, Brooklyn, and Queens, which would require tons of cleanup in Blender or Maya. My Maya skills are pretty much nonexistent, so I really wanted to find a way to extract just Manhattan from the New York City OSM file.

To make matters worse, Manhattan has some non-contiguous areas, so it would really be 3 bounding areas…

And, come to think of it, I’m not interested in exporting all of the features included in OpenStreetMaps to my 3D file. I only want buildings. So I realized that I needed to find a way to massage the OSM data a bit before I fed it to OSM2World.

The Scalpel: Osmosis

That’s when I came across Osmosis, which allows you to do all kinds of cool things with OSM files.

Relevant to my problem, osmosis allows you to feed in an OSM file and a .poly file (simply a list of points that define a polygon) and it will give you a new OSM file with all of the features that fall within the polygon.

Luckily, JamesChevalier has been collecting .poly files for tons of cities at github.com/jameschevalier/cities, so I was able to find the Manhattan one (pictured above).

Furthermore, to keep the file size as small as possible, I wanted to get rid of stuff in the OSM that wasn’t buildings. You might want to customize the –tag-filters to get your own set of features. To do so, you need to understand a bit about how OSM data is structured - specifically, how tags work.

So, after downloading osmosis using the links on the wiki page, I ran:

./osmosis-latest/bin/osmosis
--read-pbf file="new-york_new-york.osm.pbf"
--tag-filter accept-ways building=*
--tag-filter reject-relations
--used-node
--bounding-polygon file="manhattan_new-york.poly.txt"
--write-xml file="manhattan-buildings.osm"

Here’s the line-by-line of what is going on in this command.

1. ./osmosis-latest/bin/osmosis The path to the osmosis program

2. --read-pbf file="new-york_new-york.osm.pbf" Read in the PBF file that I got from Mapzen

3. --tag-filter accept-ways building=* Only accept nodes tagged as some kind of building
4. --tag-filter reject-relations This isn’t entirely necessary because they wouldn’t be included in the 3D file anyway, but it makes the resulting OSM file a bit smaller, so…
5. --used-node Restricts output of nodes to those that are used in ways and relations.
6. --bounding-polygon file="manhattan_new-york.poly.txt" Specify the bounding box that I got from jameschevalier
7. --write-xml file="manhattan-buildings.osm" Write the result to manhattan.osm

and got a nice 68MB OSM file, manhattan-buildings.osm:

Or, if you are interested, this is what Manhattan looks like without filtering out the roads: manhattan-all.osm

Detour: Direct from API

Lest you think I am overcomplicating this task, I should mention that the OSM wiki on downloading data also mentions the Overpass API, a read-only API optimized for fetching custom datasets. Sounds great! And it even seems to have a query that allows you to query by polygon.

The upside of using the Overpass query language is that it eliminates the step of downloading a huge dataset and then having to filter it into a smaller one.

I tried using the poly2request.sh method described in the Select region by polygon section with the Manhattan .poly file,

./poly2request.sh < manhattan_new-york.poly.txt > request.txt

which gave me a long request that looked something like…

(node(poly:”4.068969E+01 -7.404350E+01 4.069019E+01 -7.404389E+01 4.069056E+01 -7.404460E+01 4.069092E+01 -7.404549E+01

…

4.072545E+01 -7.402152E+01 4.069759E+01 -7.402778E+01 4.068668E+01 -7.403446E+01 “);<;);out;

…which you then feed to Overpass like so:

wget --post-file=request.txt http://overpass-api.de/api/interpreter

but I got the following error from Overpass

runtime error: Query run out of memory in “polygon-query” at line 1 using about 513 MB of RAM.

I don’t know whether it’s the complexity of the query or the size of the region that is causing Overpass to run out of RAM, but suffice to say, it’s not up for the task.

OSM2World

Now that I have the perfect OSM file that only contains the buildings in Manhattan, I’m ready to convert to a 3D format. This post about making papercraft buildings has a great description of how buildings are represented in OSM. As I learned, it doesn’t have a whole lot to do with typical 3D formats, so the task of converting would not be trivial.

So after some digging, I found OSM2World, which can convert an OSM file to OBJ.

To get a quick preview, download the program at osm2world.org/download and try:

./osm2world.sh --gui -i ~/Desktop/manhattan-buildings.osm

NOTE: I think you have to use osm2world-windows.bat instead of osm2world.sh if you are on Windows.

You might get a message like this:

But apparently 68MB isn’t too big after all (and this is just a preview, so if yours is bigger and causes some kind of error, just skip to the next step). So after some thinking, OSM2World gives you something like:

If you like what you see, you can either save out from the program using the File menu, or bypass the “preview” and just output it directly.

./osm2world.sh -i ~/Desktop/manhattan-buildings.osm -o ~/Desktop/manhattan-buildings.obj

Which gave me manhattan.obj and manhattan.obj.mtl

After converting to FBX using the Autodesk FBX tool, I was able to open it in Cheetah.

OSM2World Properties

If you are happy with how your model looks, then you are all done! But if you think that the reds and greens and beiges that OSM2World chooses are a little tacky, all you have to do is make yourself a configuration file.

You can get pretty detailed with the properties for OSM2World, including specifying not only colors, but also textures for different elements of your model. You an check out example_config.properties for a starting point, and all of the material names are listed in the Materials.java file. There are about 200 of them!

Here is mine, my_props.properties:

treesPerSquareMeter = 0.001
defaultTreeHeight = 10.0
defaultTreeHeightForest = 20.0

# I want to define all of the colors myself
useBuildingColors = false 
drawBuildingWindows = true
useBillboards = true
renderUnderground = false

material_TERRAIN_DEFAULT_color = #626262

material_BUILDING_DEFAULT_color = #949494
material_ROOF_DEFAULT_color = #B2B2B2

Then you just add --config my_props.properties to the command.

./osm2world.sh --config my_props.properties -i ~/Desktop/manhattan.osm -o ~/Desktop/manhattan.obj

Success!

This is my model opened in Cheetah3D. After running it through the FBX tool, it weighs in at 26.6MB.

Textures

I should note that it is theoretically possible to specify textures for different elements of the model using syntax like this in your properties file:

material_BUILDING_DEFAULT_texture_file = textures/concrete.jpg
material_BUILDING_DEFAULT_texture_width = 2.5
material_BUILDING_DEFAULT_texture_height = 2.5
material_BUILDING_DEFAULT_texture_colorable = true

…but I was never able to get it to work