Providing maps for your web site
There are many different options for providing maps on your own web site. The most basic and easiest for you to use is a JavaScript library called OpenLayers that provides a draggable map that you can embed on your own webpage. This is similar to the earlier section in this book describing the "Export" tab on http://osm.org/. In this chapter we'll go farther by describing how to set up your own server with a customized map style based on OpenStreetMap data. You might want to do this if you are interested in highlighting a particular kind of OpenStreetMap data like bicycle or foot paths or if you are dissatisfied with the elements present in the other available map styles.
System requirements
Serving your own maps is a fairly intensive task. Depending on the size of the area you're interested in serving and the traffic you expect the system requirements will vary. In general, requirements will range from 10-20GB of storage, 4GB of memory, and a modern dual-core processor for a city-sized region to 300GB+ of fast storage, 24GB of memory, and a quad-core processor for the entire planet.
What are tiles?
For our purposes, a web map is made up of several adjacent square images, each 256 pixels square, that cover a particular geographic region of the Earth. The OpenLayers library we are using in this example will place these tiles in a grid to form a draggable view of anywhere on the Earth.
Using OpenLayers
OpenLayers is a JavaScript library for displaying maps on your web site. It is very powerful and can be used to add advanced features to your map like buttons, overlays, popup windows and lines. In this example we'll use it simply to show the map that you render, but to learn more about extending it to include these features, please visit http://openlayers.org/.
The toolchain
We use a series of tools for generating and serving map tiles:
Apache provides the front end server that handles requests from your web browser and passes the request to mod_tile. The Apache web server can also be used to serve static web content like the HTML, JavaScript, or CSS for your map webpage.
Once Apache handles the request from the web user, it passes the request to mod_tile to deal with. Mod_tile checks if the tile has already been created and is ready for use or whether it needs to be updated due to not being in the cache already. If it is already available and doesn't need to be rendered, then it immediately sends the tile back to the client. If it does need to be rendered, then it will add it to a "render request" queue, and when it gets to the top of the queue, a tile renderer will render it and send the tile back to the client.
We use a tool called Mapnik to render tiles. It pulls requests from the work queue as fast as possible, extracts data from various data sources according to the style information, and renders the tile. This tile is passed back to the client and moves on to the next item in the queue.
For rendering, OpenStreetMap data is stored in a PostgreSQL database created by a tool called osm2pgsql. These two pieces work together to allow efficient access to the OpenStreetMap geographic data. It is possible to keep the data in the PostgreSQL database up to date using a stream of diff files produced every 60 seconds on the main OpenStreetMap server.
Setting up your own tile server
Setting up your own OpenStreetMap server is a lengthy task and best described on Richard Weait's website here: http://weait.com/content/build-your-own-openstreetmap-server.
Software Configuration
Start with an installation of Ubuntu (10.04) Lucid Lynx Server for your hardware architecture. Add the LAMP server and SSH server during installation. Once installed, this can be run as a headless server with neither monitor nor keyboard. So ssh to the new box, and let's get started. Since there may be several updates for Lucid Lynx, update the packages on your system:
sudo apt-get update sudo apt-get upgrade
Fetch some essential system tools for this tile server installation process:
sudo apt-get install subversion autoconf screen munin-node munin htop
Organize the file system to make the installation process a bit easier:
cd ~ mkdir src bin planet
Get the latest OpenStreetMap data
Retrieve a piece of OpenStreetMap data from http://planet.openstreetmap.org/. Since the whole planet is at least 18GB when compressed, there are links to smaller country or state sized extracts on that page. Since it is smaller and faster to interact with, the PBF file format is preferable when available. In this case we'll download the entire planet file by issuing the following command:
cd planet wget http://planet.openstreetmap.org/planet-latest.osm.bz2
Install and configure PostgreSQL database
Next, we need to install the PostgreSQL database system. Start by using apt-get to install the database and required extensions:
sudo apt-get install postgresql-8.4-postgis postgresql-contrib-8.4 postgresql-server-dev-8.4 build-essential libxml2-dev libtool libgeos-dev libpq-dev libbz2-dev proj
The default configuration for PostgreSQL 8.4 needs to be tuned for the amount of data you are about to add to it. Edit the file /etc/postgresql/8.4/main/postgresql.conf and make the following changes:
shared_buffers = 128MB checkpoint_segments = 20 maintenance_work_mem = 256MB autovacuum = off
These changes require a kernel configuration change. Perform that change and save it so that it is applied next time your computer is rebooted:
sudo sysctl -w kernel.shmmax=268435456 sudo sysctl -p /etc/sysctl.conf
Restart PostgreSQL to apply these new changes:
sudo /etc/init.d/postgresql-8.4 restart
It should restart without any errors and report back with this message:
* Restarting PostgreSQL 8.4 database server ...done.
Create a database called gis. Some of our future tools presume that you will use this database name. Substitute your username for username in two places below. This should be the username that will render maps with Mapnik.
sudo -u postgres -i createuser username # answer yes for superuser createdb -E UTF8 -O username gis createlang plpgsql gis exit
Set up PostGIS on the postresql database:
psql -f /usr/share/postgresql/8.4/contrib/postgis-1.5/postgis.sql -d gis
This should respond with many lines ending with:
... CREATE FUNCTION COMMIT ... DROP FUNCTION
Give your newly-created user permission to access some of the PostGIS extensions' data. Make sure you replace username with your user's name:
echo "ALTER TABLE geometry_columns OWNER TO username; ALTER TABLE spatial_ref_sys OWNER TO username;" | psql -d gis
Load OpenStreetMap data into the database
We need to set up the tool that converts OSM data into the PostgreSQL data format. Start by checking out the latest source code and compiling it:
cd ~/bin svn co http://svn.openstreetmap.org/applications/utils/export/osm2pgsql/ cd osm2pgsql ./autogen.sh ./configure make
Once compiled, we need to set the spatial reference on the newly-created database:
psql -f ~/bin/osm2pgsql/900913.sql -d gis
With the conversion tool compiled and the database prepared, the following command will insert the OpenStreetMap data you downloaded earlier into the database. This step is very disk I/O intensive and will take roughly 30 hours depending on the speed of the computer performing the conversion and the size of the data you retrieved from the server.
cd ~/bin/osm2pgsql ./osm2pgsql -S default.style --slim -d gis -C 2048 ~/planet/planet-latest.osm.bz2
Let's have a look at your data import as it progresses. The first part of the osm2pgsql output looks scary, but is normal:
Using projection SRS 900913 (Spherical Mercator) Setting up table: planet_osm_point NOTICE: table "planet_osm_point" does not exist, skipping NOTICE: table "planet_osm_point_tmp" does not exist, skipping Setting up table: planet_osm_line NOTICE: table "planet_osm_line" does not exist, skipping NOTICE: table "planet_osm_line_tmp" does not exist, skipping Setting up table: planet_osm_polygon NOTICE: table "planet_osm_polygon" does not exist, skipping NOTICE: table "planet_osm_polygon_tmp" does not exist, skipping Setting up table: planet_osm_roads NOTICE: table "planet_osm_roads" does not exist, skipping NOTICE: table "planet_osm_roads_tmp" does not exist, skipping Mid: pgsql, scale=100, cache=4096MB, maxblocks=524289*8192 Setting up table: planet_osm_nodes NOTICE: table "planet_osm_nodes" does not exist, skipping NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_nodes_pkey" for table "planet_osm_nodes" Setting up table: planet_osm_ways NOTICE: table "planet_osm_ways" does not exist, skipping NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_ways_pkey" for table "planet_osm_ways" Setting up table: planet_osm_rels NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "planet_osm_rels_pkey" for table "planet_osm_rels"
Don't be concerned by the NOTICE: entries above. All normal. Next, osm2pgsql will start reading the compressed planet file.
Reading in file: /home/user/planet/planet-latest.osm.bz2
As osm2pgsql reads the planet file it will give progress reports. The line below will refresh every few seconds and update the numbers in brackets. This part of the import takes a long time. Depending on your server, it will take between hours and days.
Processing: Node(10140k) Way(0k) Relation(0k)
As the import proceeds, the Node number will update a couple of times per second until complete, then the Way number will update roughly every second or two. Finally, the Relation number will update at roughly once per minute. As long as you can see these numbers advancing the import process is still operating normally for your server. Do not interrupt the import process unless you have decided to start over again from the beginning.
Processing: Node(593072k) Way(45376k) Relation(87k) Exception caught processing way id=110802 Exception caught processing way id=110803 Processing: Node(593072k) Way(45376k) Relation(474k)
The exceptions shown above are due to minor errors in the planet file. The planet import is still proceeding normally.
The next stage of the osm2pgsql planet import process also will take between hours and days, depending on your hardware. It begins like this:
Node stats: total(593072533), max(696096737) Way stats: total(45376969), max(55410575) Relation stats: total(484528), max(555276) Going over pending ways processing way (752k)
The processing way number should update approximately each second.
Going over pending relations node cache: stored: 515463899(86.91%), storage efficiency: 96.01%, hit rate: 85.97% Committing transaction for planet_osm_roads Committing transaction for planet_osm_line Committing transaction for planet_osm_polygon Sorting data and creating indexes for planet_osm_line Sorting data and creating indexes for planet_osm_roads Sorting data and creating indexes for planet_osm_polygon Committing transaction for planet_osm_point Sorting data and creating indexes for planet_osm_point Stopping table: planet_osm_nodes Stopping table: planet_osm_ways Stopping table: planet_osm_rels Building index on table: planet_osm_rels Stopped table: planet_osm_nodes Building index on table: planet_osm_ways Stopped table: planet_osm_rels Completed planet_osm_point Completed planet_osm_roads Completed planet_osm_polygon Completed planet_osm_line Stopped table: planet_osm_ways
At this point the import is complete and successful.
Install Mapnik library
Next, we need to install the Mapnik library. Mapnik is used to render the OpenStreetMap data into the tiles used for an OpenLayers web map. To start, we'll install the dependencies for the Mapnik library followed by downloading and compiling the source:
sudo apt-get install libltdl3-dev libpng12-dev libtiff4-dev libicu-dev libboost-python1.40-dev python-cairo-dev python-nose libboost1.40-dev libboost-filesystem1.40-dev libboost-iostreams1.40-dev libboost-regex1.40-dev libboost-thread1.40-dev libboost-program-options1.40-dev libboost-python1.40-dev libfreetype6-dev libcairo2-dev libcairomm-1.0-dev libgeotiff-dev libtiff4 libtiff4-dev libtiffxx0c2 libsigc++-dev libsigc++0c2 libsigx-2.0-2 libsigx-2.0-dev libgdal1-dev python-gdal imagemagick ttf-dejavu
Build the Mapnik library from source:
cd ~/src svn co http://svn.mapnik.org/tags/release-0.7.1/ mapnik cd mapnik python scons/scons.py configure INPUT_PLUGINS=all OPTIMIZATION=3 SYSTEM_FONTS=/usr/share/fonts/truetype/ python scons/scons.py sudo python scons/scons.py install sudo ldconfig
Verify that Mapnik has been installed correctly:
python >>> import mapnik >>>
If python replies with the second chevron prompt >>> and without errors, then Mapnik library was found by Python. Congratulations!
Install Mapnik tools
Next, we need to install the OpenStreetMap Mapnik tools, which include the default style file and tools to help Mapnik render OpenStreetMap data:
cd ~/bin svn co http://svn.openstreetmap.org/applications/rendering/mapnik
Mapnik uses prepared files to generate coastlines and ocean areas for small scale maps since it is faster than reading the entire database for this information.
cd ~/bin/mapnik mkdir world_boundaries wget http://tile.openstreetmap.org/world_boundaries-spherical.tgz tar xvzf world_boundaries-spherical.tgz wget http://tile.openstreetmap.org/processed_p.tar.bz2 tar xvjf processed_p.tar.bz2 -C world_boundaries wget http://tile.openstreetmap.org/shoreline_300.tar.bz2 tar xjf shoreline_300.tar.bz2 -C world_boundaries wget http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/10m-populated-places.zip unzip 10m-populated-places.zip -d world_boundaries wget http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/110m/cultural/110m-admin-0-boundary-lines.zip unzip 110m-admin-0-boundary-lines.zip -d world_boundaries
The database is loaded and the tools are installed. Let's test everything together. Remember to replace username with your username.
cd ~/bin/mapnik ./generate_xml.py --dbname gis --user username --accept-none ./generate_image.py
View image.png to confirm that you have rendered a map of England. Congratulations!
Install and configure Apache web server and mod_tile
Now that we've created an image using your map data, we need to set up the tile serving software to handle tile requests using Apache. To do this, we'll install the Apache web server and use the mod_tile extension to handle the tile rendering system.
Begin by installing tools necessary to build mod_tile:
sudo aptitude install apache2 apache2-threaded-dev apache2-mpm-prefork apache2-utils libagg-dev
Compile the mod_tile source code:
cd ~/src svn co http://svn.openstreetmap.org/applications/utils/mod_tile cd mod_tile make sudo make install
Change the the mod_tile settings by editing the
plugins_dir=/usr/local/lib/mapnik/input font_dir=/usr/lib/mapnik/fonts XML=/home/username/bin/mapnik/osm.xml HOST=localhost
Create the files required for the mod_tile system to run (remember to change username to your user's name):
sudo mkdir /var/run/renderd sudo chown username /var/run/renderd
Next, we need to tell the Apache web server about our new mod_tile installation by creating the file /etc/apache2/conf.d/mod_tile and adding one line:
LoadModule tile_module /usr/lib/apache2/modules/mod_tile.so
Also, Apache's default website configuration file needs to be modified to include mod_tile settings. Modify the file /etc/apache2/sites-available/default to include the following lines immediately after the admin e-mail address line:
LoadTileConfigFile /etc/renderd.conf ModTileRenderdSocketName /tmp/osm-renderd # Timeout before giving up for a tile to be rendered ModTileRequestTimeout 0 # Timeout before giving up for a tile to be rendered that is otherwise missing ModTileMissingRequestTimeout 30
You'll then need to start the mod_tile rendering application. Note that it will not output anything upon successful start.
cd ~/src/mod_tile ./renderd
Congratulations! You now have a server capable of generating tiles using your own copy of the OpenStreetMap data. Visit http://localhost/osm_tiles2/0/0/0.png to verify that you have a working map tile generated.