Skip to main content
Sign in

Vector Tiles on Google Cloud Storage: Serving the Tiles

David Hollema
  • Updated

In the first part of this series, we answer the question "Why vector tiles?" and demonstrate how to build, upload and configure a tileset to Google Cloud Storage (GCS).

Vector tiles continue to gain widespread adoption in the web mapping community as a choice format for disseminating basemaps and operational data. In addition to the often-mentioned benefits of small size and speedy transmission, vector tiles are favored because they are consumable in a variety of proprietary and open source formats and require very little supporting infrastructure. GCS' ability to host static content makes it a natural fit for serving vector tiles. Let's take a look at our journey to hosting vector tiles in GCS.

The Drivers

At the outset of our evaluation, we identified some drivers for a tile serving system. Our overall goal was to substitute or replace heavy backend GIS infrastructure at a public or private enterprise for those layers that are particularly tough to manage: large files and tables, wide spatial extents, or heavy request volumes.

Desired: Features

  • Ability to support a variety of clients including web (Leaflet, OpenLayers, ArcGIS JavaScript API) and desktop (ArcGIS Pro, QGIS) and potentially mobile.
  • Attributes available for layer styling, queries and feature inspection (pop-ups)
  • Support for non-web Mercator coordinate systems

Desired: Non-functionals

  • Scalable to support source data in tens of gigabytes
  • Minimized cost with maximized serving speed
  • Simple construction and maintenance with few moving parts

Tile Server Specs

While the vector tile file format is captured in a rigid specification document, the details of the tile server interface is not. A variety of standards have developed over time, including TMS and tilejson, but a single, well-adopted standard has yet to surface. Fortunately, most vector tile clients adhere to a two-part convention:

  • {z}/{y}/{x} - Well-known hierarchical path structure that uniquely identifies each tile in the pyramid based on zoom level, row and column (sometimes row and column are reversed)
  • EPSG:3857 - Web Mercator coordinate system

With these two parts in place, a client can calculate which tiles to request for any map extent and zoom level.

Why Cloud Storage?

Cloud Storage, Google Cloud's object store, is an attractive cloud resource for serving vector tiles.  Here's why.

  • As a fully managed service, there is no web server or other supporting infrastructure to maintain, and it's nearly always "on."
  • Unlimited storage provides for near-infinite scalability, and geo-redundancy provides disaster protection.  
  • Object versioning offers a convenient way to snapshot, update, and delete content, all at the individual tile level. 
  • By combining versioning with storage classes, object lifecycle management can transition tiles automatically from the high-performance tier to back-up and archival storage, managing costs and preserving history.
  • GCS supports hosting static content and tiles are just that, static content.

Even though GCS uses a flat namespace, we leverage the virtual hierarchy to map tiles to well-known paths in the tile pyramid. Finally, by placing a cloud load balancer in front of GCS, we get HTTPS support and the custom domain name of our choice. And, with the flip of a switch, we can enable Cloud Content Delivery Network (CDN) to cache tiles near our users for added performance.

Step by Step

You can host tiles on GCS with the goal of later consuming them from a variety of web and desktop clients. You will encounter common bash commands and vector tile utilities below. Since the focus is on GCS, the details of the ancillary commands are not explained.

Bake the Tiles

For example purposes, we are looking at contour data from Hamilton County, Ohio. Available as an Esri file geodatabase (topography.gdb.zip), the 600Mb file took roughly 10 minutes to download.

# fetch the data
#   - Hamilton county 2 ft contours, 60M, ~10 minute download
curl -o topography.gdb.zip http://cagis.org/Opendata/topography.gdb.zip
unzip topography.gdb.zip

# convert to GeoJSON
#   - filter to 50 ft contours only
ogr2ogr \
    -f GeoJSON hamilton_contours_50ft.geojson topography.gdb HAM_CONTOUR_2017 \
    -select ELEVATION \
    -nln contours_50ft \
    -where "ELEVATION%50=0" \
    -lco DESCRIPTION="Hamilton County, OH, 50 foot contours" \
    -t_srs EPSG:4326

# bake tiles
#   - uncompressed tiles as some clients do not support compressed tiles
#   - output-to-directory writes .pbf files to tile pyramid in hierarchical folder structure
tippecanoe \
    -z11 \
    --no-tile-compression \
    --output-to-directory hamilton_contours_50ft \
    hamilton_contours_50ft.geojson

At this point, the tile pyramid is built on a local disk. It consists of a set of .pbf files (2.6M) that follow the vector tile spec and the elevation attribute is embedded for each geometry. In addition, tippecanoe creates a metadata file in json format at the root of the pyramid. The tile pyramid ranges from zoom levels 0 to 11 and extends spatially across the Hamilton County, OH area.

mceclip2.png

On to GCS

Next, we created the cloud project, created the bucket, and configured that bucket with some common settings for hosting a static website. We chose the project name geoaccelerators, which appears in the bucket name. For static website hosting, the bucket name serves as the host portion of the URL, so geoaccelerators becomes the subdomain.

# create project
gcloud projects create geoaccelerators --folder=YOUR_FOLDER_ID
gcloud config set project geoaccelerators

# create bucket
gsutil mb gs://geoaccelerators.woolpert.dev

# grant public read access to bucket
gsutil iam ch allUsers:objectViewer gs://geoaccelerators.woolpert.dev

We configured CORS with a json file and allowed access from all origins.

# cors_settings.json
[
 {
   "origin": ["*"],
   "responseHeader": ["Content-Type"],
   "method": ["HEAD","GET"],
   "maxAgeSeconds": 3600
 }
]

# configure bucket CORS setttings
gsutil cors set cors_settings.json gs://geoaccelerators.woolpert.dev

Next, we uploaded the tile pyramid to GCS and set appropriate metadata. The folder structure was retained as a virtual hierarchy on the GCS side. Here, we set the max age for caching to 0 (no caching) for testing purposes, but this value should be changed for production to take advantage of caching. If left unset, cache control defaults to 3600 seconds (one hour).

# upload tile pyramid
gsutil -m cp -r hamilton_contours_50ft gs://geoaccelerators.woolpert.dev/tileservices

# metadata for content type and cache
gsutil -m setmeta -r -h "Content-Type:application/vnd.mapbox-vector-tile" \
    gs://geoaccelerators.woolpert.dev/tileservices/hamilton_contours_50ft/**/*.pbf
gsutil -m setmeta -r -h "Cache-Control:no-cache, max-age=0" \
    gs://geoaccelerators.woolpert.dev/tileservices/hamilton_contours_50ft/*

A quick word on compression: GCS supports decompressive transcoding. This means that files stored with gzip compression in GCS will be decompressed automatically at request time. This appears to be a nice fit for our needs because it saves costs while satisfying clients' requests for uncompressed vector tiles.

Finally, we placed a cloud load balancer in front of the GCS bucket and received support for the HTTPS protocol as well as easy-on Cloud CDN. The details of the set-up can be found in Google's documentation (starting at Reserving an external IP address).

mceclip3.png

Visit the next part in this series, where we take our vector tiles for a test drive in a couple of modern web mapping clients.

Related articles

Comments

0 comments

Please sign in to leave a comment.