Serverless GIS with Amazon S3, open data, and ArcGIS

If you are hosting an ArcGIS (geographic information system) web app today, then you are probably hosting it on a Windows or Linux server using traditional web server software like IIS or Apache. With the web hosting capability of Amazon Simple Storage Service (Amazon S3) you can remove the need to run these servers and the maintenance, management, and monitoring overhead that comes with it. Serverless services like Amazon S3 can scale automatically and can be as simple as copying over your website assets to get up and running in minutes. This blog focuses on web app implementations using ArcGIS API for JavaScript (as other ArcGIS web apps have additional considerations).

Solution overview

For this solution we are going to build a Serverless ArcGIS API for JavaScript web app that you can use to deliver both static and dynamic content. We use the following services and data stores as part of our deployment:

• Amazon Simple Storage Service (Amazon S3) – Object storage
• Registry of Open Data on AWS – Open dataset catalogue
• Amazon Athena – Interactive query service

To create the web app, upload some HTML and JavaScript files to S3. Enable S3 Web Hosting and setup a default page for our website. Lastly, use Athena to query an open dataset that we can plot on our ArcGIS map. Access to ArcGIS API for JavaScript is no cost for non-revenue-generating apps and requires an account which can be setup here.

Create an S3 Bucket

• Navigate to the Amazon S3 console
• Select Create bucket
• Enter a bucket name
• Uncheck Block all public access
• Check Acknowledge
• Select Create bucket

Setup and Query OpenStreetMap with Amazon Athena

In this step we use Amazon Athena to query an OpenStreetMap (OSM) dataset that is hosted on the Registry of Open Data on AWS. Amazon Athena can be pointed at a dataset that is hosted in a personal bucket or a shared bucket that is part of the Registry of Open Data. This gives you the ability to run SQL queries against petabytes of data. Run a simple query against OSM to get some restaurants between a designated latitude and longitude.

• Navigate the the Athena Console
• Select Get Started
• Select Settings in the top right corner
• Enter the path of your bucket you created (Example S3://mybucket/)
• Select Save
• Paste the below code snippet to create our Athena Table

CREATE EXTERNAL TABLE planet (
  id BIGINT,
  type STRING,
  tags MAP<STRING,STRING>,
  lat DECIMAL(9,7),
  lon DECIMAL(10,7),
  nds ARRAY<STRUCT<ref: BIGINT>>,
  members ARRAY<STRUCT<type: STRING, ref: BIGINT, role: STRING>>,
  changeset BIGINT,
  timestamp TIMESTAMP,
  uid BIGINT,
  user STRING,
  version BIGINT
)
STORED AS ORCFILE
LOCATION 's3://osm-pds/planet/';

• Select Run Query (You may need to refresh the page to enable this button)
• Remove the code above and paste this SQL Query to get some results to plot on our map

SELECT lat, lon, tags['name'] as name from planet
WHERE type = 'node'
AND tags['amenity'] = 'restaurant'
AND lon BETWEEN -97.9863 AND -97.6651
AND lat BETWEEN 30.3531 AND 30.6762
limit 10

• Select Run Query (If you receive an error make sure you see the planet table you created in the previous step, you maybe have to change the database dropdown to locate it)

We have now created a table in Athena and have queried the OpenStreetMap (OSM) data that is publicly available on AWS. When we run a query in Amazon Athena, the results are saved to the S3 bucket we specified previously. In the next step we make our results public so they can be accessed by our serverless website.

Update our dataset permissions

• Navigate to S3 console
• Select the bucket
• Select Unsaved
• Select on the folder with today’s Year, then Month, then Day.
• Select the <filename>.csv that you would like to plot. Ignore the file ending in metadata. (Note: There are multiple files if you ran multiple queries, just share the CSV you want to plot)
• Copy the “Object URL” from the details panel on the right or from the details page of the file.
• Select Make Public

Upload website assets

• Download our ArcGIS API for JavaScript web app
• Open the index.html file with a text editor and update (Line: 48) var url = “” with the “Object URL” we copied above. (Note: replace the https://<bucketname>.amazonaws.com/ with just ./ so the url will just look like “./Unsaved/Year/Month/Day/YourFile.csv)
• Save the file
• Navigate back to the root of the bucket we created.
• Select Upload
• Select Add files and select our index.html file
• Select Acknowledge on the bucket versioning disclaimer
• Select Upload
• Select the index.html file we just uploaded in your bucket
• Select Make public

Configuring your static S3 Website

• Navigate back to the root of your bucket
• Select the Properties tab
• Go to the Static website hosting section at the bottom
• Select Edit
• Select Enable
• For Index and Error document, enter index.html
• Select Save changes
• Go back to the bottom of the page and copy the endpoint that was created

You can access your website by pasting the Endpoint you copied in the previous step into your browser. You can notice a couple of data points plotted on the map. You can update these points by changing your Athena SQL query and referencing the new CSV file that is created. Note you have to mark the newly created CSV files as public. This process could be automated by scheduling a Lambda to make these modifications with an S3 event trigger. You could also use the AWS JavaScript SDK to call Athena directly from your JavaScript website.

Conclusion

You have now created a Serverless ArcGIS for JavaScript web app and have used it to plot publicly available data. Hosting this on Amazon S3 gives your website high availability and durability with little to no operational overhead. You also have the option to use AWS CloudFront as a content distribution network (CDN) in front of your serverless website. This is a best practice as you can use CloudFront to require traffic to be redirected to HTTPS. This also gives you the option to protect your website with AWS WAF, a web application firewall, and AWS Shield, a distributed denial-of-service (DDoS) protection service. Finally, adding a CDN to your website improves performance and latency by caching your website’s content closer to your users. Check out some of the other datasets in the Registry of Open Data on AWS like OpenAQ and GDELT that have latitude and longitude parameters that you can map in the same way.