How Natural Resources Canada migrated petabytes of geospatial data to the cloud

With the second largest landmass on Earth and the longest coastline in the world, Canada relies on satellite technology to monitor its geography. Providing access to this technology is a core mandate of Canada Centre for Mapping and Earth Observation (CCMEO) at Natural Resources Canada (NRCan). Since 1971, CCMEO has accumulated an Earth observation (EO) data archive in excess of two petabytes (PB). Today, the NRCan ground segment consists of four satellite antennas located in Inuvik, Prince Albert, and Gatineau, as well as the Earth Observation Data Management System (EODMS) and Emergency Geomatics Service (EGS).

NRCan wanted to modernize its geospatial offerings at a faster pace, so they turned to Amazon Web Services (AWS) to harness the agility of the cloud to be more responsive to stakeholder needs. NRCan used the AWS Snow Family to migrate the data.

Accessing the AWS Snow Family through the Government of Canada

In 2019, AWS signed a framework agreement with the government to host Government of Canada (GC) workloads that are classified up to Canada’s Cloud Protected B, Medium Integrity, Medium Availability (PBMM) Profile. GC customers can use AWS Snow Family to transfer data up to PBMM, provided the destination AWS accounts to which they are uploading the data have PBMM guardrails in place and the department has received an Authority to Operate (ATO) against the PBMM profile.

The AWS Snow Family helps customers that need to run operations in austere, non-data center environments, and in locations where there is a lack of consistent network connectivity. AWS Snow Family, comprised of AWS Snowcone, AWS Snowball, and AWS Snowmobile, offers a number of physical devices and capacity points, most with built-in computing capabilities. These services help physically transport up to exabytes of data into and out of AWS. Snow Family devices are owned and managed by AWS and integrate with AWS security, monitoring, storage management, and computing capabilities. The AWS Canadian Region currently offers the AWS Snowball Edge and AWS Snowcone devices. NRCan opted for the AWS Snowball Edge for their solution, which is a ruggedized shippable device best suited for local storage and large-scale data transfer. AWS Snow Family devices can assist with data migration, disaster recovery, data center shutdown, and remote data collection projects.

Collaborating with GC’s Shared Services Canada to migrate

For NRCan’s data transfer solution, the team needed to document the processes for receiving the Snowball Edge devices at the data center, installing the device, connecting it to the network and then finally returning the devices to AWS. Process documentation was important for the data center personnel to efficiently manage the migration using Snowball Edge devices. NRCan and AWS worked with the GC’s central IT organization, Shared Services Canada (SSC) to orchestrate all the data center activities in support of this migration. Once the workflow was documented using the first Snowball Edge devices, subsequent orders followed until the entire 1.4PB of data was migrated over the course of six months, completing in March 2022. NRCan ultimately used a total of 16 Snowball Edge devices to complete the migration. Because they documented the entire process, other GC customers who need to migrate data from SSC-managed data centers to Amazon Simple Storage Service (Amazon S3) can repeat the same process.

Setting up the Snowball Edge

SSC’s data centers contain a multi-tenant virtualization platform. SSC maintains a rigorous process to evaluate and approve the installation of software packages on SSC supported virtual machines (VMs).

Customers should always review the best practices for using the AWS Snowball Edge to support a migration. In NRCan’s case, they decided to use two separate command line interfaces (CLI) to operate a Snowball Edge for data transfer. The Snowball Edge CLI was used to access the Snowball Edge management APIs. This client was run on a departmental laptop with network access to the APIs on the Snowball Edge. However, to normally copy data to the Snowball Edge from a network attached storage (NAS) device, NRCan needed to install the AWS CLI on a VM that could Network Filesystem (NFS)-mount the storage. The AWS CLI is used to issue the Amazon S3 commands to perform the data copy to the Amazon S3 API on the Snowball Edge installed in the SSC data center.

AWS Snowball Edge (and AWS Snowcone) come with a mix of compute and storage that has been assessed by the Canadian Centre for Cyber Security (CCCS) for use with PBMM data. NRCan used an Amazon Elastic Compute Cloud (Amazon EC2) instance on Snowball Edge with a supported Amazon Machine Image (AMI), and NFS-mounted the network share containing the data to be transferred. The data copying was then initiated from the Amazon EC2 instance to the Amazon S3 bucket on the Snowball Edge without having to install anything in the virtualized compute environment. This configuration allowed the migration to start without delay.

NRCan avoids blockers on Migration Day

Any team knows how stressful it can be to migrate an operational system from its current environment to any new place, and cloud is no exception. Decisions need to be made quickly, procedures can go wrong, and sometimes teams must change course. On the day of their migration, NRCan encountered an issue: an Amazon EC2 node had only been provisioned with 8GB of ram instead of the desired 16GB and the application was choking as a result. But in only a couple of minutes and with a few clicks in the AWS console, the node was scaled appropriately and the migration continued smoothly. An issue that might have taken a year to resolve in their old on-premise environment became a one-minute fix on the day of NRCan’s migration using AWS, which helped NRCan finish their move on schedule.

EODMS also uses the AWS Registry of Open Data

Launched in 1995, the now inactive RADARSAT-1 was Canada’s first commercial Earth observation satellite. It utilized synthetic aperture radar (SAR) to obtain images of the Earth’s surface to manage natural resources and monitor global climate change. NRCan wanted to make RADARSAT-1 data available for collaborators under an open license, so NRCan turned to the Registry of Open Data on AWS.

The Registry of Open Data on AWS makes it simple to find datasets made publicly available through AWS services. The AWS Open Data Sponsorship Program covers the cost of storage for publicly available high-value cloud-optimized datasets. The open RADARSAT-1 dataset includes historical images of Earth from the satellite’s 17 years of operation, which have proven invaluable for climate change research.

Conclusion and learn more

NRCan’s migration experience shows that Government of Canada customers can use the AWS Snowball Family of devices to migrate petabytes of data to AWS. Migrating data to AWS and leveraging programs like the AWS Open Data Registry can help GC researchers collaborate and accelerate time to new discoveries.

EODMS continues to evolve its ground segment data services in line with the Government of Canada’s Directive on Service and Digital. For more information, contact eodms-sgdot@nrcan-rncan.gc.ca. If you would like to learn more about using the AWS Snowball Family of devices, reach out to your AWS account manager, or contact the AWS Public Sector team directly.

AWS Public Sector Blog