What is Data Archiving?

Data archiving is the process of moving data that is no longer in use into long-term storage, often in a more compact format. Organizations mainly use data archiving for record-keeping and compliance. Most archived data will not be retrieved before eventual retirement. Data archiving differs from data backups in that you use the data for reference instead of recovery. The data archiving lifecycle is the framework for managing data as it moves through archival systems.

Data archiving and data backups are both operations that involve storing data, but they serve different purposes.

Data archiving

Data archiving is the process of moving inactive data from primary storage to long-term storage for preservation. The goal of archiving is to make data available for historical reference and record-keeping. For example, archived data can consist of fixed records that are no longer actively used but must be kept for years or decades to meet regulatory and legal requirements. Archived data is immutable: after it is stored, it must not be altered.

Data backup

Data backup is the process of creating a point-in-time copy of actively used data. The goal of a data backup is to make the active data available for recovery in the case of a data loss event. For example, you can restore lost or corrupted files after a hardware failure or accidental data deletion. Because active data changes frequently, backups are mutable, meaning they can be regularly updated or overwritten.

Data archiving and data warehousing both involve storing large volumes of information, but they use different IT resources and have entirely different use cases.

Data archiving

Data archiving is cold storage for older, inactive data. You move data out of your primary systems into archives to free up space, reduce costs on infrastructure that is more expensive to maintain, and meet compliance requirements for data holding times. You store archived data on cost-effective infrastructure designed for long-term retention, not for active querying and analysis.

Data warehousing

A data warehouse stores massive amounts of data from multiple sources in a highly structured, query-optimized format. Data warehouses are designed for active data analysis. You use a data warehouse to run complex analytics, generate business intelligence, and quickly retrieve aggregated results. The data in a data warehouse is in use or ready for use in the near future.

Read about data warehousing »

Data storage systems designed for active, daily use can be costly and complex to maintain. This type of data storage is called primary storage. As the volumes of data that businesses use continue to grow, it can become inefficient to keep all data in primary storage. Establishing a structured data archiving strategy solves several core operational challenges for IT and data management teams:

Legal and regulatory compliance

Many industries have strict regulations requiring organizations to retain records for years or even decades. A data archive, when configured properly, can provide the immutable, secure storage necessary to meet legal standards, such as HIPAA or GDPR. The data archive makes sure that historical data is readily available for any audits or discovery requests that arise.

Cost reductions

Primary storage provides high-speed access, making it the most expensive tier to maintain. Identifying and moving inactive data to secondary, colder storage environments can significantly reduce storage costs.

Disaster recovery and business continuity

When archived data is configured for storage in a read-only, immutable format, these mechanisms help guard against events such as data theft, ransomware attacks, or accidental deletion.

Data lifecycle management

Archiving is a practical, necessary phase in managing information from its creation to its eventual disposal. Systematically moving older data out of active systems keeps them uncluttered and helps maintain system performance.

By archiving static, historical information, you also effectively remove it from your scheduled and daily backup rotation. Shrinking the size of your active datasets makes your routine backups faster, more reliable, and easier to manage over time.

Modern data archiving tools manage data in four phases:

Data classification and tiering

Data archiving is only effective if you first classify and “tier” your data. Tiers are different data storage categories defined by access needs:

• Hot: Frequently accessed primary storage
• Warm: Occasionally accessed secondary storage
• Cold: Rarely accessed archive storage

Administrators create automated data lifecycle policies that trigger the archiving process in their storage systems, moving data from hot to cold tiers. Rules for moving data between tiers are based on specific criteria such as file age, a drop in access frequency, or file size.

Ingestion and transfer

After data meets a data-shifting policy criteria, it enters the ingestion and transfer phase. Depending on an organization's needs, its archival system either moves data in scheduled batch transfers during off-peak hours or through continuous archival pipelines. Continuous archiving occurs for real-time applications such as compliance logging or transactional audits. Before the transfer occurs, the system typically performs compression and deduplication to minimize the size of the data's footprint. Compression helps maximize storage efficiency and typically reduces long-term costs.

Indexing and metadata management

Because retrieving raw archived files can be slow and costly, archiving systems build a searchable index before the data is stored. This involves attaching metadata, or labels, such as file type, creation date, author, and topic. Archiving systems use a metadata schema to ensure data is easily searchable. Archival systems can also apply automated tags and catalog entries that describe each file's contents. When an administrator needs a specific record, they can query this index without touching the underlying archive.

Retrieval

Because archive storage is optimized to be cost-effective, access is governed by retrieval latency tiers. Depending on the selected storage class, retrieving data might take from milliseconds to a few minutes or up to hours or days. Modern archiving solutions also allow for partial restores, so you extract only the specific files needed rather than restoring an entire data volume.

Data classified for long-term storage generally falls into one of three main categories:

Structured data

Structured data is highly organized and strictly formatted, making it easy to search and query, such as in a relational database. Common examples of archived structured data include historical database records, transaction logs, and financial records that need retention for multi-year compliance audits.

Read about structured data »

Read about relational databases »

Unstructured data

Unstructured data is any data that does not follow a predefined data model. The vast majority of business data falls into this category, such as scanned documents, messages, internal documents, and large media files, such as images, audio recordings, and videos.

Read about unstructured data »

Semi-structured data

Semi-structured data does not reside in a strict relational database, but it does contain internal tags, metadata, or structural markers that help organize the data. Archived semi-structured data often includes old JSON and XML logs, historical IoT (Internet of Things) sensor telemetry, and raw event log streams. Nonrelational databases, object stores, and data lakes are the systems typically used to store semi-structured business data.

Read about JSON »

Read about XML »

Read about nonrelational databases »

A data archiving service operates on several key principles.

Immutability

Data cannot be altered or overwritten after archiving. To follow the principle of immutability, most enterprise data archiving solutions use the Write Once, Read Many (WORM) storage model.

Durability

The storage environment is designed for the secure, long-term retention of records over many years or decades.

Retrievability

Archived data is thoroughly indexed so administrators can easily search the archive and selectively restore data.

Cost efficiency

The system uses low-cost, cold storage tiers optimized for holding massive volumes of inactive data.

Restricted access controls

Strict security permissions mean that only authorized personnel can access or retrieve sensitive historical records. Audit logging shows which users accessed which data.

The data archiving lifecycle helps ensure information is handled accurately from when it leaves primary storage until it is permanently deleted.

1. Identification: You classify your data based on its value, sensitivity, and specific retention requirements.
2. Define policy: You establish any necessary rules for the data, including retention periods, compliance mandates you need to follow, and access tiers for different personnel to read and retrieve it.
3. Data migration: Your data archiving service automatically moves the data from your primary storage system into designated archive storage.
4. Verification: You and the service confirm data integrity using methods such as checksum verifications.
5. Management: The data archiving service maintains the archive through ongoing indexing and access control, and applies legal holds when necessary.
6. Disposal: The system securely deletes the data after its required retention period ends, as defined by your archiving policy.

Before moving data into a long-term archival service, organizations must evaluate several critical factors to help ensure the archive meets their operational and regulatory needs:

Compliance and legal holds

Regulatory requirements for data retention vary by industry and region. For example, regulations such as GDPR, HIPAA, PCI-DSS, and SEC 17a-4 all dictate how long records must be kept and secured. Additionally, the archiving system needs to support legal holds to preserve historical data during active litigation or audits.

Retrieval Service Level Agreements (SLAs)

Evaluate the tradeoff between total storage cost and retrieval speed. For example, you can reduce data archiving costs by utilizing slower, deeper storage tiers for datasets that you do not expect to regularly access. However, you must establish clear retrieval SLAs so that when a request does arrive, you can restore the data within the specified timeframe.

Read about SLAs »

Storage format and longevity

When planning for multi-decade retention, you must consider the risk of format obsolescence. A proprietary file format used today might be entirely unreadable by software systems 20 or 30 years from now.

Security

A secure archive needs to have encryption for data at rest and in transit. Administrators must also implement role-based access controls and maintain detailed access logging to mitigate potential data tampering.

To maintain a secure and efficient archive, consider adopting the following best practices:

• Define your data retention policy: Document what data needs to be archived, how long it must be kept, and when it is time to destroy the data.
• Automate lifecycle transitions: Use automated rules to move data from primary storage to the archive tier.
• Use open data formats: Store your archives in standard, open file formats to guarantee you can read the data decades from now.
• Use encryption and access controls: Encrypt data at rest and in transit. Enforce a Zero Trust access control model to limit who has permissions to use the archive.
• Test retrieval periodically: Schedule periodic test restores.
• Maintain audit logs of access and modifications: Record who accessed the archive and when.

AWS offers a range of data archiving services designed to suit your data retention policies and storage class requirements:

• Amazon S3 Glacier is purpose-built for data archiving, providing you with the highest performance, most retrieval flexibility, and the lowest cost archive storage in the cloud. You can choose from three archive storage classes optimized for different access patterns and storage durations.
  • For archive data that needs immediate access, such as medical images, news media assets, or genomics data, choose the S3 Glacier Instant Retrieval storage class.
  • For archive data that does not require immediate access but needs the flexibility to retrieve large sets of data at no cost, such as backup or disaster recovery use cases, choose S3 Glacier Flexible Retrieval.
  • For long-lived archive storage such as compliance archives and digital media preservation, choose S3 Glacier Deep Archive, the lowest cost storage in the cloud, with data retrieval within twelve hours.
• Amazon S3 Intelligent-Tiering storage class is designed to optimize storage costs by automatically moving data to the most cost-effective access tier when access patterns change.

Get started with data archiving on AWS by creating a free account today.