Optimize data analytics in capital markets with time-series databases

Capital markets firms use vast amounts of historical and streaming market data to perform real-time analytics. They also use data to inform decision-making, from projecting asset-pricing movement and fund values to calculating risk assessments. However, traditional on-premises relational databases and data management systems can fall short on delivering the functionality, agility, speed, and cost efficiency that these institutions demand. In this post, I discuss some of the principles in the recent AWS Marketplace webinar, How to optimize data analytics in capital markets with time series databases on AWS.

Time-series data drives critical use cases in capital markets

A time-series database (TSDB) stores, catalogs, and retrieves data records that are part of a time series, or a set of data points associated with timestamps. The timestamps provide critical context for each of the data points in how it relates to others. TSDBs are purpose-built and designed to simplify the analysis of large volumes of data over blocks of time. Much of the success in using TSDBs comes from inserting new data by time horizon or granularity. Time horizons are periods where investments are held until they are needed. This new data insertion helps enterprises to generate timely results and support complex risk analysis or trading model backtesting. Backtesting is a method of evaluating the effectiveness of a trading strategy by running it against historical data to see how it would have fared.

TSDBs on AWS enable customers to innovate and scale your data-processing and analysis capabilities without requiring significant capital expenditure (CapEx) commitments. With this operational model, investment firms can simplify how they capture and monitor data and deliver improved market analysis and models in a timely fashion. TSDBs and other financial services solutions are available in AWS Marketplace, a digital catalog of independent software vendors that gives AWS customers a way to discover, buy, and deploy software that runs on AWS. Financial services customers can deploy TSDBs and start experimenting without the overhead of acquiring new hardware and with the comfort of pay-as-you-go (PAYG) services.

Time-series data requires special handling

Building new trading scenarios requires capturing all of the activity from the stock exchanges and trading venues, which presents as market data for analysts and researchers. The market data, when including the full order book, can exceed several terabytes per day. A full order book is an electronic registry of buy and sell orders organized by price level for specific securities.

The volume of this data creates an immediate ingestion challenge for database administrators, who need to consume the data and split the content into numerous databases, which adds to architectural complexity.

Although file systems can store the data, most have limited analytical capabilities. Relational databases can’t process such amounts of data at speed. NoSQL databases can solve the scalability challenge, but they have minimal analytical capabilities and can’t store time-series data efficiently due to inadequate compression.

TSDBs deliver the fastest ingestion rates with efficient storage by compressing data, preserving its coherences and lineage, then storing it in a form that dramatically simplifies for further analysis. This removes the need to split the data manually and enables transparent querying of the complete dataset. Its querying speed ensures this because it ingests gigabytes per second.

The cloud amplifies the benefits of time-series databases

Capital markets firms have been conducting time-series calculations for risk management, regulatory compliance, and product development for years. However, both the volume of these calculations and the amount of data have increased at a dramatic rate. As a result, firms have been migrating their on-premises compute farms to the cloud, where they can use on-demand capacity and scalability as well as pay-as-you-go services.

When there is limited compute capacity, as there is on-premises, analysts compete for compute time. This creates contention in the architecture and leaves them unable to act on new signals or test their research ideas. TSDBs in the cloud are ideal for specific and intermittent compute-intensive workloads, such as on-demand risk calculations in reaction to market events in real time. By offloading data-center overhead, you can focus resources on research, risk modeling, and building new investable assets for your customers.

Migrating TSDBs to the cloud provides access to virtually unlimited infrastructure, including on-demand burst capacity. This evolution has enabled financial institutions to organize and store terabytes of data and use thousands of compute instances for short periods of time. TSDBs on AWS simplify the access to data and give the quants and analysts the compute capacity to perform their calculations on demand without waiting for data access or compute grids. For example, Jefferies Group shared in the webinar that they achieved significant improvements in query performance in addition to cost reduction and process scalability.

Click-and-deploy solutions enable you to do more with your data

Solutions from AWS and technology partners such as OneTick, Kx, and QuasarDB enable you to load, store, and analyze TSDBs. These solutions, deployed on AWS, offer storage that can handle the transaction-intensive workloads, tools for real-time analysis, and data streaming capabilities to capture events as they occur.

As an example, one Kx customer, a global hedge fund, required a tool capable of analyzing trade data, tick data, and order book information in a single, scalable platform. A tick is a measure of the minimum upward or downward movement in the price of a security, and an order book lists the number of shares being bid on or offered at each price point. They needed to consolidate and replay data so that they could reconstruct trading events at a given point of time, to determine the effectiveness of their trade algorithms and demonstrate execution and compliance. The hedge fund turned to Kx’s kdb+ solution to conduct cost-effective, on-demand analysis. In creating an enriched National Best Bid and Offer, the hedge fund gained insight from multiple exchanges and got a market-wide view of liquidity.

For more information about managing and unlocking the value of time series data in the cloud, and to hear first-hand from Jefferies on their adoption of TSDBs in AWS, see the on-demand webinar.

About the author

Brian Cassin is a capital markets specialist on the Worldwide Financial Services team at AWS, where he leads business and market development efforts in North America. He provides support to investment banks, broker dealers, asset managers, and hedge funds in transforming their existing businesses with new, innovative solutions leveraging AWS services. Brian has more than 20 years of experience developing solutions for capital markets companies and holds a BA in economics and an MBA in finance from Fordham University.