Using events for secure, near-real-time integration of measurement and related data from the EnergySys platform

Energy companies operate sophisticated technical infrastructure with complex requirements, and they need to share data securely across application and company boundaries. These companies are increasingly focused on achieving agile operations, responding rapidly to change, and developing new solutions in an evolving business context, including energy transition. However, conventional approaches to data sharing are often cumbersome, time consuming, and error prone. Many products still use proprietary file formats for data exchange, and support for comprehensive application programming interfaces (APIs) is generally lacking. Securing data exchange also presents real difficulties, particularly for data sharing between organizations.

Even when high-quality web APIs exist, there are certain types of data that are produced at irregular and unpredictable intervals or in high volumes over short periods. Using a web service to retrieve this kind of data makes it necessary to resort to strategies such as polling, combined with retrieval patterns that limit the amount of data returned in each call. These strategies increase the implementation time and complexity of retrieving the data through the service. While the load from polling from a single client is rarely problematic, having hundreds or thousands of customers constantly checking for new data generates load and costs.

An example of this irregularly provisioned data is fluid samples sent from oil and gas fields to labs for analysis. The results are often returned by email to the field operator and stored for use in production reporting and hydrocarbon accounting systems. Knowledge of the composition of fluids produced is an essential part of verifying that each party within a system receives their appropriate allocation of produced fluids. However, this same sample analysis is required in many other parts of the business, including the metering systems that run in the field and measure flows and volumes.

Another example is the generation of security events in cloud services used by a business. In this case, the timing of events is irregular, but the real challenge is the unpredictable volume of events. Making calls to retrieve the data, even using changed data flags, creates unnecessary load and requires the management of code for scheduled polling and retrieval.

Making data sharing easier for customers with EnergySys

EnergySys Limited (EnergySys) is an AWS Energy Competency Partner that offers a low-code application and data management platform for the energy industry. The EnergySys Cloud Platform is a low-code to no-code platform for building enterprise-grade technical business applications, and it is available through AWS Marketplace, a place to find, buy, and deploy software that runs on AWS. The low-code environment offers dynamically generated user interfaces that help customers to configure both objects and calculations and to create solutions quickly and easily without code. The EnergySys Cloud Platform is used for production reporting, production forecasting, meter management, greenhouse gas emissions tracking, marine vetting, and invoicing. It offers a range of tools to store and calculate data, including the ability to run queries with data products using an open standard web API. These essential features are designed to support businesses in creating tools that facilitate efficient, dynamic processes. However, some data is updated infrequently or on an unpredictable schedule, and having to poll the EnergySys Cloud Platform for changes can be inefficient.

To address this challenge, EnergySys is the first independent software vendor (ISV) in the energy space to offer a Partner event source integration for Amazon EventBridge, a service that helps to build event-driven applications at scale. Available now in the AWS Marketplace, the Amazon EventBridge integration allows EnergySys customers to add a source to their account from within their own AWS account console. If they have the appropriate configuration set up in EnergySys, they will be able to receive and process data within minutes. EnergySys currently supports the publication of lab analyses in JSON format, but the integration has been designed to make it possible to add any configured EnergySys Cloud Platform object to the bus, including audit events. This ability supports the EnergySys commitment to open standard–based access to data, avoiding vendor lock-in and supporting customers in making integration choices that best suit their applications.

For customers using software as a service (SaaS), there is the opportunity to use Amazon EventBridge SaaS integration services to securely and efficiently receive data in their own accounts. New events, like a lab sample or a security event, are sent as they occur, and the scalability of Amazon EventBridge helps customers to process the data at their own rate. Further, the Amazon EventBridge integration’s pattern-matching capabilities mean that multiple event types can coexist on the same event bus and can be processed differently based on the receiving service’s requirements.

The addition of this new Amazon EventBridge SaaS integration is an important component in helping customers to maximize the value of their data. As depicted in Figure 1 below, customers can receive events from EnergySys and process them with AWS Lambda—a serverless, event-driven compute service—to update other systems. Alternatively, customers can use machine learning (ML) tools and data lake technology to combine the data, building broad-based datasets for analytics with Amazon QuickSight, which powers data-driven organizations with unified business intelligence (BI) at hyperscale, or Amazon SageMaker, which helps to build, train, and deploy ML models for any use case.

Figure 1 – an example of a client-side reference architecture of the integration

Working with AWS Partner solutions architects, the EnergySys team was able to successfully deliver the first release of the Amazon EventBridge SaaS integration within a matter of weeks. And the process for setting it up is straightforward:

1. The user logs onto the EnergySys Cloud Platform and chooses to share selected objects on the Amazon EventBridge.

2. The user then logs onto their own AWS console and selects “EnergySys” from within the Amazon EventBridge page. The user then selects the menu on the left-hand side. Within the menu, the user selects “Integration,” then “Partner event sources,” then “Setup,” which will guide the user to complete the subscription using a webpage hosted on EnergySys.

3. On the subscription page, the user adds his or her AWS account number, desired region, and object feed. As of the publishing of this blog, the feed with lab sample data is the only one available. In the future, new feeds will be available, and users will be able to add other events to the same bus.

4. On submission of the request, a confirmation page will be displayed.

5. The EnergySys service validates the request and creates a Partner event source for the account, region, and object type requested. Once this process is complete, customers are notified by email and are then able to see the Partner event source in their AWS console under Amazon EventBridge > Partner event sources.

6. The status of the newly created Partner event source will be “Pending.” Once the Partner event source is associated with an event bus in the customer account, the Partner event source status will change to “Active.” From this point onward, selected events will flow from the EnergySys Cloud Platform to the destination account.

With the integration complete, customers have complete flexibility in choosing how to interact with the event bus. Events generated by EnergySys Cloud Platform use a specific JSON structure. Every event has the same top-level envelope fields, such as the source of the event, timestamp, and region. This information is followed by a detail field, which is the body of the event. Customers create rules by building JSON patterns that match against the events to decide if the rules should fire. Customers also configure the targets for the rules, which can be API calls or one of several AWS services, including AWS Lambda, Amazon SageMaker Pipelines, Amazon Kinesis Data Firehose—an extract, transform, and load (ETL) service—and AWS Step Functions—a visual workflow service that helps developers use AWS services.

Amazon SageMaker Pipelines are particularly interesting. This feature would help customers to develop their own ML models, hosted in their own accounts, to act on data from EnergySys Cloud Platform or any other SaaS provider that supports the Amazon EventBridge Partner integration. For example, a custom model might look at measurement data over time from hundreds of wells in a single area to determine if there are any anomalous readings.

Conclusion

Customers really understand SaaS benefits, such as cost reduction, scalability, reliability, and infrastructure management. EnergySys Cloud Platform accelerates data accessibility and saves customer time spent on building custom solutions. EnergySys is committed to giving customers secure, open access to their data through standard APIs and integrations. The addition of Amazon EventBridge SaaS partner integration is a new, powerful tool to help with this goal. The ease of integration, combined with the power and scalability of Amazon EventBridge and the range of AWS service targets, makes this new offering a useful option for many customers. Reach out to EnergySys to connect for your use case.