Data is increasing exponentially. The amount of data generated daily will reach 463 exabytes globally in three years. To put that in perspective, it is estimated that all of the words ever generated by human beings total five exabytes.
To succeed in today’s digital economy, enterprises are implementing strategies to consume and analyze all of this data to make the right business decisions in real time. By 2025, it’s estimated that the average number of daily digital interactions per connected person will reach 5,000. Enterprises need to be able to meet this increasing demand for real-time data, processes, and user experiences.
One strategy is to use an event-driven architecture. Event-driven architecture is a software design pattern that allows enterprises to act in real time on business events such as customer transactions. Event-driven architecture is not a new concept; it’s been around since the 1970s. However, the technology hasn’t existed to truly serve business and consumers in real time until much more recently.
Event-driven architecture is now in the limelight and able to solve modern business problems and deliver amazing consumer experiences because we finally have the necessary capabilities.
How event-driven architecture works, and its benefits
Event-driven architecture is based on events, which signal to interested parties that a state change occurred in a business system. For example, an event could be an online purchase, the shipment of the product, or the delivery of the product to the customer’s doorstep. Events are happening all the time across all industries.
In event-driven architecture, there are producers and consumers. Producers trigger events sent to interested consumers as messages through an event channel, where they are processed asynchronously. Producers and consumers don’t have to wait for each other to begin the next task, as producers are loosely coupled or completely decoupled from recipients.
Decoupling also improves scalability, as it separates communication and business logic. A publisher can avoid bottlenecks and remain unaffected if its subscribers go offline or their consumption slows down. If a subscriber struggles to keep up with the pace of events, the event stream records the events for future retrieval. The publisher can continue to send notifications without throughput limitations and with high resilience to failure.
Publish/subscribe (pub/sub) is a common design pattern in event-driven architecture that provides a framework for exchanging messages between publishers and subscribers. A message broker receives all of the publisher’s events and routes them as they happen to the subscribers, i.e., to those who registered to receive them. A broker also records events. Consumers access the event stream at any time, where they can read the most recent message or process the series of messages received since the last time they checked the stream.
With a message broker, a publisher does not know its subscribers and is unaffected if the number of interested parties scales up. Publishing to the broker offers the producer the opportunity to deliver notifications to a range of consumers across different devices and platforms.
Challenges of implementing an event-driven architecture
Building the capabilities into applications to deliver real-time experiences at scale is risky, complex, costly, and time-consuming. Organizations could spend months initially putting together a custom solution with off-the-shelf components. This could subtly turn into years, as the team reacts to the complexities that come with scale and reliability requirements.
Many applications rely on a sequence of messages that depend on each other. However, if these messages are lost or unordered, the user experience will be broken and customer data could be damaged or lost in the process. The engineering required to provide real-time digital experiences with data integrity is complex and often leads to unacceptable tradeoffs, such as companies sacrificing performance at scale for data integrity.
There are also performance concerns. Issues with poor or unpredictable latency and high-bandwidth consumption create uncertainty for application developers when designing, building, and scaling real-time features. In addition to minimizing latency and bandwidth requirements, organizations must minimize the variances in these measures and provide predictability to developers for the experience delivered to be competitive. It’s also hard for organizations to design, build, and operate their own globally distributed, fault-tolerant, real-time infrastructure. To achieve fault tolerance, organizations need to have multiple redundant components, in multiple data centers, that can keep the system operational if other components are lost.
When organizations start to scale, they will either reap the benefits or suffer the consequences of how they built their infrastructure. Inadequate infrastructure will fail to scale or provide the elasticity necessary to meet customers’ real-time requirements. Furthermore, for most organizations, building their own infrastructure for an event-driven architecture will only distract them from developing the real-time experiences that will truly differentiate their products. Getting to the point that the infrastructure can be relied upon to deliver a competitive experience will require significant hiring and skills growth within the team that may have little relevance to the core of the business.
Ably pub/sub messaging
To overcome these challenges, the Ably edge messaging platform provides APIs that enable developers to build apps and infrastructure that communicate in real time without the organization having to managing scale, latency, message integrity, or network outages.
A message from a device publishing to Ably will be received in real time by any number of subscribing devices. To do this, Ably organizes message traffic into named channels. Once connected to Ably, clients can be publishers (pushing messages to Ably), subscribers (waiting for messages to be pushed from Ably), or both. While billions of messages may be delivered by Ably, subscribers will only receive the messages on the channels they subscribe to.
Channels offer a way to implement the pub/sub pattern, allowing publishers to push data to subscribers quickly and efficiently. New data is pushed to subscribers, so they don’t have to poll servers to check for new data.
Ably’s presence feature allows clients to announce their presence on a channel. Presence enables developers to build collaboration apps such as chat rooms, multiplayer games, or collaboration tools as Ably automatically keeps track of who is present in real time across any device. Each member present on a channel has a unique client identifier and an optional payload to describe the member’s status, such as entering, updating their state, or leaving the channel. Other devices or services subscribe to these presence events in real time.
Ably is a globally distributed system. Channels can be active in multiple regions independently, so that there is no single point of failure or congestion.
The figure below illustrates how Ably solves the challenge of efficient global routing:
The publisher-only server in New York is routed to the nearest data center (US East) using latency-based routing. Message A published to US East is routed to clients in US East, and once to every other data center that hosts clients subscribed to these messages. Subscribed clients in all other regions will receive Message A from the data center they are connected to exactly once.
The publisher and subscriber client in London is routed to the nearest data center (EU West) using latency-based routing. Message B published to EU West is routed to subscribed clients in EU West, and once to every other data center that hosts clients subscribed to these messages. Subscribed clients in all other regions receive Message B from the data center they are connected to.
The future of real-time experiences
Digital experiences are undergoing a real-time revolution. Consumers demand digital experiences to be instantaneous. As a result, organizations must synchronize data in real time. As more devices come online and businesses shuffle to adapt to a more complex, real-time data economy, they will require easier and more reliable infrastructure to meet the real-time data synchronization needs of today, and in the future, when routine services will rely on data in constant motion.
James Aley is the chief technology officer at Ably, an edge messaging platform to power synchronized, real-time digital experiences at scale.
—
New Tech Forum provides a venue to explore and discuss emerging enterprise technology in unprecedented depth and breadth. The selection is subjective, based on our pick of the technologies we believe to be important and of greatest interest to InfoWorld readers. InfoWorld does not accept marketing collateral for publication and reserves the right to edit all contributed content. Send all inquiries to newtechforum@infoworld.com.