How can cloud-centric data marketplaces help with managing data at scale?

Data is the lifeblood of financial analysis, and cloud-based data distribution platforms are creating new options for selling, procuring, and managing data at scale. I recently participated in a great discussion with a panel of senior cloud data executives to explore how firms are leveraging these new data options and the challenges and opportunities they face. The group covered a range of topics, including typical use cases, real-time data concerns, and access controls.

Use cases: more than just cost savings

Cloud migration is often associated with the potential for capital and operational cost savings, and cloud-based data distribution is no different, as companies can more easily pay for just what they need. But far more valuable are the abilities to access a broader range of data and to make the data commercially valuable as quickly as possible. Our moderator, Peenaki Dam from ICE Data Services, summarized this when he said “the value is much more in the network effect [of data] … more than just saving infrastructure or storage costs”. Cloud-based marketplaces and distribution systems are helping by making a lot more data available, and by automating many of the steps involved in consuming that data. The resulting network of datasets is giving companies greater agility and fueling new areas of creativity and innovation that contribute to competitive advantage. An interesting analogy that we used was like Netflix for data, but with a twist. Not only can you watch a movie on any device at any time, but you can also edit that movie or create and promote your own film.

Facilitating real-time data

Real-time access and the “speed of light” problem is an ongoing data issue in financial services. Companies are inventing new ways to stream and ingest data, with varying degrees of latency as needed by their clients. The cloud is very helpful for this problem, because you can co-locate data and processing on the cloud provider’s very fast internal network and reduce the amount of friction involved in data access. “There is no better way to beat the speed of light problem than to not have to move the data at all,” said Matthew Glickman of Snowflake. While there will certainly continue to be use cases that require special hardware and networking connections for speed-of-light processing, a lot of the applications that we are seeing need “eyeball latency”, or some sort of visualization and decision processing in human time. This is a sweet spot for many data and application providers, as they explore the commercial value of faster access to both new and existing datasets.

Managing access and entitlement

When moving from on-premise to cloud-based data, managing permissions, access controls, and data usage are common concerns. Are we just adding another layer of complexity to data entitlements? In the cloud world this is rapidly becoming a solved problem. When you log in to Netflix or other cloud services, your access and entitlements are effectively determined by your identity, who you login as, not by your device. And your entitlement is checked each time you login or access the data, not just for the initial download. And as Christian Schneider of Amazon Web Services said, “The tools for security and identity and access management, fine grain controls, logging, reporting, and tracking are going to continue to advance and get better.” This is giving both data owners and data consumers greater control over their usage, and creating new opportunities for making data available by user or role.

Performance, latency, security

“The disciplines [of cloud data] are the same as they were 20 years ago…performance, latency, security,” said Paul Watmough of Iowa Rocks. These are all areas where cloud hosted environments excel over traditional, on-premise data centers. Instead of being limited by the number of on-premise processors and having to schedule analytic workloads on evenings or weekends, quants and data scientists can now access seemingly limitless compute and data resources and only pay for what they use. Both the distance and friction of accessing data is substantially reduced, bringing many more dataset within reach in real-time or near-real-time. And the enhanced security and access controls of the cloud make it easier to share data that should be shared more broadly, and restrict access to data that is confidential or proprietary.

This summarizes our broader discussion, and I would like to thank our moderator and panelists who made it such an entertaining and informative call. To learn more about cloud data distribution and how firms are leveraging these new platforms to integrate data faster, check out the full panel discussion on Beacon’s YouTube channel.

About the panelists

Peenaki Dam, Head of Product Strategy and Innovation, ICE Data Services

Financial services product leader with in-depth expertise in next generation technologies and product design. Peenaki, who moderated this panel, is responsible for the development of new data delivery capabilities at ICE Data Services, including leveraging cloud and emerging technologies to meet the evolving needs of the quant trader segment.

Matthew Glickman, VP Customer Product Strategy, Snowflake Cloud Data Platform

Financial industry veteran turned passionate cloud evangelist leveraging the public cloud revolution to create the Cloud Data Exchange. Matthew is thrilled to help shape a world-class product that not only empowers enterprises to embrace the Data Cloud but also enables them to connect to one another in the world’s only cross-cloud Data Marketplace!

Christian Schneider, Global Business Development Manager, Financial Services and Hedge Funds, Amazon Web Services

Cloud services advisor and sales executive, growing teams, driving revenues, and transforming customer business models. Christian is the leader of a sales/business development team focused on growth of key strategic relationships in the capital markets industry.

Paul Watmough, Founder & Executive CEO, Iowa Rocks Global Data Marketplace

Strategic thinker who excels at identifying business opportunities and building profitable partnership networks. Paul and the IOWArocks team shared a dream which was to liberate the financial services industry from the restrictions and limitations imposed by the long-established data vendor monopolies. That dream is now a reality.