Steering The IT Convoy Around Data Pitfalls, Perils And Potholes

Cloud Steering The IT Convoy Around Data Pitfalls, Perils And Potholes Adrian Bridgwater Senior Contributor Opinions expressed by Forbes Contributors are their own. I track enterprise software application development & data management. Following New! Follow this author to stay notified about their latest stories.

Got it! Sep 8, 2022, 02:12am EDT | New! Click on the conversation bubble to join the conversation Got it! Share to Facebook Share to Twitter Share to Linkedin Spain’s Antonio Albacete (L) competes ahead of German’s driver Jochen Hahn during the 24 Hour truck . . .

[+] race in the western French city of Le Mans, on Octobrer 13, 2023. Hahn won the 2013 European championship ahead of Albacete after the Le Mans event. AFP PHOTO JEAN-FRANCOIS MONIER (Photo credit should read JEAN-FRANCOIS MONIER/AFP via Getty Images) AFP via Getty Images Data is good.

We all know that modern enterprise organizations run on data and that the digitally encoded streams of information that traverse the cloud and the web enable us to work smarter, faster, better and so on. But data, unfortunately, data also has a bad side, some negative management aspects and more than a few difficult strands of DNA. That bad side is often as simple as identifying data access competencies i.

e. organizations know that they have data inside working company systems, but these blocks of IT are disconnected, badly integrated or siloed away in some way that makes them altogether tough to work with. According to an IDC InfoBrief, sponsored by data access and security company Immuta, data access issues remain the most frequent and tangible bottleneck to achieving what C-suite directors (typically, chief experience officers known as CXOs) would define as the point where a business has become a so-called data-driven intelligent organisation.

Time-to-data delay The lack of a modern, continuously governed and well-audited data access strategy is often the problem, especially when firms look at their time-to-data measure. “Cloud migration, especially when companies have adopted multiple cloud data platforms, has created significant problems for businesses, in particular the ‘time-to-data delay’ where organisations struggle to efficiently manage data access demands whilst balancing privacy and security requirements,” explains Immuta’s EMEA general manager, Colin Mitchell . MORE FOR YOU Western Digital’s Journey To Build Business Resiliency Through Cloud And ERP Transformation Amazon Climate Pledge: Two Years In And Going Strong Microsoft Takes First Steps To Finally Kill The Password This data access bottleneck gets in the way of the business being able to use data to make intelligent, data-led decisions.

Mitchell suggests that with barriers to data access, it is important to make the C-suite aware of the problem that affects all professionals and processes across an organisation and how to address it in a way that’s both efficient and future-proof. “Organizations need to automate data access at scale – and so the technology to do that is Attribute-Based Access Control, or ABAC,” added Michell. “The move from legacy Role-Based Access Control (RBAC) to ABAC simplifies data access control and provides dynamic policy enforcement.

It enables organizations to move from a default ‘no’ to a default ‘yes,’ enabling more teams to gain value from their data. Concurring with Michell’s sentiments, but coming from a data quality rather than data access and audit perspective is Chad Sanderson . Leading the data platform team at Convoy, a company that uses data-centric technology to make freight more efficient, Sanderson and team work to deliver a platform designed to reduce shippers’ costs, increase carriers’ earnings and eliminate carbon emissions.

Existential data quality threats Sanderson has his own personal take on what factors create the real existential threats that data quality faces. If there were four horsemen of the data apocalypse , he would list them as: data omission, data waste, data divergence, data downtime. Taking them in order, Sanderson explains data omission as the point when we refer to missing metadata that would allow data consumers to understand how data is used, who owns it, what it means, and where it’s located.

“Data cataloging tools can solve a piece of this problem; some solve where data is located or who owns it, but there’s no one-size-fits-all tool today that solves the full problem, leading to users bouncing between too many tools to try to figure out data specifics like who owns the data and what it means, adding a lot of pain to the data discovery, modification and creation experience,” said Sanderson. He further points to data waste. This refers to the growth of unused, unmaintained or duplicated/similar queries that inflate cost while obscuring clarity.

Waste typically happens when the cost of recreating data to make it fit specific parameters is lower than using what already exists. “So someone might say: ‘I’m going to create my own column, I’m going to modify that column and potentially add filters on top of it. I’m also going to add columns to other data across the company, and will tweak and prod them in the way that I want.

’ But now you’ve created two different states of the world. So if you ask a relatively simple business question, such as “how many total active customers do I have?” Depending on the team you ask, you might get two different answers,” explained Sanderson. Human in the loop risk Horseman number three would be data divergence.

This refers to the growing divide between what’s happening in ‘the real world’ and what’s being reflected in the data warehouse. “When you’re consuming all of this data exhaust and reverse engineering business logic on top of it, now you have to have a human in the loop that’s thinking about how are they need to modify their queries when the business evolves to keep up, otherwise those queries are going to reflect an old-state of the world. Things will be actively changing and if someone is not modifying those queries (which may be hundreds of lines long with many joins), then a single line of SQL can cause the interpretation of a metric or table to be vastly different than what’s reflected in reality,” detailed Sanderson.

Finally, we come to data downtime. This refers to periods of time when data is partial, erroneous, absent or otherwise inaccurate. According to Sanderson, this represents a traditional data quality problem.

In this scenario, you’re aware that your data is not serving the business in the way it needs to be… and there’s a tangible (or perhaps at least perceivable) gap between when it can be fixed to continue delivering value for customers. Of all the types of data that we define today, we are not about to coin evil data or bad data with any seriousness, but we can definitely agree that enterprise data estates have a fair few pitfalls, perils and potholes to steer around. The fact that a company called Convoy should suggest we think about navigating this route is almost too good to be true.

Now we know where Martin ‘Rubber Duck’ Penwald was coming from when he said: My daddy always told me to be like a duck. Stay smooth on the surface and paddle like the devil underneath! Follow me on Twitter or LinkedIn . Adrian Bridgwater Editorial Standards Print Reprints & Permissions.