I'm a platform implementation engineer by trade. My regular work is to come into a firm and help implement purchased products into their technology stack. The very first question I ask is: What is the state of the data service medium in your data ecosystem? I define a "data service medium" as:
a platform or set of services which facilitate data interchange, retention, and/or transformation processes within a data ecosystem which are unique to the implementation and not otherwise encapsulated in another platform offering used in the system.
To be clear, the "data service medium" is different from a "data service layer." The latter is an abstracted method for accessing data in a particular application or platform. I'm talking about the collective services responsible for accessing all of the systems' data in your environment. Quick analogy: Data service layer is the goo inside a single cell organism (cytoplasm). Data service medium is the goo surrounding cells in a multi-cellular organism (ECF).
That's a wordy and technical way of putting it. I normally refer to it as the "goo" which binds all of the enterprise systems together in collaborative harmony. You see, there's no such thing as a tech stack comprised of purchased systems talking to each other and sharing data streams without some "middleware" to facilitate it. It's not necessarily something that a platform vendor will address at length, either, because they are presenting their tool as a solution, not a solution which needs other stuff to make it work. But if you ask your data scientist or business analysts, they'll tell you that they need many tools to move, modify, and analyze the data to do their jobs. And they're not the only ones with supplemental needs.
Lots of companies already know about this requirement. Virtually all companies have some form of data service medium, in fact. What varies is the quality. By quality I mean stability, security, forethought, and capability of that "goo". Let's examine the various types of data service mediums I've come across over the years, in order of sophistication:
Types of data service mediums
Network
This is flat files and FTP, spreadsheets via email, folders full of files in shared locations. Different systems can place files and pick up files via those locations. The boss of this medium is your network security team.
VBA
This is Excel and Access, used with sophisticated macros (often developed by rogue, "shadow IT" business managers) to connect to systems and grab data mainly to build reporting and analysis. The owner of this medium is the business units that build them.
SQL Server
This is SSMS, SSIS, and similar technologies in the open-source side, like MySql and cron jobs. Legacy enterprise infrastructure is built with this, when data ecosystems were mainly in-house and on the network. IT is typically the owner of this (,and if a company is lucky, the business teams have a hand in it as well).
Cloud Services
This is the new breed of mediums. AWS, Azure, Google Cloud Computing and the like are what we're talking about. They are specifically designed for distributed systems across the internet. They have security features to mitigate the risk of distributed infrastructure. It follows a new organizational structure of firms as well when it comes to ownership. DevOps and Data Engineering are the joint owners, and whether they are in IT or the business unit is up to the firm.
Network --> VBA --> SQL Server --> Cloud Services
Composition of data service mediums
A company's tech stack may employ one or more of these types of mediums. It's the distribution among these types which reveals the maturity of a stack. If a stack uses lots of Network resources and no Cloud Services resources, then it is relatively low in maturity, for instance. But it's not reasonable to expect a stack to use zero Network resources. It's a relative concept. You want to evolve toward Cloud Services, especially as you invest in stack components (vendor Platforms-as-a-Service) which are entirely based on them.
How does the data service medium affect my business?
Put simply, the more invested on the left side of the spectrum a company is, the more expensive any tech project will be. The more invested in the right side, the more likely technology projects will be completed cheaper, faster, and better. New companies benefit from investing directly into Cloud Services, while older companies following strict accounting principles (and not factoring in economic principles of opportunity cost or competitive advantage) are working with their legacy investments until they depreciate before updating.
How do I upgrade my data service medium?
This is actually quite simple. Hire a DevOps specialist and a Data Engineer resource, either directly or as a consultative service. Then follow Steve Jobs' advice: "We hire the smart people so they can tell us what to do."
A data- and ops-oriented resource will help assess you current data service medium, including your human capital, and recommend next steps to enhance the stack. This is not necessarily a race -- no need to leap to the bleeding edge. But incremental improvements in this commonly neglected and critical area will go a long way to improve all operational aspects of your business.