In just a few short years, AI has become embedded in everyday business operations, many companies are moving quickly to automate processes and rely on AI-driven systems. But according to Georg Meyer, a strategist and systems expert, speed and convenience can come at a cost, especially when businesses lose visibility into how their most important systems actually work.
Meyer warns that one of the biggest risks is a false sense of understanding. “Companies may think they know what’s going on because of the prompts they gave,” he explains, “but they don’t have true visibility of what happens inside the box.”
Modern AI systems, particularly large language models, are powerful but difficult to fully inspect. Their internal processes are not always transparent, and their outputs can vary due to built-in randomness. This creates challenges around consistency and reliability, especially when AI is used in core business functions. “There are hidden risks to reproducibility and reliability,” Meyer says, “which can be catastrophic when they affect the heart of the company.”
For Meyer, the solution isn’t to avoid AI, but to ensure that human understanding remains central. Businesses still need people who know how systems work, even if those systems are supported by AI.
Short-Term Gains, Long-Term Risks
Another concern Meyer highlights is the long-term impact of AI on expertise within organizations. While AI can dramatically speed up workflows and help turn knowledge into action, over-reliance on it may come with unintended consequences.
“Both over- and under-reliance can be risky,” Meyer says. “AI can reduce the time it takes to apply expertise, but it raises an important question, how do we train the next generation of experts if they never develop the fundamentals?”
In the short term, AI can make teams more productive. But if employees begin to depend on it for core thinking and problem-solving, businesses risk losing the expertise needed to handle complex, unexpected, or situations that need a human touch. Meyer believes companies and educators alike need to be intentional about maintaining foundational knowledge, especially for scenarios where AI cannot be relied upon.
Accountability Still Belongs to Humans
Perhaps the most important issue Meyer raises is accountability. As AI systems take on more responsibility, some organizations may be tempted to shift blame when things go wrong. Meyer is clear that this is not a viable approach.
“Accountability, and its legal counterpart, liability, are fundamentally attached to humans,” he says. “You cannot wash your hands of a bad outcome by saying ‘AI did it.’”
He points out that many companies are already trying to navigate this gray area. AI providers often include disclaimers that their systems can make mistakes, while industries like automotive promote “self-driving” features but still hold the human driver responsible. However, Meyer argues that if systems are truly autonomous, responsibility must shift to those who design and deploy them.
This applies even in smaller, everyday use cases. For example, businesses using AI to build applications that handle sensitive data cannot assume the technology will manage security correctly. “The AI is not entrusted with the data,” Meyer says. “The company is.”
