This insight explores how opaque AI models and design-induced automation errors undermine trust, create new safety risks, and complicate human–AI collaboration in maritime operations.
Many AI models, especially deep neural networks, function as black boxes. This means that while users can view the inputs and outputs, the internal processes that generate those outputs remain unclear. This makes it difficult to understand how and why an AI model made the decision it did.
Black boxes also tend to hide deficiencies in AI systems, such as the existence of bias, inaccuracies, or hallucinations. This black box effect can have one of two outcomes: it either reduces trust or leads to an over-reliance on AI systems.
Lisanne Bainbridge wrote in 1983 about the Ironies of Automation and how automated systems can lull humans into a passive monitoring role that they are bad at, meaning they may fail to notice or respond properly when the automation reaches its limits or fails. This still holds true today and can be seen in ship operations when humans, who are fundamentally poor passive monitors of automated systems, rely on them, leading to loss of skills, out-of-loop syndrome, mental workload and subsequently and ironically, a lack of trust in these systems, which then contributes to accidents.
A Fuel Performance Analyst at a global shipping and logistics company explained that his company is piloting two AI tools. The main challenge he sees is not the willingness of people to engage with the tools but measuring real benefit. The models work like black boxes, suggesting actions, but it is difficult to understand why these suggestions were made and to quantify the impact, which complicates adoption and trust.
According to Todd Lanouette, a meteorologist and Business Manager of Shipping at StormGeo, an overreliance and assumption that the AI must be right has severe implications. Even just one AI-related casualty would undermine confidence, slow adoption, trigger regulatory pushback, and reinforce the idea that AI is unsafe without humans.
One of the problems with advanced technologies like AI and automation today is that errors have been baked into the design of a system or model. This happens when designers make assumptions about how people will work, fail to account for real-world operating conditions, or build systems without sufficient human-factors expertise. These errors may lie dormant for many years until an unusual situation arises, at which point the design flaw surfaces and causes an accident or operational failure.
Design-induced errors show that neither AI nor humans alone can guarantee safe and effective operations. AI offers speed and data processing, but lacks real-world judgement, while humans provide context and adaptability but are vulnerable to overload when automation fails. Achieving the right balance ensures each compensates for the other’s weaknesses.
To explore how a balanced human-AI approach can strengthen voyage optimisation in safety-critical conditions, read the full Thought Leadership report ‘Co-Pilots of the Sea: Exploring the Human Intelligence Behind Maritime AI’ produced in partnership with StormGeo.
