A related issue: Let’s say that it is crunch time for the finances to be prepared for filing, and the CFO asks for — and is granted — permission for an additional 30 people from different company business units to temporarily help with the filings. Does someone think to reprogram the LLM to grant temporary data access to those 30 temporary resources? And does someone remember to go back and remove their access once they return to their regular roles?
Unrecognized glitches
Another LLM concern is more practical. Veteran IT managers have many years of experience working with all manner of software. Their experience teaches them how systems look when they crash, such as slowing down, halting, generating error messages, and throwing out screens of garbage characters. But when an LLM glitches — its version of crashing — it doesn’t act that way.
“When traditional software is broken, it’s obvious: screens don’t load, error messages are everywhere. When [LLM] software is broken, it’s much more opaque: you don’t get glaring errors, you just get a model with bad predictions,” said Kevin Walsh, head of artificial intelligence at HubSpot. “It may take weeks or months of having the LLM out in the real world before hearing from users that it’s not solving the problem it is supposed to.”
