Eight months into a three-year contract, Software developer Ibrahim Diallo found himself being escorted out of his workplace by security. However, no one, not even the managers, knew why. It was later discovered that Mr Diallo had, in fact, been fired by an automated computer system used by the company. 
The ordeal began in late June this year, when Mr Diallo was unable to access his office when his keycard failed. Having eventually made it to his desk, he found his login was not working. Then, colleagues began pointing out that his account was labelled as ‘inactive’ on the work system. Finally, he was contacted by his recruiter who received an email saying his contract was terminated. She promised to fix the problem however the next day Mr Diallo was locked out of every single system except for his Linux machine. Despite being told by colleagues and managers that the issue would be fixed and to keep coming to work, it became impossible for Mr Diallo to do so. Eventually, he was escorted from the building by security staff who had received an email saying he was to be removed.
After some digging, the company found all emails had come from the automated computer system. After going through a recent transition, Mr Diallo’s previous manager had forgotten to transfer his name into the new system. When the final transition date came around, the computer did not ask for any opinion or confirmation, but instead began the process of terminating an employee. Unfortunately for Mr Diallo, it was three weeks before the system had thoroughly done its job and he could return with a clean slate as a new employee.
While automation in companies has seen huge growth over recent years and will likely become a key aspect of business operations, it is far from perfect. In their 2017 report, IT corporation Cognizant warns that “while automation’s benefits are well established, many companies find themselves on a creaky IT foundation due to flawed implementation or unrealistic expectations from their investment in automation”.
Examples of failed automation systems are increasing. In 2014, a ‘subtle bug’ in a Dropbox script was automatically applied to various updates during a planned upgrade, affecting thousands of production servers and causing the company’s live services to fail. In September 2015, some of Amazon’s automated infrastructure processes timed out, causing a severe network outage in their cloud platform. In 2017, Facebook scaled back on its investment in the chatbot initiative after it was found that the program failed to fulfil 70% of user’s requests.
One of the worst automation disasters was suffered by Knight Capital in 2012. A human error led to a misconfigured server going unnoticed and making a decommissioned code available across the server. Then, the automated server system began sending orders to certain trading centres for execution, which triggered a domino effect around algorithmic stock trading and eventually cost Knight Capital $465 million in trading loss.
Companies must be wary of relying on large automated systems that are run and re-run without being checked for defects and having their decisions confirmed by humans. The Cognizant report stresses that “it is always better and especially important nowadays to apply proper diligence and be selective in automation”. Author and expert Geoff Colvin believes the issue is that executives are confused about automation and the impact it will have on their companies, stating that “most boards are not considering this sufficiently because it may seem too abstract a problem to them right now”.
However, automation is undeniably growing. For example, Bartier Perry Solicitor Darren Gardner explains that many businesses are using automated AI software to make decisions about job applications. These systems are designed to rapidly scan applications, select candidates and even conduct video interviews. The algorithms analyse keywords, language, facial responses and other personal information to make recruitment decisions. However, many including Mr Gardner and Mr Diallo warn that these systems could enhance bias in the recruitment process, by basing decisions on information such as surnames, addresses, which school they attended and other key socio-economic factors. After surveying over 2,500 executives across 36 economies, a report in 2015 found that 56% of firms were automating their processes or planning to do so over the next year. Another report by McKinsey & Company estimated that half of today’s work activities could be automated by 2055. As this report emphasizes, we are living in a new automation age in which robots and computers can perform various physical work activities better and more cheaply than humans, but are also “increasingly capable of accomplishing activities that include cognitive abilities…activities that used to be considered too difficult to automate successfully”.
At the end of the day, it is guaranteed that automation will continue to rise and be heavily integrated into various operations of companies. This is not a bad thing, as automation has many well-established benefits and can greatly enhance both employee and customer experiences. Automation systems are capable of improving quality, enhancing speed, reducing error and even, according to one study, raising global productivity growth by 0.8-1.4% annually. Indeed, some have actually pointed out that in Mr Diallo’s case, the error was human and have stressed that a more comprehensive automatic AI system could have avoided the disaster. Nevertheless, executives must properly understand these systems when utilising them, and approach them with caution and moderation. If they fail to do so, companies will never be capable of harnessing the full potential of automation and will continue to experience embarrassing and costly failures.