ID verification and authentication unreliable due to AI Deepfakes

Last week represented what could be a turning point in peoples trust in ID verification tools across the internet.

ID verification unreliable due to AI Deepfakes

A finance worker at a multinational firm was tricked into paying out $25 million to fraudsters using deepfake technology to pose as the company’s chief financial officer in a video conference call.

The elaborate scam saw the worker duped into attending a video call with what he thought were several other members of staff, but all of whom were in fact deepfake recreations, according to Hong Kong police.

“(In the) multi-person video conference, it turns out that everyone [he saw] was fake,” said senior superintendent Baron Chan Shun-ching.

Chan said the worker had grown suspicious after he received a message that was purportedly from the company’s UK-based chief financial officer. Initially, the worker suspected it was a phishing email, as it talked of the need for a secret transaction to be carried out.

However, the worker put aside his early doubts after the video call because other people in attendance had looked and sounded just like colleagues he recognized, Chan said.

Gartner say 2026, attacks using AI-generated deepfakes on face biometrics will mean that 30% of enterprises will no longer consider such ID verification and authentication solutions to be reliable in isolation.

“In the past decade, several inflection points in fields of AI have occurred that allow for the creation of synthetic images,” said Akif Khan, VP Analyst at Gartner.

“These artificially generated images of real people’s faces, known as deepfakes, can be used by malicious actors to undermine biometric authentication or render it inefficient.

As a result, organizations may begin to question the reliability of identity verification and authentication solutions, as they will not be able to tell whether the face of the person being verified is a live person or a deepfake.”

ID verification and authentication processes using face biometrics today rely on presentation attack detection (PAD) to assess the user’s liveness.

“Current standards and testing processes to define and assess PAD mechanisms do not cover digital injection attacks using the AI-generated deepfakes that can be created today,” said Khan.

Gartner research said presentation attacks are the most common attack vector, but injection attacks increased 200% in 2023.  Preventing such attacks will require a combination of PAD, injection attack detection (IAD) and image inspection.

Combine tools to mitigate Deepfake threats

To assist organizations in protecting themselves against AI-generated deepfakes beyond face biometrics, chief information security officers (CISOs) and risk management leaders must choose vendors who can demonstrate they have the capabilities and a plan that goes beyond current standards and are monitoring, classifying and quantifying these new types of attacks.

“Organizations should start defining a minimum baseline of controls by working with vendors that have specifically invested in mitigating the latest deepfake-based threats using IAD coupled with image inspection,” said Khan.

Once the strategy is defined and the baseline is set, CISOs and risk management leaders must include additional risk and recognition signals, such as device identification and behavioural analytics, to increase the chances of detecting attacks on their identity verification processes.

Above all, security and risk management leaders responsible for identity and access management should take steps to mitigate the risks of AI-driven deepfake attacks by selecting technology that can prove genuine human presence and by implementing additional measures to prevent account takeover.