The phishing implications of AI chatbots
Phishing evolution in the era of advanced AI
State-of-the-art deep neural networks — a type of machine learning technology used to implement artificial intelligence (AI) — have grabbed headlines for several years because of their ability to translate languages, write poetry, and generate tweets, among other creative tasks.
More recently, GPT-4 and its predecessor, ChatGPT — two conversational AI models that use deep learning — have been touted as “game changers” that will “transform the way all of us work.” Both GPT-4 (Generative Pretrained Transformer 4) and ChatGPT are much more versatile than traditional chatbots. They produce more human-like text answers to questions and requests, and can ‘understand’ the context of a search query or written ‘conversation’ and interpret the intent behind a user’s query. ChatGPT’s unique features helped make it the fastest-growing consumer application in history.
AI chatbots can help people gather impressive human-like responses or even create certain applications — to the extent that their capabilities have raised concerns that they could be used by hackers to create phishing campaigns.
To better understand the security implications of GPT-4 and ChatGPT, it’s important to understand how they could be ‘trained’ to write content for phishing campaigns, as well as the differences between simple and highly personalized, targeted phishing campaigns.
Attackers have always exploited the latest trends and technologies, from cloud storage services to cryptocurrency. As such, the emergence of generative AI serves as an important reminder for organizations to ensure their email security is capable of blocking advanced phishing campaigns, regardless of who — or what — wrote the message.
Getting machines to understand and generate text
The models powering today’s AI chatbots represent new milestones in natural language processing (NLP), a branch of AI that enables machines to ‘understand’ and respond to text or spoken words in much the same way that humans can. NLP combines rule-based modeling of human language with various models to help computers make sense of what they are processing.
A variety of applications — such as social media monitoring tools and voice assistants like Siri — have been using NLP for years. But ChatGPT and GPT-4, which were trained on billions of text and image parameters, are unquestionably more advanced.
Each represents a so-called "large language model" — a neural network-based NLP model that has been trained to make predictions about what is the most logical next word to follow a given phrase. This training technique has been found to produce NLP models that are good at many other tasks, as well.
How OpenAI (the creator of both ChatGPT and GPT-4) has applied this technique represents a significant milestone. OpenAI took the training one step further than other applications, using novel techniques to incorporate human opinions on text or images produced, and specialized training to follow instructions in prompts. As a result, their models are fine-tuned to generate more nuanced, human-like conversation.
The articulate responses generated by ChatGPT and GPT-4 are intended for good. However, cyber criminals can exploit their capabilities as a tool in developing phishing campaigns.
Abusing AI chatbots
Phishing is the most common cause of data breaches, and a common entry point for ransomware.
Because phishing emails are socially engineered to imitate legitimate entities, they can be difficult to identify at first glance. However, historically, common identifiers of phishing messages (especially those generated by criminals who don’t speak/write the victim’s native language) have included poor grammar, misspelled or misused words, and improper sentence structures.
In January 2023, the threat intelligence company Recorded Future reported that ChatGPT can be used by cyber criminals for activities such as creating authentic-looking phishing messages.
Recorded Future found that within weeks of ChatGPT’s launch, threat actors on the dark web and special-access sources were sharing proof-of-concept ChatGPT conversations that enable malware development, social engineering, and disinformation.
There have also been reports of attackers exploiting the popularity of ChatGPT and GPT-4, for example:
Researchers have found several cases where ChatGPT’s name and images from OpenAI were spoofed in phishing websites to spread malware or steal credit card information.
There have been fake ChatGPT apps that, upon download, deploy phishing campaigns to steal users’ information.
Immediately after the GPT-4 launch, scammers began sending phishing emails and tweeting phishing links about a fake OpenAI token.
Technically, OpenAI prohibits use of its models for the “generation of malware,” “activity that has high risk of economic harm,” “fraudulent or deceptive activity,” and any other illegal activity. Their models won’t write phishing emails or help create phishing websites if asked; however, they can simplify how hackers create phishing campaigns. If nothing else, AI chatbots can make it possible for everyone, including attackers, to sharpen their writing skills quickly.
In the wrong hands, ChatGPT and GPT-4 could be exploited to create more authentic-looking, well-written phishing messages and websites that can evade traditional email security or anti-phishing filters.
Could AI chatbots tailor phishing campaigns?
Attackers know they just need to lure one victim into one click or conversation in order to steal credentials, information or money. This is evident in ‘fake job’ phishing attacks targeting job-seekers, charity impersonation scams targeting donors, and romance scams targeting online daters.
Today, the most powerful neural networks aren’t capable of ‘knowing’ the personal details of average citizens, or the specific organizational and communication structure of any one company. But an attacker combining the power of AI chatbots with sufficient research on their intended victim could tailor phishing messages at scale — making it even more difficult for users to spot malicious emails.
Cyber criminals already use highly targeted, low volume business email compromise (BEC) attacks to successfully defraud organizations. BECs typically impersonate a specific employee or executive that the intended victim corresponds with regularly. Vendor email compromise (VEC), a form of BEC, is based on compromising the accounts of a trusted third-party (like a vendor or supplier), and will mirror previously-exchanged messages. Because both BEC and VEC exploit ‘trusted’ relationships, they can evade traditional secure email gateways and authentication. BEC attacks have already cost businesses over $43 billion worldwide.
Block phishing across all channels with Zero Trust
Attackers will always exploit new technology to their advantage. Fortunately, security innovations can identify malicious messages that bypass legacy defenses or user awareness. Sophisticated machine learning models have been created and trained over the years to examine many signals — beyond just text or images — to detect and block phishing.
Email messages contain significant amounts of additional information in header and metadata fields, including information about where the email was sent from, the originating server infrastructure, and its transmission path. Beyond headers, other details in a message, such as specific URLs and links, attachments, distribution list members, tone, and more need to be assessed.
Cloudflare’s preemptive email security, part of its Zero Trust platform, looks at multiple signals that email content generates, including:
Sentiment analysis to detect changes in patterns and behaviors (writing patterns and expressions)
Structural analysis of headers, body copy, images, links, and payloads using heuristics and machine learning models specifically designed for these signals
Trust graphs that evaluate partner social graphs, sending history, and potential partner impersonations
Cloudflare also utilizes intelligence gleaned from the average ~165 billion cyber threats blocked each day and from 3.6 billion daily DNS queries. With this intelligence, Cloudflare customers can block malicious domains, insulate their users from suspicious web content, prevent users from divulging credentials on phishing websites, and stop phishing across different attack vectors.
These and other techniques help stop attackers from exploiting users’ implicit trust in business communications. The overall approach — referred to as extending Zero Trust to threat defense — is based on three core tenets:
Assume breach: Assume that phishing campaigns are always being set up; scan the Internet to proactively hunt for attacker infrastructure and block phishing attacks before they hit the inbox.
Never trust: Don’t just trust business communications because they have email authentication set up, are from reputable domains, or are from someone a corporate user has communicated with before.
Always verify: Continually verify every user and request, even if they are inside the corporate network.
It is clear that attackers will use any readily-available tool, like new AI chatbots, to improve their tactics. Constantly playing defense, or waiting to determine whether new cyber threats are reality can put an organization at greater risk. Rather, “assume breach,” “never trust,” and “always verify” to be better protected against any phishing campaign.
This article is part of a series on the latest trends and topics impacting today’s technology decision-makers.
Dive deeper into this topic.
To explore the types of phishing campaigns Cloudflare detects and blocks, take the self-guided email security demo.
Key takeaways
After reading this article you will be able to understand:
How conversational AI chatbots generate text
How attackers could exploit ChatGPT or GPT-4 to create phishing campaigns
How to protect against AI-generated phishing threats