Detecting AI-Generated Text and LLM Bots on Telegram
Disclaimer 1: This post makes no political claims and serves solely as a demo of an AI-bot detector on real data.
Disclaimer 2: AI tools helped draft parts of the text and supporting code.
TL;DR
I applied the open AI-text detector mbert-ai-bot-detector to ~80K Telegram comments from July–September 2026. Combined with behavioural signals and an LLM review, it helped flag 62 likely bot accounts, two thirds of them caught only by the AI-text detector. Not every bot writes AI text, so the detector works best alongside other signals.
Коротко українською: Я застосував відкритий детектор ШІ-тексту mbert-ai-bot-detector до ~80 тис. коментарів у телеграм-чаті за липень–вересень 2026 року. Разом із поведінковими ознаками та перевіркою мовною моделлю він допоміг виявити 62 ймовірні бот-акаунти, дві третини з яких знайдено лише завдяки детектору. Проте не всі боти пишуть ШІ-текстом, тож детектор найкраще працює разом з іншими сигналами.
Motivation
Can you tell whether a comment on social media was written by a real person or generated by an AI? Well, you definitely can have a good guess, but it is not easy. This case study is for anyone who works with comment sections: researchers, moderators and journalists. It shows how the open model mbert-ai-bot-detector, from our CIKM 2026 paper Adversarial Creation and Detection of AI-Generated Social Bot Content, can help find AI-bot accounts in a real Ukrainian-language Telegram chat.
Writing a fluent, on-topic comment used to take a person minutes. A large language model does it in a fraction of a second, in any language and in any tone. That makes it cheap to fill a comment section with accounts that look like ordinary people.
What are the consequences? Chats can be flooded with AI-generated comments that push specific narratives, drown out dissent and make it look like a topic is more popular than it really is. A single AI comment can be enough to start a conversation, and a few hundred can make it look like a movement.
About the detector
mbert-ai-bot-detector is an open text classifier on Hugging Face. It reads one text and returns the probability that it was written by an AI-powered social bot (LABEL_1) rather than a person (LABEL_0). In this post I call that probability the AI score: 0 looks human, 1 looks generated.
- Base model: multilingual BERT (
bert-base-multilingual-cased), covering 104 languages, fine-tuned on data in 17 languages. - Training idea: the paper generates adversarial bot content with LLMs and trains the detector to catch it. On the paper’s Fox-8-23 benchmark it reaches a user-level AUC of 0.989.
- Intended use: judge accounts by aggregating scores over many of their texts. The model card recommends 20 or more texts per account. A single short text is a weak signal.
Why this model for this case: it is open, runs on a laptop GPU (or even a CPU), handles 17 languages including Ukrainian and Russian without translation, and was built for social-media bot text rather than essays. Short, slangy chat in two mixed languages is also a fair stress test.
How to use the detector?
pip install transformers torch
from transformers import pipeline
detector = pipeline("text-classification",
model="trokhymovych/mbert-ai-bot-detector",
truncation=True, max_length=512)
def ai_score(texts):
"""Probability that each text was written by an AI-powered bot."""
return [r["score"] if r["label"] == "LABEL_1" else 1 - r["score"]
for r in detector(texts, batch_size=32)]
# Two (lightly edited) comments from the case study (both replies to the same channel post)
comments = [
"Ще один завод на цвинтар, а влада мобілізацію посилює, капець",
"Так житомир то попереджали . Незабувайте нас в це втягнув зелений.",
]
for text, score in zip(comments, ai_score(comments)):
print(f"{score:.2f} {text}")
For an account, score all of its comments, keep the longer ones and look at the mean:
long_comments = [c for c in account_comments if len(c) >= 40]
scores = ai_score(long_comments)
mean_score = sum(scores) / len(scores) # >= 0.5 over several long messages: worth a closer look
The data
The chat history was exported with Telegram Desktop’s built-in export. It covers July–September 2026 of the public open discussion chat attached to a large Ukrainian-language political channel.
| Comments analysed | ~80K |
| Commenting accounts | ~7.3K |
| Accounts classified (5+ comments) | ~2.4K (they wrote 90% of comments) |
| Channel posts (excluded) | ~500 |
| Period | July–September 2026 |
| Languages | Ukrainian and Russian, often mixed |
The channel is not named, accounts become pseudonyms, @mentions are masked and channel posts presented here are paraphrased. Comments are quoted in the original language (lightly edited, so they cannot be found by exact search) with an English translation, because the original text is what the model reads. All data is processed exclusively locally and removed after analysis.
How the pipeline works
Why each step
Each step exists because of something I saw in the data:
- Score every message. It is cheap, automatic and the strongest single signal I have.
- Judge accounts, not messages. Short messages are unreliable. I classify accounts with at least 5 comments, compute AI features on messages of 40+ characters, and compare each score with messages of similar length. This follows the model card: aggregate over many texts.
- Look beyond AI text. Not every bot uses an LLM. Some paste the same human-written slogan a hundred times. For each account I therefore also measure repetition, identical or rephrased texts shared with other accounts, bursts of activity, activity while the chat is quiet, account lifetime and co-commenting with the same partners.
- Flag candidates. An account is flagged when it shows content evidence (AI text, repetition or copying) or matches a concrete rule. One rule relies on the detector alone: if the long messages of an account average an AI score of 0.5 or more, it is always flagged. Behaviour alone, such as posting a lot, never flags anyone, because very active people are common.
- Review the full account. A local LLM (gemma4:26b, run with Ollama) gets a dossier of each flagged account. The dossier contains every measurement with its percentile in the chat, the highest- and lowest-scoring messages, timing, copies and co-commenting partners side by side, topics and a comment timeline. It answers bot, not bot or unsure. This removes false alarms that numbers alone cannot. For example, many people in Kyiv comment at 3 a.m. during air-raid alerts, so night activity looks suspicious until you see the whole chat is awake. The reviewer never decides whether a text is AI-written; that comes only from the detector.
Review outcome by detector tier
| Detector tier | Accounts | LLM-confirmed bots |
|---|---|---|
| Likely bot | 58 | 31 (53%) |
| Suspicious | ~360 | 30 (8%) |
| Normal (random sample of 150) | ~2K | 1 of 150 (1%) |
By design, this is outlier detection (rule-based, not trained on this chat): it catches the obvious cases and keeps precision high, but more sophisticated bots can be missed.
Results
Examples are more convincing than numbers. Below are real (slightly modified) comments from the chat. Each card shows the original text, an English translation and the AI score. Could you spot the bots?
Show the answer
Show the answer
Show the answer
Three account stories
The burner. Account 3155 appeared, posted 10 long comments in about six minutes and was never seen again. Their mean AI score is 0.92. The LLM reviewer labelled it automated cadence.
The switcher. Account 5363 first wrote human-like comments. Then, at one point, the mean AI score of their long messages jumped from 0.13 to 0.72 and stayed there. Nobody told the detector about dates; the change shows up in its scores alone. The LLM reviewer labelled the account as a bot pushing one narrative.
The persistent agent. Account 5841 posted ~140 comments in ~5 weeks, about 20 per active day. Their long messages average 0.86, and 80% score above 0.8. The LLM reviewer found near-identical content shared with other suspicious accounts and labelled it coordinated.
Not every bot is an AI bot
Among the 62 LLM-confirmed bots, 73% were identified through AI-like text and 27% through repetition, copying or other rules only. Without the detector, 42 of the 62 LLM-confirmed bots (68%) would not have been flagged at all. Other signals catch copy-paste and template bots. Only the text model catches a well-written, one-off generated comment.
A few more patterns that stood out:
- AI-like text: 27.9% of long messages from likely-bot accounts score above 0.8, against 6.5% for everyone else.
- Account age: 24% of LLM-confirmed bots are among the newest 10% of account IDs, against 10% of normal accounts. Telegram IDs grow over time, so this is a rough age proxy.
- People mostly ignore them: bots wrote 3.6% of comments but received only 1.1% of replies and 2.6% of reactions.
The hidden part: deleted messages
The export shows the chat after moderation. Telegram numbers the messages of a chat sequentially, and in my export the IDs increase with time in 100% of cases. So every gap in the numbering is a message that no longer exists.
- 16.8% of message IDs in the period are missing (~16K of ~98K). This is an upper bound: it includes deletions by authors, admins and antispam.
- ~2.4K comments come from accounts that have since been deleted. That happened to 8% of LLM-confirmed bots, against 3% of other accounts.
- On the “worst” day, 55% of the messages were removed.
Why it matters: everything in this post is a lower bound. Moderators and Telegram had already removed part of the bot activity before I looked.
Interpreting the AI score
- Single messages: treat a score above 0.8 as “AI-like”, but only for messages of 40+ characters. On 20 to 39 characters, the share of high scores is the same (17%) for bots and everyone else, so very short texts carry no signal.
- Accounts: a mean of 0.5 or more over several long messages is a good reason to review an account, not a verdict. Here, 12% of accounts flagged by the AI signal alone were LLM-confirmed after review. With 5+ comments per account I stayed below the model card’s recommendation of 20+ texts; more text per account gives more reliable decisions.
- Calibration: compare scores within the same length band, and against the community’s own baseline.
Typical failure modes:
- Very short messages, even from regular human members, can score high.
- Humans who write with an AI assistant look like bots in the scores.
- Human-written spam and copy-paste is not AI text; use repetition and copying signals for it.
- Out-of-domain text, such as quoted news or code, is not what the model was trained on.
Limitations and responsible use
- Lower bound, not a census. Accounts with fewer than 5 comments are not classified, and part of the activity was deleted before export.
- No external ground truth. “LLM-confirmed” means confirmed by an LLM reviewer that saw the detector’s verdict, not verified independently. A blinded re-run and a human-labelled sample are needed.
- One chat, three months. The results describe this community, not Telegram as a whole.
- Generated is not malicious. Do not use the AI score alone to ban or name anyone. Use it to decide where a human should look.
Paper & citation
- Paper page: Adversarial Creation and Detection of AI-Generated Social Bot Content (CIKM 2026; arXiv:2606.07219)
- Model: trokhymovych/mbert-ai-bot-detector on Hugging Face
If you use the model, please cite:
@inproceedings{trokhymovych2026adversarial,
title = {Adversarial Creation and Detection of {AI}-Generated Social Bot Content},
author = {Trokhymovych, Mykola and Baeza-Yates, Ricardo and Flammini, Alessandro and Saez-Trumper, Diego and Menczer, Filippo},
booktitle = {Proceedings of the 35th ACM International Conference on Information and Knowledge Management (CIKM '26)},
year = {2026},
url = {https://arxiv.org/abs/2606.07219}
}
