{"id":3415,"date":"2025-04-28T10:51:51","date_gmt":"2025-04-28T07:51:51","guid":{"rendered":"https:\/\/factcheck.lt\/?p=3415"},"modified":"2025-05-02T09:34:52","modified_gmt":"2025-05-02T06:34:52","slug":"","status":"publish","type":"post","link":"https:\/\/factcheck.lt\/eng\/news\/nltk_openai_tiktok_content\/","title":{"rendered":"Artificial Intelligence vs. Manipulation: Comparing Approaches to Detecting Manipulative Content","raw":"Artificial Intelligence vs. Manipulation: Comparing Approaches to Detecting Manipulative Content"},"content":{"rendered":"

In the era of information wars and mass disinformation, automatic content analysis systems are becoming an indispensable tool for fact-checkers and media researchers. In this article, we compare two common approaches to identifying manipulative content \u2014 classical natural language processing (NLP) methods and the latest technologies based on large language models (LLM).
\nAs part of the research, a corpus of video content with the hashtag #\u0431\u0435\u043b\u0430\u0440\u0443\u0441\u044c<\/strong> from the TikTok social network was analyzed. The analysis period was from March 20 to April 21, 2025. Exolyt – TikTok Social Intelligence Platform<\/a> was used for data collection. The titles, descriptions, and tags of the videos were analyzed. But the main interest is not the content itself, but how different technological approaches evaluate the same data.<\/p>\n

\n
\n
\n

Research Methodology<\/h2>\n

To compare the two approaches, we used a common dataset \u2014 a corpus of 12,252 videos with the hashtag #\u0431\u0435\u043b\u0430\u0440\u0443\u0441\u044c<\/strong>, including titles, descriptions, and metadata. Two different algorithms were applied to the same data, which allowed us to directly compare their effectiveness, strengths, and weaknesses. The main analysis parameters included keyword extraction (tokens), frequency analysis of the most used words and phrases, assessment of emotional coloring in three categories (negative, positive, neutral), and determining the level of content manipulativeness.<\/p>\n

\nNLTK (Natural Language Toolkit) is a leading open-source platform for natural language processing in Python, created in 2001 and actively developed by the community. The library provides easy-to-use interfaces for working with more than 50 text corpora and lexical resources, as well as a rich set of tools for text processing: tokenization, stemming, lemmatization, part-of-speech tagging, syntactic analysis, named entity extraction, and semantic analysis. NLTK includes extensive documentation and educational materials, making it popular among both researchers and developers who apply NLP methods to solve practical problems \u2014 from sentiment analysis and text classification to building chatbots and information retrieval systems. Thanks to its modular architecture and flexibility, NLTK continues to be one of the main tools in the arsenal of specialists in computational linguistics and text analysis, despite the emergence of new libraries focused on deep learning.<\/p><\/blockquote>\n

Detailed Methodology of the Two Approaches<\/h2>\n

Classical Approach Using NLTK<\/h3>\n

Technology<\/strong>: Open Natural Language Toolkit (NLTK) library with additional tokenization and analysis algorithms.<\/p>\n

Data Processing<\/strong>:<\/p>\n

The first stage of text preprocessing included converting text to lowercase, removing punctuation and special characters, removing numbers and extra spaces, as well as combining text fields (title, description, tags).<\/p>\n

This was followed by tokenization \u2013 breaking text into individual words using NLTK word_tokenize, removing stop words from Russian and English languages (prepositions, conjunctions, etc.), and filtering words shorter than 3 characters.<\/p>\n

Emotional analysis was carried out by searching for words from predefined dictionaries of emotional markers, counting the number of words in each category, and determining the dominant emotion by the maximum number of markers.<\/p>\n

For manipulation assessment, the formula was applied: manipulation_score = emotion_negative * 1.5 - emotion_positive<\/code> with subsequent normalization of the indicator relative to the average value and determination of the manipulation threshold (mean + standard deviation).<\/p>\n

The final stage was the process of forming the results, which included frequency analysis of words and bigrams (word pairs), classification of content by emotional coloring, and identification of potentially manipulative content.<\/p>\n

Implementation Features<\/strong>: Data processing was carried out in parts to save memory, a backup tokenization system was provided in the absence of NLTK, and visualization of results was carried out using matplotlib\/plotly.<\/p>\n

Modern Approach Using OpenAI Models<\/h3>\n

Technology<\/strong>: OpenAI API with GPT-4o-mini model<\/p>\n

Data Processing<\/strong>:<\/p>\n

The initial stage included query formation \u2013 combining text fields (title, description, tags) and sending the full text without preprocessing.<\/p>\n

This was followed by working with a system prompt for the model, which contained instructions for text analysis, keyword extraction, emotional coloring assessment, and determining the level of manipulativeness.<\/p>\n

Processing the model’s response involved extracting structured data from the JSON response, saving tokens, emotional ratings, and manipulativeness indicator, as well as determining the dominant emotion based on the model’s ratings.<\/p>\n

The final stage \u2013 aggregation of results included counting the frequency of tokens and bigrams, calculating average indicators for emotional categories, determining the manipulation threshold, and identifying content with a high level of manipulativeness.<\/p>\n

Implementation Features<\/strong>: A structured response format (JSON) was used, pauses between requests were provided to comply with API limits, and error handling mechanisms with return to neutral values in case of failures.<\/p>\n

Comparative Analysis of Results<\/h2>\n

Keywords and Frequency Analysis<\/h3>\n

Both methods identified similar keywords in the content, which indicates a basic level of consistency. The word clouds from the reports show that the word “belarus” is the most frequent in both cases, followed by “news”, “recommendations”, “minsk”, “lukashenko”.<\/p>\n

NLTK:
\n\"\"<\/span><\/div>\n

OpenAI:
\n\"\"<\/span><\/p>\n

Notably, the visual representation of word clouds looks very similar in both approaches, despite the difference in methodology. This may be explained by the fact that the word cloud visualization algorithm was identical in both scripts, and the main frequently occurring words indeed coincide. This coincidence confirms the basic reliability of both approaches in highlighting key content themes, even with significant differences in assessing emotional coloring and manipulativeness.<\/p>\n

However, on closer examination of frequency histograms, differences are noticeable:<\/p>\n

In the NLTK method, the word “belarus” occurs about 14,025 times, and the next word “news” \u2014 3,189 times. In the OpenAI method, the word “belarus” occurs about 8,258 times, “news” \u2014 2,652 times.<\/p>\n

NLTK:
\n\"\"<\/span>
\nOpenAI:
\n\"\"<\/span><\/p>\n

These differences are explained by different principles of tokenization and keyword extraction. The OpenAI model seeks to highlight more meaningful words with information load, while the NLTK method works with the formal frequency of occurrence.<\/p>\n

In the top 15 word pairs, there are also significant differences: the NLTK method highlights pairs: “brest minsk” (385), “economy politics” (371), “politics MFA” (369), while the OpenAI method highlights: “belarus belarus” (1,723), “belarus lukashenko” (615), “minsk belarus” (527).<\/p>\n

NLTK:
\n\"\"<\/span>
\nOpenAI:
\n\"\"<\/span><\/p>\n

This demonstrates different approaches to forming bigrams: OpenAI takes into account the proximity of words in the text, while the NLTK method analyzes thematic connectivity rather.<\/p>\n

Striking Differences in Emotional Coloring Assessment<\/h3>\n

The most dramatic divergence between the two systems concerns the assessment of the emotional coloring of content. When we analyzed the same dataset of videos with the hashtag #belarus<\/strong>, the results were distributed as follows:<\/p>\n

NLTK:
\n\"\"<\/span><\/div>\n

NLTK method<\/strong> showed an extremely unbalanced distribution with a predominance of negative assessment:<\/p>\n