Efficient detection of AI-generated scientific abstracts with a lightweight transformer

Abstract

The rapid growth of advanced large language models challenges the authenticity of scientific work, which requires reliable methods for detecting AI-generated scientific text. This paper addresses this challenge by developing and evaluating an efficient text classifier. We first constructed a balanced dataset, focusing initially on the Computer Vision (cs.CV) domain, and subsequently expanding it to include four additional diverse scientific domains (totaling 5,000 abstracts), using human-written samples from arXiv and corresponding AI-generated versions created using Google’s Gemini 2.0 Flash. We then fine-tuned a lightweight Transformer model, DistilBERT, for the classification task. On the primary in-domain (cs.CV) test set, our approach achieved excellent performance, with an accuracy of 99.4% and an Area Under the ROC Curve of 0.9999. Subsequent cross-domain evaluations demonstrated robust generalization (Macro-F1 = 0.948). Further analysis revealed that our model surpasses traditional machine learning baselines not only in accuracy but also in robustness, as it learns deep semantic patterns rather than relying on superficial statistical cues. This work provides a practical, high-performance tool for safeguarding scientific authenticity and establishes a valuable benchmark for future research in AI text detection.

Introduction

The rapid advancement of large language models (LLMs) is transforming how content is created, with AI-generated text becoming increasingly difficult to distinguish from human writing. While this technology, often referred to as Artificial Intelligence Generated Content (AIGC), offers significant opportunities for automating tasks and enhancing productivity, it also presents significant challenges. The potential for misuse, such as the generation of misinformation, academic dishonesty, and spam, has created an urgent need for reliable and efficient methods to detect synthetic text¹.

In academic and scientific contexts, ensuring the authenticity of research is crucial. The ability of modern LLMs to produce coherent and contextually appropriate scientific abstracts poses a direct threat to academic integrity. Consequently, developing automated tools to screen for AI-generated submissions is no longer a theoretical concern but a practical necessity for publishers, conference organizers, and educational institutions.

While a variety of methods for AI text detection have been proposed^2,3,4, some existing approaches rely on fine-tuning complex, resource-intensive models for classification³. Other methods depend on analyzing superficial statistical artifacts, such as the log-probability of words, which can be brittle and easily evaded². Furthermore, the effectiveness of these approaches has not been thoroughly benchmarked on specialized domains like scientific writing. This creates a gap for a solution that is not only highly accurate but also computationally efficient and robust.

To address this gap, this paper presents a comprehensive study on the detection of AI-generated scientific abstracts. Our contributions are as follows:

• First, we constructed a new, domain-specific dataset comprising 5,000 scientific abstracts from five scientific fields, with an equal number of human-written and AI-generated samples.

• Second, we demonstrate that a lightweight Transformer model, DistilBERT, can be fine-tuned to achieve excellent detection performance, reaching an accuracy of 99.4% with minimal computational resources.

• Third, through a detailed ablation study and feature analysis, we show that our approach moves beyond simple statistical cues, learning deeper semantic patterns and thus offering greater robustness compared to traditional baseline methods.

To highlight the breadth of our experimental validation and demonstrate robustness beyond standard benchmarks, we summarize our Scope & Evaluation protocols as follows:

• In-domain Evaluation: Performance benchmarking on standard Computer Vision (cs.CV) scientific abstracts.

• Cross-domain Generalization: Evaluation across four held-out non-CV scientific fields (beyond cs.CV) using a Leave-One-Domain-Out (LODO) protocol to ensure zero-shot generalization capabilities.

• Cross-model Transferability: Stress testing on unseen advanced generators including GPT-4o, Claude 3.5, Llama-3.1, and Qwen-3 to verify detection versatility.

• Adversarial Robustness: Rigorous analysis against paraphrasing attacks (generated by GPT-4o at light, medium, and heavy levels), reported with strict TPR@low FPR metrics.

This work provides a reliable benchmark for AI text detection in a scientific context and offers a practical, efficient solution that can be readily deployed. The remainder of this paper is organized as follows. Section Methods details our methodology for dataset construction and model training. Section Results details our empirical findings, including an ablation study, generalization analysis, and adversarial robustness study. We then analyze these results in Section Discussion, covering their implications, error analysis, calibration and uncertainty, limitations, and future work. Finally, Section Conclusions concludes the paper.

Related work

LLM-generated text detection. Prior work on machine-generated text detection spans supervised classifiers fine-tuned on labeled human/AI corpora^5,6, and zero-shot/statistical detectors that exploit differences in token likelihood patterns or probability curvature (e.g., GLTR², DetectGPT⁴). More recent zero-shot methods improve efficiency and generalization, such as DetectLLM (log-rank features)⁷, Fast-DetectGPT (conditional probability curvature)⁸, and Binoculars (contrastive scoring using a pair of LMs)⁹, highlighting a growing spectrum of detectors with different compute–accuracy trade-offs.

Robustness, distribution shift, and shared benchmarks. A recurring finding is that detectors that perform extremely well in-domain can degrade substantially under distribution shift (new domains, new generators) or adversarial rewriting^5,10. Recent benchmarks emphasize robustness evaluation across models/domains/attacks, e.g., MAGE and RAID^11,12, and stress-test studies demonstrate that even moderate paraphrasing can reduce detection rates¹⁰. These results motivate reporting deployment-relevant operating points such as TPR at fixed low false positive rates (TPR@FPR), rather than relying solely on aggregate metrics like accuracy or AUROC.

Provenance and watermarking. Complementary to post-hoc detection, watermarking aims to embed algorithmically detectable signals into generated text with minimal perceptual impact, and recent surveys summarize watermarking techniques and evaluation dimensions (detectability, quality impact, robustness under attacks)^6,13. In this work, we focus on a practical, lightweight supervised detector for scientific abstracts and explicitly evaluate cross-domain (five scientific fields), cross-model (four strong unseen LLMs), and paraphrase robustness under a unified protocol, positioning our approach within the above detection and robustness landscape.

Methods

Our methodology for detecting AI-generated text is structured into three main stages, which are dataset construction, model architecture selection, and a rigorous experimental setup. This section details the procedures and configurations employed in each stage to ensure the validity and reproducibility of our findings. An overview of the end-to-end pipeline is shown in Fig. 1, from data curation and preprocessing through modeling and unified evaluation (including low-FPR operating points). Algorithm 1 summarizes our unified training and evaluation protocol, including document-level near-duplicate removal, explicit-cue stripping, stratified splits, threshold selection at fixed FPRs (1%/5%), and the stress-test suite (cross-domain, cross-model, and paraphrase levels).

Fig. 1

End-to-end pipeline for data curation, modeling, and evaluation. The Data & Preparation block aggregates human-written and AI-generated abstracts, followed by curation and de-duplication. Preprocessing performs tokenization, length normalization, and cleaning. The Modeling block fine-tunes a lightweight Transformer to output a score $s\in [0,1]$ with operating points selected at fixed false-positive-rate (FPR) levels (1% and 5%). The Evaluation & Analysis block reports AUROC, PR-AUC, macro-F1, and TPR@FPR, and compares against DetectGPT, GLTR, and a representative online detector.

Algorithm 1

Unified training and evaluation protocol.

Dataset construction

Our initial model development and main performance evaluation (reported in Sections Main Model Performance and Ablation Study and Baseline Comparison) utilized a primary dataset focused specifically on the Computer Vision domain (cs.CV). We collected human-written abstracts via the official arXiv API¹⁹ and generated AI counterparts from the paper titles using Gemini 2.0 Flash²⁰ (API; prompts and sampling parameters released).

To rigorously evaluate generalization capabilities (Section Generalization Analysis), we expanded our curation to include balanced human/AI abstract sets from five arXiv subject areas: cs.CV, eess.SP (Signal Processing), q-bio (Quantitative Biology), Physics, and CS (non-CV). For each domain, we collected 500 human-written abstracts and generated 500 AI counterparts using the same methodology. This yields 1,000 documents per domain (N=5,000 in total).

Leakage audit and confounder control. We remove near-duplicates via 5-gram MinHash/Jaccard with threshold $\tau =0.82$, strip explicit cues (URLs/code/metadata), and match length distributions via stratified sampling and truncation to 256 tokens. Splits are stratified by document ID to avoid cross-contamination. Thresholds are selected on validation only, and we report deployment-relevant operating points $\textrm{TPR}@\textrm{FPR}\in \{1\%,5\%\}$ alongside AUROC/PR-AUC/macro-F1 (Fig. 2).

Fig. 2

Leakage-audit and data-integrity workflow. The audit enforces (i) document-level near-duplicate removal using MinHash/Jaccard similarity (threshold $\tau =0.82$); (ii) explicit-cue stripping (URLs, code, metadata) and length matching; (iii) stratified dataset splits with no cross-contamination; and (iv) a unified evaluation protocol. Stress tests include cross-model/domain transfer and paraphrasing at light/medium/heavy strengths.

Model architecture

For the task of classifying text, we selected DistilBERT²¹ as our primary model architecture. DistilBERT is a distilled version of the BERT (Bidirectional Encoder Representations from Transformers) model²², specifically engineered for high performance with lower computational overhead.

Our choice was motivated by two key factors:

• DistilBERT’s smaller footprint, achieved through knowledge distillation (reducing BERT-base size by 40% while retaining 97% capability), makes it exceptionally suitable for fine-tuning on consumer-grade hardware.

• As a Transformer-based model, DistilBERT captures complex semantic relationships and subtle stylistic nuances, moving beyond superficial statistical features to better distinguish sophisticated AI-generated text.

Generative AI tools were used to (i) produce synthetic abstracts for the positive class and (ii) generate paraphrased variants for robustness testing, as detailed in Results. No generative AI system was listed as an author.

Experimental setup

To ensure the reliability and reproducibility of our results, we established a rigorous experimental protocol.

Data splitting

The final, shuffled dataset was partitioned into a training set and a testing set. We used 80% of the data for fine-tuning the model, while holding out the remaining 20% exclusively for final performance evaluation. We further split the training set into train/validation (e.g., 90/10) for threshold selection. We used a fixed random seed (random_state=42) during this process to guarantee that the exact same data split can be reproduced in future work .

Hyperparameter configuration

The model was fine-tuned using the hyperparameters listed in Table 1. These parameters were chosen based on established best practices for fine-tuning BERT-like models for text classification tasks.

Table 1 Hyperparameter settings for DistilBERT fine-tuning.

Evaluation metrics and statistical testing

To provide a comprehensive assessment, we report Accuracy, Precision, Recall, macro-F1, AUROC, and PR-AUC. For deployment-relevant regimes, we additionally report TPR at fixed FPR levels of 1% and 5%, with thresholds selected on the validation set. Uncertainty is quantified via stratified bootstrap over documents (95% CIs), and AUROC differences are assessed with DeLong’s test. Calibration is summarized by ECE and Brier score (with post-hoc temperature scaling)^24,25.

Algorithm 2

Training and Evaluation Protocol for the Lightweight Transformer.

Results

This section presents the empirical results of our study. We first report the in-domain performance of our fine-tuned DistilBERT model and compare it against several baselines. Subsequently, we detail the generalization analysis across different domains and models, followed by an evaluation of the model’s adversarial robustness against paraphrasing attacks.

Main model performance

Our primary model, a DistilBERT classifier fine-tuned on our custom dataset (cs.CV domain), demonstrated exceptional performance in distinguishing between human-written and AI-generated scientific abstracts. The model achieved an overall accuracy of 99.40% and a F1-score of approximately 99.40% on the held-out test set.

A more detailed breakdown of the performance was provided by the confusion matrix in Figure 3. The matrix showed that the model correctly identified 499 out of 500 AI-generated abstracts and 495 out of 500 human-written abstracts. Notably, the model exhibited a very high recall of 99.80% for the AI class, indicating its capability to detect nearly all instances of synthetic text with only a single false negative.

At low false-positive regimes, the classifier retains strong recall: $\textrm{TPR}@\textrm{FPR}{=}1\%=0.90$ and $\textrm{TPR}@\textrm{FPR}{=}5\%=0.97$ (on the non-paraphrased test set with thresholds fixed on validation).

Fig. 3

Confusion matrix for the DistilBERT model on the test set. The model shows high accuracy for both Human (0) and AI (1) classes.

To further assess the model’s discriminative power, we plotted the Receiver Operating Characteristic (ROC) curve, shown in Figure 4. The Area Under the Curve (AUC) reached an outstanding 0.9999, a value extremely close to the ideal of 1.0. This confirms that the model is highly robust and maintains excellent performance across all classification thresholds.

Fig. 4

ROC curve for the DistilBERT model.

Ablation study and baseline comparison

To contextualize the performance of our DistilBERT model, we conducted an ablation study comparing it against a wide range of baselines, including other Transformer architectures, zero-shot methods (DetectGPT, GPTZero, GLTR), and traditional machine learning models (Logistic Regression, Random Forest, SVM). Traditional models were trained on features extracted using TF-IDF²⁷ and implemented with Scikit-learn²⁸.

Table 2 Comprehensive performance comparison of our proposed DistilBERT model against a wide range of baseline models on the test set. The models are grouped by their architectural class. Our model demonstrates superior performance across all primary evaluation metrics.

The comparative results are summarized in Table 2. Our proposed DistilBERT model outperformed all baseline models across all key metrics. While the SVM model also achieved a high accuracy of 98.3%, our model provided a significant performance improvement, establishing a new excellent result on our dataset.

Interestingly, an analysis of the feature importance for the Logistic Regression model reveals that the baseline models heavily rely on superficial statistical cues, as shown in Table 3. For instance, words such as ‘https‘ and ‘github‘ were strong indicators of human-written text, likely because authors often include code repository links in their abstracts. Conversely, formulaic academic terms like ‘novel‘, ‘leverages‘, and ‘addresses the challenge‘ were highly indicative of AI-generated text. This suggests that while traditional models can achieve high performance, they do so by exploiting shallow heuristics rather than deep semantic understanding. Our DistilBERT model, in contrast, is designed to move beyond such cues and learn the underlying linguistic patterns.

Table 3 Top 10 most important features for the Logistic Regression model, based on TF-IDF weights.

Generalization analysis

A practical detector must be able to identify text from various sources, not just the specific model or domain it was trained on. To rigorously evaluate the real-world applicability of our DistilBERT-based detector, we conducted experiments to assess its generalization capabilities across different LLMs (cross-model) and different scientific fields (cross-domain).

Cross-model generalization

We tested our detector’s performance on datasets consisting of abstracts from the original ‘cs.CV‘ domain but generated by unseen LLMs. We evaluated on Qwen-3³², GPT-4o, Claude 3.5, and Llama-3.1-70B under the same unified protocol.

The results are summarized in Table 4. The detector demonstrated strong cross-model generalization, maintaining a Macro-F1 score of 95.0% against Qwen-3, and consistently high performance against other advanced models (e.g., 94.5% against GPT-4o). The high AUC scores (e.g., 0.9880 for Qwen-3) further indicate that the model learned features that are not specific to the Gemini architecture, confirming its robust generalization capabilities across different LLMs.

Table 4 Robustness and generalization analysis of the detector. We report in-domain performance on cs.CV, cross-model generalization to unseen LLMs, and robustness under GPT-4o paraphrasing attacks. Cross-domain (held-out domain) results are summarized separately in Table 5.

Cross-domain generalization (LODO)

To evaluate out-of-domain generalization beyond the in-domain cs.CV setting, we used the expanded multi-domain dataset spanning five scientific categories. We report held-out results on four non-CV target domains (Biology/q-bio, Physics, CS non-CV, and Signal Processing/eess.SP) using a LODO-style protocol: for each target domain d, we train on the remaining domains (including cs.CV) and evaluate on d. Results are summarized in Table 5.

LODO evaluation. Following distribution-shift best practice (e.g., WILDS³⁶), we adopted a leave-one-domain-out (LODO) protocol: for each target domain d, we train on all other source domains, tune hyperparameters on the union of their validation sets (no target data used), and report AUROC, macro-F1, and PR-AUC on d with 95% confidence intervals computed via stratified bootstrap over documents. AUC differences are assessed with DeLong’s test.

Per-domain results (LODO). Table 5 summarizes the domain-specific performance under the unified protocol. Each domain contains 500 human and 500 AI abstracts (total $N{=}1000$ per domain). Under LODO, cross-domain averages reach AUROC= 0.982 and macro-F1= 0.948 with tight 95% CIs (typically $\le$ 0.02 on AUROC and $\le$ 2 pp on macro-F1). The worst-domain macro-F1 remains $\ge$ 0.92. These results indicate consistent out-of-domain generalization beyond the original cs.CV domain.

Table 5 Per-domain performance under the unified LODO protocol. Values are mean ± 95% CI over documents.

Adversarial robustness

To simulate realistic evasion attempts, we evaluated the detector’s resilience against paraphrasing attacks of varying strengths (Light, Medium, Heavy). We employed GPT-4o to paraphrase the AI-generated abstracts in our original test set. This approach realistically mimics an adversary’s attempt to alter surface-level stylistic features while preserving the core scientific meaning.

As shown in Table 4, the detector’s performance significantly degraded under attack. Under the medium attack, the F1-score dropped sharply to 82.1%, primarily driven by a substantial decrease in recall to 70.4%. This indicates that nearly 29.6% of the paraphrased AI texts successfully evaded detection.

The degradation is visualized in Figure 5, which compares the ROC curves of the detector on the baseline test set versus the medium paraphrased test set¹⁰. The area under the curve (AUC) dropped from 0.9999 to 0.9765, confirming that the model’s discriminative power is compromised by sophisticated paraphrasing. At fixed operating points, the medium paraphrase setting yields TPR@1%FPR $=0.62$ and TPR@5%FPR $=0.78$.

Comparative Robustness. While the performance of our model degrades under attack, it is crucial to assess its robustness relative to other detection methods. We subjected the baseline models–including other Transformers, DetectGPT, GLTR, and GPTZero–to the identical moderate paraphrasing attack and evaluated them using the same low-FPR metrics.

The results, summarized in Table 6, demonstrate that robustness challenges are pervasive across all methods, yet our DistilBERT-based detector maintains a significant advantage. For instance, at 1% FPR, our model achieves a TPR of 0.62, whereas the next best Transformer (DeBERTa-v3-small) achieves 0.58. Zero-shot and statistical methods are significantly less robust, with DetectGPT and GPTZero achieving TPRs of only 0.14 and 0.19, respectively. This comparison highlights that although supervised methods are vulnerable to sophisticated paraphrasing, they currently offer substantially better robustness than alternative approaches.

Table 6 Comparative robustness against moderate paraphrasing attacks. Our model maintains superior performance at low False Positive Rates (FPR) compared to baselines.

Fig. 5

ROC curves for the detector on the baseline test set versus the paraphrased (attacked) test set.

Discussion

The results presented in the previous section demonstrate that our DistilBERT-based classifier can detect AI-generated scientific abstracts with exceptionally high accuracy (99.4%). In this section, we interpret these findings, discuss the implications of our ablation study, acknowledge the limitations of our work, and propose directions for future research.

Our model’s high performance, particularly its AUC score of 0.9999, suggests that current generative models, despite their sophistication, still leave behind subtle, detectable artifacts in the text they produce. The model’s high recall for the AI class (99.8%) is especially noteworthy. It implies that our approach is highly effective at identifying synthetic text, making it a reliable tool for screening purposes where minimizing false negatives is critical.

The most insightful findings, however, emerge from our ablation study. The baseline models demonstrated strong performance, with SVM, for example, achieving a high accuracy of 98.3%, as shown in Table 2. To understand how these traditional models succeed, we analyzed the feature importance of the Logistic Regression model, as presented in Table 3, which revealed that their success is likely predicated on superficial and potentially brittle heuristics. The analysis shows these models learn to associate explicit keywords like https and github with human authorship, and formulaic academic phrases like novel approach and paper addresses with AI authorship. This reliance on statistical shortcuts poses a significant risk. Such a detector could be easily evaded if AI models begin to incorporate placeholder links or if human authors adopt a more formulaic writing style.

In contrast, our DistilBERT model’s strength lies in its ability to move beyond these surface-level cues. By leveraging its deep, contextual understanding of language, it learns the more essential semantic and syntactic patterns that differentiate human and machine-generated text. This suggests that our model is not merely a keyword spotter but a more robust classifier that has learned a more fundamental representation of linguistic style. This inherent robustness is the primary advantage of our approach over traditional methods, even when the headline performance metrics appear numerically close.

Implications of robustness and generalization tests

Our comprehensive testing, detailed in Section 3, reveals a nuanced picture of the detector’s real-world capabilities. The model’s strong performance in cross-domain and cross-model scenarios suggests that it has successfully learned to identify intrinsic, generalizable artifacts of AI-generated text, rather than merely overfitting to the stylistic conventions of a single domain or the specific quirks of one LLM. The ability to maintain an AUC about 0.9880 when faced with an unseen model like Qwen-3 is particularly encouraging, indicating that the underlying patterns of machine-generated language share enough commonalities to be detected by a well-trained classifier.

Fig. 6

Feature analysis of misclassified samples. The boxplots show the distribution for the five false positives (Human $\rightarrow$ AI), while the orange marker represents the single false negative (AI$\rightarrow$ Human). The analysis compares (a) text length in characters, (b) lexical diversity, (c) average sentence length in words, and (d) the confidence distribution of the model’s incorrect predictions.

However, the most critical insight comes from the model’s response to a moderate paraphrasing attack. While the overall AUC remained high at 0.9765, this figure masks a critical vulnerability, namely a dramatic drop in recall from 99.8% to 70.4%. This demonstrates that while the detector remains highly precise (i.e., its positive predictions are reliable), its sensitivity is severely compromised, allowing nearly 30% of adversarial examples to go undetected. This suggests that the features learned by the model, while generalizable, are still heavily reliant on surface-level linguistic patterns, such as sentence structure and word choice that can be effectively obscured by a sophisticated paraphrasing agent like GPT-4o.

This vulnerability has profound implications. It suggests that for high-stakes applications, such as ensuring academic integrity, relying solely on a single detection tool is insufficient, as it may create a false sense of security. Future research must therefore pivot towards more fundamentally robust detection paradigms. This could include models that focus on semantic inconsistency, logical fallacies, or other deeper cognitive artifacts that are more difficult to manipulate through simple text rewriting. Our findings serve as a crucial benchmark, quantifying the performance gap that must be closed to create truly reliable and trustworthy AI detection systems.

Error analysis

To better understand the model’s limitations and failure modes, we conducted a detailed error analysis on the six misclassified samples in the test set. Our findings reveal distinct patterns for false positives (human-written text classified as AI) and the single false negative (AI-written text classified as human), as illustrated in Figure 6.

The five false positive samples shared a common characteristic. They were written in a highly structured, formulaic academic style. For instance, these abstracts used phrases common in AI-generated text, such as “we propose,” “novel framework,” “significant improvements,” and “extensive experiments.” As shown in our feature analysis (Figure 6a-c), these human-written texts exhibited longer average sentence lengths and high lexical diversity, closely mimicking the stylistic patterns of the AI-generated content in our training data. The model’s high confidence in these misclassifications (Figure 6d, four out of five with >99% confidence) suggests it has learned to associate this formal, polished writing style strongly with AI generation.

Conversely, the single false negative sample, an AI-generated abstract that evaded detection–was notably shorter and had lower lexical diversity compared to the false positives. The model’s lower confidence (0.425) in this incorrect prediction indicates that this sample represents a “boundary case” that does not fit neatly into the learned patterns.

This analysis suggests that while our model is highly effective, its primary challenge lies in distinguishing human-written abstracts that adopt a formal and almost “template-like” structure from genuine AI-generated text. Future work could address this by incorporating more diverse examples of human academic writing into the training set to improve the model’s ability to discern more subtle authorship cues beyond stylistic formality.

Calibration and uncertainty

We plot reliability diagrams (SI) and summarize calibration using the expected calibration error (ECE) and Brier score. As detailed in Table 7, the scores are well-calibrated overall (ECE = 0.022) and remain consistent across different text length buckets. Post-hoc temperature scaling provides only minor improvements, indicating the model’s inherent calibration is strong.

Table 7 Calibration and uncertainty (overall and by text length).

Limitations and future work

Despite the promising results, we acknowledge several limitations. Firstly, while our study represents a significant step towards multi-domain generalization by evaluating across five diverse scientific fields, the model’s performance on texts from disciplines with vastly different writing conventions, such as the humanities or law, remains unexplored. Similarly, although we demonstrated robustness against four powerful unseen LLMs, a broader spectrum of models, especially those not based on the Transformer architecture, would be required for a truly comprehensive assessment.

Second, our investigation into adversarial robustness, while systematic across three levels of paraphrasing intensity (light, medium, and heavy), was focused on automated rewriting attacks. The model’s resilience against different types of attacks, such as human-AI collaborative writing or more sophisticated semantic alterations, was not assessed.

These limitations pave the way for several exciting avenues for future research. A crucial next step is to build upon our findings by developing detectors that are fundamentally more robust against various adversarial paraphrasing attacks³⁷. Our results pinpoint a significant drop in recall as the primary failure mode, suggesting that future work should focus specifically on techniques to improve model sensitivity to manipulated texts without sacrificing precision. Furthermore, expanding the evaluation to encompass a much wider array of academic disciplines is essential for creating a truly universal detection tool. Finally, exploring feature attribution methods to understand precisely which linguistic patterns are obscured during paraphrasing could yield valuable insights for designing next-generation, attack-resistant detectors.

Conclusions

In this study, we addressed the growing challenge of detecting AI-generated text within the specialized domain of scientific writing. We successfully developed and validated a highly effective method for distinguishing between human-written and AI-generated scientific abstracts, providing a valuable tool for upholding academic integrity.

Our primary contribution is the demonstration that a lightweight Transformer model, DistilBERT, can be fine-tuned to achieve excellent performance, reaching an accuracy of 99.4% and an AUC of 0.9999 on a custom-built dataset of computer vision abstracts. More significantly, our ablation study revealed that while traditional machine learning models can achieve high accuracy, they often rely on superficial and fragile heuristics, such as the presence of URLs. In contrast, our DistilBERT-based approach learns deeper semantic and syntactic patterns, offering greater robustness and a more genuine understanding of the linguistic differences between human and machine-generated text.

This work provides not only a highly accurate detector but also a reproducible benchmark for future research in this area. The findings confirm that despite the increasing sophistication of LLMs, their outputs retain detectable artifacts. The proposed model is both efficient and effective, making it a practical solution for deployment by academic publishers and institutions. Future work will focus on expanding the dataset to encompass more diverse academic domains and testing the model’s resilience against sophisticated adversarial attacks.