Detecting AI-Generated Code Comments
A classifier helps identify AI-authored documentation by distinguishing synthetic styles from human writing.
A newly released code comment detector provides a way to distinguish between human and machine-authored documentation with a balanced accuracy of 77%. Unlike previous versions built on opaque private data, this release relies on public repository data, making the detection mechanism verifiable. The tool offers a feature activation view that allows users to inspect specific segments of text. This view highlights which tokens or linguistic patterns triggered a robot or human label, exposing the model's reliance on specific stylistic markers.
Performance and Calibration
The classifier achieves its 77% accuracy by balancing the likelihood of human and machine inputs. More useful than the raw accuracy score is its output of a calibrated probability percentage. The underlying model uses Platt scaling, a method that fits a logistic regression model to the outputs of the classifier, to map raw scores into a probability space. When the confidence is 80% or more, the empirical risk of a false positive drops to 5%. If the model is uncertain and outputs a probability around 50%, the verdict is effectively a coin flip.
| Metric | Human-Written (TPR/TNR) | Robot-Generated (TPR/TNR) |
|---|---|---|
| Correct Judgment | 73% | 80% |
| Error Rate | 27% | 20% |
These metrics were derived from cross-validation and tested against non-synthetic, real-world comments. The classifier performed well on these samples. This suggests that the synthetic data generation process—stripping existing comments from files and asking LLMs to re-comment them—successfully isolates the style of machine writing without being tethered to specific, unrepresentative code bases.
Avoiding Subject Matter Leakage
The primary engineering challenge in building this detector is preventing subject matter leakage. This occurs when the classifier ignores the style of the comment and instead learns to recognize the repository or programming language itself. If the model sees code-specific keywords like std::vector or interface as features for AI-generated because those files happened to have AI comments, the detector becomes a code classifier rather than a comment detector. The builder must minimize these effects to ensure the model focuses on stylistic markers such as the overuse of transition words, repetitive sentence structures, or the lack of technical nuance that often accompanies generated prose.
Practical Diagnostic Usage
A developer reviewing a pull request for a legacy Java project can use the tool to confirm the automated nature of comments. If the AI-assisted assistant adds comments that are technically correct but structurally monotonous, the tool provides a classification. Because you can inspect feature activation, you can see if a block of code is being flagged because of common robot-isms or because of actual substance. The tool is calibrated specifically for code comments, which are significantly different in structure and intent from general prose. The model is currently distributed as an open-source library, and users can load it via a standard Python package manager to run inference locally on their repositories. What remains unknown is how the model will handle emerging LLMs with evolving prompting styles, as the current training set is limited to earlier generation models.