How To Detect AI-Generated Code with CodeLeaks

With LLMs running rampant in education, teachers are forced to adapt by implementing AI detection tools into their arsenal. However, most AI detectors only extend to text, but we all know that there’s more than one kind of assignment.

For instance, what about code? 

No worries — CopyLeaks has teachers covered with their feature called CodeLeaks. The only question is, how accurate is it actually? That’s what we’ll discuss in this article, along with how to use CodeLeaks and my overall opinions about it. Stay tuned!

What is CopyLeaks?

CopyLeaks is a platform made to ensure AI misuse and plagiarism gets contained to a minimum. It’s a suite of tools that utilizes advanced algorithms and emerging technologies to dissect text, documents, and even code. 

True to their slogan of “Empowering Originality and Inspiring Authenticity,” CopyLeaks’ most popular features are their plagiarism checker and AI content detector. We’ve tested the latter using our own dataset and found it to be 75% accurate in true positive tests (beating the likes of Content at Scale and Originality) and 80% in false positive tests (which is the second highest score across eight detectors).

What is CodeLeaks?

CodeLeaks is a specific feature of CopyLeaks that targets plagiarized code either from pre-existing codebases or an LLM. Every code input will generate a full report complete with a highlight on copied code and where they’re from, percentage plagiarized, and more. We’ll dive deeper into this later.

How To Detect AI Code Using CodeLeaks?

Step #1: Create An Account

To start detecting code using CodeLeaks, you need an account. Simply head to their dashboard, and then select the “Login” or “Create Account” button on the top-left side of the screen.

Step #2: Upload Your Code

Now, you should have full access to their dashboard. To confirm, you should see these six choices on the center of your screen. From there, select the “Code” option.

Once you’re in, simply drag a code file into the dashboard and all that’s left to do now is the last step.

Step #3: Get A Detailed Report

Before we proceed, let me generate a Python code using ChatGPT and save it as a .py file. So, I asked ChatGPT to create a code based on Fizzbuzz, a popular Leetcode question. 

The exercise goes like this: You need to efficiently print all numbers from 1 to 100, but for multiples of 3, there must be a “FIZZ” instead of the number; for multiples of 5, there must be a “BUZZ,” and for multiples of both, the output must be “FIZZBUZZ.”

Here’s what ChatGPT gave me:

Let’s save that as a .py file and upload it to CodeLeaks. Here’s the output:

Compared to code plagiarism analysis, AI code analysis only gives you one key information about the input: the percentage likelihood that it came from an AI. 

How Accurate is CodeLeaks?

Now that you know how CodeLeaks works, it’s time to test and find out how accurate it is at detecting AI code. This test will be divided into two parts: true positive and false positive. The latter is for AI-generated code, while the latter will measure if CodeLeaks can detect human code. So, without further ado…

True Positive Tests

Test #1 — AI successfully detected!
AI Likelihood Score: 100%

Test #2 — AI successfully detected!
AI Likelihood Score: 100%

Test #3 — AI successfully detected!
AI Likelihood Score: 100%

Test #4 — AI successfully detected!
AI Likelihood Score: 100%

Test #5 — AI successfully detected!
AI Likelihood Score: 100%

False Positive Tests

Test #1 — Failed, AI detected in human content.
AI Likelihood Score: 100%

Test #2 — Human content successfully detected!
AI Likelihood Score: 0%

Test #3 — Human content successfully detected!
AI Likelihood Score: 0%

Tallied Score and Thoughts on CodeLeaks’ Accuracy

I didn’t expect CodeLeaks to be this accurate, but it is. Despite having one false positive result, the fact that it successfully detected the sample data as AI or human 7 out of 8 times is a remarkable feat on its own. What’s more is that CodeLeaks was absolutely certain (0% or 100% AI likelihood scores) of their analysis, which mostly turned out to be correct.

It’s also interesting to see that CopyLeaks seems to be more accurate in detecting AI in code than traditional text. I believe that comments play a huge factor in these results, as the only thing that the AI-generated codes and the one false positive test had in common was an abundance of comments and annotations.

The Bottom Line

In a world where AI detection receives so much scrutiny, CopyLeaks continues to not disappoint. We already know that it’s a capable AI detector for text, but who knew it was this good at detecting AI code too?

It’s a good sign that AI detection, whether it’s text or code, is heading in a more positive direction. OpenA caught flack for saying that detection isn’t reliable, even though they were absolutely right. But now, AI detection tools are evolving along with LLMs — and CopyLeaks might be at the forefront of that change.

Want to learn more about CopyLeaks? You can read more about it in our articles like this one. Good luck!