Building BlueSkills: An agent skill scanner tested in public

An agent skill can look like a small Markdown file. Once installed, it may guide an agent that can read your files, run commands, and use your connected tools. BlueSkills is a free online security scanner for AI agent skills built by Bluethroat Labs. I built it to give people a check before that handoff: submit a SKILL.md, a public repository, or a ZIP package at BlueSkills, then read the findings and coverage before deciding whether to install it.

The internship assignment that became a product

In July 2026, I was between jobs when I saw Rahul Saxena's invitation to a technical internship at Bluethroat Labs. One assignment option was to build a security scanner for agent skills. I chose it because the question was immediate: what could go wrong when someone downloads a stranger's skill and lets an agent follow it?

The assignment came from Rahul's own close call. He had installed a skill he found on X and run it in Codex before reading what it asked the agent to do. Codex could access his laptop. The skill turned out to be benign, but he had already made the trust decision without checking it. For me, that made the problem concrete: the check had to fit into the moment before someone installs a new skill.

My first answers were credential-stealing code and instructions that redirect the agent. I started building before researching every existing scanner. For code, I looked for paths from sensitive files to outbound requests. For instructions, I tried pattern checks and a prompt-injection classifier, then added multilingual similarity checks against examples of attacks. None of those methods was enough on its own. I also added an LLM review stage, but gave it a strict limit: it could raise a suspicious result to malicious, never talk an earlier finding away.

The interview made me nervous. Python was not my primary language, and this was only my second interview while between jobs. It became one of the best interviews I have had: Rahul and I spent most of it discussing the scanner, crypto security, and what it would take to ship something people could actually use. After the interview, I joined Bluethroat Labs and kept building BlueSkills.

People often discover skills through a GitHub link, a social post, or a recommendation from another agent. Popularity is a reason to look; it is not evidence that a particular revision is safe to run. BlueSkills is meant to give them an accessible check at that moment of trust.

What changed after the prototype

The assignment prototype was largely a static check. A real skill can carry scripts, hooks, dependencies, and supporting files beyond SKILL.md; it can also behave differently once an agent runs it. We widened the unit of review to the submitted package and added an optional isolated runtime observation stage. I worked on a cloud design that did not require a virtual machine running all day.

Rahul pushed me to measure what each layer contributed. When is runtime testing necessary? When does it add cost without changing a decision? What happens if a run times out or never reaches the suspicious behavior? Our standard became clearer: a quiet run cannot erase a finding or turn incomplete coverage into a claim of safety. A report has to say what it found, where, why it matters, and what it actually examined.

That scope matters when you use the tool. Pasting SKILL.md submits one file; it does not fetch every script mentioned inside it. A repository or ZIP lets BlueSkills examine bundled files. A CLEAN result means this scan found no supported issue in the material it covered. It is not a certificate that the skill will behave safely every time, on every platform, with every future version.

The challenge that humbled me

We first made BlueSkills available through a Telegram bot. Internal tests used malicious samples and benign controls, and I felt confident enough that I did not expect many successful submissions when we opened a small public challenge. I was wrong. Three prize-winning submissions exposed different blind spots: a poisoned address table that led agents to alter an EVM deployment, an SSH-key theft path written to activate on macOS while our runtime used Linux, and instructions to archive hidden files before uploading them. The last case also exposed a limit in how our runtime looked for planted credentials leaving the test environment.

Those results changed how I think about the product. The scanner must give useful evidence, but its confidence cannot exceed its coverage. I explain the three challenge findings and what they taught us in a companion article.

Kruz built the web interface so people could scan without Telegram and inspect the report more easily. We later added a CLI so an agent could submit the exact skill revision before installation.

Why build this when other scanners exist?

I found NVIDIA's SkillSpector after starting the prototype. Its public work is useful, and we evaluated it alongside our own approach. We still wanted a free, no-install way to review the submitted package, show evidence and coverage clearly, and improve against failures we could reproduce ourselves. Existing tools gave us things to learn from; they did not settle the question for someone about to trust a particular skill.

BlueSkills' exact analyzers and thresholds remain closed source. If we published the complete check list, an attacker could feed those rules to a capable model and ask it to rewrite a malicious skill around them. That choice does not make BlueSkills unbreakable: anyone can still test a public service, and the challenge showed that we can miss things. We can be open about the threat model, the limits, and what testing teaches us without handing out every detection rule. Other public-good tools we build at Bluethroat Labs can be open source where publishing their internals does not weaken their purpose.

How do I scan an AI agent skill before installing it?

Before installing an unfamiliar skill, submit the exact version you intend to use at blueskills.bluethroatlabs.com. Prefer the repository or ZIP option if you want bundled files examined; pasted SKILL.md is a one-file check. Read the findings and coverage notes, inspect the skill yourself, and consider what access your agent will have. Submit only public material; do not upload private packages or secrets.

If you build or operate an agent, make scanning a checkpoint before it installs a third-party skill. The Bluethroat CLI can submit a public repository revision, local skill directory, ZIP, or SKILL.md, wait for the report, and print its findings. After a GitHub device login approved by the user, an agent can run bluethroat blueskills scan <source>, show the completed result, and ask the user to approve that exact revision before installing it. The website's human verification is for manual scans; automated scans should use the CLI. BlueSkills improves the decision; it does not make it for you.

The scanner is free to use. Our public repository provides project information and a place to report problems.