I Tested Hoppscotch With AI and No Code Selectors - Here's What I Learned

When I first heard about Passmark, my reaction was: "Plain English tests? That sounds too good to be true."

So I decided to put it to the test, literally. I picked Hoppscotch, one of the most popular open-source API clients, and wrote a full Playwright test suite using nothing but human-readable descriptions. No CSS selectors. No XPath. No page objects.

Here's everything I built, everything that broke, and what it taught me about the future of QA.

What Is Passmark?

Passmark is an open-source AI testing library by Bug0. Instead of writing brittle selectors like div.button-wrapper > button:nth-child(2), you write things like:

"Click the Send button next to the URL bar"

Passmark uses AI (via OpenRouter) and Playwright under the hood to interpret those instructions and execute them in a real browser. Your assertions are also plain English:

"A response panel shows Status 200 and JSON data"

It feels like writing a user story, except the computer actually runs it.

Why I Picked Hoppscotch

Hoppscotch is a public web app with no login required for basic use, no CAPTCHA, and rich UI interactions; perfect for AI-driven testing. It has distinct flows: REST requests, GraphQL, Collections, Headers, Parameters. Lots to test.

My Test Suite — 9 Tests, 5 Files

Here's what I built: github.com/yusfate4/breaking-apps-hoppscotch

File 1 — REST Client (01-rest-client.spec.ts)

test("Send a GET request and see the response", async ({ page }) => {
  await runSteps({
    page,
    userFlow: "Send a GET request using Hoppscotch REST client",
    steps: [
      { description: "Navigate to https://hoppscotch.io" },
      {
        description: "Click the URL input bar and clear it, then type a new URL",
        data: { value: "https://jsonplaceholder.typicode.com/posts/1" },
      },
      { description: "Click the Send button" },
    ],
    assertions: [
      { assertion: "A response panel shows Status 200 and JSON data" },
    ],
    test, expect,
  });
});

This was my first test and it passed immediately. Passmark found the URL bar, typed the endpoint, clicked Send, and confirmed the 200 response, all without me writing a single selector.

File 2 — HTTP Methods (02-http-methods.spec.ts)

Two tests: switching the method from GET to POST, and opening the Body tab to add a JSON payload. Both passed after I made the step descriptions more specific.

Key learning: The more specific your step description, the better Passmark performs. "Click the HTTP method dropdown that shows GET" works much better than "Change the method."

File 3 — Headers & Parameters (03-headers-params.spec.ts)

Adding custom headers and query parameters. These were the trickiest tests. The Headers tab caused repeated timeouts, Passmark kept getting confused because the step "Click on REST in the left sidebar" was unnecessary (Hoppscotch opens on REST by default).

Fix: Remove any step that assumes a state that's already true. Once I removed the redundant navigation step and described the Headers tab location precisely, it passed cleanly.

File 4 — GraphQL (04-graphql.spec.ts)

Navigating to the GraphQL section and entering an endpoint URL. These passed first try, GraphQL has clear navigation links that the AI identified easily.

File 5 — Collections (05-collections.spec.ts)

Opening the Collections panel and creating a new collection named "Hackathon Suite." The Collections button on the right sidebar was easy to find once I described it as "the button labeled Collections on the right side of the screen" rather than "the Collections icon or tab."

What Broke and Why

The most interesting failures were not bugs in Hoppscotch, they were lessons in how to communicate with AI.

Problem 1: Parallel test execution. When I ran tests with 2 workers, they crashed each other. The fix was setting workers: 1 in playwright.config.ts. AI-driven tests need sequential execution, each test takes over the full browser context.

Problem 2: Token limit errors. Some tests failed mid-step with "Provider returned error." The cause: Hoppscotch's DOM is large, and multi-step conversations with lots of back-and-forth hit the AI model's context limit. The fix was reducing step count per test and removing unnecessary intermediate steps.

Problem 3: Redundant steps confuse the AI. Steps like "Click on REST in the left sidebar" when you're already on the REST page caused the AI to click the link, which triggered a page reload, which then timed out waiting for state that was already there.

The pattern across all failures: over-describing the path instead of describing the destination.

Setup in 5 Minutes

npm init playwright@latest my-tests
cd my-tests
npm install passmark dotenv

Create .env:

OPENROUTER_API_KEY=your_key_here

Add to playwright.config.ts:

import { configure } from "passmark";
configure({ ai: { gateway: "openrouter" } });

Register at the Breaking Apps Hackathon page to get a free OpenRouter API key.

What This Means for QA

Traditional Playwright tests break every time a developer renames a class or restructures the DOM. Passmark tests describe intent, not implementation. If the Send button moves but still says "Send," the test still passes.

This is a fundamentally different relationship between tests and UI. It's slower to run (AI calls take seconds each), but the tests are dramatically more resilient and readable by non-engineers.

The question isn't whether AI testing will replace traditional testing. It's how quickly teams will adopt it for the flows that matter most, the ones that get manually tested before every release because nobody trusts the automated suite.

Final Results

✅ 9 tests across 5 files — all passing ⏱ Total run time: ~8.5 minutes (sequential) 📁 Repo: github.com/yusfate4/breaking-apps-hoppscotch

Built for the Breaking Apps Hackathon by Bug0. Every registered participant gets free AI credits via OpenRouter. Go build something.

#BreakingAppsHackathon #breakingappshackathon #Playwright #AITesting #OpenSource #ai #SoftwareTesting #Passmark #Hoppscotch #Bug0 #bug0inc