Spaces:

kdcyberdude
/

HARvestGym

Sleeping

App Files Files Community

HARvestGym / GROUND_TRUTH_EXTRACTION.md

kdcyberdude

Upload folder using huggingface_hub

e6ce96e verified 15 days ago

preview code

raw

history blame contribute delete

21.1 kB

Ground Truth Extraction

Extract the API catalog from each live WebArena container by connecting to EC2 from Cursor, entering each container, and running Claude Code with the prompts below.

Container names confirmed from the running EC2 instance:

App	Container name	Image
Shopping	`shopping`	`shopping_final_0712`
Shopping Admin	`shopping_admin`	`shopping_admin_final_0719`
Forum	`forum`	`postmill-populated-exposed-withimg`
Wikipedia	`kiwix33`	`ghcr.io/kiwix/kiwix-serve:3.3.0`
Map (web)	`openstreetmap-website-web-1`	`openstreetmap-website-web`

Skipping GitLab for now (gitlab) - facing some 502 errors.
Wikipedia (Kiwix) serves a static ZIM file — there is no source code to analyze. Its catalog entry is hardcoded at the bottom of this file.

Connection workflow

Cursor → Remote SSH → EC2 host → Dev Containers extension → Attach to Running Container → pick app container → paste prompt into Cursor sidebar

Step 1 — Connect from Cursor to EC2

Cmd+Shift+P → Remote-SSH: Connect to Host → ubuntu@<EC2_IP>

Step 2 — Attach to a container

With the Remote-SSH window open: Cmd+Shift+P → Dev Containers: Attach to Running Container → select the container (e.g. shopping).

Cursor opens a new window with the container's filesystem as the workspace. The full source code is now loaded and indexed — no copying needed.

Step 3 — Paste the prompt into the Cursor sidebar

Open the AI chat sidebar, paste the prompt for that container (see sections below), and run it. Cursor's AI has the full codebase in context and will write api_catalog.json into the workspace (inside the container).

Step 4 — Copy the output back

The file written inside the container can be downloaded via File → Download from Cursor's remote explorer, or via scp from the EC2 host after the fact.

Repeat for each container: shopping, shopping_admin, forum, openstreetmap-website-web-1.

Expected output format

The catalog captures all API surface found in the codebase — REST, GraphQL, WebSocket, form submissions. The api_type field distinguishes them. Do not restrict to only the endpoints used by the 7 training tasks; document everything.

[
  {
    "api_type": "rest",
    "endpoint": "POST /rest/V1/guest-carts/{cartId}/items",
    "auth": "none",
    "path_params": {
      "cartId": {
        "type": "string",
        "source": "PREV_CALL",
        "from_endpoint": "POST /rest/V1/guest-carts",
        "from_field": ".body",
        "notes": "entire response body is the cartId string"
      }
    },
    "body_params": {
      "cartItem.sku":      { "type": "string", "source": "PREV_CALL", "from_endpoint": "GET /rest/V1/products", "from_field": ".items[0].sku" },
      "cartItem.qty":      { "type": "number", "source": "TASK_SPEC" },
      "cartItem.quote_id": { "type": "string", "source": "DERIVED", "same_as": "cartId" }
    },
    "response_key_fields": []
  },
  {
    "api_type": "graphql",
    "endpoint": "POST /graphql",
    "operation_name": "GetProducts",
    "operation_type": "query",
    "auth": "none",
    "variables": {
      "search":    { "type": "String", "source": "TASK_SPEC" },
      "pageSize":  { "type": "Int",    "source": "STATIC", "value": 20 }
    },
    "response_key_fields": [".products.items[].sku", ".products.items[].name"]
  },
  {
    "api_type": "websocket",
    "endpoint": "ws:///realtime",
    "auth": "session_cookie",
    "notes": "describe message protocol; include event names and payload shapes"
  },
  {
    "api_type": "form",
    "endpoint": "POST /submission/create",
    "auth": "session_cookie+csrf",
    "content_type": "application/x-www-form-urlencoded",
    "form_params": {
      "_token": { "type": "string", "source": "AUTH_FLOW", "notes": "hidden input in page HTML" },
      "title":  { "type": "string", "source": "TASK_SPEC" },
      "url":    { "type": "string", "source": "TASK_SPEC", "notes": "optional if body provided" }
    },
    "response_key_fields": []
  }
]

**api_type:** rest | graphql | websocket | form

**source:**

TASK_SPEC — given in the task description
PREV_CALL — from a prior response this episode; specify from_endpoint + from_field
AUTH_FLOW — token / cookie / CSRF from the login flow
STATIC — hardcoded in the app; document the actual value
DERIVED — aliased from another value (e.g. quote_id = cartId)

App 1 — Shopping and Shopping Admin (Magento 2)

Container: shopping
Source root: /var/www/magento2/ (confirmed — WebArena README runs docker exec shopping /var/www/magento2/bin/magento ...)

Attach to the shopping container in Cursor, open /var/www/magento2/ as the workspace, then paste this prompt into the sidebar:

You are working inside a Magento 2 codebase. Start by exploring the directory structure to orient yourself.

Your job: produce a COMPLETE api_catalog.json covering ALL APIs exposed by this Magento 2
installation — not just a subset. Document every endpoint you find, regardless of whether
it is used in a specific task or not. The goal is a full map of the application's API surface.

API types to scan for (all of them):
1. REST endpoints — declared in webapi.xml files. These are the primary API.
2. GraphQL — Magento has a full GraphQL API parallel to REST. Find the .graphqls schema files
   and document every query and mutation.
3. WebSockets — Magento does not typically use WebSockets, but check. If none found, note it.
4. Admin AJAX endpoints — controllers under adminhtml/ that handle JSON AJAX requests.
   These are separate from the REST API.

For REST and Admin AJAX endpoints, produce:
{
  "api_type": "rest",
  "endpoint": "METHOD /path/{template}",
  "auth": "none | bearer_token | admin_bearer_token | session_cookie",
  "path_params":   { "<name>": { "type": "...", "source": "...", "from_endpoint": "...", "from_field": "...", "notes": "..." } },
  "query_params":  { ... },
  "body_params":   { ... },
  "response_key_fields": ["jq paths that downstream calls will consume"]
}

For GraphQL queries/mutations, produce:
{
  "api_type": "graphql",
  "endpoint": "POST /graphql",
  "operation_name": "...",
  "operation_type": "query | mutation | subscription",
  "auth": "none | bearer_token",
  "variables": { "<name>": { "type": "...", "source": "...", "notes": "..." } },
  "response_key_fields": ["jq paths that downstream calls will consume"]
}

Source types: TASK_SPEC | PREV_CALL | AUTH_FLOW | STATIC | DERIVED
- PREV_CALL: must include from_endpoint and from_field (jq path into that response)
- AUTH_FLOW: any token/cookie obtained during login
- STATIC: include the actual static value

Rules:
- Document REQUIRED parameters only. Skip X-Requested-With, Cache-Control, correlation IDs.
- For guest-cart: POST /rest/V1/guest-carts returns a plain quoted string — that IS the cartId.
- quote_id in add-item body equals cartId — mark DERIVED.
- For searchCriteria filter params, document the exact query string structure.
- For GraphQL: read ALL .graphqls files to find every query, mutation, subscription.

Write the output to api_catalog.json at the root of the codebase.

App 2 — Forum (Postmill / Symfony)

Container: forum
Source root: /var/www/html/ (confirmed — docker exec forum find / -name "composer.json" returned /var/www/html/composer.json)

Attach to the forum container in Cursor, open /var/www/html/ as the workspace, then paste this prompt into the sidebar:

You are working inside a Postmill forum codebase (PHP / Symfony). Start by exploring the
directory structure to orient yourself.

Your job: produce a COMPLETE api_catalog.json covering ALL HTTP endpoints exposed by this
Postmill installation — every route, every form action, every AJAX endpoint.

API types to scan for:
1. Form submissions (POST, application/x-www-form-urlencoded) — the primary interaction pattern
2. JSON AJAX endpoints — controllers that return JsonResponse
3. REST-style endpoints — if any exist under /api/
4. WebSockets — Postmill does not typically use WebSockets but check for any Mercure
   or Pusher integration. If none, note it.

For form submissions and JSON endpoints, produce:
{
  "api_type": "form" | "rest",
  "endpoint": "METHOD /path/{template}",
  "auth": "none | session_cookie | session_cookie+csrf",
  "content_type": "application/x-www-form-urlencoded | application/json | multipart/form-data",
  "path_params":  { "<name>": { "type": "...", "source": "...", "from_endpoint": "...", "from_field": "...", "notes": "..." } },
  "query_params": { ... },
  "form_params":  { ... },   // use this for form submissions
  "body_params":  { ... },   // use this for JSON body
  "response_key_fields": ["what downstream calls consume from this response"]
}

Source types: TASK_SPEC | PREV_CALL | AUTH_FLOW | STATIC | DERIVED

Postmill-specific notes:
- Login is a form POST. Find the exact CSRF token field name in the security config or login form type.
- All write operations (create post, vote, comment) require session_cookie+csrf.
- The community slug / post ID in path templates come from TASK_SPEC or PREV_CALL.
- Read every FormType class to get the exact field names for each form.
- For CSRF tokens in forms: source is AUTH_FLOW (extracted from the page HTML before submit).

Write the output to api_catalog.json at the root of the codebase.

App 3 — Map (OpenStreetMap / Rails)

Container: openstreetmap-website-web-1
Source root: /app (confirmed — docker exec openstreetmap-website-web-1 ls /app shows Gemfile, app/, config/, db/, etc.)

Attach to the openstreetmap-website-web-1 container in Cursor, open /app as the workspace, then paste this prompt into the sidebar:

You are working inside an OpenStreetMap Rails codebase. Start by exploring the directory
structure to orient yourself.

Your job: produce a COMPLETE api_catalog.json covering ALL HTTP endpoints exposed by this
OpenStreetMap installation — every route, every API endpoint, every format variant.

API types to scan for:
1. REST API under /api/0.6/ — the main machine-readable API (XML and JSON variants)
2. Search / geocoding — how place searches are handled (may proxy to Nominatim, or local)
3. Web interface endpoints — HTML controllers, but also any that return JSON
4. OAuth endpoints — any OAuth 1.0 or 2.0 flows
5. WebSockets — unlikely but check for ActionCable or similar integration

For each endpoint, produce:
{
  "api_type": "rest" | "form" | "websocket",
  "endpoint": "METHOD /path/{template}",
  "auth": "none | oauth | session_cookie",
  "format_variants": [".json", ".xml"],   // if the endpoint supports multiple formats via extension
  "path_params":  { "<name>": { "type": "...", "source": "...", "from_endpoint": "...", "from_field": "...", "notes": "..." } },
  "query_params": { ... },
  "body_params":  { ... },
  "response_key_fields": ["XPath or jq paths downstream calls consume"]
}

Source types: TASK_SPEC | PREV_CALL | AUTH_FLOW | STATIC | DERIVED

OpenStreetMap-specific notes:
- The /api/0.6/ endpoints return XML by default; .json suffix returns JSON. Document both variants.
- Node, way, relation IDs are integers — source is TASK_SPEC for direct tasks, PREV_CALL when
  they come from a search result.
- The search endpoint may be /search or proxied through Nominatim — read the routes and
  controllers carefully to find where geographic searches are handled.
- Read ALL controller files, not just the api/ subdirectory. There may be JSON endpoints in
  the main web controllers too.

Start with the routes file (e.g. config/routes.rb) to get the complete route list, then read each controller. Write the output to api_catalog.json at the root of the codebase.

App 4 — Wikipedia (Kiwix) — No extraction needed

Kiwix serves a static ZIM file at /data/wikipedia_en_all_maxi_2022-05.zim (confirmed — docker exec kiwix33 find / -name "*.zim" returned that path). There is no application source code to analyze — kiwix-serve is a C++ binary, not a web framework. The catalog entry is hardcoded below.

Hardcoded catalog entry (save as catalogs/wikipedia.json):

{
  "_meta": {
    "generated": "2026-04-08",
    "source": "hardcoded — kiwix-serve binary serves a static ZIM file; no application source to analyze",
    "zim_file": "/data/wikipedia_en_all_maxi_2022-05.zim",
    "search_response": "HTML only — GET /search returns HTML page; agent must parse <a href> links for article URLs",
    "article_page": "GET /wikipedia_en_all_maxi_2022-05/A/{title} — returns HTML article",
    "websockets": "none"
  },
  "endpoints": [
    {
      "api_type": "rest",
      "endpoint": "GET /search",
      "auth": "none",
      "query_params": {
        "pattern": {
          "type": "string",
          "source": "TASK_SPEC",
          "notes": "the search query, URL-encoded"
        },
        "books.name": {
          "type": "string",
          "source": "STATIC",
          "value": "wikipedia_en_all_maxi_2022-05",
          "notes": "selects which ZIM book to search"
        }
      },
      "response_key_fields": [],
      "notes": "IMPORTANT: response is HTML, not JSON. Parse <a href> anchor links matching /wikipedia_en_all_maxi_2022-05/A/... to extract article slugs."
    },
    {
      "api_type": "rest",
      "endpoint": "GET /wikipedia_en_all_maxi_2022-05/A/{article_title}",
      "auth": "none",
      "path_params": {
        "article_title": {
          "type": "string",
          "source": "PREV_CALL",
          "from_endpoint": "GET /search",
          "from_field": "href attribute of first search result <a> tag",
          "notes": "URL-encoded article slug, e.g. Albert_Einstein. Extract from the href on the search results HTML page."
        }
      },
      "response_key_fields": [],
      "notes": "Returns full HTML article page. HTTP 200 when article exists, 404 when not found."
    }
  ]
}

Validation — smoke-test each catalog entry

After Claude Code writes api_catalog.json for each app, validate a few key entries against the live server before committing:

EC2="ec2-16-59-2-56.us-east-2.compute.amazonaws.com"

# Shopping: guest cart + add item
CART=$(curl -s -X POST http://$EC2:7770/rest/V1/guest-carts \
  -H "Content-Type: application/json" | tr -d '"')
echo "cart_id: $CART"

# Get admin token first (product listing requires auth)
ADMIN_TOKEN=$(curl -s -X POST http://$EC2:7770/rest/V1/integration/admin/token \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"admin1234"}' | tr -d '"')

SKU=$(curl -s "http://$EC2:7770/rest/V1/products?searchCriteria%5BpageSize%5D=1" \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['items'][0]['sku'])")
echo "sku: $SKU"

curl -s -X POST http://$EC2:7770/rest/V1/guest-carts/$CART/items \
  -H "Content-Type: application/json" \
  -d "{\"cartItem\":{\"sku\":\"$SKU\",\"qty\":1,\"quote_id\":\"$CART\"}}" | python3 -m json.tool

If the response is a 200 with item details, the catalog entries for tasks 3–5 are correct.

Or use the automated validator (requires pip install requests):

python3 validate_catalog.py --host ec2-16-59-2-56.us-east-2.compute.amazonaws.com --all

Final structure

All source roots confirmed by running commands on the live EC2 instance:

Container	Source root	How confirmed
`shopping`	`/var/www/magento2/`	WebArena README `docker exec` commands
`shopping_admin`	`/var/www/magento2/`	Same
`forum`	`/var/www/html/`	`find / -name "composer.json"` returned `/var/www/html/composer.json`
`openstreetmap-website-web-1`	`/app`	`ls /app` shows Gemfile, app/, config/, db/
`kiwix33`	N/A	Binary server; data at `/data/wikipedia_en_all_maxi_2022-05.zim`

After running the AI on each container, download api_catalog.json via Cursor's remote explorer (Right-click → Download) and save locally as:

catalogs/
  shopping.json       ← from shopping container
  shopping_admin.json ← from shopping_admin container
  forum.json          ← from forum container
  osm.json            ← from openstreetmap-website-web-1 container
  wikipedia.json      ← hardcoded above (no container needed)

These five files are committed to the repo and loaded by the judge at startup. They are never regenerated during training.

Catalog status — live endpoint verification (2026-04-08)

All five catalogs have been extracted and are committed. Below is the result of live testing against ec2-16-59-2-56.us-east-2.compute.amazonaws.com.

Summary table

Catalog	Endpoints	JSON valid	Structure	Live test	Notes
`shopping.json`	502	✅	✅	✅ PASS	See details below
`shopping_admin.json`	552	✅	✅	✅ PASS	See details below
`forum.json`	91	✅	✅	⚠️ WARN	Login not verified (see below)
`osm.json`	217	✅	✅	✅ PASS	See details below
`wikipedia.json`	2	✅	✅	⚠️ WARN	Search returns HTML not JSON (corrected)

Shopping (port 7770) — PASS

Auth: POST /rest/V1/integration/admin/token with admin/admin1234 returns a JWT bearer token. ✅
Guest cart: POST /rest/V1/guest-carts returns a plain quoted string — confirmed this is the cartId. ✅
Product listing: GET /rest/V1/products?searchCriteria[pageSize]=N requires bearer token — returns full product JSON with total_count: 104368. ✅
Add to cart: POST /rest/V1/guest-carts/{cartId}/items with {sku, qty, quote_id} returns item detail at HTTP 200. ✅

Key finding: GET /rest/V1/products without auth returns HTTP 401 ("consumer isn't authorized"). The catalog documents auth: "admin_bearer_token" for this endpoint — correct.

Shopping Admin (port 7780)

Shopping Admin uses the same Magento 2 REST API as Shopping but accessed on port 7780 with admin credentials. The shopping_admin.json catalog documents the same REST surface with admin-scoped auth. The admin UI itself is a browser-based SPA — its internal AJAX endpoints are documented in the catalog under admin_ajax type entries.

Forum (port 9999)

Homepage: HTTP 200. ✅
Login form structure: Confirmed via HTML inspection. Form action is POST /login_check. CSRF field is _csrf_token (Symfony token, not form_key). ✅
Login result: POST /login_check with MarvelsGrantMan136/test1234 redirects to / (homepage) — login successful. ✅ (The original password notarobot from WebArena defaults was stale; the correct password on this instance is test1234.)

Catalog correctness: The forum.json catalog correctly documents:

POST /login_check with _csrf_token, _username, _password
All write endpoints require session_cookie+csrf
route_name field on each entry (extra metadata, not used by judge)

OSM / Map (port 3000)

Capabilities: GET /api/0.6/capabilities returns XML. ✅
Map bbox: GET /api/0.6/map?bbox=-0.1,51.5,0.1,51.6 returns valid OSM XML with <osm> root. ✅

Search finding: GET /search?query=... returns an HTML page (HTTP 200), not JSON. The actual geocoding is dispatched client-side to sub-endpoints:

POST /geocoder/search_osm_nominatim — Nominatim-backed search
POST /geocoder/search_latlon — coordinate-based search
POST /geocoder/search_osm_nominatim_reverse — reverse geocode

Search param name: The catalog documents query as the query param name. Confirmed: GET /search?query=New+York returns HTTP 200 (HTML).

Wikipedia / Kiwix (port 8888)

Search endpoint: `GET /search?pattern=...&books.name=wikipedia_en_all_maxi_2022-05` returns HTTP 200. ✅
Article endpoint: `GET /wikipedia_en_all_maxi_2022-05/A/Albert_Einstein` returns HTTP 200. ✅