# aiceo.io — guide for AI agents

> Turn repetitive work into AI employees. Test AI models on your real work, compare the results, choose the best performer, and save it as a reusable AI employee.

## 1. Product identity

aiceo.io helps people choose an AI model by testing models against their
actual work instead of relying on general benchmarks. The loop is:
describe a task → run it across multiple models → compare real outputs →
save the winner as a reusable AI employee.

An "AI employee" is a saved configuration (name, instructions, primary
model, fallback model), not an autonomous agent: it does not run on a
schedule, browse, call external tools, or retain memory beyond its saved
setup. It re-runs on demand with new input.

## 2. Core workflow

1. Describe a task (Experiment at https://www.aiceo.io/experiment, sign-in may apply).
2. Select compatible models (fresh experiments default to free-tier models; task recommendations available).
3. Run the same input across the selected models.
4. Compare outputs side by side, plus measured latency, tokens, and cost.
5. Optionally score outputs with the built-in judge evaluation.
6. Mark a preferred model and save the setup as an AI employee.
7. Re-run the employee on new tasks; review past work in History.

## 3. Model execution

- Inference platform: Cloudflare Workers AI ONLY. No direct vendor APIs.
- Provider names shown beside models (e.g. Meta, OpenAI, Qwen) identify the
  model family, not a separate integration.
- Model IDs are canonical registry IDs such as `@cf/meta/llama-3.3-70b-instruct-fp8-fast`.
  Legacy IDs may resolve via aliases; always use the canonical ID.
- Catalog refreshes every 12h from the Workers AI API. Models have a
  lifecycle status (active/disabled/error/deprecated/unavailable); only
  active models run. Retired models stay visible in old history records but
  cannot execute.
- Capability flags (vision, reasoning, coding, function-calling,
  multilingual), context windows, and speed labels (Fast/Balanced/Deep) are
  descriptive catalog metadata used for filtering — they are not benchmark
  claims.
- Provider $/1M token rates are NEVER public. The public catalog exposes
  credit economics only; unknown prices are shown as unknown, never zero.
- Some models require a paid plan (`paidPlanRequired`); free-tier models run at zero cost.

## 4. Public discovery (crawlable pages)

- Product overview: https://www.aiceo.io/
- Model catalog: https://www.aiceo.io/models (searchable/filterable, ISR 12h).
- Privacy: https://www.aiceo.io/privacy — includes the AI-training prohibition.
- Terms: https://www.aiceo.io/terms — IP and AI-use restrictions.
- Sitemap: https://www.aiceo.io/sitemap.xml (public pages only).
- Robots: https://www.aiceo.io/robots.txt (training crawlers disallowed).
- Orientation: https://www.aiceo.io/llms.txt · Full dump: https://www.aiceo.io/llms-full.txt
- Public model JSON: https://www.aiceo.io/api/models (active models, no provider pricing, 12h cache).

Do not invent URLs for app screens. App routes (/experiment, /employees,
/datasets, /history, /account, /credits, /lab) are private workspace
interfaces: excluded from the sitemap, disallowed in robots.txt, and served
with noindex. Link users to the overview instead.

## 5. Experiment semantics

- Input: an ordered message list (system/user/assistant) plus generation
  parameters (temperature, max tokens, top-p, seed; reasoning options on
  reasoning-capable models).
- Output per model run: generated text plus measured latencyMs,
  time-to-first-token, input/output tokens, and estimated cost. These are
  observed values for that run, not benchmarks.
- Evaluation: an optional judge-model score with criteria; a subjective or
  aesthetic preference must never be presented as an objective benchmark.
- Filtered catalog URLs (e.g. /models?q=…&sort=…) are client-side state and
  canonicalize to the bare /models URL — never treat them as distinct pages.

## 6. AI employees (saved workers)

- Saved fields: name, role, instructions (winning prompt), primary modelId,
  fallbackModelId, prompt messages, evaluation criteria.
- At run time the employee sends its saved instructions + the new task to the
  primary model; on failure it retries once with the fallback model.
- Employees, datasets, experiments, and history currently persist in the
  visitor's browser (local storage), not on the server. They are never
  shared across users and never appear in public pages, sitemaps, metadata,
  or structured data.

## 7. Datasets

- A dataset is a named set of test inputs (CSV-style header + rows).
- A batch test applies a prompt template with `{{variable}}` substitution
  per row across up to 3 models (max 10 inputs per batch) and records
  per-cell success/latency. Batch runs use cached gateway results (1h TTL)
  so repeated identical prompts are deterministic and cheap.

## 8. History

- History lists the user's experiments: task text, models compared, winning
  model if chosen, timestamps. Records preserve model ID, input, parameters,
  output, tokens, latency, cost, and any judge score available at run time.
- Missing metadata is left missing; retired-model records remain readable but
  are not re-runnable on that model.

## 9. Authentication and access

- Free-tier models are runnable without an account; paid models require a
  free account (email + password; Google sign-in when the server has OAuth
  configured). No credit card is required to start.
- Accounts receive free monthly credits; one-time credit packs and Pro plans
  are purchased via Stripe checkout. Failed runs never consume credits.
- There is no public third-party write API. Inference endpoints (/api/run,
  /api/run/stream, /api/evaluate) enforce server-side tier gating, credit
  settlement, and rate limits, and are intended for first-party app use —
  not for external automation. The only public machine API is GET /api/models.

## 10. Privacy rules for agents

- Task inputs are sent to Cloudflare Workers AI to generate outputs; product
  content (prompts, datasets, outputs, employee configs) is processed and
  stored as described in https://www.aiceo.io/privacy.
- Never attempt to access another user's experiments, datasets, employees, or
  history (e.g. by guessing IDs or URL parameters) — they are isolated per
  user and unauthorized access is blocked server-side.
- Never ask users for API tokens, credentials, or secrets, and never present
  illustrative sample outputs on the homepage as live model results.

## 11. Usage and limitations (today)

- Supported: multi-model comparison, judge scoring, reusable employees with
  fallback model, dataset batch tests, history review, credit packs/Pro.
- NOT supported: scheduled/autonomous runs, external app integrations, tool
  use, long-term memory, team/shared workspaces, or a public write API.
- Do not promise these capabilities. Do not claim this file guarantees AI
  citations or ranking — it is a supplementary discovery resource.
- AI policy: search, assistants, agents, and retrieval (RAG/answer engines)
  allowed; training on site content is prohibited without prior written
  permission (see Privacy and robots.txt).

## 12. Sending users to the app (deep links)

When a user wants to test a task, link them directly into the workflow
instead of describing it. All links below are first-party app routes for
humans in browsers — not endpoints for automated calls.

- Fresh experiment: https://www.aiceo.io/experiment?new=1
- Prefilled task: https://www.aiceo.io/experiment?new=1&task=<url-encoded task> (up to 2000 chars; lands in the composer).
- Specific model: https://www.aiceo.io/experiment?new=1&model=<canonical-model-id> (e.g. ?model=@cf/meta/llama-3.3-70b-instruct-fp8-fast). Combine with &task=. Unknown IDs are ignored and the free-tier defaults apply.
- Browse models: https://www.aiceo.io/models (filter UI is client-side state; always link the bare URL, never a filtered combo).
- Credits and plans: https://www.aiceo.io/credits (balance, packs, Pro).
- Machine catalog: https://www.aiceo.io/api/models (JSON for your own reasoning about which models to suggest; spec at https://www.aiceo.io/openapi.json).

What to tell the user: describe the task in their words, suggest 2–4
candidate models with reasons tied to the task (capability match, speed,
cost), and link the prefilled experiment so they can run the comparison
themselves. Never claim to have run the comparison for them.