Silktrace.

The custom work lives between provider calls: deciding which result becomes the next request, bounding that fan-out, reconciling inconsistent fields and retaining enough evidence to review the output. This chapter describes the hosted implementation reviewed on 8 September 2026. Legacy local storage and sample data paths are separate.

Implementationsrc/app/api/research/route.ts src/server/services/hosted-*-service.ts src/server/sync/queue.ts

stc1.<base64url payload>.<HMAC-SHA256 signature>

{
  format, version, keyVersion,
  workspaceHash,          // SHA-256 of server-resolved workspace
  operation, requestHash,
  idempotencyHash,        // binds approval to this submission
  costMicros,             // server-computed amount
  issuedAt, expiresAt,    // five-minute validity
  nonce                  // random value
}

A server-only, versioned signing key authenticates the payload with a domain-separated HMAC. Verification checks strict schema, canonical encoding, constant-time signature equality, expiry and each binding against freshly computed server context. Modifying the amount, request, credential-dependent hash or submission key invalidates approval. A client boolean alone cannot authorize hosted work.

The initial challenge returns HTTP 409 without reserving a task or calling DataForSEO. Confirmation resubmits the token and the same idempotency key. Credential rotation or changed parameters require a new valid quote; an invalid/expired token receives a replacement challenge.

The atomic quota update succeeds only when reserved + used + 1 ≤ limit. These records reserve permission to execute within usage limits. They do not hold money or reserve funds in a DataForSEO account. Settlement subtracts reserved slots, consumes daily slots if work may have reached the provider, releases concurrency, records cost and closes the idempotency record in one transaction.

For dead-link research, the task remains submitting while the complete live pipeline runs. Only afterward does it move through submitted and collecting, save the result and settle. Its individual live calls are not persisted as separate provider tasks. The asynchronous OnPage crawl does persist its remote task ID for later polling and collection.

Keyword Mode starts with a topic, an optional replacement site and an optional specific replacement URL. DeadLinkOpportunitiesClient posts to /api/explore/dead-links. It creates and retains an idempotency key through submission and confirmation; changed input invalidates that key. An in-flight guard prevents overlapping client submissions. There is no language/location selector or user-configurable research budget in this request.

POST /api/explore/dead-links
Idempotency-Key: <one key retained through confirmation>

{
  "mode": "keyword",
  "keyword": "sustainable building materials",
  "replacementTarget": "example.com",
  "replacementUrl": "https://example.com/materials-guide"
}

// The confirmation submission adds:
// confirmationToken: <signed server quote>
// confirmCost: true

The route rejects unknown fields and bodies over 8 KiB. normalizeDeadLinkKeywordInput() normalizes Unicode and whitespace and accepts a 2–200 character topic. Replacement URLs must be public HTTP(S), without embedded credentials; fragments are removed. Credentials, workspace identity, endpoint choices and execution limits come from the server.

{
  "keyword": "sustainable building materials",
  "search_mode": "one_per_domain",
  "limit": 20,
  "rank_scale": "one_hundred",
  "order_by": ["domain_rank,desc"]
}

{
  "url": "https://publisher.example/old-resource",
  "store_raw_html": false,
  "load_resources": false,
  "enable_javascript": false,
  "return_despite_timeout": true
}

{
  "target": "https://publisher.example/old-resource",
  "limit": 20,
  "exclude_internal_backlinks": true,
  "backlinks_status_type": "live",
  "rank_scale": "one_hundred",
  "order_by": ["rank,desc"]
}

[{ "url": "https://source.example/article-linking-to-resource" }]

Implementationsrc/app/api/explore/dead-links/route.ts src/server/services/hosted-dead-link-service.ts src/server/providers/dataforseo/hosted-dead-link-prospecting.ts

relevance() lowercases the topic, splits on non-ASCII letters/numbers and keeps unique terms longer than two characters. It checks whether each term appears as a substring in the destination title, destination URL, anchor or source URL. Relevance is the rounded percentage of matched terms, or zero if there are no qualifying terms. Body content, embeddings and an LLM are not used.

Numeric rank/spam evidence is bounded to 0–100. Unknown authority contributes zero; unknown spam assumes 35. Deduplicated rows sort by score descending and stop at fifty. There is no minimum-score cutoff or high/medium/low category in this dead-link path. A 24-character SHA-256 prefix supplies the stable destination/source row ID.

1. Wanted termsTopic + dead title + anchor + dead URL path
2. Eligible pagesSame host · HTTP 200 · indexable · not the dead URL
3. Available termsCandidate title + H1 + URL path
4. Rank candidatesExact term overlap → top three → user selection

wanted = uniqueTokens(keyword + deadTitle + anchorText + deadPath)
available = uniqueTokens(page.title + page.h1 + page.path)
matchedTerms = wanted.filter(term => available.has(term))

score = round(100 * matchedTerms.length / wanted.length)

// Reject candidates with no matched terms.
// Sort by score descending, then URL for deterministic ties.
// Retain the top three; selection remains with the user.

The tokenizer normalizes Unicode, lowercases, splits on non-letter/non-number characters and removes short terms plus a small stop-word list. Eligible pages must be on the replacement host, HTTP 200 and indexable in the saved crawl. Duplicate URLs and the dead destination itself are excluded. Candidate titles, H1s and paths contribute exact tokens; body content, semantic embeddings and LLM judgments do not.

For four wanted terms, matching three gives 75% overlap. That number is an inspectable lexical fit, not a probability that the replacement satisfies the same intent. Each suggestion retains matchedTerms and evidence:"saved-crawl"; the response also exposes crawl date/context. There is no high/medium replacement-confidence classifier.

A manual URL is placed first unless it is the dead destination itself. If it is already an eligible scored candidate, its saved evidence is retained; otherwise it is labeled manual-unverified with a null score. Up to three other suggestions can follow. With no matches and no manual URL, the list stays empty and the UI offers a crawl refresh or a supplied URL. Suggested pages are not selected automatically. The provider cache stores base opportunities; replacement suggestions are recomputed from stored crawl evidence on response, so a new page choice does not invalidate paid discovery.

Implementationhosted-dead-link-prospecting.ts · relevance(), contact(), executeHostedDeadLinkKeywordProspecting() src/server/services/dead-link-replacement-service.ts · enrichDeadLinkReplacements(), rankDeadLinkReplacements() src/lib/contracts/page-vms.ts · DeadLinkReplacementVM

/api/research resolves settings and optional project ownership before execution. Region presets become concrete provider codes: US/English is 2840/en, Canada/English is 2124/en, and Canada/French is 2124/fr. Missing/invalid selection falls back through workspace defaults. Overview splits comma, newline or semicolon input, collapses whitespace and deduplicates case-insensitively, up to fifty keywords.

HostedResearchRawPayload is a discriminated union: single overview carries overview + SERP; cluster carries expansion + suggestions + enrichment; gap carries ranked + intersection; matrix carries results by domain. mapHostedResearchPayload() selects the matching mapper rather than asking the UI to interpret different provider structures.

Input: { tab: "overview", query: "where to stay in Bishop", region: "us-en" }
  → contract: keywords[], locationCode: 2840, languageCode: "en"
  → workspace cache lookup; on miss, estimate/reserve/execute
  → keyword_info.search_volume / cpc + keyword_properties + intent
  → normalized metrics: monthlyVolume: 90, cpc: 2.96,
                        keywordDifficulty: 2, intent: "commercial"
  → buildOpportunity(): score 73, priority "high"
  → mapped JSONB result + best-effort saved lookup
  → KeywordResearchPageVM → metric views, plan rows and trend charts

The mapper tolerates metrics nested under keyword_info, root fields or keyword_data.keyword_info. Volume/difficulty become bounded numeric values or null; unknown intent is explicit. Cluster merging fills missing values across sources by normalized keyword. Monthly trends are sorted by year/month and retain the latest twelve entries. A requested bulk keyword absent from the response gets an explicit no-data row instead of disappearing.

For the illustrated commercial keyword, volume 90 contributes 15.67, difficulty 2 contributes 24.5, intent contributes 18, missing client position contributes 15 and missing competitor position contributes zero. The total rounds to 73.

Recommended actions are a separate ordered rule set: difficulty ≥75 with volume <500 gives “Low priority”; a client at 4–20 gives “Review existing fit”; top-three client rankings give “Validate manually”; informational intent with missing difficulty or difficulty ≤55 gives “Group with supporting content”; missing difficulty or difficulty ≤65 gives “Map to best page”; otherwise, “Validate manually.” Thus a score category and an action can differ. Existing ranking URLs can be carried forward, but no semantic page matcher is invoked.

The mapped result is cached for thirty days under its market and credential identity. A single overview’s built-in estimate is US$0.0121, below the confirmation threshold; a cluster’s is US$0.125, above it. Both still use durable reservations and usage recording when they execute paid work.

Implementationsrc/server/providers/dataforseo/hosted-keyword-research.ts src/server/services/research-service.ts · mapHostedResearchPayload(), buildOpportunity() src/lib/keyword-research.ts src/components/research/keyword-research-client.tsx

Neon Auth handles sign-in and session cookies. On the server, getCurrentUser() calls auth.getSession() and requires a user ID, session ID and valid session creation time. getHostedWorkspaceSession() calls bootstrapWorkspace(), then creates WorkspaceScope.fromAuthenticatedContext() from that user, owner membership, workspace, session and a generated request identity. The browser does not choose the workspace ID for a research operation.

After sign-in, /auth/continue resolves the workspace and reads its settings. Incomplete setup goes to onboarding; completed setup goes to Domain Explorer. Missing sessions return to login, while deleted or suspended accounts receive their own access outcome. Paid API handlers resolve authorization independently rather than trusting that a page was previously opened.

Bootstrap checks existing active membership and workspace status. A transaction takes a user-specific advisory lock and rechecks before creating the workspace, owner membership and settings. This avoids two simultaneous first requests creating two owner workspaces. WorkspaceScope has a private constructor and validates owner/session context; repository predicates still provide the actual data-access boundary.

Queries filter on scope.workspaceId. Composite foreign keys such as (workspace_id, credential_id) prevent a child record from referring to another workspace’s credential; equivalent constraints protect projects, tasks and crawl evidence. This is application-enforced tenant scoping backed by relational constraints, not a claim that database row-level security is enabled.

Implementationsrc/server/auth/session.ts src/server/auth/hosted-workspace-session.ts WorkspaceScope.fromAuthenticatedContext() postgres-hosted-repository.ts · bootstrapWorkspace() src/server/db/schema/hosted.ts

(workspace_id,
 credential_id,
 credential_version,
 endpoint_key,
 request_hash,
 data_mode)

AND expires_at > now()

Workspace and live/sandbox mode are separate key components even when they are not inside the hash. A global cache could expose one owner’s research or incorrectly treat another owner’s credential-bound request as already paid. Credential versions also prevent reusing old provider-account context after rotation.

Expired rows, different markets, changed inputs, a new endpoint version or rotated credentials produce a miss. Normalization is deliberately bounded: dead-link keyword case is preserved for identity, so two case variants can still miss the cache even though the relevance calculation lowercases both.

Implementationpostgres-hosted-provider-store.ts · getProviderCache(), saveProviderCache() hosted-keyword-research-service.ts · getHostedResearchRequestContract() src/server/db/schema/hosted.ts · providerCache, savedResearchLookups

1. Construct and validate. The client sends topic, optional replacement and submission key. The route resolves the authenticated workspace and validates the bounded request.
2. Resolve identity. The service loads configured, verified credentials, builds the normalized request hash and checks the scoped seven-day cache. A hit returns immediately.
3. Approve the cold request. The server issues the US$0.25 signed challenge. The client resubmits it with the original key. The server verifies all bindings.
4. Reserve before spending. reserveProviderTask() atomically claims the attempt and quota slots. The service resolves that exact credential version and enters submitting.
5. Execute the dependency graph. Discover resources, normalize/deduplicate candidates, verify status, enrich up to five failed destinations, then parse up to ten referring domains.
6. Derive the internal result. Normalize and deduplicate provider rows, associate contact evidence, calculate relevance/score and retain the top fifty opportunities.
7. Persist and settle. Save rows, coverage counters, warnings and capture time in provider_cache, settle task/quota state and record the usage receipt.
8. Join replacement evidence. The route reads the replacement site’s saved crawl, ranks eligible pages and adds suggestions plus any manual URL. This happens outside the paid discovery cache.
9. Render and reuse. Return DeadLinkOpportunitiesVM. React renders rows, warnings, contact provenance and replacement choices. Matching future requests reuse provider evidence; replacement changes use storage-only enrichment.