Marvia

Sign inSign up

Marvia-09

One crawl, one JSON, every extension

The snapshot engine is the single source of truth for website knowledge. It collects, normalizes and structures data into immutable versioned JSON. It never scores, never runs AI and never writes a report.

Schema version

1.0

Pipeline stages

6

Extraction categories

9

Repositories

5

PrincipleImplementation
Snapshot firstStore structured JSON once, reuse forever.
JSON firstExtensions consume snapshot JSON, never HTML.
Schema firstStrong typing against one universal schema.
Immutable historyNever overwrite; keep the full version history.
Workspace isolationEvery snapshot is scoped to a workspace.
Versioned dataEach refresh writes a version, timestamp and hash.
Extension friendlyOne clean API surface for consumption.
AI readyStructured data doubles as AI context.
Queue drivenNon-blocking processing via Cloudflare Queues.
Provider agnosticAbstract fetch layer, swappable providers.

Pipeline

Six replaceable stages, one fixed contract

Extensions may swap a stage. They may never skip one, and no stage is allowed to interpret meaning.

fetch

01

Normalized URL → FetchResponse

Provider-agnostic; timeouts and robots.txt enforced here.

parse

02

FetchResponse → Raw extraction

Extraction only — no scoring, no analysis, no AI.

normalize

03

Raw extraction → Normalized extraction

Standardize casing, URLs, units and empty values.

validate

04

Normalized extraction → Validated extraction

Schema compliance; invalid data never reaches storage.

build

05

Validated extraction → Snapshot JSON

Hash the content and stamp version, source and workspace.

knowledge

06

Snapshot JSON → KnowledgeGraph

Derive nodes and edges; no new facts are invented.

Intake

Normalization and the fetch layer

A URL becomes canonical before anything is fetched, and fetching stays behind a provider interface.

URL normalization

  • Upgrade the scheme to HTTPS.
  • Lowercase the hostname and strip the default port.
  • Remove trailing slashes from non-root paths.
  • Sort query parameters; drop the fragment.
  • Resolve the canonical URL and follow the redirect chain to the final URL.

Example

HTTP://Marvia.app/Pricing/?b=2&a=1#plans

https://marvia.app/Pricing?a=1&b=2

Fetch layer

nativecloudflare-workerbrowserremote
  • Time out after 30s by default; a slow host must never block the queue.
  • Respect robots.txt and crawl delay before requesting a document.
  • Support gzip and deflate transfer encodings.
  • Preserve the original HTML for debugging, never as the stored record.
  • Validate the response and degrade gracefully on malformed HTML.

default timeout 30s · sources: user · extension · scheduler

Extraction

Nine categories, structure only

The parser pulls facts out of HTML. Judgement belongs to extensions reading the snapshot.

metadata

Title, description, language, robots, Open Graph, Twitter Card, JSON-LD, microdata, RDFa

SEO and social sharing

dom

Depth, headings, semantic elements, navigation, sections

Content hierarchy and navigation

links

Internal/external, anchor text, nofollow, sponsored, UGC, broken links

Link graph analysis

images

Source, alt, width, height, lazy, responsive, format

Image optimization and accessibility

structured data

Organization, Article, Product, Breadcrumb, FAQ, Event, Recipe, Video

Rich snippets and knowledge graph

technology

CMS, framework, analytics, ads, server, security headers

Technology stack analysis

performance

Asset counts, compression, caching headers

Performance insights

accessibility

Alt text, heading order, landmarks

Accessibility compliance

security

HTTPS, HSTS, CSP, cookies

Security posture assessment

Storage

Repositories, versioning and the knowledge graph

Snapshots are immutable rows. History and relationships are first-class, not derived at read time.

Repositories

  • SnapshotRepository

    Immutable snapshot storage with versioning.

  • KnowledgeRepository

    Knowledge graph nodes and edges per snapshot.

  • VersionRepository

    Version history per URL, per workspace.

  • DiffRepository

    Computed differences between two versions.

  • CacheRepository

    Short-lived snapshots for frequent reads.

Versioning & diffing

  • Every refresh writes a new snapshot; existing rows are never mutated.
  • Content is hashed with SHA-256; an equal hash records a version with no diff.
  • History is per URL and per workspace, ordered by timestamp.
  • Diffs are computed on demand from two versions and stored in the diff repository.
addedremovedchangedmoved

Knowledge graph

Nodes

pageheadingimagemetalinktechnologyasset

Edges

containsreferenceslinks-toparentchilduses

Caching

workspace scope · default TTL 24h

  • Manual refresh request

    A member asks for a fresh snapshot.

  • Extension request

    An extension needs current knowledge.

  • Scheduled refresh

    Cron-driven refresh. Planned, not shipped in v1.0.

Events

snapshot.createdsnapshot.completedsnapshot.failedsnapshot.version.createdknowledge.updated

Extensions subscribe. The engine publishes. There is no direct call path from an extension into the pipeline.

Consumption

The extension API

Six methods. No HTML, no parsers, no private access to storage.

  • getSnapshot(snapshotId)Retrieve one immutable snapshot.
  • getKnowledgeGraph(snapshotId)Nodes and edges for semantic queries.
  • getDiff(snapshotId)What changed since the previous version.
  • getHistory(url)Version history for a URL in this workspace.
  • refresh(url)Queue a manual refresh.
  • subscribe(event, handler)React to snapshot lifecycle events.

Consumer rules

  • Never parse HTML — read snapshot JSON.
  • Use the published schema; unknown fields are ignored, never guessed.
  • Handle versioning gracefully: an older snapshot is valid data, not an error.
  • Authenticate and stay inside the workspace boundary; rate limits apply.

Operations

Monitoring, alerting and security

A knowledge engine is only trustworthy when its failures are visible.

Dashboard metrics

  • Last snapshottimestamp
  • Processing durationms
  • Queue backlogjobs
  • Storage usageGB
  • Error rate%

Alerts

  • Queue backlog above threshold

    Scale consumers, pause scheduled refreshes.

  • Processing time above SLA

    Investigate the fetch provider before the parser.

  • Storage growth above 80% capacity

    Spill snapshot bodies to object storage.

  • Failed snapshots above 5% of total

    Halt scheduled refreshes and page the on-call.

Security

  • Never store sensitive HTML content; sanitize everything extracted.
  • Validate all inputs before they enter the pipeline.
  • Enforce workspace isolation on every read and write.
  • Authenticate extensions and rate-limit the API.
  • Validate fetch responses and handle malformed HTML gracefully.