Documentation

Periplus exposes a ClickHouse catalogue through native ClickHouse SQL. Run queries against the configured catalogue or through the query service. Each row in captures has retained HTML. The same URL can appear more than once, and retained HTML HTTP error responses are included. Captured HTML may reflect browser rendering rather than the original HTTP response body.

The query API defaults to public_v1: use captures or public_v1.captures. Requests may specify schema_version: "public_v1"; unsupported versions are rejected. Responses report the resolved schema version separately from the data snapshot. Each query uses one consistent materialization version while new captures continue arriving.

Periplus observes a page in a browser after configured loading and interaction steps. That document records one observed state of a page, rather than every possible interactive or personalised state.

Each capture owns an immutable archive of its exact HTML. Retention policies protect capture history; identical bytes share a content identity but retain independent archives. It records captures, then builds HTML and link projections separately. Collection can finish before those projections are queryable.

Coverage includes selected sources. It offers neither a representative sample of the web nor a uniform refresh schedule. Collection dates can be missing; the latest date for a site does not establish that all its pages are fresh.

Submit finite discovery through the control API. A link found in a page does not mean its destination has been collected. Check available captures and projections before relying on a source in your dataset.

Discovery starts from a list of URLs, with maximum depth and a total page limit. Depth zero captures starting pages only; deeper discovery follows links within their sites.

A page limit means “up to N”, including starting pages. Discoveries can share acquisitions while retaining their own selection and accounting. Discovery is one-off and does not create retention or refresh policies.

Request membership and readiness are available in coverage APIs and the request view. The public SQL schema contains page, capture and content evidence.

Independent page policies

Use Retention to protect exact capture IDs until a fixed expiry measured from claim activation. The latest expiry applies on overlapping claims, with a seven-day service baseline. No future captures are automatically included. This does not fetch pages or restore deleted history.

Use Refresh to request another capture every N days, without following links. The shortest active interval applies; successful captures from other work also satisfy freshness. Failures retry with backoff. Refresh does not extend retention.

Freshness accepts returned URLs in Explore, or SQL returning a url column. SQL runs once and saves up to 1,000 fixed pages. Retention accepts up to 64 exact capture IDs per claim. Review the draft before activating it. Later SQL results do not change membership. Pausing or removing policies does not immediately delete captures; already-started refresh captures may finish.

Reproducibility

SQL preserves the extraction method. Identical results also require the same input data, schema, and deterministic semantics. A source snapshot reference records the input version; it does not pin a future rerun.

Different captures can share one document. A document_id identifies the retained bytes together with their HTML interpretation. Joining a document to captures repeats it for each matching capture.

Use a LEFT JOIN to retain captures without matching HTML structure. An INNER JOIN keeps only matches. Counting joined HTML rows counts elements, not pages; choose distinct URLs, capture IDs, or document IDs according to your question.

For a parent element, match both document_id and child.parent_index = parent.node_index. A subtree occupies node indices from node_index inclusive to subtree_end_index exclusive.

The complete parsed tree includes document, element, text, comment and template-fragment nodes. Indices are contiguous preorder positions. Sibling positions count every node kind, and parent_index references the immediate parent. Node 0 is the document root at depth 0. Attributes remain attached to elements as an object. Filter nodes by kind, name and namespace to inspect headings, metadata or scripts. Node positions belong to the selected materialization; original byte syntax remains in the archived HTML.

Retrieve original captured bytes with GET /api/documents/{document_id}/content. The download returns the original bytes for that document. The catalogue exposes capture metadata and parsed HTML. A source URL identifies the live site; its current content may differ from the retained capture.

Replace the illustrative document IDs below with document_id values from your own captures. Retention and catalogue coverage affect whether an example returns rows. Exact document filters keep extraction focused; a final LIMIT alone does not.

Text relations preserve source text order without automatic trimming or inserted separators. Apply normalization explicitly. Missing projections can mean indexing is still pending; missing rows alone do not prove a page has no headings, forms, or tables.

The public_v1 namespace exposes these five views. A page is a normalized URL, including linked destinations without captures. Redirects and canonical declarations do not merge page identities. These are public views. Their keys describe logical identity; they are not declarations of database-enforced primary keys on the views. The SQL console schema explorer shows the deployed relations.

The query API accepts SQL and positional parameters through POST /api/v1/sql. The public website does not proxy SQL or schema requests. Set operation to prepare to explain a SELECT; execution returns rows. GET /api/v1/schema describes the installed helpers. Responses preserve submitted SQL and report any applied compiler optimizations. ClickHouse optimizes the execution plan.

These limits apply to the query service. Direct ClickHouse connections use their connection settings and do not inherit the service’s HTTP result caps.

A LIMIT bounds output, not necessarily work. Sorting, aggregation, and joins may still scan substantial data. Filter captures and select the columns you need before expanding HTML. Preparation also consumes a bounded execution slot.

Acquisitions without retained HTML do not appear in captures. A published capture references a completed document tree. A missing matching element means the selection found no match in that tree. HTML projections may become available after collection finishes. Capture times describe captured data, not a guarantee of freshness; live website links can show different content.

Use the SDK from the same Periplus source checkout as this deployment, with Python 3.11 or newer. These public_v1 examples require the updated source SDK; the previously published 0.2.0 release predates this contract. Configure the query API origin, not the public website URL. See the source SDK README for its authentication contract.

python -m pip install ./clients/periplus-python-sdk

from periplus_sdk import Client

with Client("https://your-periplus-host") as client:
    result = client.execute(
        "SELECT capture_id FROM public_v1.captures LIMIT ?", [10]
    )
    print(result.columns, result.rows)
    print(result.schema_version, result.source_snapshot, result.truncated)

prepare(sql, parameters) returns diagnostics and a plan. execute(sql, parameters) independently validates and runs the query. helpers() describes installed SQL helpers. Use AsyncClient with an async context manager and await these methods for asynchronous Python.

The same public access settings and query limits apply. Results preserve SQL types and explicit truncation; the SDK never retries or fetches more rows automatically. SQL and parameters are recorded as SDK operations in private, best-effort query history for 30 days; result rows are not stored.