Skip to Content
Data LayerOverview

Data Layer

The data layer is what turns a CSV or JSON file into a live, browsable, monetizable product. You bring already-shaped data; the platform derives a per-tenant DataSchema for it, validates every row against that schema, and makes the result searchable and presentable — without per-dataset custom code.

Ingest flow

The canonical flow is CLI-first, designed to be driven by an agent or by a human at the terminal:

shipmore schema infer  →  schema apply  →  import
  1. Infer — sample the file, propose a DataSchema + import mapping. Output is a fields[] JSON the operator can edit.
  2. Apply — persist the schema and the source mapping for the tenant.
  3. Import — project raw rows into records, validate against the schema, and write in batches via the job runner.

Bulk imports use @payloadcms/plugin-import-export underneath. The job runner is enabled via jobs.autoRun in payload.config.ts (tune with PAYLOAD_JOB_AUTORUN_CRON).

See Import for the full CLI flow and the agent’s 4-phase workflow.

Records

Every item becomes a record with a stable set of root fields (id, slug, status, location, …) plus a data blob shaped by the DataSchema — so datasets of any shape are handled the same way.

When source rows include lat / lng columns, they’re paired automatically into the record’s location at import time — no CLI flag needed.

DataSchema

The DataSchema describes the shape of each record’s data — it’s what validation and presentation are driven from. It covers the data blob only; the stable root fields aren’t part of it.

Each field in the DataSchema drives both data validation and presentation. A field carries a label, a type, whether it’s searchable (full-text indexed) and facetable (exposed as a sidebar filter), an optional role (where it renders), and an optional card-visibility hint (what shows on the card vs. the detail page).

For the roles you can assign, run shipmore schema field set --help.

See DataSchema for the full field reference.

Search & Explore

Once records are in, a complete browse surface ships out of the box: a per-tenant search index, faceted sidebar, search bar, sort, and pagination. Records render as browsable, searchable, filterable cards (grid or list) or a table, with a map view when the data carries geo coordinates.

See Search & Explore.

Where the data layer is NOT

ShipMore is not a data sourcing or cleaning pipeline. You bring already-shaped data (or generate it via the generation pipeline). Sourcing, scraping, and messy ETL stay upstream — ShipMore maps to schema, presents, monetizes.

Multi-TenancyDataSchema