Skip to Content
Data LayerImport

Import

The canonical import path is the shipmore CLI. The plugin (@payloadcms/plugin-import-export) handles the underlying batch / queue / ops UI; the CLI is what humans and agents drive.

Quickstart (operator)

# 1. Make sure you have the CLI + an API key
shipmore --version
export SHIPMORE_API_KEY=<your-payload-api-key>
export SHIPMORE_API_URL=https://your-shipmore-box
 
# 2. Create a tenant
shipmore tenant create --name "Acme Ramen" --domain ramen.example.com
 
# 3. Infer a schema from a sample
shipmore schema infer --tenant ramen.example.com --data-file sample.csv > /tmp/schema.json
 
# 4. (optional) Edit fields
shipmore schema field set  --fields-file /tmp/schema.json --key addr_postal_code --type string
shipmore schema field drop --fields-file /tmp/schema.json --key can_claim
 
# 5. Apply + import
shipmore schema apply --tenant ramen.example.com --fields-file /tmp/schema.json
shipmore import       --tenant ramen.example.com           # reads the staging table
shipmore imports list --tenant ramen.example.com           # confirm batches

See Agents & CLI → CLI for full flag tables and authentication.

The 4-phase agent workflow

When an agent is driving (via the shipmore-cli skill), the flow is structured into four phases. Each phase has a specific output and a clear hand-off:

  1. Understand the dataset (data analyst) — read the inferred output, build a mental model of what the records are, identify pause-criteria (compound enums, opaque IDs marked searchable, identifier typed as number, …).
  2. Propose annotations and ASK (product designer) — render a single markdown table of every field with proposed edits, then stop and wait for the operator.
  3. Edit + apply + import (operator) — mechanical pipeline once the operator confirms.
  4. Build the page (optional) — only if the user asked.

The operator confirmation between phase 2 and 3 is the safety mechanism. The agent should never schema apply without it.

Full agent protocol lives in SKILL.md at the repo root.

Staging session

schema infer opens a per-tenant staging session that lives across the full infer → apply → import flow (max 10-min TTL).

  • schema apply is metadata-only — it doesn’t touch the staging table.
  • import with no --data-file reads the staged table; passing --data-file overwrites it.
  • The session is released by a non-dry import (success or failure).
  • --dry-run preserves staging.
  • shipmore staging status --tenant X inspects the session; shipmore staging clear --tenant X drops it.

Geo pairing

Columns named lat / latitude and lng / lon / long / longitude (numeric) are paired into record.location at import time. No CLI primitive needed; no operator confirmation required. The lat/lng fields stay in the schema as plain numbers — assign them cardVisibility: detail-only if you don’t want them surfaced as facets.

Bulk JSON via the admin

For pure operator UX (no CLI), the @payloadcms/plugin-import-export plugin exposes Imports / Exports under Data in the Payload admin. Recommended format: array of records-shaped objects (already projected into the DataSchema).

For idempotent re-imports, use upsert mode and matchField: sourceId.

Queued imports need a job runner:

  • Long-lived Node (Railway) — jobs.autoRun is enabled in payload.config.ts. Tune cadence with PAYLOAD_JOB_AUTORUN_CRON.
  • Serverless (Vercel) — use cron + the jobs run API with Authorization: Bearer ${CRON_SECRET}.

Memory & file size

Import peak memory is roughly 2–3× the file size. Capped by IMPORT_MAX_FILE_SIZE_MB (default 100). Raising this above 100 requires confirming free RAM on the deployed box exceeds 3× the new value during peak tenant activity.

Large files are auto-chunked into multiple Payload jobs.

Trust the API counts

When checking import status, trust the API payload, not shell heuristics:

  • Row counts come from rowCount in schema infer, not wc -l. RFC 4180 CSV cells can embed newlines inside quotes; line counts lie.
  • After import, import status <id> returns { status, summary: { imported, total, issues }, … }. Counts are nested under summary, not at the top level.
  • imports list returns Payload pagination — read the batch list from .docs, not .batches.
DataSchemaSearch & Explore