Skip to main content

Bulk exports

Exports are for data too large for a paged query. You create a job, poll it, then download its files. The request fields and response shapes are in the API reference; this page explains the workflow and what to expect.

Create a job​

H="Authorization: Bearer $VEINMESH_API_KEY"
API=https://data.cyberun.cloud

curl -s -H "$H" -H "Content-Type: application/json" \
  -X POST "$API/v2/exports" \
  -d '{
    "dataset": "book.snapshot",
    "symbols": ["SHFE.rb2701", "SHFE.rb2705"],
    "from_day": "2026-09-01",
    "to_day": "2026-10-20",
    "format": "parquet"
  }' | jq '{export_id, status}'

The answer is a job with an export_id and a status: queued, running, done, failed or expired. The job belongs to the key that created it.

Follow it​

curl -s -H "$H" "$API/v2/exports/<export_id>" | jq '{status, rows, files, error}'

While the job is running, pending shows the instrument-days still to read. Poll a few seconds apart; do not create the same job repeatedly.

When it is done, the response lists:

  • rows: total rows written;
  • files: one or more files with their index, row count and SHA-256;
  • partitions: the partition versions the job read, for reproducibility.

Download​

GET /v2/exports/{export_id}/files/{file_index}

Files are streamed through the gateway; you never receive a bucket URL. Use the file index from the job response.

  • If the job is still running, the download answers 503 data_pending.
  • If the job has expired, files answer 404 not_found.
  • If the file index is outside the job's file list, 404 not_found.

Formats and size​

  • parquet: typed columns, decimals as DECIMAL(38,9), timestamps in UTC nanoseconds.
  • csv: one header row; list fields become one column per level (bid_px_1, bid_px_2, …).

A large job is split into several files. Download every file; the row counts in the job are the authoritative total. Exports stay available for a limited time after they complete; see Limits.

What you can export​

The job's dataset, symbols and range are validated when you create it:

  • the dataset must be one the instrument has;
  • the symbols must resolve on the range's as_of day; a continuous symbol (.main, .cN) is resolved per trading day into contract segments (use one continuous symbol per export; instant ranges are not accepted). Rows keep the continuous symbol; instrument_id is the resolved contract and changes at a roll;
  • the range must be within the export limits.

See Limits for symbol counts, range and retention.

Python​

job = client.export("book.snapshot", ["SHFE.rb2701", "SHFE.rb2705"],
                    from_day="2026-09-01", to_day="2026-10-20", format="parquet")
df = job.wait().to_pandas()       # downloads and combines the files
files = job.download("./data")    # or keep the files

job.wait() polls until the job is done; client.get_export(id) reopens a job from another process. to_pandas(exact=True) keeps Decimal values.

Failures​

A failed job carries an error string. Submit it again if the range is still needed; a failed job does not retry itself. A not_found for the job means the id is wrong or the key does not own it.