Skip to main content
Probe keeps a catalog of the team’s data: sources (where outside data came from), datasets (named, versioned, made by runs) and checkpoints (every save every run made). Each one links to the runs that made and used it, so you can answer “what exactly did this model train on, and how was that data made?” months later.

​
Sources

A source is where a chain of data starts: a Hugging Face dataset or model, a bucket prefix, a URL, a git repo.
The same uri always gives the same source (hf:datasets/allenai/c4@1588ec4). A uri that carries a credential (a presigned signature, a token, a password) is refused before it leaves your machine. A source a run used cannot be deleted.

​
Datasets

A dataset is a team-wide name with versions: dataset:c4-clean@3. Register what a run wrote:
  • Bytes stay where they are by default. A folder is identified by its manifest: each file’s path, size and SHA-256 (hashed within the run’s I/O budget; past it, a fingerprint). Probe stores the manifest, never the data.
  • Same bytes, same version. Registering identical bytes again adds no version; it records the run as a check on the existing one.
  • Store it in Probe with --upload / upload=True, file or folder. Unchanged files are never uploaded twice. Hosted data has its own quota: 50 GB on the free plan (datasets and checkpoints together), no limit on paid plans.
  • A step done by hand: probe dataset add OUT --name N --from dataset:raw@1 --step "dedup by hand" --project P records the step as a finished run.
  • A mixture: log_dataset(..., composition={"dataset:a@2": 24000, "dataset:b@1": 10000}) (rows or shares) records what the new dataset is made of.
  • Read-restricted projects: registering from a run in one needs --publish / publish=True, because the catalog is team-wide.
  • A dataset any run used cannot be deleted.

​
What a run used

Say what a run trained on, and with what role (train, validation, test, eval, init, tokenizer, other):
dataset:<name> without a version means the version current at that moment, stored by its id: history never moves when @4 lands. The Hugging Face integration records the base model (init) and Hub datasets for you.

​
Checkpoints

Every checkpoint a run saved, team-wide, with what it was trained on and what it started from:
A checkpoint freezes the run’s declared training inputs at the moment it was saved, so switching datasets later never rewrites what an old checkpoint learned from. After a rewind (--rewind-to-step), checkpoints from the discarded range are marked rewound and are never picked by @latest or @best.

​
Where it came from

The walk also follows files nobody registered: a file one run wrote and another read is matched by its hash, so the chain exists without log_dataset anywhere. For data Probe must not read, use the hash level: the same chain, from hashes and row counts only.

​
Downloading

Data kept outside Probe is not downloaded; the command prints where it lives.
The catalog needs CLI 0.253.0 or later. Every SDK call warns once and carries on if the server or network says no: it never stops a training job.