{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-03-10-introducing-storage-buckets-on-the-hugging-face-hub/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "Hugging Face Models and Datasets repos are great for publishing final artifacts. But production ML generates a constant stream of intermediate files (checkpoints, optimizer states, processed shards, logs, traces, etc.) that change often, arrive from many jobs at once, and rarely need version control.\n\n**Storage Buckets** are built exactly for this: mutable, S3-like object storage you can browse on the Hub, script from Python, or manage with the `hf` CLI. And because they are backed by [Xet](https://huggingface.co/docs/hub/en/xet), they are especially efficient for ML artifacts that share content across files.\n\n![](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/buckets/buckets-annoucement.png)\n\n## Why we built Buckets\n\nGit starts to feel like the wrong abstraction pretty quickly when you're dealing with:\n\n- Training clusters writing checkpoints and optimizer states throughout a run\n- Data pipelines processing raw datasets iteratively\n- Agents storing traces, memory, and shared knowledge graphs\n\nThe storage need in all these cases is the same: write fast, overwrite when needed, sync directories, remove stale files, and keep things moving.\n\nA Bucket is a non-versioned storage container on the Hub. It lives under a user or organization namespace, has standard Hugging Face permissions, can be private or public, has a page you can open in your browser, and can be addressed programmatically with a handle like `hf://buckets/username/my-training-bucket`.\n\n## Why Xet matters\n\nBuckets are built on [Xet](https://huggingface.co/docs/hub/en/xet), Hugging Face’s chunk-based storage backend, and this matters more than it might seem.\n\nInstead of treating files as monolithic blobs, Xet breaks content into chunks and deduplicates across them. Upload a processed dataset that’s mostly similar to the raw one? Many chunks already exist. Store successive checkpoints where large parts of the model are frozen? Same story. Buckets skip the bytes that are already there, which means less bandwidth, faster transfers, and more efficient storage.\n\nThis is a natural fit for ML workloads. Training pipelines constantly produce families of related artifacts — raw and processed data, successive checkpoints, Agent traces and derived summaries — and Xet is designed to take advantage of that overlap.\n\nFor Enterprise customers, billing is based on deduplicated storage, so shared chunks directly reduce the billed footprint. Deduplication helps with both speed and cost.\n\n![](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/buckets/buckets-xet-dedup.png)\n\n## Pre-warming: bringing data close to compute\n\nBuckets live on the Hub, which means global storage by default. But not every workload can afford to pull data from wherever it happens to live, for distributed training and large-scale pipelines, storage location directly affects throughput.\n\nPre-warming lets you bring hot data closer to the cloud provider and region where your compute runs. Instead of data traveling across regions on every read, you declare where you need it and Buckets make sure it's already there when your jobs start. This is especially useful for training clusters that need fast access to large datasets or checkpoints, and for multi-region setups where different parts of a pipeline run in different clouds.\n\nWe are partnering with AWS and GCP to start with, more more cloud providers coming in the future.\n\n![](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/buckets/buckets-cdn-prewarm.png)\n\n## Getting started\n\nYou can get a bucket up and running in under 2 minutes with the `hf` CLI. First thing is to install it and log in:\n\n```bash\ncurl -LsSf https://hf.co/cli/install.sh | bash\nhf auth login\n```\n\nCreate a bucket for your project:\n\n```bash\nhf buckets create my-training-bucket --private\n```\n\nSay your training job is writing checkpoints locally to `./checkpoints`. Sync that directory into the Bucket:\n\n```bash\nhf buckets sync ./checkpoints hf://buckets/username/my-training-bucket/checkpoints\n```\n\nFor large transfers, you might want to see what will happen before anything moves. `--dry-run` prints the plan without executing anything:\n\n```bash\nhf buckets sync ./checkpoints hf://buckets/username/my-training-bucket/checkpoints --dry-run\n```\n\nYou can also save the plan to a file for review and apply it later:\n\n```bash\nhf buckets sync ./checkpoints hf://buckets/username/my-training-bucket/checkpoints --plan sync-plan.jsonl\nhf buckets sync --apply sync-plan.jsonl\n```\n\nOnce done, inspect the Bucket from the CLI:\n\n```bash\nhf buckets list username/my-training-bucket -h\n```\n\nor browse it directly on the Hub at `https://huggingface.co/buckets/username/my-training-bucket`.\n\nThat is the whole loop. Create a bucket, sync your working data into it, check on it when you need to, and save the versioned repo for when something is worth publishing. For one-off operations, `hf buckets cp` copies individual files and `hf buckets remove` cleans up stale objects.\n\n## Using Buckets from Python\n\nEverything above also works from Python via [`huggingface_hub`](https://github.com/huggingface/huggingface_hub) (available since [v1.5.0](https://github.com/huggingface/huggingface_hub/releases/tag/v1.5.0)). The API follows the same pattern: create, sync, inspect.\n\n```python\nfrom huggingface_hub import create_bucket, list_bucket_tree, sync_bucket\n\ncreate_bucket(\"my-training-bucket\", private=True, exist_ok=True)\n\nsync_bucket(\n \"./checkpoints\",\n \"hf://buckets/username/my-training-bucket/checkpoints\",\n)\n\nfor item in list_bucket_tree(\n \"username/my-training-bucket\",\n prefix=\"checkpoints\",\n recursive=True,\n):\n print(item.path, item.size)\n```\n\nThis makes it straightforward to integrate Buckets into training scripts, data pipelines, or any service that manages artifacts programmatically. The Python client also supports batch uploads, selective downloads, deletes, and bucket moves for when you need finer control.\n\nBucket support is also available in JavaScript via [`@huggingface/hub`](https://www.npmjs.com/package/@huggingface/hub) (since v2.10.5), so you can integrate Buckets into Node.js services and web applications as well.\n\n## Filesystem integration\n\nBuckets also work through `HfFileSystem`, the [fsspec](https://filesystem-spec.readthedocs.io/)-compatible filesystem in `huggingface_hub`. This means you can list, read, write, and glob Bucket contents using standard filesystem operations — and any library that supports fsspec can access Buckets directly.\n\n```python\nfrom huggingface_hub import hffs\n\n# List files in a bucket directory\nhffs.ls(\"buckets/username/my-training-bucket/checkpoints\", detail=False)\n\n# Glob for specific files\nhffs.glob(\"buckets/username/my-training-bucket/**/*.parquet\")\n\n# Read a file directly\nwith hffs.open(\"buckets/username/my-training-bucket/config.yaml\", \"r\") as f:\n print(f.read())\n```\n\nBecause fsspec is the standard Python interface for remote filesystems, libraries like pandas, Polars, and Dask can read from and write to Buckets using `hf://` paths with no extra setup:\n\n```python\nimport pandas as pd\n\n# Read a CSV directly from a Bucket\ndf = pd.read_csv(\"hf://buckets/username/my-training-bucket/results.csv\")\n\n# Write results back\ndf.to_csv(\"hf://buckets/username/my-training-bucket/summary.csv\")\n```\n\nThis makes it easy to plug Buckets into existing data workflows without changing how your code reads or writes files.\n\n## From Buckets to versioned repos\n\nBuckets are the fast, mutable place where artifacts live while they are still in motion. Once something becomes a stable deliverable, it usually belongs to a versioned model or dataset repo.\n\nOn the roadmap, we plan to support direct transfers between Buckets and repos in both directions: promote final checkpoint weights into a model repo, or commit processed shards into a dataset repo once a pipeline completes. The working layer and the publishing layer stay separate, but fit into one continuous Hub-native workflow.\n\n## Trusted by launch partners\n\nBefore opening Buckets to everyone, we ran a private beta with a small group of launch partners.\n\n![](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/buckets/buckets-launch-partners.png)\n\nA huge thank you to Jasper, Arcee, IBM, and PixAI for testing early versions, surfacing bugs, and sharing feedback that directly shaped this feature.\n\n## Conclusion and resources\n\nStorage Buckets bring a missing storage layer to the Hub. They give you a Hub-native place for the mutable, high-throughput side of ML: checkpoints, processed data, Agent traces, logs, and everything else that is useful before it becomes final.\n\nBecause they are built on Xet, Buckets are not just easier to use than forcing everything through Git. They are also more efficient for the kinds of related artifacts AI systems produce all the time. That means faster transfers, better deduplication, and on Enterprise plans, billing that benefits from the deduplicated footprint.\n\nIf you already use the Hub, Buckets let you keep more of your workflow in one place. If you come from S3-style storage, they give you a familiar model with better alignment to AI artifacts and a clear path toward final publication on the Hub.\n\nBuckets are included in existing [Hub storage plans](https://huggingface.co/docs/hub/en/storage-limits#storage-plans). Free accounts come with storage to get started, and PRO and Enterprise plans offer higher limits. See the [storage page](https://huggingface.co/storage) for details.\n\nRead more and try it yourself:\n\n- [Buckets guide](https://huggingface.co/docs/huggingface_hub/en/guides/buckets)\n- [Hub documentation](https://huggingface.co/docs/hub/storage-buckets)\n- CLI [Installation guide](https://huggingface.co/docs/huggingface_hub/en/installation)\n- CLI [guide](https://huggingface.co/docs/huggingface_hub/en/guides/cli) and [reference](https://huggingface.co/docs/huggingface_hub/en/package_reference/cli)\n- [Example Bucket on the Hub](https://huggingface.co/buckets/julien-c/my-training-bucket)\n- [Storage pricing](https://huggingface.co/pricing#storage)\n", "authors": [], "captured_at": "2026-10-04T12:50:30.804479567+00:00", "category": "ai", "content_html": "

Hugging Face Models and Datasets repos are great for publishing final artifacts. But production ML generates a constant stream of intermediate files (checkpoints, optimizer states, processed shards, logs, traces, etc.) that change often, arrive from many jobs at once, and rarely need version control.

\n

Storage Buckets are built exactly for this: mutable, S3-like object storage you can browse on the Hub, script from Python, or manage with the hf CLI. And because they are backed by Xet, they are especially efficient for ML artifacts that share content across files.

\n

\"\"

\n

Why we built Buckets

\n

Git starts to feel like the wrong abstraction pretty quickly when you're dealing with:

\n\n

The storage need in all these cases is the same: write fast, overwrite when needed, sync directories, remove stale files, and keep things moving.

\n

A Bucket is a non-versioned storage container on the Hub. It lives under a user or organization namespace, has standard Hugging Face permissions, can be private or public, has a page you can open in your browser, and can be addressed programmatically with a handle like hf://buckets/username/my-training-bucket.

\n

Why Xet matters

\n

Buckets are built on Xet, Hugging Face’s chunk-based storage backend, and this matters more than it might seem.

\n

Instead of treating files as monolithic blobs, Xet breaks content into chunks and deduplicates across them. Upload a processed dataset that’s mostly similar to the raw one? Many chunks already exist. Store successive checkpoints where large parts of the model are frozen? Same story. Buckets skip the bytes that are already there, which means less bandwidth, faster transfers, and more efficient storage.

\n

This is a natural fit for ML workloads. Training pipelines constantly produce families of related artifacts — raw and processed data, successive checkpoints, Agent traces and derived summaries — and Xet is designed to take advantage of that overlap.

\n

For Enterprise customers, billing is based on deduplicated storage, so shared chunks directly reduce the billed footprint. Deduplication helps with both speed and cost.

\n

\"\"

\n

Pre-warming: bringing data close to compute

\n

Buckets live on the Hub, which means global storage by default. But not every workload can afford to pull data from wherever it happens to live, for distributed training and large-scale pipelines, storage location directly affects throughput.

\n

Pre-warming lets you bring hot data closer to the cloud provider and region where your compute runs. Instead of data traveling across regions on every read, you declare where you need it and Buckets make sure it's already there when your jobs start. This is especially useful for training clusters that need fast access to large datasets or checkpoints, and for multi-region setups where different parts of a pipeline run in different clouds.

\n

We are partnering with AWS and GCP to start with, more more cloud providers coming in the future.

\n

\"\"

\n

Getting started

\n

You can get a bucket up and running in under 2 minutes with the hf CLI. First thing is to install it and log in:

\n
curl -LsSf https://hf.co/cli/install.sh | bash\nhf auth login\n
\n

Create a bucket for your project:

\n
hf buckets create my-training-bucket --private\n
\n

Say your training job is writing checkpoints locally to ./checkpoints. Sync that directory into the Bucket:

\n
hf buckets sync ./checkpoints hf://buckets/username/my-training-bucket/checkpoints\n
\n

For large transfers, you might want to see what will happen before anything moves. --dry-run prints the plan without executing anything:

\n
hf buckets sync ./checkpoints hf://buckets/username/my-training-bucket/checkpoints --dry-run\n
\n

You can also save the plan to a file for review and apply it later:

\n
hf buckets sync ./checkpoints hf://buckets/username/my-training-bucket/checkpoints --plan sync-plan.jsonl\nhf buckets sync --apply sync-plan.jsonl\n
\n

Once done, inspect the Bucket from the CLI:

\n
hf buckets list username/my-training-bucket -h\n
\n

or browse it directly on the Hub at https://huggingface.co/buckets/username/my-training-bucket.

\n

That is the whole loop. Create a bucket, sync your working data into it, check on it when you need to, and save the versioned repo for when something is worth publishing. For one-off operations, hf buckets cp copies individual files and hf buckets remove cleans up stale objects.

\n

Using Buckets from Python

\n

Everything above also works from Python via huggingface_hub (available since v1.5.0). The API follows the same pattern: create, sync, inspect.

\n
from huggingface_hub import create_bucket, list_bucket_tree, sync_bucket\n\ncreate_bucket("my-training-bucket", private=True, exist_ok=True)\n\nsync_bucket(\n    "./checkpoints",\n    "hf://buckets/username/my-training-bucket/checkpoints",\n)\n\nfor item in list_bucket_tree(\n    "username/my-training-bucket",\n    prefix="checkpoints",\n    recursive=True,\n):\n    print(item.path, item.size)\n
\n

This makes it straightforward to integrate Buckets into training scripts, data pipelines, or any service that manages artifacts programmatically. The Python client also supports batch uploads, selective downloads, deletes, and bucket moves for when you need finer control.

\n

Bucket support is also available in JavaScript via @huggingface/hub (since v2.10.5), so you can integrate Buckets into Node.js services and web applications as well.

\n

Filesystem integration

\n

Buckets also work through HfFileSystem, the fsspec-compatible filesystem in huggingface_hub. This means you can list, read, write, and glob Bucket contents using standard filesystem operations — and any library that supports fsspec can access Buckets directly.

\n
from huggingface_hub import hffs\n\n# List files in a bucket directory\nhffs.ls("buckets/username/my-training-bucket/checkpoints", detail=False)\n\n# Glob for specific files\nhffs.glob("buckets/username/my-training-bucket/**/*.parquet")\n\n# Read a file directly\nwith hffs.open("buckets/username/my-training-bucket/config.yaml", "r") as f:\n    print(f.read())\n
\n

Because fsspec is the standard Python interface for remote filesystems, libraries like pandas, Polars, and Dask can read from and write to Buckets using hf:// paths with no extra setup:

\n
import pandas as pd\n\n# Read a CSV directly from a Bucket\ndf = pd.read_csv("hf://buckets/username/my-training-bucket/results.csv")\n\n# Write results back\ndf.to_csv("hf://buckets/username/my-training-bucket/summary.csv")\n
\n

This makes it easy to plug Buckets into existing data workflows without changing how your code reads or writes files.

\n

From Buckets to versioned repos

\n

Buckets are the fast, mutable place where artifacts live while they are still in motion. Once something becomes a stable deliverable, it usually belongs to a versioned model or dataset repo.

\n

On the roadmap, we plan to support direct transfers between Buckets and repos in both directions: promote final checkpoint weights into a model repo, or commit processed shards into a dataset repo once a pipeline completes. The working layer and the publishing layer stay separate, but fit into one continuous Hub-native workflow.

\n

Trusted by launch partners

\n

Before opening Buckets to everyone, we ran a private beta with a small group of launch partners.

\n

\"\"

\n

A huge thank you to Jasper, Arcee, IBM, and PixAI for testing early versions, surfacing bugs, and sharing feedback that directly shaped this feature.

\n

Conclusion and resources

\n

Storage Buckets bring a missing storage layer to the Hub. They give you a Hub-native place for the mutable, high-throughput side of ML: checkpoints, processed data, Agent traces, logs, and everything else that is useful before it becomes final.

\n

Because they are built on Xet, Buckets are not just easier to use than forcing everything through Git. They are also more efficient for the kinds of related artifacts AI systems produce all the time. That means faster transfers, better deduplication, and on Enterprise plans, billing that benefits from the deduplicated footprint.

\n

If you already use the Hub, Buckets let you keep more of your workflow in one place. If you come from S3-style storage, they give you a familiar model with better alignment to AI artifacts and a clear path toward final publication on the Hub.

\n

Buckets are included in existing Hub storage plans. Free accounts come with storage to get started, and PRO and Enterprise plans offer higher limits. See the storage page for details.

\n

Read more and try it yourself:

\n\n", "content_markdown": "Hugging Face Models and Datasets repos are great for publishing final artifacts. But production ML generates a constant stream of intermediate files (checkpoints, optimizer states, processed shards, logs, traces, etc.) that change often, arrive from many jobs at once, and rarely need version control.\n\n**Storage Buckets** are built exactly for this: mutable, S3-like object storage you can browse on the Hub, script from Python, or manage with the `hf` CLI. And because they are backed by [Xet](https://huggingface.co/docs/hub/en/xet), they are especially efficient for ML artifacts that share content across files.\n\n![](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/buckets/buckets-annoucement.png)\n\n## Why we built Buckets\n\nGit starts to feel like the wrong abstraction pretty quickly when you're dealing with:\n\n- Training clusters writing checkpoints and optimizer states throughout a run\n- Data pipelines processing raw datasets iteratively\n- Agents storing traces, memory, and shared knowledge graphs\n\nThe storage need in all these cases is the same: write fast, overwrite when needed, sync directories, remove stale files, and keep things moving.\n\nA Bucket is a non-versioned storage container on the Hub. It lives under a user or organization namespace, has standard Hugging Face permissions, can be private or public, has a page you can open in your browser, and can be addressed programmatically with a handle like `hf://buckets/username/my-training-bucket`.\n\n## Why Xet matters\n\nBuckets are built on [Xet](https://huggingface.co/docs/hub/en/xet), Hugging Face’s chunk-based storage backend, and this matters more than it might seem.\n\nInstead of treating files as monolithic blobs, Xet breaks content into chunks and deduplicates across them. Upload a processed dataset that’s mostly similar to the raw one? Many chunks already exist. Store successive checkpoints where large parts of the model are frozen? Same story. Buckets skip the bytes that are already there, which means less bandwidth, faster transfers, and more efficient storage.\n\nThis is a natural fit for ML workloads. Training pipelines constantly produce families of related artifacts — raw and processed data, successive checkpoints, Agent traces and derived summaries — and Xet is designed to take advantage of that overlap.\n\nFor Enterprise customers, billing is based on deduplicated storage, so shared chunks directly reduce the billed footprint. Deduplication helps with both speed and cost.\n\n![](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/buckets/buckets-xet-dedup.png)\n\n## Pre-warming: bringing data close to compute\n\nBuckets live on the Hub, which means global storage by default. But not every workload can afford to pull data from wherever it happens to live, for distributed training and large-scale pipelines, storage location directly affects throughput.\n\nPre-warming lets you bring hot data closer to the cloud provider and region where your compute runs. Instead of data traveling across regions on every read, you declare where you need it and Buckets make sure it's already there when your jobs start. This is especially useful for training clusters that need fast access to large datasets or checkpoints, and for multi-region setups where different parts of a pipeline run in different clouds.\n\nWe are partnering with AWS and GCP to start with, more more cloud providers coming in the future.\n\n![](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/buckets/buckets-cdn-prewarm.png)\n\n## Getting started\n\nYou can get a bucket up and running in under 2 minutes with the `hf` CLI. First thing is to install it and log in:\n\n```bash\ncurl -LsSf https://hf.co/cli/install.sh | bash\nhf auth login\n```\n\nCreate a bucket for your project:\n\n```bash\nhf buckets create my-training-bucket --private\n```\n\nSay your training job is writing checkpoints locally to `./checkpoints`. Sync that directory into the Bucket:\n\n```bash\nhf buckets sync ./checkpoints hf://buckets/username/my-training-bucket/checkpoints\n```\n\nFor large transfers, you might want to see what will happen before anything moves. `--dry-run` prints the plan without executing anything:\n\n```bash\nhf buckets sync ./checkpoints hf://buckets/username/my-training-bucket/checkpoints --dry-run\n```\n\nYou can also save the plan to a file for review and apply it later:\n\n```bash\nhf buckets sync ./checkpoints hf://buckets/username/my-training-bucket/checkpoints --plan sync-plan.jsonl\nhf buckets sync --apply sync-plan.jsonl\n```\n\nOnce done, inspect the Bucket from the CLI:\n\n```bash\nhf buckets list username/my-training-bucket -h\n```\n\nor browse it directly on the Hub at `https://huggingface.co/buckets/username/my-training-bucket`.\n\nThat is the whole loop. Create a bucket, sync your working data into it, check on it when you need to, and save the versioned repo for when something is worth publishing. For one-off operations, `hf buckets cp` copies individual files and `hf buckets remove` cleans up stale objects.\n\n## Using Buckets from Python\n\nEverything above also works from Python via [`huggingface_hub`](https://github.com/huggingface/huggingface_hub) (available since [v1.5.0](https://github.com/huggingface/huggingface_hub/releases/tag/v1.5.0)). The API follows the same pattern: create, sync, inspect.\n\n```python\nfrom huggingface_hub import create_bucket, list_bucket_tree, sync_bucket\n\ncreate_bucket(\"my-training-bucket\", private=True, exist_ok=True)\n\nsync_bucket(\n \"./checkpoints\",\n \"hf://buckets/username/my-training-bucket/checkpoints\",\n)\n\nfor item in list_bucket_tree(\n \"username/my-training-bucket\",\n prefix=\"checkpoints\",\n recursive=True,\n):\n print(item.path, item.size)\n```\n\nThis makes it straightforward to integrate Buckets into training scripts, data pipelines, or any service that manages artifacts programmatically. The Python client also supports batch uploads, selective downloads, deletes, and bucket moves for when you need finer control.\n\nBucket support is also available in JavaScript via [`@huggingface/hub`](https://www.npmjs.com/package/@huggingface/hub) (since v2.10.5), so you can integrate Buckets into Node.js services and web applications as well.\n\n## Filesystem integration\n\nBuckets also work through `HfFileSystem`, the [fsspec](https://filesystem-spec.readthedocs.io/)-compatible filesystem in `huggingface_hub`. This means you can list, read, write, and glob Bucket contents using standard filesystem operations — and any library that supports fsspec can access Buckets directly.\n\n```python\nfrom huggingface_hub import hffs\n\n# List files in a bucket directory\nhffs.ls(\"buckets/username/my-training-bucket/checkpoints\", detail=False)\n\n# Glob for specific files\nhffs.glob(\"buckets/username/my-training-bucket/**/*.parquet\")\n\n# Read a file directly\nwith hffs.open(\"buckets/username/my-training-bucket/config.yaml\", \"r\") as f:\n print(f.read())\n```\n\nBecause fsspec is the standard Python interface for remote filesystems, libraries like pandas, Polars, and Dask can read from and write to Buckets using `hf://` paths with no extra setup:\n\n```python\nimport pandas as pd\n\n# Read a CSV directly from a Bucket\ndf = pd.read_csv(\"hf://buckets/username/my-training-bucket/results.csv\")\n\n# Write results back\ndf.to_csv(\"hf://buckets/username/my-training-bucket/summary.csv\")\n```\n\nThis makes it easy to plug Buckets into existing data workflows without changing how your code reads or writes files.\n\n## From Buckets to versioned repos\n\nBuckets are the fast, mutable place where artifacts live while they are still in motion. Once something becomes a stable deliverable, it usually belongs to a versioned model or dataset repo.\n\nOn the roadmap, we plan to support direct transfers between Buckets and repos in both directions: promote final checkpoint weights into a model repo, or commit processed shards into a dataset repo once a pipeline completes. The working layer and the publishing layer stay separate, but fit into one continuous Hub-native workflow.\n\n## Trusted by launch partners\n\nBefore opening Buckets to everyone, we ran a private beta with a small group of launch partners.\n\n![](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/buckets/buckets-launch-partners.png)\n\nA huge thank you to Jasper, Arcee, IBM, and PixAI for testing early versions, surfacing bugs, and sharing feedback that directly shaped this feature.\n\n## Conclusion and resources\n\nStorage Buckets bring a missing storage layer to the Hub. They give you a Hub-native place for the mutable, high-throughput side of ML: checkpoints, processed data, Agent traces, logs, and everything else that is useful before it becomes final.\n\nBecause they are built on Xet, Buckets are not just easier to use than forcing everything through Git. They are also more efficient for the kinds of related artifacts AI systems produce all the time. That means faster transfers, better deduplication, and on Enterprise plans, billing that benefits from the deduplicated footprint.\n\nIf you already use the Hub, Buckets let you keep more of your workflow in one place. If you come from S3-style storage, they give you a familiar model with better alignment to AI artifacts and a clear path toward final publication on the Hub.\n\nBuckets are included in existing [Hub storage plans](https://huggingface.co/docs/hub/en/storage-limits#storage-plans). Free accounts come with storage to get started, and PRO and Enterprise plans offer higher limits. See the [storage page](https://huggingface.co/storage) for details.\n\nRead more and try it yourself:\n\n- [Buckets guide](https://huggingface.co/docs/huggingface_hub/en/guides/buckets)\n- [Hub documentation](https://huggingface.co/docs/hub/storage-buckets)\n- CLI [Installation guide](https://huggingface.co/docs/huggingface_hub/en/installation)\n- CLI [guide](https://huggingface.co/docs/huggingface_hub/en/guides/cli) and [reference](https://huggingface.co/docs/huggingface_hub/en/package_reference/cli)\n- [Example Bucket on the Hub](https://huggingface.co/buckets/julien-c/my-training-bucket)\n- [Storage pricing](https://huggingface.co/pricing#storage)\n", "dateCreated": "2026-10-04T12:50:30.804479567+00:00", "date_created": "2026-03-10T00:00:00+00:00", "date_modified": "2026-03-10T00:00:00+00:00", "date_published": "2026-03-10T00:00:00+00:00", "description": "Hugging Face Models and Datasets repos are great for publishing final artifacts. But production ML generates a constant stream of intermediate files (checkpoints, optimizer states, processed shards, logs, traces, etc.) that change often,…", "external_url": "https://huggingface.co/blog/storage-buckets", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/huggingface-co/2026/03/2026-03-10-introducing-storage-buckets-on-the-hugging-face-hub.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/huggingface-co/2026/03/2026-03-10-introducing-storage-buckets-on-the-hugging-face-hub.md" }, "id": "urn:aggr:item:c47ae7254a4eac1efb5a26736e41dbf30e5c3664", "isBasedOn": { "@id": "https://huggingface.co/blog/storage-buckets" }, "mainEntity": { "@id": "https://huggingface.co/blog/storage-buckets", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-03-10-introducing-storage-buckets-on-the-hugging-face-hub/#webpage" }, "dateModified": "2026-03-10T00:00:00+00:00", "datePublished": "2026-03-10T00:00:00+00:00", "headline": "Introducing Storage Buckets on the Hugging Face Hub", "timeRequired": "PT6M", "url": "https://huggingface.co/blog/storage-buckets", "wordCount": 1341 }, "name": "Archived snapshot: Introducing Storage Buckets on the Hugging Face Hub", "preview": null, "reading_minutes": 6, "replicated_at": null, "source": "huggingface-co", "source_name": "Hugging Face - Blog", "summary": null, "tags": [], "temporalCoverage": "2026-10-04T12:50:30.804479567+00:00", "title": "Introducing Storage Buckets on the Hugging Face Hub", "url": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-03-10-introducing-storage-buckets-on-the-hugging-face-hub/", "word_count": 1341 }