Missing files when downloading subdirectory via huggingface-cli download

[Raibows]

Describe the bug

Env

I'm using WSL2 Ubuntu22.04 with huggingface-hub==1.1.4.

The repo is a private model repo enabled with Xet.

Files are missing

The command I used looks like

hf download --repo-type model --token xxxx--local-dir ./ xxxxx/a-private-repo --include 'xxxxx/*'

It did not throw any errors, but I found many files and subdirectories are missing.

I also tried rm -rf ~/.cache/huggingface/hub/models--xxxx--a-private-repo but still cannot download those files and subdirectories.

Git works well

However, the following git command works well

git clone --filter=blob:none --no-checkout https://xxxxxxx
cd <repo>
git sparse-checkout init --cone
git sparse-checkout set path/to/subdir/xxxxxx
git checkout main

Please let me know how to provide more useful information to help and I cannot make this repo public since it has many sensitive information.

Reproduction

---

Logs

I set the HF_DEBUG=1, however, it does not print more logs.


Fetching 864 files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████| 864/864 [00:00<00:00, 1027.44it/s]
Download complete: : 0.00B [00:00, ?B/s]              /tmp█████████████████████████████████████████████████████████████████████████▎   | 835/864 [00:00<00:00, 1005.63it/s]
Download complete: : 0.00B [00:00, ?B/s]

System info

- huggingface_hub version: 1.1.4
- Platform: Linux-6.6.87.2-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.18
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Running in Google Colab Enterprise ?: No
- Token path ?: /home/XXXXX/.cache/huggingface/token
- Has saved token ?: True
- Who am I ?: XXXXX
- Configured git credential helpers: cache
- Installation method: unknown
- httpx: 0.28.1
- hf_xet: 1.2.0
- gradio: N/A
- tensorboard: N/A
- ENDPOINT: https://huggingface.co
- HF_HUB_CACHE: /home/XXXX/.cache/huggingface/hub
- HF_ASSETS_CACHE: /home/XXXX/.cache/huggingface/assets
- HF_TOKEN_PATH: /home/XXXX/.cache/huggingface/token
- HF_STORED_TOKENS_PATH: /home/XXXX/.cache/huggingface/stored_tokens
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_DISABLE_XET: False
- HF_HUB_ETAG_TIMEOUT: 10
- HF_HUB_DOWNLOAD_TIMEOUT: 10
- HF_XET_HIGH_PERFORMANCE: False

[hanouticelina]

Hi @Raibows,
Sorry about that. How many files are in the subdirectory you're trying to download?
I'm not able to reproduce this issue on my side with a private repo. would you be able to try reproducing it with a minimal public repo that mimics the same structure (but without any sensitive content)? that would be the easiest way for us to understand what might be going wrong.

thank you!

[Raibows]

Less than 10 subdirectories. And each subdirectory contains about 100 files.

Actually, this repo worked well before I uploaded some subdirectories.

I cannot reproduce it in other repos. It is really weird. Could you please provide some diagnosis scripts for me running it?

The bug exists and I want to help fix it. 😂

[hanouticelina]

Hey @Raibows,
Sorry for the delay! Could you try installing the latest version of huggingface_hub v1.1.5 and rerunning the command with HF_DEBUG enabled?

pip install -U huggingface_hub
export HF_DEBUG=1
hf download --repo-type model --token xxxx--local-dir ./ xxxxx/a-private-repo --include 'xxxxx/*'

Another thing you can try is deleting the cache inside the download directory (i.e. ./.cache/huggingface) before running the command again.

[Raibows]

Hi @hanouticelina

These are the full commands I run

pip install -U huggingface_hub
export HF_DEBUG=1
rm -rf ~/.cache/huggingface 
rm -rf ./xxxxxx
hf download --repo-type model --token xxxx--local-dir ./ xxxxx/a-private-repo --include 'xxxxx/*'

The results are still not complete, many subdirectories are missing.

The truncated log is here:

Error while checking for CLI update.
Traceback (most recent call last):
  File "XXXXXX/mambaforge/envs/webarena/lib/python3.10/site-packages/huggingface_hub/cli/_cli_utils.py", line 127, in check_cli_update
    _check_cli_update()
  File "XXXXXX/mambaforge/envs/webarena/lib/python3.10/site-packages/huggingface_hub/cli/_cli_utils.py", line 147, in _check_cli_update
    Path(constants.CHECK_FOR_UPDATE_DONE_PATH).touch()
  File "XXXXXX/mambaforge/envs/webarena/lib/python3.10/pathlib.py", line 1168, in touch
    self._accessor.touch(self, mode, exist_ok)
  File "XXXXXX/mambaforge/envs/webarena/lib/python3.10/pathlib.py", line 331, in touch
    fd = os.open(path, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: 'XXXXXX/.cache/huggingface/.check_for_update_done'
Request b5d27632-9f5d-4b08-9be7-aad3dd07a89f: GET https://huggingface.co/api/models/XXXXXX/A-PRIVATE-REPO/revision/main (authenticated: True)
Send: curl -X GET -H 'accept: */*' -H 'accept-encoding: gzip, deflate' -H 'authorization: <TOKEN>' -H 'connection: keep-alive' -H 'host: huggingface.co' -H 'user-agent: huggingface-cli/None; hf_hub/1.1.5; python/3.10.18; torch/2.8.0+cpu' -H 'x-amzn-trace-id: b5d27632-9f5d-4b08-9be7-aad3dd07a89f' -d https://huggingface.co/api/models/XXXXXX/A-PRIVATE-REPO/revision/main
Request 830ef64a-c938-4fa1-a0ae-d965d60e4069: HEAD https://huggingface.co/XXXXXX/A-PRIVATE-REPO/resolve/d2c48e6e654ec2fd1eefcf83844ab3a4388f09a2/ICRL/results/tot/AMC-TTT_Qwen/Qwen2.5-Math-7B_20251116_014451_232686/.prompt_cache/02c489c6bf2aae5f3637bef388625fe642993ed2.json (authenticated: True)
Send: curl -X HEAD -H 'accept: */*' -H 'accept-encoding: identity' -H 'authorization: <TOKEN>' -H 'connection: keep-alive' -H 'host: huggingface.co' -H 'user-agent: huggingface-cli/None; hf_hub/1.1.5; python/3.10.18; torch/2.8.0+cpu' -H 'x-amzn-trace-id: 830ef64a-c938-4fa1-a0ae-d965d60e4069' -d https://huggingface.co/XXXXXX/A-PRIVATE-REPO/resolve/d2c48e6e654ec2fd1eefcf83844ab3a4388f09a2/ICRL/results/tot/AMC-TTT_Qwen/Qwen2.5-Math-7B_20251116_014451_232686/.prompt_cache/02c489c6bf2aae5f3637bef388625fe642993ed2.json

Fetching 735 files:   0%|          | 0/735 [00:00<?, ?it/s]Request afa6a3f2-6e6b-420a-9926-17864e101bcf: HEAD https://huggingface.co/XXXXXX/A-PRIVATE-REPO/resolve/d2c48e6e654ec2fd1eefcf83844ab3a4388f09a2/ICRL/results/tot/AMC-TTT_Qwen/Qwen2.5-Math-7B_20251116_014451_232686/.prompt_cache/00aa1b4b5d7370b28059e5638aeccc60e29fac5b.json (authenticated: True)
Send: curl -X HEAD -H 'accept: */*' -H 'accept-encoding: identity' -H 'authorization: <TOKEN>' -H 'connection: keep-alive' -H 'host: huggingface.co' -H 'user-agent: huggingface-cli/None; hf_hub/1.1.5; python/3.10.18; torch/2.8.0+cpu' -H 'x-amzn-trace-id: afa6a3f2-6e6b-420a-9926-17864e101bcf' -d https://huggingface.co/XXXXXX/A-PRIVATE-REPO/resolve/d2c48e6e654ec2fd1eefcf83844ab3a4388f09a2/ICRL/results/tot/AMC-TTT_Qwen/Qwen2.5-Math-7B_20251116_014451_232686/.prompt_cache/00aa1b4b5d7370b28059e5638aeccc60e29fac5b.json
Request cd7a96e2-5308-4ba7-8799-5c2223b27f77: HEAD https://huggingface.co/XXXXXX/A-PRIVATE-REPO/resolve/d2c48e6e654ec2fd1eefcf83844ab3a4388f09a2/ICRL/results/tot/AMC-TTT_Qwen/Qwen2.5-Math-7B_20251116_014451_232686/.prompt_cache/0293970cee012a7aa269130fd6a4865c1b1adc46.json (authenticated: True)
Request 7165e591-aac2-4a63-a107-93f9a7d44788: HEAD https://huggingface.co/XXXXXX/A-PRIVATE-REPO/resolve/d2c48e6e654ec2fd1eefcf83844ab3a4388f09a2/ICRL/results/tot/AMC-TTT_Qwen/Qwen2.5-Math-7B_20251116_014451_232686/.prompt_cache/032a60fc90225bd42a06f2e0ca16ced88fb0572c.json (authenticated: True)
Send: curl -X HEAD -H 'accept: */*' -H 'accept-encoding: identity' -H 'authorization: <TOKEN>' -H 'connection: keep-alive' -H 'host: huggingface.co' -H 'user-agent: huggingface-cli/None; hf_hub/1.1.5; python/3.10.18; torch/2.8.0+cpu' -H 'x-amzn-trace-id: cd7a96e2-5308-4ba7-8799-5c2223b27f77' -d https://huggingface.co/XXXXXX/A-PRIVATE-REPO/resolve/d2c48e6e654ec2fd1eefcf83844ab3a4388f09a2/ICRL/results/tot/AMC-TTT_Qwen/Qwen2.5-Math-7B_20251116_014451_232686/.prompt_cache/0293970cee012a7aa269130fd6a4865c1b1adc46.json
Send: curl -X HEAD -H 'accept: */*' -H 'accept-encoding: identity' -H 'authorization: <TOKEN>' -H 'connection: keep-alive' -H 'host: huggingface.co' -H 'user-agent: huggingface-cli/None; hf_hub/1.1.5; python/3.10.18; torch/2.8.0+cpu' -H 'x-amzn-trace-id: 7165e591-aac2-4a63-a107-93f9a7d44788' -d https://huggingface.co/XXXXXX/A-PRIVATE-REPO/resolve/d2c48e6e654ec2fd1eefcf83844ab3a4388f09a2/ICRL/results/tot/AMC-TTT_Qwen/Qwen2.5-Math-7B_20251116_014451_232686/.prompt_cache/032a60fc90225bd42a06f2e0ca16ced88fb0572c.json
Request 948eb842-3df7-4a01-9ef0-72b0c7fa1551: HEAD https://huggingface.co/XXXXXX/A-PRIVATE-REPO/resolve/d2c48e6e654ec2fd1eefcf83844ab3a4388f09a2/ICRL/results/tot/AMC-TTT_Qwen/Qwen2.5-Math-7B_20251116_014451_232686/.prompt_cache/0003e0b28787235a338336ffcc12e4fc10fbc0f8.json (authenticated: True)
Request da296549-0e3a-463f-a9b4-67155fd39a52: HEAD https://huggingface.co/XXXXXX/A-PRIVATE-REPO/resolve/d2c48e6e654ec2fd1eefcf83844ab3a4388f09a2/ICRL/results/tot/AMC-TTT_Qwen/Qwen2.5-Math-7B_20251116_014451_232686/.prompt_cache/05b8d2e57121e3dfebd98d756032547ce93a371e.json (authenticated: True)
Request b222f616-dd97-45e4-8e9c-ba8a066cc69d: HEAD https://huggingface.co/XXXXXX/A-PRIVATE-REPO/resolve/d2c48e6e654ec2fd1eefcf83844ab3a4388f09a2/ICRL/results/tot/AMC-TTT_Qwen/Qwen2.5-Math-7B_20251116_014451_232686/.prompt_cache/0349b6ae49d6b5022cb262bd272d9dd6c9fe608c.json (authenticated: True)
Request d02425e1-c29e-44ee-a390-1b40943d8a9e: HEAD https://huggingface.co/XXXXXX/A-PRIVATE-REPO/resolve/d2c48e6e654ec2fd1eefcf83844ab3a4388f09a2/ICRL/results/tot/AMC-TTT_Qwen/Qwen2.5-Math-7B_20251116_014451_232686/.prompt_cache/0116df6c8b8c54197619d51a87a02a0c5540edf3.json (authenticated: True)
Send: curl -X HEAD -H 'accept: */*' -H 'accept-encoding: identity' -H 'authorization: <TOKEN>' -H 'connection: keep-alive' -H 'host: huggingface.co' -H 'user-agent: huggingface-cli/None; hf_hub/1.1.5; python/3.10.18; torch/2.8.0+cpu' -H 'x-amzn-trace-id: 948eb842-3df7-4a01-9ef0-72b0c7fa1551' -d https://huggingface.co/XXXXXX/A-PRIVATE-REPO/resolve/d2c48e6e654ec2fd1eefcf83844ab3a4388f09a2/ICRL/results/tot/AMC-TTT_Qwen/Qwen2.5-Math-7B_20251116_014451_232686/.prompt_cache/0003e0b28787235a338336ffcc12e4fc10fbc0f8.json
Send: curl -X HEAD -H 'accept: */*' -H 'accept-encoding: identity' -H 'authorization: <TOKEN>' -H 'connection: keep-alive' -H 'host: huggingface.co' -H 'user-agent: huggingface-cli/None; hf_hub/1.1.5; python/3.10.18; torch/2.8.0+cpu' -H 'x-amzn-trace-id: da296549-0e3a-463f-a9b4-67155fd39a52' -d https://huggingface.co/XXXXXX/A-PRIVATE-REPO/resolve/d2c48e6e654ec2fd1eefcf83844ab3a4388f09a2/ICRL/results/tot/AMC-TTT_Qwen/Qwen2.5-Math-7B_20251116_014451_232686/.prompt_cache/05b8d2e57121e3dfebd98d756032547ce93a371e.json
Send: curl -X HEAD -H 'accept: */*' -H 'accept-encoding: identity' -H 'authorization: <TOKEN>' -H 'connection: keep-alive' -H 'host: huggingface.co' -H 'user-agent: huggingface-cli/None; hf_hub/1.1.5; python/3.10.18; torch/2.8.0+cpu' -H 'x-amzn-trace-id: b222f616-dd97-45e4-8e9c-ba8a066cc69d' -d https://huggingface.co/XXXXXX/A-PRIVATE-REPO/resolve/d2c48e6e654ec2fd1eefcf83844ab3a4388f09a2/ICRL/results/tot/AMC-TTT_Qwen/Qwen2.5-Math-7B_20251116_014451_232686/.prompt_cache/0349b6ae49d6b5022cb262bd272d9dd6c9fe608c.json
........

Full log can be found in the attached log

log.log

file.

[hanouticelina]

hi @Raibows,
thanks for all the logs! Unfortunately there isn't anything in there that points to a clear culprit, and without being able to reproduce locally it's difficult to dig deeper. Here's another batch of checks that would maybe help us narrow things down :

I suggest running the commands with HF_DEBUG enabled and the local cache cleaned.

• Download a single missing file directly (just to confirm the repo is ok):

   hf download xxxxx/a-private-repo path/to/missing/file --local-dir ./single-file-debug --token xxxx 

make sure the file is downloaded locally.

• Compare the API's file counts (to be sure nothing is silently filtered):

   from huggingface_hub import list_repo_tree, model_info
   from huggingface_hub.hf_api import RepoFile

   repo_id = "xxxxx/a-private-repo"

   info = model_info(repo_id, files_metadata=False)
   siblings = len(info.siblings or [])
   print("siblings (model_info):", siblings)

   tree_count = sum(1 for obj in list_repo_tree(repo_id, recursive=True) if isinstance(obj, RepoFile))
   print("files (list_repo_tree):", tree_count)

Let me know if the counts match.

• Run the original command with --dry-run so we can inspect the matched files:

   hf download xxxxx/a-private-repo  --token xxxx --include 'xxxxx/*' --dry-run

check the summary line ("Will download X files (out of Y)") to see if X/Y matches what you expect, and check whether the subdirectories you're missing locally are actually listed in that dry-run output.

I hope this helps, beyond these checks it will be hard to investigate more without a reproducible example 😕

[Raibows]

Hi @hanouticelina

Download a single file

I used the following commands to download a single file from the repo:

rm -rf ~/.cache/huggingface
export HF_DEBUG=1
hf download xxxxx/xxxxrepo a-path-to-a-single-file-that-previously-failed-to-download --local-dir ./another-single-file-debug --token xxxxxxx

This successfully downloaded that file to the local. The log is as follows:

Error while checking for CLI update.
Traceback (most recent call last):
  File "/home/xxxx/mambaforge/envs/webarena/lib/python3.10/site-packages/huggingface_hub/cli/_cli_utils.py", line 127, in check_cli_update
    _check_cli_update()
  File "/home/xxxx/mambaforge/envs/webarena/lib/python3.10/site-packages/huggingface_hub/cli/_cli_utils.py", line 147, in _check_cli_update
    Path(constants.CHECK_FOR_UPDATE_DONE_PATH).touch()
  File "/home/xxxx/mambaforge/envs/webarena/lib/python3.10/pathlib.py", line 1168, in touch
    self._accessor.touch(self, mode, exist_ok)
  File "/home/xxxx/mambaforge/envs/webarena/lib/python3.10/pathlib.py", line 331, in touch
    fd = os.open(path, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/home/xxxx/.cache/huggingface/.check_for_update_done'
Request c32c2fd4-31ad-4b37-9230-d813a5e63854: HEAD https://huggingface.co/xxxx/xxxrepo/resolve/main/ICRL/results/tot/AMC-TTT_Qwen/Qwen2.5-Math-7B_20251116_065747_781460/metrics.json (authenticated: True)
Send: curl -X HEAD -H 'accept: */*' -H 'accept-encoding: identity' -H 'authorization: <TOKEN>' -H 'connection: keep-alive' -H 'host: huggingface.co' -H 'user-agent: huggingface-cli/None; hf_hub/1.1.5; python/3.10.18; torch/2.8.0+cpu' -H 'x-amzn-trace-id: c32c2fd4-31ad-4b37-9230-d813a5e63854' -d https://huggingface.co/xxxx/xxxrepo/resolve/main/ICRL/results/tot/AMC-TTT_Qwen/Qwen2.5-Math-7B_20251116_065747_781460/metrics.json
Downloading 'ICRL/results/tot/AMC-TTT_Qwen/Qwen2.5-Math-7B_20251116_065747_781460/metrics.json' to 'another-single-file-debug/.cache/huggingface/download/ICRL/results/tot/AMC-TTT_Qwen/Qwen2.5-Math-7B_20251116_065747_781460/dwv9JJd8GoYAm6ZCo6wNY6Qwsw0=.cb4b19e8942a92e5b0f43629944917046f4a0520.incomplete'
Request 56db41ed-a168-4e2a-80d1-ebe04842132a: GET https://huggingface.co/xxxx/xxxrepo/resolve/main/ICRL/results/tot/AMC-TTT_Qwen/Qwen2.5-Math-7B_20251116_065747_781460/metrics.json (authenticated: True)
Send: curl -X GET -H 'accept: */*' -H 'accept-encoding: gzip, deflate' -H 'authorization: <TOKEN>' -H 'connection: keep-alive' -H 'host: huggingface.co' -H 'user-agent: huggingface-cli/None; hf_hub/1.1.5; python/3.10.18; torch/2.8.0+cpu' -H 'x-amzn-trace-id: 56db41ed-a168-4e2a-80d1-ebe04842132a' -d https://huggingface.co/xxxx/xxxrepo/resolve/main/ICRL/results/tot/AMC-TTT_Qwen/Qwen2.5-Math-7B_20251116_065747_781460/metrics.json
metrics.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 366/366 [00:00<00:00, 1.48MB/s]
Download complete. Moving file to another-single-file-debug/ICRL/results/tot/AMC-TTT_Qwen/Qwen2.5-Math-7B_20251116_065747_781460/metrics.json
another-single-file-debug/ICRL/results/tot/AMC-TTT_Qwen/Qwen2.5-Math-7B_20251116_065747_781460/metrics.json

Compare the API's file counts

For the provided python code, I noticed that you did not specify the subdirectory path when calling list_repo_tree method. Thus, I slightly modified the code with the path ICRL/results/tot (please do let me know if this not your intention):

from huggingface_hub import list_repo_tree, model_info
from huggingface_hub.hf_api import RepoFile

repo_id = "xxxxx/a-private-repo"

info = model_info(repo_id, files_metadata=False)
siblings = len(info.siblings or [])
print("siblings (model_info):", siblings)

tree_count = sum(1 for obj in list_repo_tree(repo_id, path_in_repo="ICRL/results/tot", recursive=True) if isinstance(obj, RepoFile))
print("files (list_repo_tree):", tree_count)

The code outputs: files (list_repo_tree): 5345.

---

For comparison, I run the following command to count the files under the same path ICRL/results/tot:

hf download xxxxx/a-private-repo --token xxxxxx --include 'ICRL/results/tot/*' --dry-run

The command outputs: Will download 735 files (out of 735), which DOES NOT match the count through python code.

For each run whatever the command line or the python code, I did clean the cache by rm -rf ~/.cache/huggingface and enabled HF_DEBUG=1.

---

I wonder if it is possible to connect the team via email or slack, thus I could provide the name of this private repo, and also may provide a token for access to that repo. I just don't want to expose it in the public github.