Spaces:
Paused
Paused
Commit History
-
fix: chown /home/tao-shen after restore (old uid mapping)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>444d40a -
feat: sync /var/log for system log persistence
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>a0dcc38 -
feat: upload huggingrun.log to dataset for debugging
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>d111a78 -
feat: tao-shen user with sudo, hostname=huggingrun, ttyd login
- Create tao-shen user (sudo NOPASSWD) instead of generic 'user' - Set hostname to 'huggingrun' for clean prompt - ttyd uses 'login -f tao-shen' for proper login shell - Copy root's authorized_keys to tao-shen on restore Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>ca96613 -
feat: sync /etc with proper excludes + pip package persistence
- Restore /etc/ sync with relative-path excludes for container-managed files (hostname, hosts, resolv.conf, alternatives, ld.so.cache) - Add pip freeze/install cycle for pip package persistence - Upload pip-packages.txt alongside user-packages.list Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>0cdf7b0 -
fix: chmod SSH host keys after restore (HF loses permissions)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>03209c8 -
fix: use Popen for sshd restart, match original start_sshd() pattern
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>9a274e2 -
fix: use apt --reinstall to handle restored dpkg state
Restoring /var/lib/dpkg/ makes apt think packages are installed when the actual binaries are not synced. Use --reinstall to force. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>b56f91e -
fix: sync /etc/ssh only, not all of /etc/
Restoring all of /etc/ was overwriting system configs (PAM, nsswitch, alternatives) which broke sshd after restart. Now only sync /etc/ssh/ for SSH host keys. Also restart sshd after restore to pick up restored host keys, and add error logging to rsync. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>0db5b04 -
v2: complete rewrite — sync only user data, not system dirs
Sync scope: /home, /root, /usr/local, /opt, /var/lib, /etc - System binaries (/usr/bin, /usr/lib, etc.) NOT synced - apt packages saved as name list, restored via apt install - No symlink/permission issues (no /bin, /etc/alternatives) - ~500 files instead of 20000+, all uploads succeed Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>269f150 -
fix: exclude /etc/alternatives from restore (symlinks not preserved by HF)
Root cause: /bin/sh → /usr/bin/sh → /etc/alternatives/sh → /usr/bin/dash. HF datasets don't preserve Unix symlinks, so /etc/alternatives/sh gets restored as a regular file without execute permission, breaking the entire /bin/sh symlink chain with "Permission denied: /bin/sh". Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>4314e7c -
fix: replace hanging upload_large_folder with per-dir upload + rate limiting
upload_large_folder hangs after ~50 commits (known issue). Reverted to per-directory upload_folder approach with: - Split dirs >3000 files into sub-dirs recursively - 3s delay between uploads (rate limiting) - Retry with backoff (3 attempts, 5s/10s/15s) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>5c0994f -
fix: use upload_large_folder for entire /data/ directory
upload_large_folder doesn't support path_in_repo — must upload from the root folder. Simplified to single upload_large_folder call for all of /data/ instead of per-directory uploads. It handles chunking, hash-based dedup, retries, and multi-commit internally. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>b97023a -
fix: remove --delete from restore rsync to prevent /bin/sh deletion
Root cause: rsync --delete /data/ → / was deleting /bin/sh from the container because it wasn't in the dataset (uploads failed or never completed for system dirs). --delete removes destination files not present in the source. - Restore rsync: no --delete (only add/update, never remove system files) - Save rsync: keeps --delete (dataset should be exact mirror of /) - Also exclude huggingrun.log from restore to prevent old log overwrite Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>63934c3 -
fix: skip system dirs during restore to prevent /bin/sh permission loss
HF datasets don't preserve Unix execute permissions. Restoring /bin/sh from the dataset removes +x, breaking all shell=True subprocess calls. - Save: still syncs full disk to dataset (user's full mirror requirement) - Restore: skips /bin, /sbin, /lib, /usr/bin, /usr/lib, /usr/share etc. (system packages reinstalled via user-packages.list + apt install) - Only user-modifiable dirs (/home, /root, /etc, /opt, /var, /usr/local) are restored from dataset Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>8bddfd6 -
fix: use upload_large_folder for big dirs + retry with backoff
- Directories >2000 files now use upload_large_folder (handles chunking internally) instead of recursively splitting into sub-dir upload_folder calls - Added retry with exponential backoff (3 attempts, 5s/10s/15s delays) - Added 2s rate limit between top-level directory uploads - Eliminates the rapid-fire API calls that overwhelmed HF API Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>9f98401 -
fix: sync thread waits for restore to finish before starting
Prevent race condition where sync rsync writes to /data/ while snapshot_download is still populating it. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>54cbddc -
fix: start nginx first, restore in background
snapshot_download of 7000+ files takes too long before port 7860 opens, causing HF startup timeout (RUNTIME_ERROR). Fix: start all services + nginx immediately, then run restore (download + rsync) in a background thread. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>80c4d9b -
fix: auto-split large dirs + exclude huggingrun.env from upload
- Dirs with >2000 files are recursively split into sub-dirs (avoids 504 timeout on usr/ which has 10K+ files) - Exclude huggingrun.env (contains HF_TOKEN, rejected by secret scanner) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>980afb9 -
fix: upload per top-level directory to avoid 504 timeout
20K files in one commit causes HF API 504. Split into separate commits per top-level dir (bin/, etc/, home/, usr/, ...). Each dir is a manageable upload_folder call. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>7f87cb0 -
feat: direct file sync — dataset root = filesystem root
No more tar.zst. Dataset mirrors the container's / directly: - SAVE: rsync / → /data/ → upload_folder to dataset - RESTORE: snapshot_download → rsync /data/ → / Dataset browsable: bin/, etc/, home/, usr/, etc. match the OS. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>501b164 -
feat: full-disk persistence — tar entire / filesystem every 60s
Sync everything on disk except virtual (/proc, /sys, /dev), our persist path (/data), and temp dirs (/tmp, /run). Interval changed from 120s to 60s, first sync after 30s. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>20de61f -
feat: full filesystem persistence (not just /home + /root)
Persist all user-modifiable dirs: /home, /root, /etc, /opt, /var, /usr/local — with tar -p preserving all permissions. Exclude Docker-managed files (hostname, hosts, resolv.conf), transient runtime (var/run, var/lock, *.sock, *.pid), and caches. Dataset = disk. Container state is fully mirrored. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>acafb8d -
fix: run nginx as subprocess (not exec) to keep sync threads alive
os.execvp replaces the Python process, killing all daemon threads (sync, heartbeat, log streamer). Run nginx as Popen instead and wait for it, keeping Python as PID 1 with threads running. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>d09e48b -
fix: split Dockerfile apt-get calls (match proven build pattern)
Use openssl passwd instead of chpasswd, split apt-get into two separate calls matching the old Dockerfile that built successfully. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>8e73dda -
v2: complete redesign — single entrypoint, tar.zst persistence
Replace 13 scripts with 3 files (+ Dockerfile): - entrypoint.py: single Python PID 1 (init, sync, services, log streamer) - nginx.conf: reverse proxy (7860 → ttyd, ssh, logs) - ws_ssh_bridge.py: WebSocket-to-SSH bridge (unchanged) Key changes: - Persistence: tar.zst of /home + /root via upload_file (not folder sync) - No more rsync permission corruption (tar -p preserves modes) - No more .cache path rejections (single archive file) - No more upload_large_folder hangs (upload_file is fast) - Log streamer runs as thread (not separate process) - Sync runs as thread (not separate daemon) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>52013b5 -
fix: remove .cache dirs from rootfs before upload (HF API rejects them)
The .cache dirs already existed in rootfs from previous syncs. rsync --exclude only prevents NEW .cache from being synced but doesn't delete existing ones. Clean up before upload. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>7098b1b -
fix: exclude .cache from rsync save (HF API rejects .cache/ paths)
upload_folder fails with: cannot update files under a '.cache/' folder. Add .cache to SAVE_EXCLUDES so these paths never reach the dataset. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>9f8f146 -
fix: restore original logging pattern + fix sync hanging
Root cause analysis: 1. sync daemon log() changed from stderr to stdout (broke HF log capture) 2. stdbuf -oL -eL removed from RUN_CMD (broke line buffering) 3. start-server.sh used exec > >(tee >&2) instead of direct echo >&2 4. upload_large_folder hangs for hours on 1.8GB rootfs Fixes: - Restore log() to use file=sys.stderr (original design from ea21d45) - Restore stdbuf -oL -eL in Dockerfile RUN_CMD - Simplify start-server.sh to direct echo >&2 + logfile write - Switch from upload_large_folder to upload_folder (doesn't hang) - Restore exec nginx (original PID 1 = nginx, not bash+wait) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>ea03e2a
-
fix: replace awk with cut/grep (awk permission denied) + fix restore permissions
- Replace awk usage in start-server.sh with cut/grep/sed (awk loses execute permission after dataset restore) - Fix restore_system() to use F755 for /usr/local and /opt dirs (these contain executables that need execute permission) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>bfc7c5f -
fix: output logs to stderr (HF captures stderr for Docker SDK spaces)
Research shows HF Spaces may only capture stderr, not stdout, for Docker SDK spaces. Route entrypoint.sh log() to stderr directly. Route start-server.sh through tee to both logfile and stderr. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>291e860 -
fix: remove /proc/1/fd/1 redirect (infinite loop when entrypoint IS PID 1)
Use simple `exec > >(tee -a logfile) 2>&1` instead. Since entrypoint.sh is PID 1, its stdout IS already Docker's log pipe. Redirecting stdout to /proc/1/fd/1 created an infinite loop causing RUNTIME_ERROR. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>8d64fb2 -
feat: redirect all output through /proc/1/fd/1 for HF log capture
Route entrypoint.sh and start-server.sh output through Docker's PID 1 stdout pipe via process substitution. Update git_sync_daemon.py to also write to /proc/1/fd/1 directly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>01c6e53 -
fix: move ALL app scripts to /scripts/ (outside PERSIST_DIRS)
Dataset restore overwrites files in /opt/ and /etc/ with stale versions from the dataset. Move all Docker image scripts to /scripts/ which is NOT in PERSIST_DIRS, so they survive restore. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>6adea9c -
fix: use nginx -c /scripts/nginx.conf to bypass /etc/nginx restore
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>7152d8b -
fix: copy nginx.conf to /scripts/ AFTER COPY scripts (dir replace)
COPY scripts /scripts replaces the entire directory, deleting any files previously copied there. Move nginx.conf copy after it. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>bdc0fe0 -
fix: store nginx.conf outside PERSIST_DIRS, restore before nginx start
Dataset restore overwrites /etc/nginx/nginx.conf with stale version. Fix: keep canonical config at /scripts/nginx.conf (not in PERSIST_DIRS), copy it to /etc/nginx/ before starting nginx. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>e081bc5 -
fix: exclude /etc/nginx from persist to prevent config overwrite
The dataset restore was overwriting /etc/nginx/nginx.conf with the old version (containing /logs paths), overriding the Docker image's current config (with /runlog paths). Add /etc/nginx to SAVE_EXCLUDES. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>4233ced -
fix: use /runlog endpoints (HF intercepts /logs and /api/*)
HF's reverse proxy intercepts /logs and /api/* paths. Use /runlog and /runlog/stream instead. Also remove test delay before nginx. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>3dc8487 -
test: add 30s delay before nginx to verify HF SSE during STARTING
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>baf6f79 -
fix: start nginx LAST so startup logs appear in HF SSE
HF's runtime SSE only captures container output during STARTING state (before port 7860 responds). Move nginx from first to last service so all boot info + service status appears in the SSE stream. Also: - Rename /logs → /api/logs (HF intercepts /logs path) - Revert to bash entrypoint (Python wrapper didn't help) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>f18461b -
fix: use Python PID 1 wrapper to capture all output for HF SSE
HF's runtime log SSE captures stdout from PID 1. Gradio (Python) apps work because Python is PID 1. Bash entrypoints don't seem to work. This Python wrapper becomes PID 1, runs entrypoint.sh as a subprocess, and forwards all output to stdout with flush. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>9c66a5d -
diag: remove exec, add 10s stdout ping to test HF SSE capture
Keep PID 1 as original bash (no exec), add frequent stdout ping to diagnose whether HF's runtime log SSE captures stdout. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>faf2454 -
fix: use direct stdout (not stderr) for HF runtime log capture
HF's SSE log API captures stdout only, not stderr. Previous change to stderr broke log visibility. Now: direct echo to stdout (no pipe, no tee) + separate file append. This avoids both: - tee pipe block-buffering - stderr being invisible to HF Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>0c16381 -
fix: switch all log output from tee (pipe-buffered) to direct stderr
HF's runtime log SSE captures stderr unbuffered. Using `echo | tee` creates a pipe that block-buffers when stdout is not a TTY in Docker, preventing real-time delivery to the SSE stream. Switch to: - bash: echo >&2 (direct, unbuffered) + echo >> file - python: print(file=sys.stderr, flush=True) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>59be1b8 -
fix: targeted restore — only sync user dirs, not system dirs
Previous approach (rsync entire rootfs → /) kept breaking /bin/sh because HF dataset corrupts permissions and directory metadata. New approach: only save and restore specific user directories: /home, /root, /etc, /opt, /var, /usr/local System directories (/bin, /sbin, /usr/bin, /usr/lib, etc.) come from Docker image and are never touched. User-installed packages are tracked via dpkg package list for reinstallation on restore. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>9161deb -
fix: use --no-perms --chmod=D755,F644 in rsync restore
Root cause confirmed: HF Dataset/git strips Unix permissions from files. rsync was restoring /usr/ with wrong mode (e.g. 0644 without execute), making /usr/bin/ inaccessible → /bin/sh (symlink to /usr/bin/sh) fails with PermissionError. Fix: use --no-perms --chmod=D755,F644 to force directories=755 and files=644 during restore, regardless of source permissions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>c44ae59 -
fix: remove ACL/xattr flags from rsync, fix permissions after restore
Root cause: HF Dataset (git) doesn't preserve Unix ACLs, xattrs, or file permissions. rsync -AX restores corrupted metadata that breaks /bin/sh (PermissionError: [Errno 13]). Changes: - Use rsync -rlptgoD instead of -aAX (skip ACLs and extended attrs) - After restore, explicitly chmod 755 critical system binaries (/bin/sh, /bin/bash, /usr/bin/python3, /usr/bin/env) using os.chmod() which doesn't require /bin/sh to work Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>0278b97 -
fix: exclude system binaries from rsync to prevent /bin/sh permission crash
HF Dataset (git-based) doesn't preserve Unix file permissions, so executables restored from backup lose their execute bits. /bin/sh gets mode 0644 instead of 0755, causing PermissionError in subprocess. Changes: - Exclude /bin, /sbin, /lib*, /usr/bin, /usr/sbin, /usr/lib from rsync (these come from Docker image build and shouldn't be overwritten) - Exclude Docker-mounted files (/etc/hostname, hosts, resolv.conf) - Add package list save/restore: save dpkg package list to dataset, reinstall user-added packages on restore via apt-get - This preserves user-installed packages without breaking system binaries Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>17404b5 -
fix: restore foreground sync + stdout logging for HF runtime log API
- Restore foreground sync (init runs before port 7860 opens) so HF captures sync progress in its runtime log SSE stream - Switch all log() functions from stderr to stdout via tee (writes to both stdout and logfile simultaneously) - Add startup_duration_timeout: 1h to README.md YAML to prevent HF from timing out during init/restore - Remove diagnostic echo/python test lines from entrypoint.sh - Add scripts/stream_logs.py for JWT-based log streaming - Update scripts/curl_logs.sh with cached token fallback Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>9c623ef