Add user guide documentation
ober
aa296cafedb2bdc02fc31bbe9988a94ffbc84986
--- a/README.md +++ b/README.md @@ -6,6 +6,13 @@ This project is intended to become a single deployable Jerboa application for Proton Drive, with YubiKey-gated access and no rclone or Go helper in the runtime path. +## User Documentation + +Start with [docs/user-guide.md](docs/user-guide.md) for installation, local +state layout, authentication, encrypted credential vault setup, listing remote +files, reading and writing Drive content, FUSE mounts, configuration variables, +and troubleshooting. + ## Runtime Status Current runtime code is Scheme/Jerboa only: new file mode 100644 --- /dev/null +++ b/docs/user-guide.md @@ -0,0 +1,754 @@ +# jerboa-protonstorage User Guide + +This guide covers day-to-day use of `jerboa-protonstorage`: how to run it, +where it stores local state, how to authenticate, how to list/read/write Proton +Drive data, and how to mount the Drive tree through FUSE. + +The project is a Jerboa-native Proton Drive client. It does not use rclone or a +Go helper at runtime. + +## Current Scope + +Implemented today: + +- Native Proton SRP login. +- FIDO2/YubiKey challenge flow when Proton requests it. +- TOTP fallback for accounts that use TOTP. +- Decrypted root listing and recursive decrypted tree listing. +- Plaintext file reads with Proton block hash verification. +- Create folder, upload/replace file, trash, rename, and move operations. +- Read-only and writable FUSE mounts through `jerboa-fuse`. +- Local profile directories under `~/.jproton`. +- Optional encrypted Proton username/password vault protected by a separate + password you provide. +- Host binary build, host bundle build, and Linux/FreeBSD cross-build targets. + +Important current limitations: + +- `drive daemon start` currently runs in the foreground. It is suitable for + supervision by a terminal, launchd, systemd, or another service manager, but + the background control socket is still planned work. +- Cache directories exist and are reported, but durable cache indexing, + pinning, eviction, and sync queues are still planned work. +- Token-only auth can call raw Proton API metadata endpoints. Decrypted Drive + listing, reads, writes, and mounts need fresh login key material, so use + `--username`, stored credentials, or password environment variables. + +## Install Layout + +Expected source layout: + +```text +~/mine/jerboa-protonstorage/ +~/mine/jerboa/ +~/mine/jerboa-proton-bridge/ +~/mine/jerboa-yubikey/ +~/mine/jerboa-pgp/ +~/mine/jerboa-crypto/ +~/mine/jerboa-fuse/ +~/mine/jerboa-mail/ +~/mine/jerboa-https/ +~/mine/jerboa-ssl/ +``` + +The `Makefile` defaults point at those sibling directories. Override them with +environment variables if your checkout lives elsewhere: + +```sh +JERBOA_HOME=/path/to/jerboa make test +JERBOA_FUSE_DIR=/path/to/jerboa-fuse make binary +``` + +## Running From Source + +Use `make run` during development: + +```sh +make run ARGS='help' +make run ARGS='doctor' +make run ARGS='drive status' +``` + +Equivalent direct form: + +```sh +./bin/protonstorage help +./bin/protonstorage drive status +``` + +`make` itself is intentionally help-only. It does not build or mutate files. + +## Building + +Build for the current host: + +```sh +make binary +./protonstorage-bin help +``` + +Build a deployable host bundle: + +```sh +make bundle +tar -tzf dist/protonstorage-$(uname -s | tr A-Z a-z)-$(uname -m).tar.gz +``` + +The bundle contains a wrapper named `protonstorage` that sets runtime library +paths for the included native Jerboa support libraries. + +Cross-build targets: + +```sh +make linux-amd64 +make linux-arm64 +make freebsd-amd64 +``` + +These use the local Jerboa/Chez cross-build setup and local cross C toolchains. +The project does not use Docker or Podman for builds. + +## Local State + +By default, local state lives under: + +```text +~/.jproton/ +``` + +Each profile gets its own directory: + +```text +~/.jproton/profiles/<profile>/ + credentials.vault.json + metadata.db + op-log/ + cache/ + staging/ + locks/ + logs/ +``` + +The current code creates and reports these paths. `metadata.db`, indexed cache +state, and durable operation queues are planned persistence layers. + +The default profile name is `default`. + +Inspect the default profile: + +```sh +make run ARGS='drive cache status --profile default' +make run ARGS='drive daemon status --profile default' +``` + +Create the local profile layout: + +```sh +make run ARGS='drive cache init --profile default' +``` + +Use another profile: + +```sh +make run ARGS='drive cache init --profile work' +make run ARGS='drive credentials store --profile work' +``` + +Use another state root: + +```sh +make run ARGS='drive cache status --state-dir /secure/jproton --profile default' +``` + +Equivalent environment variables: + +```sh +export PROTONSTORAGE_STATE_DIR=/secure/jproton +export PROTONSTORAGE_PROFILE=work +``` + +`PROTONDRIVE_STATE_DIR` and `PROTONDRIVE_PROFILE` are also accepted. + +Profile names may contain only letters, digits, `.`, `-`, and `_`. + +## Authentication Options + +There are three practical authentication modes. + +### Fresh Interactive Login + +Use this for normal encrypted Drive work: + +```sh +make run ARGS='drive login --username you@example.com' +make run ARGS='drive root-children-decrypted --username you@example.com' +``` + +The command prompts for the Proton password if no password environment variable +is set. If Proton requests a FIDO2 security-key assertion, touch your YubiKey +when prompted. + +For accounts with a separate mailbox password: + +```sh +export PROTONDRIVE_MAILBOX_PASSWORD='...' +make run ARGS='drive root-children-decrypted --username you@example.com' +``` + +### Fresh Login With Environment Variables + +This avoids typing the Proton password into the prompt: + +```sh +export PROTONDRIVE_USERNAME='you@example.com' +export PROTONDRIVE_PASSWORD='your-proton-password' +make run ARGS='drive tree-decrypted --max-depth 2' +``` + +Or read the password from a named environment variable: + +```sh +export MY_PROTON_PASSWORD='your-proton-password' +make run ARGS='drive tree-decrypted --username you@example.com --password-env MY_PROTON_PASSWORD' +``` + +FIDO2 PIN options: + +```sh +export PROTONDRIVE_FIDO2_PIN='123456' +make run ARGS='drive login --username you@example.com' +``` + +or: + +```sh +make run ARGS='drive login --username you@example.com --prompt-pin' +``` + +TOTP fallback: + +```sh +make run ARGS='drive login --username you@example.com --totp-code 123456' +``` + +or: + +```sh +export PROTONDRIVE_TOTP_CODE='123456' +make run ARGS='drive login --username you@example.com' +``` + +### Token-Only API Calls + +Raw metadata endpoints can use existing Proton session material: + +```sh +export PROTONDRIVE_UID='...' +export PROTONDRIVE_ACCESS_TOKEN='...' +export PROTONDRIVE_REFRESH_TOKEN='...' +make run ARGS='drive user' +make run ARGS='drive volumes' +make run ARGS='drive shares' +``` + +Token-only auth does not provide mailbox key material. Commands that decrypt +Drive names or content will reject token-only auth and ask for fresh login key +material. + +## Encrypted Credential Vault + +The credential vault stores your Proton username and Proton password in the +selected local profile. The vault is encrypted with `scrypt` and +`chacha20-poly1305`, protected by a separate vault password you provide. + +Default vault path: + +```text +~/.jproton/profiles/default/credentials.vault.json +``` + +Store credentials interactively: + +```sh +make run ARGS='drive credentials store --profile default' +``` + +Check whether a vault exists: + +```sh +make run ARGS='drive credentials status --profile default' +``` + +Verify that the vault password can unlock the vault. This prints the username +and `Unlocked:true`; it never prints the Proton password: + +```sh +make run ARGS='drive credentials unlock-test --profile default' +``` + +Use stored credentials: + +```sh +make run ARGS='drive root-children-decrypted --profile default --use-stored-credentials' +``` + +Noninteractive vault unlock: + +```sh +export PROTONSTORAGE_VAULT_PASSWORD='your-vault-password' +make run ARGS='drive tree-decrypted --profile default --use-stored-credentials --max-depth 2' +``` + +Or choose the vault password variable explicitly: + +```sh +export MY_VAULT_PASSWORD='your-vault-password' +make run ARGS='drive tree-decrypted --use-stored-credentials --vault-password-env MY_VAULT_PASSWORD' +``` + +Store credentials noninteractively: + +```sh +export PROTONDRIVE_USERNAME='you@example.com' +export PROTONDRIVE_PASSWORD='your-proton-password' +export PROTONSTORAGE_VAULT_PASSWORD='your-vault-password' +make run ARGS='drive credentials store --profile default' +``` + +Delete the vault: + +```sh +make run ARGS='drive credentials clear --profile default' +``` + +Security notes: + +- The vault stores the Proton login password, not an OAuth token. +- The vault password is not stored. +- The vault password is still required every time stored credentials are used. +- Store `~/.jproton` on an encrypted disk if your threat model includes local + disk access while the machine is powered off. + +## Listing Remote Files + +List root children with decrypted names: + +```sh +make run ARGS='drive root-children-decrypted --username you@example.com' +``` + +List recursively: + +```sh +make run ARGS='drive tree-decrypted --username you@example.com --max-depth 3' +``` + +Using stored credentials: + +```sh +export PROTONSTORAGE_VAULT_PASSWORD='your-vault-password' +make run ARGS='drive tree-decrypted --profile default --use-stored-credentials --max-depth 3' +``` + +`--max-depth -1` means unlimited depth. The default for recursive listing and +mount construction is unlimited unless you pass a different value. + +Discover active Drive root metadata: + +```sh +make run ARGS='drive discover --username you@example.com' +``` + +Raw API listing by share and link ID: + +```sh +make run ARGS='drive children --share SHARE_ID --link LINK_ID --uid UID --access-token TOKEN' +``` + +The raw `children` command returns Proton JSON and does not decrypt names by +itself. + +## Reading Files + +Read plaintext bytes for a file link ID: + +```sh +make run ARGS='drive read-file --username you@example.com --link FILE_LINK_ID --offset 0 --size 4096' > chunk.bin +``` + +Read from stored credentials: + +```sh +export PROTONSTORAGE_VAULT_PASSWORD='your-vault-password' +make run ARGS='drive read-file --use-stored-credentials --link FILE_LINK_ID' > file.bin +``` + +`--size -1` reads the whole file. `--offset 0` starts at the beginning. + +The read path resolves the active root, unlocks Drive keys, verifies encrypted +block hashes, decrypts content session keys, and writes plaintext bytes to +stdout. Redirect stdout when reading binary files. + +## Writing Files And Folders + +Write commands require fresh login key material. + +Create a folder: + +```sh +make run ARGS='drive create-folder --username you@example.com --parent-path / --name docs' +``` + +Upload a file: + +```sh +make run ARGS='drive upload-file --username you@example.com --parent-path /docs --path ./notes.txt --mime-type text/plain' +``` + +Override the remote name: + +```sh +make run ARGS='drive upload-file --username you@example.com --parent-path /docs --path ./notes.txt --name remote-notes.txt' +``` + +Rename a child: + +```sh +make run ARGS='drive rename-child --username you@example.com --parent-path /docs --name remote-notes.txt --new-name notes-renamed.txt' +``` + +Move a child: + +```sh +make run ARGS='drive move-child --username you@example.com --parent-path /docs --name notes-renamed.txt --new-parent-path /archive' +``` + +Move and rename: + +```sh +make run ARGS='drive move-child --username you@example.com --parent-path /docs --name notes.txt --new-parent-path /archive --new-name notes-archived.txt' +``` + +Trash a child: + +```sh +make run ARGS='drive trash-child --username you@example.com --parent-path /archive --name notes-archived.txt' +``` + +Root-folder aliases: + +```sh +make run ARGS='drive create-root-folder --username you@example.com --name docs' +make run ARGS='drive upload-root-file --username you@example.com --path ./notes.txt' +``` + +Upload behavior: + +- Default MIME type is `application/octet-stream`. +- Default block size is `4194304` bytes. +- Default modification time is the current UTC time in Proton format. +- If an unfinished draft revision already exists, upload stops by default. + Pass `--replace-existing-draft` to delete that draft and create a new one. + +## FUSE Mounts + +Create a mountpoint: + +```sh +mkdir -p /tmp/proton-drive +``` + +Read-only mount: + +```sh +make run ARGS='drive mount --username you@example.com --mountpoint /tmp/proton-drive --read-only' +``` + +Writable mount: + +```sh +make run ARGS='drive mount --username you@example.com --mountpoint /tmp/proton-drive --writable' +``` + +Legacy aliases: + +```sh +make run ARGS='drive mount-root --username you@example.com --mountpoint /tmp/proton-drive' +make run ARGS='drive mount-root-write --username you@example.com --mountpoint /tmp/proton-drive' +``` + +Using stored credentials: + +```sh +export PROTONSTORAGE_VAULT_PASSWORD='your-vault-password' +make run ARGS='drive mount --profile default --use-stored-credentials --mountpoint /tmp/proton-drive --writable' +``` + +The command remains attached. Press `Ctrl-C` to unmount through the normal FUSE +shutdown path. + +Current writable FUSE behavior: + +- `mkdir` creates Proton Drive folders. +- Create/write buffers file data and commits it through the encrypted upload + path when the handle flushes or releases. +- `unlink` and empty `rmdir` move entries through Proton trash/remove paths. +- Rename in the same folder and cross-folder move use Proton move semantics. +- Cross-folder move re-encrypts the moved node passphrase for the destination + parent key. + +Mount status: + +```sh +make run ARGS='drive daemon status --profile default' +``` + +Stop marker state: + +```sh +make run ARGS='drive daemon stop --profile default --force' +``` + +`drive daemon stop` currently clears local marker files only. Use `Ctrl-C`, +`umount`, or your OS FUSE unmount command for a running foreground mount. + +Foreground daemon command: + +```sh +make run ARGS='drive daemon start --username you@example.com --mountpoint /tmp/proton-drive --writable --foreground' +``` + +Without `--foreground`, this still runs in the foreground and prints a warning. + +## Raw Proton API Commands + +These are useful for debugging endpoint behavior: + +```sh +make run ARGS='drive user' +make run ARGS='drive salts' +make run ARGS='drive addresses' +make run ARGS='drive address --address ADDRESS_ID' +make run ARGS='drive volumes' +make run ARGS='drive shares' +make run ARGS='drive link --share SHARE_ID --link LINK_ID' +make run ARGS='drive children --share SHARE_ID --link LINK_ID' +make run ARGS='drive revisions --share SHARE_ID --link FILE_LINK_ID' +make run ARGS='drive revision --share SHARE_ID --link FILE_LINK_ID --revision REVISION_ID' +make run ARGS='drive check-hash --share SHARE_ID --link FOLDER_LINK_ID --hash NAME_HASH' +``` + +These commands accept token-only auth via `PROTONDRIVE_UID` and +`PROTONDRIVE_ACCESS_TOKEN`, or fresh login options. + +## Configuration Reference + +Command options: + +```text +--username USER +--password-env ENV +--mailbox-password-env ENV +--pin-env ENV +--prompt-pin +--totp-code CODE +--uid UID +--access-token TOKEN +--refresh-token TOKEN +--base-url URL +--profile NAME +--state-dir PATH +--vault-password-env ENV +--use-stored-credentials +--mountpoint PATH +--writable +--read-only +--foreground +--force +--max-depth N +--offset N +--size N +--share SHARE_ID +--link LINK_ID +--address ADDRESS_ID +--revision REVISION_ID +--from-block N +--page-size N +--hash HASH +--path PATH +--parent-path PATH +--name NAME +--new-name NAME +--new-parent-path PATH +--mime-type TYPE +--modification-time ISO +--block-size N +--replace-existing-draft +``` + +Environment variables: + +```text +PROTONDRIVE_USERNAME +PROTON_USERNAME +PROTONDRIVE_PASSWORD +PROTON_PASSWORD +PROTONDRIVE_MAILBOX_PASSWORD +PROTON_MAILBOX_PASSWORD +PROTONDRIVE_FIDO2_PIN +PROTON_FIDO2_PIN +PROTONDRIVE_TOTP_CODE +PROTON_TOTP_CODE +PROTONDRIVE_UID +PROTON_UID +PROTONDRIVE_ACCESS_TOKEN +PROTON_ACCESS_TOKEN +PROTONDRIVE_REFRESH_TOKEN +PROTON_REFRESH_TOKEN +PROTONDRIVE_API_BASE_URL +PROTONSTORAGE_PROFILE +PROTONDRIVE_PROFILE +PROTONSTORAGE_STATE_DIR +PROTONDRIVE_STATE_DIR +PROTONSTORAGE_VAULT_PASSWORD +PROTONDRIVE_VAULT_PASSWORD +PROTONSTORAGE_USE_STORED_CREDENTIALS +PROTONDRIVE_USE_STORED_CREDENTIALS +PROTONDRIVE_MOUNTPOINT +PROTONDRIVE_PARENT_PATH +PROTONDRIVE_NEW_PARENT_PATH +PROTONDRIVE_NAME +PROTONDRIVE_NEW_NAME +PROTONDRIVE_UPLOAD_PATH +PROTONDRIVE_SHARE_ID +PROTONDRIVE_LINK_ID +PROTONDRIVE_ADDRESS_ID +PROTONDRIVE_REVISION_ID +``` + +Default API base URL: + +```text +https://drive-api.proton.me +``` + +Override only for testing or development: + +```sh +make run ARGS='drive status --base-url https://drive-api.proton.me' +export PROTONDRIVE_API_BASE_URL='https://drive-api.proton.me' +``` + +## Integration Test + +Smoke tests: + +```sh +make test +``` + +Live integration test is opt-in: + +```sh +export PROTONDRIVE_INTEGRATION=1 +export PROTONDRIVE_USERNAME='you@example.com' +export PROTONDRIVE_PASSWORD='your-proton-password' +make integration +``` + +Optional integration settings: + +```sh +export PROTONDRIVE_TEST_PARENT_PATH='/' +export PROTONDRIVE_FIDO2_PIN='123456' +export PROTONDRIVE_TOTP_CODE='123456' +``` + +The integration test creates a disposable folder, uploads a small file, moves it +between disposable folders, and trashes the disposable root during cleanup. + +## Troubleshooting + +Show implementation and native backend status: + +```sh +make run ARGS='doctor' +``` + +Show Drive API capability status: + +```sh +make run ARGS='drive status' +``` + +Check local profile paths: + +```sh +make run ARGS='drive cache status --profile default' +``` + +Common errors: + +```text +Drive login requires --username or PROTONDRIVE_USERNAME +``` + +Pass `--username` or set `PROTONDRIVE_USERNAME`. + +```text +command requires fresh login key material; use --username instead of token-only auth +``` + +The command needs mailbox key material. Use fresh login or stored credentials. + +```text +credentials vault does not exist +``` + +Run `drive credentials store --profile <profile>` first, or use the profile +that already contains the vault. + +```text +credential vault passwords did not match +``` + +The interactive vault confirmation failed. Re-run `drive credentials store`. + +```text +file link not found in decrypted tree +``` + +Increase `--max-depth`, verify the link ID, or list the tree first. + +For FUSE unmount issues, use the platform unmount command for the mountpoint: + +```sh +umount /tmp/proton-drive +``` + +On macOS with macFUSE, the command may be: + +```sh +diskutil unmount /tmp/proton-drive +``` + +## Deployment Notes + +For a single-machine deployment, build a bundle and move the generated +`dist/protonstorage-<os>-<arch>.tar.gz` to the target machine: + +```sh +make bundle +``` + +On the target: + +```sh +tar -xzf protonstorage-<os>-<arch>.tar.gz +./protonstorage-<os>-<arch>/protonstorage help +``` + +Keep `~/.jproton` private. The credential vault is encrypted, but the profile +directory also contains operational state such as cache, staging, logs, and +lock files as those layers are completed. +