Currently Building · Go
HashVault
Content-addressable cloud storage. A Dropbox-like storage platform in Go that stores identical bytes exactly once. Files are addressed by their SHA-256 hash, uploads go browser-to-S3 on presigned URLs, and the Go server never touches the data plane.
Filed June 2026 — ongoing · AnkitSinha0/HashVault ↗
# 1 · client hashes the file, asks to upload POST /files/init-upload { "name": "thesis.pdf", "size": 2481152, "checksum": "9f2c…e41a" } 200 { "deduplicated": true } # same bytes already stored → no presigned URL, no upload POST /files/confirm 201 { "object": "objects/9f2c…e41a", "ref_count": 3 }
Overview
HashVault is a production-grade cloud storage platform built as a modular monolith: one Go binary, organised into repository, service and handler layers behind interfaces, so any module can be extracted into its own service later without rewriting it.
Users get folders, uploads, downloads, share links and a 10 GB quota. Underneath, every file is a reference to a content-addressed storage object, which is what makes deduplication possible.
The Problem
Two users upload the same file. Most systems store it twice, and pay for it twice. At scale, duplicate bytes are a real share of storage cost.
The second problem is bandwidth. If every byte passes through the API server, the server becomes the bottleneck and the most expensive part of the upload path.
HashVault
Cloud storage platform · content-addressable deduplication · Go / Gin / PostgreSQL / Redis / RabbitMQ / S3
Every upload is hashed with SHA-256 before a byte leaves the client. Identical content resolves to one S3 key; reference counts increment and decrement atomically in SQL, and the object is purged only when the last reference drops to zero.
A two-token scheme pairs 15-minute HS256 access tokens with 64-character opaque refresh tokens, stored only as SHA-256 hashes in Redis. The prior key is deleted on every refresh — a stolen token cannot be replayed.
Presigned S3 URLs move file bytes browser-to-S3 directly. The Go server never touches the data plane.
Key Features
- Content-addressable deduplication
- The client supplies a SHA-256 checksum. A dedup hit skips the upload entirely; the file row simply points at the existing object.
- Presigned uploads
- The API issues a presigned PUT URL. File bytes travel browser → S3 directly; the server stays on the control plane.
- Atomic reference counting
- Each storage object carries a ref count, incremented and decremented in SQL. The object is purged from S3 and Postgres only when it reaches zero.
- Rotating refresh tokens
- 15-minute HS256 access tokens paired with 64-character opaque refresh tokens, stored only as SHA-256 hashes in Redis and rotated on every refresh.
- Google OAuth
- OAuth2 sign-in with the CSRF state held in Redis for ten minutes.
- Async email
- Welcome and OTP emails go through a durable RabbitMQ topic exchange to a worker with prefetch 1.
Technical Decisions
- 01Modular monolith, not microservices
- One deployable keeps local development and transactions simple; clean interfaces keep the door open to extraction.
- 02Objects keyed by checksum
- S3 keys are objects/{sha256}, so re-uploading identical bytes always resolves to the same key.
- 03Sentinel errors at the boundary
- Repositories translate gorm.ErrRecordNotFound into their own ErrNotFound, so persistence details never leak into business logic.
- 04Quota checked twice
- Storage quota is enforced at both init-upload and confirm, so a slow upload can't slip past a limit that changed meanwhile.
Reliability
- Redis degrades deliberately: fail-closed for refresh-token validation, fail-open for rate limiting.
- PostgreSQL connection pool capped at 25 connections.
- 30-second graceful shutdown on SIGTERM drains in-flight requests.
- Message publishing is fire-and-forget; a bad payload is nacked without requeue, headed for a dead-letter queue.
Lessons
Deduplication is a concurrency problem before it is a storage problem: fifty clients confirming the same checksum at once must produce one object with a ref count of fifty. That is why the next phase is an integration test suite against a real Postgres, not mocks.
Keeping the server off the data plane changes every other decision — quotas, confirmation, and failure handling all move to the edges of the upload.