Rohan Verma
ba9ef916bc
"Verify the multipart upload implementation works correctly by running integration tests against MinIO and validating file integrity, parallel uploads, and error handling." Core APIs: - Add InitiateMultipartUpload for starting multipart uploads - Add UploadPart for uploading individual parts (5MB-5GB) - Add CompleteMultipartUpload to finalize uploads - Add AbortMultipartUpload to cancel and cleanup uploads - Add ListParts to query uploaded parts High-Level Features: - Add FileUploadMultipart API with automatic chunking - Add uploadPartWithRetry with exponential backoff (3 retries default) - Add parallel upload worker pool with configurable concurrency - Add progress callbacks for real-time upload tracking - Add GeneratePresignedUploadPartURL for browser-based uploads Advanced Features: - Automatic file chunking with configurable part sizes (5MB-5GB) - Parallel upload worker pool with configurable concurrency - Retry logic with exponential backoff (configurable retries) - Real-time progress tracking with upload speed calculation - Automatic cleanup on errors via AbortMultipartUpload - Support for resumable uploads via ListParts - Comprehensive input validation for all parameters - File integrity verification across all upload modes Testing: - Add comprehensive integration tests (500+ lines) - Test sequential uploads (15MB, 3 parts, ~45 MB/s) - Test parallel uploads (20MB, 4 parts, concurrency=3) - Test small files (3MB, single part) - Validate abort and ListParts functionality - Test input validation and error cases - Verify file integrity with hash comparison - All tests passing against MinIO Documentation: - Update README with multipart upload examples - Add high-level and low-level API usage - Document browser-based multipart uploads - Mark multipart upload implementation as complete in ROADMAP - Update gitignore to exclude *.bin and test-output.log Verified against MinIO with complete test coverage.
6.4 KiB
Project Roadmap
Current: v0.x (Pre-Stable)
Completed Features
- File Upload (PUT via FilePut, POST multipart via FileUpload)
- File Download (GetObject)
- File Delete (DeleteObject)
- File Details (HeadObject)
- List Objects (ListObjectsV2) with pagination
- ListAll iterator (Go 1.23+) for memory-efficient iteration
- Presigned URLs (GET/PUT with expiry)
- Upload policies for browser POST uploads
- IAM Role Support (EC2 instances with IMDSv2)
- IAM token auto-renewal
- Custom Metadata
- Custom Endpoints (MinIO, DigitalOcean Spaces, etc.)
- Content-Disposition and ACL support
- Zero dependencies (stdlib only)
v0.10.2 - Code Organization (Refactor) ✅ COMPLETED
Scope: Internal restructuring, no API changes Size: Medium (refactor ~1064 LOC) Released: Nov 26, 2025 (v0.10.2)
Split simples3.go into logical modules:
simples3.go- Core (S3 struct, New(), config methods: SetEndpoint/SetToken/SetClient)iam.go- IAM (fetchIMDSToken, fetchIAMData, NewUsingIAM, renewIAMToken, SetIAMData)object.go- Object ops (FileUpload, FilePut, FileDownload, FileDelete, FileDetails)list.go- List ops (List, ListAll)helpers.go- Utils (encodePath, detectFileSize, getFirstString, getURL)- Ensure all tests pass after refactor
Why: 1064-line file hard to navigate. Clean foundation before adding features.
v0.11.0 - Bucket & Object Operations ✅ COMPLETED
Scope: Essential bucket management + object manipulation for CLI Size: Medium (5 APIs, ~400 LOC) Released: Nov 26, 2025
Bucket Operations:
- ListBuckets
- CreateBucket (with region support)
- DeleteBucket
Object Operations:
- CopyObject (within/across buckets)
- DeleteObjects (batch delete up to 1000 objects)
Why: These operations unblock CLI development (mv, sync, batch operations). Combined release for efficiency.
Note: DeleteObjects has a known compatibility issue with MinIO that requires further investigation. The implementation follows AWS S3 specification correctly.
v0.12.0 - Multipart Upload ✅ COMPLETED
Scope: Large file support with parallel uploads, progress tracking, and presigned URLs Size: Large (10 APIs/features, ~800 LOC) Released: Nov 27, 2025
Core APIs:
- InitiateMultipartUpload
- UploadPart
- CompleteMultipartUpload
- AbortMultipartUpload
- ListParts
High-Level Features:
- FileUploadMultipart (automatic chunking)
- uploadPartWithRetry (exponential backoff retry logic)
- uploadPartsParallel (concurrent part uploads with worker pool)
- Progress callbacks (real-time upload progress)
- GeneratePresignedUploadPartURL (browser-based multipart uploads)
Advanced Features Implemented:
- Automatic file chunking with configurable part sizes (5MB-5GB)
- Parallel upload worker pool (configurable concurrency)
- Retry logic with exponential backoff (3 retries default, configurable)
- Real-time progress tracking with upload speed calculation
- Automatic cleanup on errors (AbortMultipartUpload)
- Support for resumable uploads via ListParts
- Comprehensive input validation (part size, part numbers, etc.)
- Integration tests with MinIO (500+ LOC)
- File integrity verification across all upload modes
Why: Critical for large files (>100MB). Enables resumable uploads, parallel transfers, and better performance.
v0.13.0 - Server-Side Encryption
Scope: Security basics Size: Small (~150 LOC)
- SSE-S3 support (AES256)
- SSE-KMS support (key ARN)
- Encryption headers in Put/Upload/Copy
Why: Security is table stakes. Simple to add to existing operations.
v0.14.0 - Object Tagging
Scope: Metadata management Size: Small (4 APIs, ~200 LOC)
- PutObjectTagging
- GetObjectTagging
- DeleteObjectTagging
- Tagging support in Put/Upload/Copy
Why: Common requirement for organization/billing. Simple XML operations.
v0.15.0 - Object Versioning
Scope: Version control Size: Medium (5 APIs, ~300 LOC)
- PutBucketVersioning
- GetBucketVersioning
- ListObjectVersions
- GetObjectVersion
- DeleteObjectVersion (with version ID)
Why: Important for data safety. More complex due to version handling.
v0.16.0 - ACLs & Lifecycle
Scope: Advanced management Size: Medium (9 APIs, ~500 LOC)
ACLs:
- PutObjectAcl
- GetObjectAcl
- PutBucketAcl
- GetBucketAcl
Lifecycle:
- PutBucketLifecycle
- GetBucketLifecycle
- DeleteBucketLifecycle
Why: Grouped advanced features. Completes library API surface.
v0.17.0 - CLI Complete
Scope: Full-featured CLI with all library features Size: Large (~2000 LOC)
CLI (cmd/simples3/):
- Project structure (cobra or custom parser)
- S3 URI parsing (
s3://bucket/key) - Config (env vars: AWS_*, flags: --region/--profile)
- AWS credentials file support (~/.aws/credentials)
- Basic commands:
- ls (list buckets/objects)
- cp (upload/download, with multipart for large files)
- rm (delete single/batch)
- mb (make bucket)
- rb (remove bucket)
- mv (move/rename using copy+delete)
- presign (generate URL)
- sync (local ↔ S3 with diff algorithm)
- Flags:
- --recursive (cp/rm)
- --sse/--sse-kms-key-id (encryption)
- --tags (tagging)
- --version-id (versioning)
- --acl (canned ACLs)
- --delete (sync)
- --exclude/--include (patterns)
- --json (output mode)
- --dry-run
- Features:
- Progress indicators
- Parallel transfers
- Better error messages
- Subcommands:
- acl get/set
- lifecycle get/set/delete
- tags get/set/delete
Why: Library complete. Build full-featured CLI supporting all operations.
v1.0.0 - Stable Release
Focus: Production-ready stability Size: Documentation, tests, polish
Library:
- All tests passing with 80%+ coverage
- API documentation complete
- Performance benchmarks
- Security audit of signing/crypto code
CLI:
- CLI tests with real/mocked S3
- CLI documentation (man pages, --help)
- Installation guides (brew, apt, etc.)
- Shell completions (bash, zsh, fish)
Release:
- CHANGELOG.md complete
- Migration guide from v0.x
- Examples for all features
- API stability guarantee
Why: v1.0 = stability commitment. Library + CLI production-ready.