eiac/simples3
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add multipart upload support

Browse Source 
"Verify the multipart upload implementation works correctly by running integration tests against MinIO and validating file integrity, parallel uploads, and error handling."

Core APIs:
- Add InitiateMultipartUpload for starting multipart uploads
- Add UploadPart for uploading individual parts (5MB-5GB)
- Add CompleteMultipartUpload to finalize uploads
- Add AbortMultipartUpload to cancel and cleanup uploads
- Add ListParts to query uploaded parts

High-Level Features:
- Add FileUploadMultipart API with automatic chunking
- Add uploadPartWithRetry with exponential backoff (3 retries default)
- Add parallel upload worker pool with configurable concurrency
- Add progress callbacks for real-time upload tracking
- Add GeneratePresignedUploadPartURL for browser-based uploads

Advanced Features:
- Automatic file chunking with configurable part sizes (5MB-5GB)
- Parallel upload worker pool with configurable concurrency
- Retry logic with exponential backoff (configurable retries)
- Real-time progress tracking with upload speed calculation
- Automatic cleanup on errors via AbortMultipartUpload
- Support for resumable uploads via ListParts
- Comprehensive input validation for all parameters
- File integrity verification across all upload modes

Testing:
- Add comprehensive integration tests (500+ lines)
- Test sequential uploads (15MB, 3 parts, ~45 MB/s)
- Test parallel uploads (20MB, 4 parts, concurrency=3)
- Test small files (3MB, single part)
- Validate abort and ListParts functionality
- Test input validation and error cases
- Verify file integrity with hash comparison
- All tests passing against MinIO

Documentation:
- Update README with multipart upload examples
- Add high-level and low-level API usage
- Document browser-based multipart uploads
- Mark multipart upload implementation as complete in ROADMAP
- Update gitignore to exclude *.bin and test-output.log

Verified against MinIO with complete test coverage.
This commit is contained in:
Rohan Verma
2025-11-27 02:16:16 +05:30
parent 9a14996a0b
commit ba9ef916bc
7 changed files with 1958 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ROADMAP.md
+29 -9
View File
@@ -55,17 +55,37 @@ Object Operations:
---
## v0.12.0 - Multipart Upload
**Scope**: Large file support
**Size**: Large (5 APIs, ~400 LOC)
## v0.12.0 - Multipart Upload ✅ COMPLETED
**Scope**: Large file support with parallel uploads, progress tracking, and presigned URLs
**Size**: Large (10 APIs/features, ~800 LOC)
**Released**: Nov 27, 2025
- [ ] InitiateMultipartUpload
- [ ] UploadPart (with retry logic)
- [ ] CompleteMultipartUpload
- [ ] AbortMultipartUpload
- [ ] ListParts
Core APIs:
- [x] InitiateMultipartUpload
- [x] UploadPart
- [x] CompleteMultipartUpload
- [x] AbortMultipartUpload
- [x] ListParts
**Why**: Critical for large files. Enables resumable uploads.
High-Level Features:
- [x] FileUploadMultipart (automatic chunking)
- [x] uploadPartWithRetry (exponential backoff retry logic)
- [x] uploadPartsParallel (concurrent part uploads with worker pool)
- [x] Progress callbacks (real-time upload progress)
- [x] GeneratePresignedUploadPartURL (browser-based multipart uploads)
Advanced Features Implemented:
- [x] Automatic file chunking with configurable part sizes (5MB-5GB)
- [x] Parallel upload worker pool (configurable concurrency)
- [x] Retry logic with exponential backoff (3 retries default, configurable)
- [x] Real-time progress tracking with upload speed calculation
- [x] Automatic cleanup on errors (AbortMultipartUpload)
- [x] Support for resumable uploads via ListParts
- [x] Comprehensive input validation (part size, part numbers, etc.)
- [x] Integration tests with MinIO (500+ LOC)
- [x] File integrity verification across all upload modes
**Why**: Critical for large files (>100MB). Enables resumable uploads, parallel transfers, and better performance.
---