Reading the body into a zero-length buffer returned io.EOF on an
already-exhausted reader, so empty files and directory markers always
failed. Use io.ReadFull and send an explicit Content-Length: 0 for
empty bodies instead of falling back to chunked encoding. Also fix an
error variable mix-up in the NewRequest error check.
S3-compatible services such as Alibaba Cloud OSS, Tencent COS and
Huawei OBS reject path-style URLs with 'SecondLevelDomainForbidden'
and only accept virtual-hosted style (https://bucket.endpoint/key).
This adds a UseVirtualHostedStyle field and SetVirtualHostedStyle()
setter. When enabled and a custom Endpoint is set, the first path
segment is treated as the bucket and prepended to the host as a
subdomain. AWS S3 and MinIO accept both styles, so this is a no-op
for them.
Includes tests for OSS-style endpoint, http/https scheme preservation
and the empty-path fallback used by ListBuckets-style operations.
filepath.Clean converts forward slashes to backslashes on Windows,
which breaks the AWS V4 signature calculation by corrupting the
canonical URI. Use path.Clean which always uses forward slashes,
matching the S3 API expectation.
Implements server-side encryption capabilities (v0.13.0) including:
- Core Support: SSE-S3 (AES256) and SSE-KMS
- Integration: Added encryption fields to UploadInput, CopyObjectInput, InitiateMultipartUploadInput, and MultipartUploadInput
- Operations: Updated FilePut, FileUpload (POST), CopyObject, and multipart upload flows to support encryption headers
- API: Updated FileDetails (HeadObject) to return ServerSideEncryption and SSEKMSKeyId in DetailsResponse
Testing:
- Configured MinIO with built-in KMS (via MINIO_KMS_SECRET_KEY) in docker-compose.yml to enable SSE-S3 support
- Added comprehensive integration tests verifying headers are correctly sent and returned
- Verified file integrity and encryption status across all upload methods
Updated ROADMAP.md to mark v0.13.0 as completed.
Implements ACL and Lifecycle management capabilities (v0.16.0) including:
- Bucket APIs: PutBucketAcl, GetBucketAcl, PutBucketLifecycle, GetBucketLifecycle, DeleteBucketLifecycle
- Object APIs: PutObjectAcl, GetObjectAcl
- Integration: Support for Canned ACLs (private, public-read, etc.) and Full Access Control Policies
- Lifecycle Configuration: Rules, Filters, Expiration, Transitions
Verified integration with MinIO:
- Note: MinIO returns 501 Not Implemented for Canned ACLs via headers, so tests for those are skipped while Custom ACLs (XML) are fully verified.
Includes comprehensive integration tests covering all operations.
Updated ROADMAP.md to mark v0.16.0 as completed.
Implements object versioning capabilities (v0.15.0) including:
- Bucket APIs: PutBucketVersioning, GetBucketVersioning
- Object Listing: ListVersions (lists versions and delete markers)
- Object Operations: Updated FileDownload, FileDelete, and FileDetails to support VersionId
- Integration tests covering the full versioning workflow (enable, upload versions, list, download specific version, delete specific version)
Documentation updated in README.md and ROADMAP.md.
"I have implemented Object Versioning (v0.15.0) as per the roadmap."
Implements object tagging capabilities (v0.14.0) including:
- Core APIs: PutObjectTagging, GetObjectTagging, DeleteObjectTagging
- Integration with FilePut, FileUpload, and CopyObject operations
- Helper functions for URL-encoded tag header formatting
- Comprehensive input validation (max 10 tags, required fields)
Addresses specific compatibility issues with MinIO and Cloudflare R2:
- Implements a 2-step workaround for CopyObject (copy first, then tag) to avoid SignatureDoesNotMatch errors caused by the x-amz-tagging-directive header on non-AWS providers.
Includes comprehensive integration tests and documentation updates.
"Verify the multipart upload implementation works correctly by running integration tests against MinIO and validating file integrity, parallel uploads, and error handling."
Core APIs:
- Add InitiateMultipartUpload for starting multipart uploads
- Add UploadPart for uploading individual parts (5MB-5GB)
- Add CompleteMultipartUpload to finalize uploads
- Add AbortMultipartUpload to cancel and cleanup uploads
- Add ListParts to query uploaded parts
High-Level Features:
- Add FileUploadMultipart API with automatic chunking
- Add uploadPartWithRetry with exponential backoff (3 retries default)
- Add parallel upload worker pool with configurable concurrency
- Add progress callbacks for real-time upload tracking
- Add GeneratePresignedUploadPartURL for browser-based uploads
Advanced Features:
- Automatic file chunking with configurable part sizes (5MB-5GB)
- Parallel upload worker pool with configurable concurrency
- Retry logic with exponential backoff (configurable retries)
- Real-time progress tracking with upload speed calculation
- Automatic cleanup on errors via AbortMultipartUpload
- Support for resumable uploads via ListParts
- Comprehensive input validation for all parameters
- File integrity verification across all upload modes
Testing:
- Add comprehensive integration tests (500+ lines)
- Test sequential uploads (15MB, 3 parts, ~45 MB/s)
- Test parallel uploads (20MB, 4 parts, concurrency=3)
- Test small files (3MB, single part)
- Validate abort and ListParts functionality
- Test input validation and error cases
- Verify file integrity with hash comparison
- All tests passing against MinIO
Documentation:
- Update README with multipart upload examples
- Add high-level and low-level API usage
- Document browser-based multipart uploads
- Mark multipart upload implementation as complete in ROADMAP
- Update gitignore to exclude *.bin and test-output.log
Verified against MinIO with complete test coverage.
Implements object manipulation capabilities for v0.11.0:
- CopyObject: server-side copy within/across buckets with metadata control
- DeleteObjects: batch delete up to 1000 objects in single request
Both operations follow AWS S3 API specification with proper:
- AWS Signature V4 signing
- XML request/response handling
- Content-MD5 and SHA256 headers
- Comprehensive input validation
Fixed DeleteObjects signature mismatch with MinIO by correcting the canonical
query string format for empty query parameter values (delete= instead of delete).
The writeQuery function now properly appends = to keys with empty values per
AWS Signature V4 specification.
CopyObject enables CLI mv commands. DeleteObjects enables efficient
batch operations for CLI sync/rm.
Tests: All CopyObject and DeleteObjects tests passing with MinIO.
"lets work on the object level operations"
Adds bucket management capabilities:
- ListBuckets: list all buckets for AWS account with owner info
- CreateBucket: create bucket with optional region support
- DeleteBucket: delete empty buckets
Includes comprehensive tests and README documentation.
Prompt: "Please review the changes we have made for this bucket api"
"we never used to have like so many emojis earlier and the look of the readme on github was really professional. can you please fix that"
"can you critique the readme"
- Remove emojis from features list and code examples for professional appearance
- Fix error handling in Quick Start example
- Add proper error checks for io.ReadAll
- Fix escaped newlines in printf statements (\\n to \n)
- Remove confusing SetEndpoint line from Quick Start
- Add ROADMAP.md with planned features through v1.0.0
- Update Contributing section to reference ROADMAP.md
- Add List method with ListInput and ListResponse structs following simples3 patterns
- Implement pagination, prefix filtering, and delimiter grouping
- Add comprehensive test suite for List functionality
- Update tests to use environment variables instead of hardcoded values
- Fix deprecated io/ioutil usage throughout codebase
- Add comprehensive README with API documentation and examples
- Add Nix development environment with flake.nix
- Add Justfile for modern command runner
- Add Docker Compose setup for local MinIO testing
- Update GitHub Actions to use Nix and Justfile for CI consistency
- Fix test naming and setup flow issues
New API:
s3.List(ListInput{
Bucket: "my-bucket",
Prefix: "documents/",
MaxKeys: 100,
})
All tests pass with new simplified API design.
Adds renewIAMToken() call in GeneratePresignedURL.
On token expiry, GeneratePresignedURL would not renew it until
other methods that renew tokens are called.
Instead of using the errors to figure out whether IMDSv2 needs to be used,
we return a boolean value for useIMDSv2 from the function itself, and
propagate the error to upstream function calls in the library.
IMDSv2 tokens can be used in all cases, even when IMDSv2 is set to `Optional`
in the EC2 Instance configuration. As such, calls with token will not fail and
we should be able to send IMDSv2 requests with token even when using IMDSv1.
Signed-off-by: Chinmay D. Pai <chinmaydpai@gmail.com>
Currently, IMDSv2 is not supported by simple-s3's `NewUsingIAM` method, causing it
to fail on EC2 instances with IMDSv2 enabled in the EC2 metadata configuration.
This patch adds a failsafe method that executes PUT request on the IMDSv2 token endpoint,
and based on the response decides whether IMDSv2 support needs to be enabled. Doing so
will not break or require any code-level changes to existing deployments, and allows the
same binaries to run on both IMDSv2 enabled and optional instances.
ref: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-metadata-v2-how-it-works.html
Signed-off-by: Chinmay D. Pai <chinmaydpai@gmail.com>
This commit adds a defaultHTTPClient to have a timeout of 10s
instead of the one provided by Go. It introduces a few new methods such
as
- FetchAndSetIAMData which is a helper method to be able to use the
custom http client set by the user.
Closes#19
* feat: Adds support for custom metadata
assigning metadata for files will be beneficial to
find files.
ref: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/add-origin-custom-headers.html
* fix: review suggestions by @rhnvrm
- metadata moved to UploadInput as optional field.
- changed x-amz-meta as constant
- changed sprintf to simple string concat
* FIX: changed amazon meta prefix
changed variable name for amazon meta prefix to the top,
prefix variable is exported.
removed redundant comment for metadata
ys-ll
ys-ll
zhanglq
Naphat Amundsen
Rohan Verma
Abhinav Raut
Joice M. Joseph
Pierre-Olivier Mercier
Chinmay D. Pai
Joice
Jack Henschel