feat: implement Simple List API for S3 objects
- Add List method with ListInput and ListResponse structs following simples3 patterns
- Implement pagination, prefix filtering, and delimiter grouping
- Add comprehensive test suite for List functionality
- Update tests to use environment variables instead of hardcoded values
- Fix deprecated io/ioutil usage throughout codebase
- Add comprehensive README with API documentation and examples
- Add Nix development environment with flake.nix
- Add Justfile for modern command runner
- Add Docker Compose setup for local MinIO testing
- Update GitHub Actions to use Nix and Justfile for CI consistency
- Fix test naming and setup flow issues
New API:
s3.List(ListInput{
Bucket: "my-bucket",
Prefix: "documents/",
MaxKeys: 100,
})
All tests pass with new simplified API design.
This commit is contained in:
Rohan Verma
@@ -1,82 +1,334 @@
|
||||
# simples3 : Simple no frills AWS S3 Library using REST with V4 Signing
|
||||
|
||||
## Overview [](https://godoc.org/github.com/rhnvrm/simples3) [](https://goreportcard.com/report/github.com/rhnvrm/simples3) [](https://gocover.io/_badge/github.com/rhnvrm/simples3) [](https://zerodha.tech)
|
||||
## Overview [](https://godoc.org/github.com/rhnvrm/simples3) [](https://goreportcard.com/report/github.com/rhnvrm/simples3) [](https://gocover.io/_badge/github.com/rhnvrm/simples3) [](https://zerodha.tech)
|
||||
|
||||
SimpleS3 is a Go library for manipulating objects
|
||||
in S3 buckets using REST API calls or Presigned URLs signed
|
||||
SimpleS3 is a Go library for manipulating objects
|
||||
in S3 buckets using REST API calls or Presigned URLs signed
|
||||
using AWS Signature Version 4.
|
||||
|
||||
**Features:**
|
||||
- 🚀 **Simple, intuitive API** following Go idioms
|
||||
- 🪣 **Complete S3 operations** - Upload, Download, Delete, List, Details
|
||||
- 🔐 **AWS Signature Version 4** signing
|
||||
- 🌐 **Custom endpoint support** (MinIO, DigitalOcean Spaces, etc.)
|
||||
- 📋 **Simple List API** with pagination, prefix filtering, and delimiter grouping
|
||||
- 🔗 **Presigned URL generation** for secure browser uploads/downloads
|
||||
- 🪪 **IAM credential support** for EC2 instances
|
||||
- ✅ **Comprehensive test coverage**
|
||||
- 🎯 **Zero dependencies** - uses only Go standard library
|
||||
|
||||
## Install
|
||||
|
||||
```sh
|
||||
go get github.com/rhnvrm/simples3
|
||||
```
|
||||
|
||||
## Example
|
||||
## Quick Start
|
||||
|
||||
```go
|
||||
testTxt, _ := os.Open("testdata/test.txt")
|
||||
defer testTxt.Close()
|
||||
package main
|
||||
|
||||
// Create an instance of the package
|
||||
// You can either create by manually supplying credentials
|
||||
// (preferably using Environment vars)
|
||||
s3 := simples3.New(Region, AWSAccessKey, AWSSecretKey)
|
||||
// or you can use this on an EC2 instance to
|
||||
// obtain credentials from IAM attached to the instance.
|
||||
s3, _ := simples3.NewUsingIAM(Region)
|
||||
import (
|
||||
"log"
|
||||
"os"
|
||||
"github.com/rhnvrm/simples3"
|
||||
)
|
||||
|
||||
// You can also set a custom endpoint to a compatible s3 instance.
|
||||
s3.SetEndpoint(CustomEndpoint)
|
||||
func main() {
|
||||
// Initialize S3 client
|
||||
s3 := simples3.New("us-east-1", "your-access-key", "your-secret-key")
|
||||
|
||||
// Note: Consider adding a testTxt.Seek(0, 0)
|
||||
// in case you have read
|
||||
// the body, as the pointer is shared by the library.
|
||||
// Use MinIO or other S3-compatible services
|
||||
s3.SetEndpoint("https://s3.amazonaws.com")
|
||||
|
||||
// File Upload is as simple as providing the following
|
||||
// details.
|
||||
// Upload a file
|
||||
file, _ := os.Open("my-file.txt")
|
||||
defer file.Close()
|
||||
|
||||
resp, err := s3.FileUpload(simples3.UploadInput{
|
||||
Bucket: "my-bucket",
|
||||
ObjectKey: "my-file.txt",
|
||||
ContentType: "text/plain",
|
||||
FileName: "my-file.txt",
|
||||
Body: file,
|
||||
})
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
log.Printf("File uploaded successfully: %+v", resp)
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### File Operations
|
||||
|
||||
#### Upload Files
|
||||
```go
|
||||
// POST upload (recommended for browsers)
|
||||
resp, err := s3.FileUpload(simples3.UploadInput{
|
||||
Bucket: AWSBucket,
|
||||
ObjectKey: "test.txt",
|
||||
Bucket: "my-bucket",
|
||||
ObjectKey: "path/to/file.txt",
|
||||
ContentType: "text/plain",
|
||||
FileName: "test.txt",
|
||||
Body: testTxt,
|
||||
FileName: "file.txt",
|
||||
Body: file,
|
||||
})
|
||||
|
||||
// Similarly, Files can be deleted.
|
||||
// PUT upload (simpler for programmatic use)
|
||||
resp, err := s3.FilePut(simples3.UploadInput{
|
||||
Bucket: "my-bucket",
|
||||
ObjectKey: "path/to/file.txt",
|
||||
ContentType: "text/plain",
|
||||
Body: file,
|
||||
})
|
||||
```
|
||||
|
||||
#### Download Files
|
||||
```go
|
||||
// Download file
|
||||
file, err := s3.FileDownload(simples3.DownloadInput{
|
||||
Bucket: "my-bucket",
|
||||
ObjectKey: "path/to/file.txt",
|
||||
})
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
defer file.Close()
|
||||
|
||||
// Read the content
|
||||
data, err := io.ReadAll(file)
|
||||
```
|
||||
|
||||
#### Delete Files
|
||||
```go
|
||||
err := s3.FileDelete(simples3.DeleteInput{
|
||||
Bucket: os.Getenv("AWS_S3_BUCKET"),
|
||||
ObjectKey: "test.txt",
|
||||
Bucket: "my-bucket",
|
||||
ObjectKey: "path/to/file.txt",
|
||||
})
|
||||
```
|
||||
|
||||
// You can also download the file.
|
||||
file, _ := s3.FileDownload(simples3.DownloadInput{
|
||||
Bucket: AWSBucket,
|
||||
ObjectKey: "test.txt",
|
||||
#### Get File Details
|
||||
```go
|
||||
details, err := s3.FileDetails(simples3.DetailsInput{
|
||||
Bucket: "my-bucket",
|
||||
ObjectKey: "path/to/file.txt",
|
||||
})
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
data, _ := ioutil.ReadAll(file)
|
||||
file.Close()
|
||||
log.Printf("File size: %s bytes", details.ContentLength)
|
||||
log.Printf("Last modified: %s", details.LastModified)
|
||||
log.Printf("Content type: %s", details.ContentType)
|
||||
```
|
||||
|
||||
// You can also use this library to generate
|
||||
// Presigned URLs that can for eg. be used to
|
||||
// GET/PUT files on S3 through the browser.
|
||||
var time, _ = time.Parse(time.RFC1123, "Fri, 24 May 2013 00:00:00 GMT")
|
||||
### List Objects
|
||||
|
||||
url := s.GeneratePresignedURL(PresignedInput{
|
||||
Bucket: AWSBucket,
|
||||
ObjectKey: "test.txt",
|
||||
SimpleS3 provides a clean, easy-to-use List API that follows the same pattern as other library methods:
|
||||
|
||||
#### Simple Listing
|
||||
```go
|
||||
// List all objects in a bucket
|
||||
result, err := s3.List(simples3.ListInput{
|
||||
Bucket: "my-bucket",
|
||||
})
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
fmt.Printf("Found %d objects:\n", len(result.Objects))
|
||||
for _, obj := range result.Objects {
|
||||
fmt.Printf("- %s (%d bytes)\n", obj.Key, obj.Size)
|
||||
}
|
||||
```
|
||||
|
||||
#### Advanced Listing with Options
|
||||
```go
|
||||
// List with prefix filtering and pagination
|
||||
result, err := s3.List(simples3.ListInput{
|
||||
Bucket: "my-bucket",
|
||||
Prefix: "documents/",
|
||||
Delimiter: "/",
|
||||
MaxKeys: 100,
|
||||
})
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
// Process objects
|
||||
for _, obj := range result.Objects {
|
||||
fmt.Printf("📄 %s (%d bytes)\n", obj.Key, obj.Size)
|
||||
}
|
||||
|
||||
// Process "directories" (common prefixes)
|
||||
for _, prefix := range result.CommonPrefixes {
|
||||
fmt.Printf("📁 %s/\n", prefix)
|
||||
}
|
||||
```
|
||||
|
||||
#### Handle Pagination
|
||||
```go
|
||||
// Handle large result sets with pagination
|
||||
func listAllObjects(s3 *simples3.S3, bucket string) ([]simples3.Object, error) {
|
||||
var allObjects []simples3.Object
|
||||
var continuationToken string
|
||||
|
||||
for {
|
||||
// List objects with continuation token
|
||||
result, err := s3.List(simples3.ListInput{
|
||||
Bucket: bucket,
|
||||
ContinuationToken: continuationToken,
|
||||
MaxKeys: 1000, // Maximum page size
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Add current page objects to our collection
|
||||
allObjects = append(allObjects, result.Objects...)
|
||||
|
||||
// Check if there are more pages
|
||||
if !result.IsTruncated {
|
||||
break
|
||||
}
|
||||
|
||||
// Set token for next page
|
||||
continuationToken = result.NextContinuationToken
|
||||
}
|
||||
|
||||
return allObjects, nil
|
||||
}
|
||||
|
||||
// Usage example
|
||||
func main() {
|
||||
s3 := simples3.New("us-east-1", "your-access-key", "your-secret-key")
|
||||
|
||||
allObjects, err := listAllObjects(s3, "my-bucket")
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
|
||||
fmt.Printf("Total objects: %d\\n", len(allObjects))
|
||||
for _, obj := range allObjects {
|
||||
fmt.Printf("- %s (%d bytes)\\n", obj.Key, obj.Size)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Presigned URLs
|
||||
|
||||
Generate secure URLs for browser-based uploads/downloads:
|
||||
|
||||
```go
|
||||
// Generate presigned URL for download
|
||||
url := s3.GeneratePresignedURL(simples3.PresignedInput{
|
||||
Bucket: "my-bucket",
|
||||
ObjectKey: "private-file.pdf",
|
||||
Method: "GET",
|
||||
Timestamp: time,
|
||||
ExpirySeconds: 86400,
|
||||
ExpirySeconds: 3600, // 1 hour
|
||||
})
|
||||
|
||||
// Users can now download directly: <a href="{{url}}">Download</a>
|
||||
```
|
||||
|
||||
### Custom Endpoints
|
||||
|
||||
Use with MinIO, DigitalOcean Spaces, or other S3-compatible services:
|
||||
|
||||
```go
|
||||
// MinIO
|
||||
s3 := simples3.New("us-east-1", "minioadmin", "minioadmin")
|
||||
s3.SetEndpoint("http://localhost:9000")
|
||||
|
||||
// DigitalOcean Spaces
|
||||
s3 := simples3.New("nyc3", "your-access-key", "your-secret-key")
|
||||
s3.SetEndpoint("https://nyc3.digitaloceanspaces.com")
|
||||
```
|
||||
|
||||
### IAM Credentials
|
||||
|
||||
On EC2 instances, use IAM roles automatically:
|
||||
|
||||
```go
|
||||
s3, err := simples3.NewUsingIAM("us-east-1")
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
// Automatically uses instance IAM role
|
||||
```
|
||||
|
||||
## Development
|
||||
|
||||
### Setup Development Environment
|
||||
|
||||
```sh
|
||||
# Clone the repository
|
||||
git clone https://github.com/rhnvrm/simples3.git
|
||||
cd simples3
|
||||
|
||||
# Using Nix (recommended)
|
||||
nix develop
|
||||
|
||||
# Or using direnv
|
||||
direnv allow
|
||||
|
||||
# Start local MinIO for testing
|
||||
just setup
|
||||
|
||||
# Run tests
|
||||
just test-local
|
||||
```
|
||||
|
||||
### Testing
|
||||
|
||||
The library includes comprehensive tests that run against a local MinIO instance:
|
||||
|
||||
```sh
|
||||
# Run all tests (without MinIO)
|
||||
just test
|
||||
|
||||
# Run tests with local MinIO
|
||||
just test-local
|
||||
|
||||
# Run specific test
|
||||
go test -v -run TestList
|
||||
```
|
||||
|
||||
### Available Commands
|
||||
|
||||
```sh
|
||||
just # List all commands
|
||||
just test # Run tests without MinIO
|
||||
just test-local # Run tests with MinIO (includes setup)
|
||||
just setup # Setup development environment
|
||||
just minio-up # Start MinIO container
|
||||
just minio-down # Stop MinIO container
|
||||
just clean # Clean up everything
|
||||
just status # Check development environment status
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
For development and testing:
|
||||
|
||||
```sh
|
||||
export AWS_S3_REGION="us-east-1"
|
||||
export AWS_S3_ACCESS_KEY="minioadmin"
|
||||
export AWS_S3_SECRET_KEY="minioadmin"
|
||||
export AWS_S3_ENDPOINT="http://localhost:9000"
|
||||
export AWS_S3_BUCKET="testbucket"
|
||||
```
|
||||
|
||||
## Contributing
|
||||
|
||||
You are more than welcome to contribute to this project. Fork and make
|
||||
a Pull Request, or create an Issue if you see any problem or want to
|
||||
propose a feature.
|
||||
We welcome contributions! Please:
|
||||
|
||||
1. Fork the repository
|
||||
2. Create a feature branch
|
||||
3. Add tests for new functionality
|
||||
4. Ensure all tests pass: `just test-local`
|
||||
5. Submit a pull request
|
||||
|
||||
## Author
|
||||
|
||||
|
||||
Reference in New Issue
Block a user