simples3/README.md

# simples3 : Simple no frills AWS S3 Library using REST with V4 Signing

## Overview [![GoDoc](https://godoc.org/github.com/rhnvrm/simples3?status.svg)](https://godoc.org/github.com/rhnvrm/simples3) [![Go Report Card](https://goreportcard.com/badge/github.com/rhnvrm/simples3)](https://goreportcard.com/report/github.com/rhnvrm/simples3) [![GoCover](https://gocover.io/_badge/github.com/rhnvrm/simples3)](https://gocover.io/_badge/github.com/rhnvrm/simples3) [![Zerodha Tech](https://zerodha.tech/static/images/github-badge.svg)](https://zerodha.tech)

SimpleS3 is a Go library for manipulating objects
in S3 buckets using REST API calls or Presigned URLs signed
using AWS Signature Version 4.

**Features:**
- **Simple, intuitive API** following Go idioms
- **Complete S3 operations** - Upload, Download, Delete, List, Details
- **Multipart upload** - Large file support with parallel uploads, progress tracking, and resumability
- **Bucket management** - Create, Delete, and List buckets
- **AWS Signature Version 4** signing
- **Custom endpoint support** (MinIO, DigitalOcean Spaces, etc.)
- **Simple List API** with pagination, prefix filtering, and delimiter grouping
- **Iterator-based ListAll** for memory-efficient large bucket iteration (Go 1.23+)
- **Presigned URL generation** for secure browser uploads/downloads (including multipart)
- **IAM credential support** for EC2 instances
- **Comprehensive test coverage**
- **Zero dependencies** - uses only Go standard library

## Install

```sh
go get github.com/rhnvrm/simples3
```

## Quick Start

```go
package main

import (
    "log"
    "os"
    "github.com/rhnvrm/simples3"
)

func main() {
    // Initialize S3 client
    s3 := simples3.New("us-east-1", "your-access-key", "your-secret-key")

    // Upload a file
    file, err := os.Open("my-file.txt")
    if err != nil {
        log.Fatal(err)
    }
    defer file.Close()

    resp, err := s3.FileUpload(simples3.UploadInput{
        Bucket:      "my-bucket",
        ObjectKey:   "my-file.txt",
        ContentType: "text/plain",
        FileName:    "my-file.txt",
        Body:        file,
    })
    if err != nil {
        log.Fatal(err)
    }

    log.Printf("File uploaded successfully: %+v", resp)
}
```

## API Reference

### Bucket Operations

#### List All Buckets
```go
// List all buckets for the AWS account
result, err := s3.ListBuckets(simples3.ListBucketsInput{})
if err != nil {
    log.Fatal(err)
}

fmt.Printf("Found %d buckets:\n", len(result.Buckets))
for _, bucket := range result.Buckets {
    fmt.Printf("- %s (created: %s)\n", bucket.Name, bucket.CreationDate)
}

// Owner information is also available
fmt.Printf("Owner: %s (%s)\n", result.Owner.DisplayName, result.Owner.ID)
```

#### Create a Bucket
```go
// Create a new bucket
output, err := s3.CreateBucket(simples3.CreateBucketInput{
    Bucket: "my-new-bucket",
    Region: "us-east-1", // Optional, defaults to client region
})
if err != nil {
    log.Fatal(err)
}

fmt.Printf("Bucket created at: %s\n", output.Location)
```

#### Delete a Bucket
```go
// Delete an empty bucket
err := s3.DeleteBucket(simples3.DeleteBucketInput{
    Bucket: "my-old-bucket",
})
if err != nil {
    log.Fatal(err)
}

// Note: The bucket must be empty before deletion
```

### File Operations

#### Upload Files
```go
// POST upload (recommended for browsers)
resp, err := s3.FileUpload(simples3.UploadInput{
    Bucket:      "my-bucket",
    ObjectKey:   "path/to/file.txt",
    ContentType: "text/plain",
    FileName:    "file.txt",
    Body:        file,
})

// PUT upload (simpler for programmatic use)
resp, err := s3.FilePut(simples3.UploadInput{
    Bucket:      "my-bucket",
    ObjectKey:   "path/to/file.txt",
    ContentType: "text/plain",
    Body:        file,
})
```

#### Download Files
```go
// Download file
file, err := s3.FileDownload(simples3.DownloadInput{
    Bucket:    "my-bucket",
    ObjectKey: "path/to/file.txt",
})
if err != nil {
    log.Fatal(err)
}
defer file.Close()

// Read the content
data, err := io.ReadAll(file)
if err != nil {
    log.Fatal(err)
}
```

#### Delete Files
```go
err := s3.FileDelete(simples3.DeleteInput{
    Bucket:    "my-bucket",
    ObjectKey: "path/to/file.txt",
})
```

#### Copy Objects
```go
// Copy object within same bucket
output, err := s3.CopyObject(simples3.CopyObjectInput{
    SourceBucket: "my-bucket",
    SourceKey:    "original/file.txt",
    DestBucket:   "my-bucket",
    DestKey:      "copied/file.txt",
})

// Copy across buckets
output, err := s3.CopyObject(simples3.CopyObjectInput{
    SourceBucket: "source-bucket",
    SourceKey:    "file.txt",
    DestBucket:   "dest-bucket",
    DestKey:      "file.txt",
})

// Copy with metadata replacement
output, err := s3.CopyObject(simples3.CopyObjectInput{
    SourceBucket:      "my-bucket",
    SourceKey:         "file.txt",
    DestBucket:        "my-bucket",
    DestKey:           "file-copy.txt",
    MetadataDirective: "REPLACE",
    ContentType:       "application/json",
    CustomMetadata:    map[string]string{"version": "2"},
})
```

#### Batch Delete Files
```go
// Delete multiple objects in one request (up to 1000)
output, err := s3.DeleteObjects(simples3.DeleteObjectsInput{
    Bucket:  "my-bucket",
    Objects: []string{"file1.txt", "file2.txt", "file3.txt"},
    Quiet:   false,
})
if err != nil {
    log.Fatal(err)
}

// Check results
for _, deleted := range output.Deleted {
    fmt.Printf("Deleted: %s\n", deleted.Key)
}
for _, errItem := range output.Errors {
    fmt.Printf("Failed to delete %s: %s\n", errItem.Key, errItem.Message)
}
```

#### Get File Details
```go
details, err := s3.FileDetails(simples3.DetailsInput{
    Bucket:    "my-bucket",
    ObjectKey: "path/to/file.txt",
})
if err != nil {
    log.Fatal(err)
}

log.Printf("File size: %s bytes", details.ContentLength)
log.Printf("Last modified: %s", details.LastModified)
log.Printf("Content type: %s", details.ContentType)
```

### List Objects

SimpleS3 provides a clean, easy-to-use List API that follows the same pattern as other library methods:

#### Simple Listing
```go
// List all objects in a bucket
result, err := s3.List(simples3.ListInput{
    Bucket: "my-bucket",
})
if err != nil {
    log.Fatal(err)
}

fmt.Printf("Found %d objects:\n", len(result.Objects))
for _, obj := range result.Objects {
    fmt.Printf("- %s (%d bytes)\n", obj.Key, obj.Size)
}
```

#### Advanced Listing with Options
```go
// List with prefix filtering and pagination
result, err := s3.List(simples3.ListInput{
    Bucket:    "my-bucket",
    Prefix:    "documents/",
    Delimiter: "/",
    MaxKeys:   100,
})
if err != nil {
    log.Fatal(err)
}

// Process objects
for _, obj := range result.Objects {
    fmt.Printf("%s (%d bytes)\n", obj.Key, obj.Size)
}

// Process "directories" (common prefixes)
for _, prefix := range result.CommonPrefixes {
    fmt.Printf("%s/\n", prefix)
}
```

#### Iterator-based Listing (Go 1.23+)
```go
// Use the new ListAll method for memory-efficient iteration
s3 := simples3.New("us-east-1", "your-access-key", "your-secret-key")

// Iterate over all objects with automatic pagination and error handling
seq, finish := s3.ListAll(simples3.ListInput{
    Bucket: "my-bucket",
    Prefix: "documents/", // Optional filtering
})

for obj := range seq {
    fmt.Printf("%s (%d bytes)\n", obj.Key, obj.Size)
}

// Check for any errors that occurred during iteration
if err := finish(); err != nil {
    log.Fatalf("Error during iteration: %v", err)
}
```

#### Handle Pagination (Legacy)
```go
// Handle large result sets with pagination
func listAllObjects(s3 *simples3.S3, bucket string) ([]simples3.Object, error) {
    var allObjects []simples3.Object
    var continuationToken string

    for {
        // List objects with continuation token
        result, err := s3.List(simples3.ListInput{
            Bucket:            bucket,
            ContinuationToken: continuationToken,
            MaxKeys:          1000, // Maximum page size
        })
        if err != nil {
            return nil, err
        }

        // Add current page objects to our collection
        allObjects = append(allObjects, result.Objects...)

        // Check if there are more pages
        if !result.IsTruncated {
            break
        }

        // Set token for next page
        continuationToken = result.NextContinuationToken
    }

    return allObjects, nil
}

// Usage example
func main() {
    s3 := simples3.New("us-east-1", "your-access-key", "your-secret-key")

    allObjects, err := listAllObjects(s3, "my-bucket")
    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf("Total objects: %d\n", len(allObjects))
    for _, obj := range allObjects {
        fmt.Printf("- %s (%d bytes)\n", obj.Key, obj.Size)
    }
}
```

### Multipart Upload

For large files (>100MB), use multipart upload for better performance, resumability, and parallel uploads.

#### High-Level API (Recommended)

The easiest way to upload large files:

```go
file, err := os.Open("large-video.mp4")
if err != nil {
    log.Fatal(err)
}
defer file.Close()

// FileUploadMultipart automatically handles chunking and parallel uploads
output, err := s3.FileUploadMultipart(simples3.MultipartUploadInput{
    Bucket:      "my-bucket",
    ObjectKey:   "videos/large-video.mp4",
    Body:        file,
    ContentType: "video/mp4",
    PartSize:    10 * 1024 * 1024, // 10MB parts (optional, default 5MB)
    Concurrency: 5,                 // Upload 5 parts in parallel (optional, default 1)
    OnProgress: func(info simples3.ProgressInfo) {
        fmt.Printf("\rProgress: %.1f%% (%d/%d parts)",
            float64(info.UploadedBytes)/float64(info.TotalBytes)*100,
            info.CurrentPart, info.TotalParts)
    },
})
if err != nil {
    log.Fatal(err)
}

fmt.Printf("\nUploaded: %s (ETag: %s)\n", output.Key, output.ETag)
```

#### Low-Level API (Advanced)

For more control over the upload process:

```go
// 1. Initiate multipart upload
initOutput, err := s3.InitiateMultipartUpload(simples3.InitiateMultipartUploadInput{
    Bucket:      "my-bucket",
    ObjectKey:   "large-file.bin",
    ContentType: "application/octet-stream",
})
if err != nil {
    log.Fatal(err)
}

// 2. Upload parts (can be done in parallel or resumed later)
var completedParts []simples3.CompletedPart
partSize := int64(10 * 1024 * 1024) // 10MB

for partNum := 1; partNum <= totalParts; partNum++ {
    // Read part data
    partData := getPartData(partNum, partSize) // Your function to get part data

    output, err := s3.UploadPart(simples3.UploadPartInput{
        Bucket:     "my-bucket",
        ObjectKey:  "large-file.bin",
        UploadID:   initOutput.UploadID,
        PartNumber: partNum,
        Body:       bytes.NewReader(partData),
        Size:       int64(len(partData)),
    })
    if err != nil {
        // Abort on error
        s3.AbortMultipartUpload(simples3.AbortMultipartUploadInput{
            Bucket:    "my-bucket",
            ObjectKey: "large-file.bin",
            UploadID:  initOutput.UploadID,
        })
        log.Fatal(err)
    }

    completedParts = append(completedParts, simples3.CompletedPart{
        PartNumber: output.PartNumber,
        ETag:       output.ETag,
    })
}

// 3. Complete the upload
result, err := s3.CompleteMultipartUpload(simples3.CompleteMultipartUploadInput{
    Bucket:    "my-bucket",
    ObjectKey: "large-file.bin",
    UploadID:  initOutput.UploadID,
    Parts:     completedParts,
})
if err != nil {
    log.Fatal(err)
}

fmt.Printf("Upload completed: %s\n", result.ETag)
```

#### List Uploaded Parts

Query which parts have been uploaded (useful for resuming uploads):

```go
output, err := s3.ListParts(simples3.ListPartsInput{
    Bucket:    "my-bucket",
    ObjectKey: "large-file.bin",
    UploadID:  uploadID,
})
if err != nil {
    log.Fatal(err)
}

for _, part := range output.Parts {
    fmt.Printf("Part %d: %d bytes (ETag: %s)\n",
        part.PartNumber, part.Size, part.ETag)
}
```

#### Abort Multipart Upload

Cancel an in-progress upload and clean up parts:

```go
err := s3.AbortMultipartUpload(simples3.AbortMultipartUploadInput{
    Bucket:    "my-bucket",
    ObjectKey: "large-file.bin",
    UploadID:  uploadID,
})
```

#### Browser-Based Multipart Upload

Generate presigned URLs for each part to enable direct browser uploads:

```go
// Backend: Initiate and generate presigned URLs
initOutput, _ := s3.InitiateMultipartUpload(simples3.InitiateMultipartUploadInput{
    Bucket:    "my-bucket",
    ObjectKey: "client-upload.bin",
})

// Generate presigned URL for part 1
presignedURL := s3.GeneratePresignedUploadPartURL(simples3.PresignedMultipartInput{
    Bucket:        "my-bucket",
    ObjectKey:     "client-upload.bin",
    UploadID:      initOutput.UploadID,
    PartNumber:    1,
    ExpirySeconds: 3600,
})

// Frontend: Upload directly to S3 using the presigned URL (via PUT request)
// Then send ETags back to backend for completion
```

### Presigned URLs

Generate secure URLs for browser-based uploads/downloads:

```go
// Generate presigned URL for download
url := s3.GeneratePresignedURL(simples3.PresignedInput{
    Bucket:        "my-bucket",
    ObjectKey:     "private-file.pdf",
    Method:        "GET",
    ExpirySeconds: 3600, // 1 hour
})

// Users can now download directly: <a href="{{url}}">Download</a>
```

### Custom Endpoints

Use with MinIO, DigitalOcean Spaces, or other S3-compatible services:

```go
// MinIO
s3 := simples3.New("us-east-1", "minioadmin", "minioadmin")
s3.SetEndpoint("http://localhost:9000")

// DigitalOcean Spaces
s3 := simples3.New("nyc3", "your-access-key", "your-secret-key")
s3.SetEndpoint("https://nyc3.digitaloceanspaces.com")
```

### IAM Credentials

On EC2 instances, use IAM roles automatically:

```go
s3, err := simples3.NewUsingIAM("us-east-1")
if err != nil {
    log.Fatal(err)
}
// Automatically uses instance IAM role
```

## Development

### Setup Development Environment

```sh
# Clone the repository
git clone https://github.com/rhnvrm/simples3.git
cd simples3

# Using Nix (recommended)
nix develop

# Or using direnv
direnv allow

# Start local MinIO for testing
just setup

# Run tests
just test-local
```

### Testing

The library includes comprehensive tests that run against a local MinIO instance:

```sh
# Run all tests (without MinIO)
just test

# Run tests with local MinIO
just test-local

# Run specific test
go test -v -run TestList
```

### Available Commands

```sh
just              # List all commands
just test         # Run tests without MinIO
just test-local   # Run tests with MinIO (includes setup)
just setup        # Setup development environment
just minio-up      # Start MinIO container
just minio-down    # Stop MinIO container
just clean         # Clean up everything
just status        # Check development environment status
```

## Environment Variables

For development and testing:

```sh
export AWS_S3_REGION="us-east-1"
export AWS_S3_ACCESS_KEY="minioadmin"
export AWS_S3_SECRET_KEY="minioadmin"
export AWS_S3_ENDPOINT="http://localhost:9000"
export AWS_S3_BUCKET="testbucket"
```

## Contributing

Contributions welcome! Check [ROADMAP.md](ROADMAP.md) for planned features. Please add tests and ensure `just test-local` passes before submitting PRs.

## Author

Rohan Verma <hello@rohanverma.net>

## License

BSD-2-Clause-FreeBSD