genro-storage Documentation

Universal storage abstraction for Python with pluggable backends

genro-storage provides a unified interface for accessing files across local filesystems, cloud storage (S3, GCS, Azure), and remote protocols (HTTP, SFTP). Built on top of fsspec, it adds an intuitive mount-point system and user-friendly API inspired by Unix filesystems.

Features

Async/await support - The same StorageManager is awaitable from async code via @smartasync
Native permission control - Configure readonly, readwrite, or delete permissions for any backend
Powered by fsspec - Leverage 20+ battle-tested storage backends
Mount point system - Organize storage with logical names like home:, uploads:, s3:
Intuitive API - Pathlib-inspired interface that feels natural and Pythonic
Dynamic paths - Callable paths that resolve at runtime for user-specific directories
External tool integration - local_path() for seamless ffmpeg, imagemagick integration
Cloud metadata - Get/set custom metadata on S3, GCS, Azure files
URL generation - Generate presigned URLs, data URIs for sharing
S3 versioning - Access historical file versions when versioning enabled
Flexible configuration - Load mounts from YAML, JSON, databases, or code
Test-friendly - In-memory backend for fast, isolated testing
Production-ready - Built on 6+ years of Genropy production experience
Cross-storage operations - Copy/move files between different storage types seamlessly

Quick Example

Synchronous Usage

from genro_storage import StorageManager

# Configure storage backends
storage = StorageManager()
storage.configure([
    {'name': 'home', 'protocol': 'local', 'path': '/home/user'},
    {'name': 'uploads', 'protocol': 's3', 'bucket': 'my-app-uploads'},
    {'name': 'backups', 'protocol': 'gcs', 'bucket': 'my-backups'}
])

# Work with files using a unified API
node = storage.node('uploads:users/123/avatar.jpg')
if node.exists:
    # Copy from S3 to local
    node.copy_to(storage.node('home:cache/avatar.jpg'))

    # Read and process
    data = node.read(mode='rb')

    # Backup to GCS
    node.copy_to(storage.node('backups:avatars/user_123.jpg'))

Async Usage

The same StorageManager works in both contexts: every I/O method carries @smartasync, so it is awaitable when called from async code.

from genro_storage import StorageManager

storage = StorageManager()
storage.configure([
    {'name': 'uploads', 'protocol': 's3', 'bucket': 'my-bucket'}
])

async def process_file(filepath: str):
    node = storage.node(f'uploads:{filepath}')

    if await node.exists():
        data = await node.read_bytes()
        size = await node.size()
        return data

Installation

# Basic installation (local storage only)
pip install genro-storage

# With cloud storage support
pip install genro-storage[s3]      # Amazon S3
pip install genro-storage[gcs]     # Google Cloud Storage
pip install genro-storage[azure]   # Azure Blob Storage

# With network protocols
pip install genro-storage[smb]     # SMB/CIFS (Windows shares)
pip install genro-storage[sftp]    # SFTP/SSH

# With at-rest encryption
pip install genro-storage[encryption]

# All backends
pip install genro-storage[all]     # Everything

Documentation Contents

User Guide

API Reference

API Reference

Development

Appendices