1# Cloudflare Pipelines
2 
3ETL streaming platform for ingesting, transforming, and loading data into R2 with SQL transformations.
4 
5## Overview
6 
7Pipelines provides:
8- **Streams**: Durable event buffers (HTTP/Workers ingestion)
9- **Pipelines**: SQL-based transformations
10- **Sinks**: R2 destinations (Iceberg tables or Parquet/JSON files)
11 
12**Status**: Open beta (Workers Paid plan)  
13**Pricing**: No charge beyond standard R2 storage/operations
14 
15## Architecture
16 
17```
18Data Sources → Streams → Pipelines (SQL) → Sinks → R2
19                 ↑          ↓                ↓
20            HTTP/Workers  Transform     Iceberg/Parquet
21```
22 
23| Component | Purpose | Key Feature |
24|-----------|---------|-------------|
25| Streams | Event ingestion | Structured (validated) or unstructured |
26| Pipelines | Transform with SQL | Immutable after creation |
27| Sinks | Write to R2 | Exactly-once delivery |
28 
29## Quick Start
30 
31```bash
32# Interactive setup (recommended)
33npx wrangler pipelines setup
34```
35 
36**Minimal Worker example:**
37```typescript
38interface Env {
39  STREAM: Pipeline;
40}
41 
42export default {
43  async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
44    const event = { user_id: "123", event_type: "purchase", amount: 29.99 };
45    
46    // Fire-and-forget pattern
47    ctx.waitUntil(env.STREAM.send([event]));
48    
49    return new Response('OK');
50  }
51} satisfies ExportedHandler<Env>;
52```
53 
54## Which Sink Type?
55 
56```
57Need SQL queries on data?
58  → R2 Data Catalog (Iceberg)
59    ✅ ACID transactions, time-travel, schema evolution
60    ❌ More setup complexity (namespace, table, catalog token)
61 
62Just file storage/archival?
63  → R2 Storage (Parquet)
64    ✅ Simple, direct file access
65    ❌ No built-in SQL queries
66 
67Using external tools (Spark/Athena)?
68  → R2 Storage (Parquet with partitioning)
69    ✅ Standard format, partition pruning for performance
70    ❌ Must manage schema compatibility yourself
71```
72 
73## Common Use Cases
74 
75- **Analytics pipelines**: Clickstream, telemetry, server logs
76- **Data warehousing**: ETL into queryable Iceberg tables
77- **Event processing**: Mobile/IoT with enrichment
78- **Ecommerce analytics**: User events, purchases, views
79 
80## Reading Order
81 
82**New to Pipelines?** Start here:
831. [configuration.md](./configuration.md) - Setup streams, sinks, pipelines
842. [api.md](./api.md) - Send events, TypeScript types, SQL functions
853. [patterns.md](./patterns.md) - Best practices, integrations, complete example
864. [gotchas.md](./gotchas.md) - Critical warnings, troubleshooting
87 
88**Task-based routing:**
89- Setup pipeline → [configuration.md](./configuration.md)
90- Send/query data → [api.md](./api.md)
91- Implement pattern → [patterns.md](./patterns.md)
92- Debug issue → [gotchas.md](./gotchas.md)
93 
94## In This Reference
95 
96- [configuration.md](./configuration.md) - wrangler.jsonc bindings, schema definition, sink options, CLI commands
97- [api.md](./api.md) - Pipeline binding interface, send() method, HTTP ingest, SQL function reference
98- [patterns.md](./patterns.md) - Fire-and-forget, schema validation with Zod, integrations, performance tuning
99- [gotchas.md](./gotchas.md) - Silent validation failures, immutable pipelines, latency expectations, limits
100 
101## See Also
102 
103- [r2](../r2/) - R2 storage backend for sinks
104- [queues](../queues/) - Compare with Queues for async processing
105- [workers](../workers/) - Worker runtime for event ingestion
106