1# Cloudflare R2 Data Catalog Skill Reference
2 
3Expert guidance for Cloudflare R2 Data Catalog - Apache Iceberg catalog built into R2 buckets.
4 
5## Reading Order
6 
7**New to R2 Data Catalog?** Start here:
81. Read "What is R2 Data Catalog?" and "When to Use" below
92. [configuration.md](configuration.md) - Enable catalog, create tokens
103. [patterns.md](patterns.md) - PyIceberg setup and common patterns
114. [api.md](api.md) - REST API reference as needed
125. [gotchas.md](gotchas.md) - Troubleshooting when issues arise
13 
14**Quick reference?** Jump to:
15- [Enable catalog on bucket](configuration.md#enable-catalog-on-bucket)
16- [PyIceberg connection pattern](patterns.md#pyiceberg-connection-pattern)
17- [Permission errors](gotchas.md#permission-errors)
18 
19## What is R2 Data Catalog?
20 
21R2 Data Catalog is a **managed Apache Iceberg REST catalog** built directly into R2 buckets. It provides:
22 
23- **Apache Iceberg tables** - ACID transactions, schema evolution, time-travel queries
24- **Zero-egress costs** - Query from any cloud/region without data transfer fees
25- **Standard REST API** - Works with Spark, PyIceberg, Snowflake, Trino, DuckDB
26- **No infrastructure** - Fully managed, no catalog servers to run
27- **Public beta** - Available to all R2 subscribers, no extra cost beyond R2 storage
28 
29### What is Apache Iceberg?
30 
31Open table format for analytics datasets in object storage. Features:
32- **ACID transactions** - Safe concurrent reads/writes
33- **Metadata optimization** - Fast queries without full scans
34- **Schema evolution** - Add/rename/delete columns without rewrites
35- **Time-travel** - Query historical snapshots
36- **Partitioning** - Organize data for efficient queries
37 
38## When to Use
39 
40**Use R2 Data Catalog for:**
41- **Log analytics** - Store and query application/system logs
42- **Data lakes/warehouses** - Analytical datasets queried by multiple engines
43- **BI pipelines** - Aggregate data for dashboards and reports
44- **Multi-cloud analytics** - Share data across clouds without egress fees
45- **Time-series data** - Event streams, metrics, sensor data
46 
47**Don't use for:**
48- **Transactional workloads** - Use D1 or external database instead
49- **Sub-second latency** - Iceberg optimized for batch/analytical queries
50- **Small datasets (<1GB)** - Setup overhead not worth it
51- **Unstructured data** - Store files directly in R2, not as Iceberg tables
52 
53## Architecture
54 
55```
56┌─────────────────────────────────────────────────┐
57│  Query Engines                                  │
58│  (PyIceberg, Spark, Trino, Snowflake, DuckDB)  │
59└────────────────┬────────────────────────────────┘
60                 │
61                 │ REST API (OAuth2 token)
62                 ▼
63┌─────────────────────────────────────────────────┐
64│  R2 Data Catalog (Managed Iceberg REST Catalog)│
65│  • Namespace/table metadata                     │
66│  • Transaction coordination                     │
67│  • Snapshot management                          │
68└────────────────┬────────────────────────────────┘
69                 │
70                 │ Vended credentials
71                 ▼
72┌─────────────────────────────────────────────────┐
73│  R2 Bucket Storage                              │
74│  • Parquet data files                           │
75│  • Metadata files                               │
76│  • Manifest files                               │
77└─────────────────────────────────────────────────┘
78```
79 
80**Key concepts:**
81- **Catalog URI** - REST endpoint for catalog operations (e.g., `https://<account-id>.r2.cloudflarestorage.com/iceberg/<bucket>`)
82- **Warehouse** - Logical grouping of tables (typically same as bucket name)
83- **Namespace** - Schema/database containing tables (e.g., `logs`, `analytics`)
84- **Table** - Iceberg table with schema, data files, snapshots
85- **Vended credentials** - Temporary S3 credentials catalog provides for data access
86 
87## Limits
88 
89| Resource | Limit | Notes |
90|----------|-------|-------|
91| Namespaces per catalog | No hard limit | Organize tables logically |
92| Tables per namespace | <10,000 recommended | Performance degrades beyond this |
93| Files per table | <100,000 recommended | Run compaction regularly |
94| Snapshots per table | Configurable retention | Expire >7 days old |
95| Partitions per table | 100-1,000 optimal | Too many = slow metadata ops |
96| Table size | Same as R2 bucket | 10GB-10TB+ common |
97| API rate limits | Standard R2 API limits | Shared with R2 storage operations |
98| Target file size | 128-512 MB | After compaction |
99 
100## Current Status
101 
102**Public Beta** (as of Jan 2026)
103- Available to all R2 subscribers
104- No extra cost beyond standard R2 storage/operations
105- Production-ready, but breaking changes possible
106- Supports: namespaces, tables, snapshots, compaction, time-travel, table maintenance
107 
108## Decision Tree: Is R2 Data Catalog Right For You?
109 
110```
111Start → Need analytics on object storage data?
112         │
113         ├─ No → Use R2 directly for object storage
114         │
115         └─ Yes → Dataset >1GB with structured schema?
116                  │
117                  ├─ No → Too small, use R2 + ad-hoc queries
118                  │
119                  └─ Yes → Need ACID transactions or schema evolution?
120                           │
121                           ├─ No → Consider simpler solutions (Parquet on R2)
122                           │
123                           └─ Yes → Need multi-cloud/multi-tool access?
124                                    │
125                                    ├─ No → D1 or external DB may be simpler
126                                    │
127                                    └─ Yes → ✅ Use R2 Data Catalog
128```
129 
130**Quick check:** If you answer "yes" to all:
131- Dataset >1GB and growing
132- Structured/tabular data (logs, events, metrics)
133- Multiple query tools or cloud environments
134- Need versioning, schema changes, or concurrent access
135 
136→ R2 Data Catalog is a good fit.
137 
138## In This Reference
139 
140- **[configuration.md](configuration.md)** - Enable catalog, create API tokens, connect clients
141- **[api.md](api.md)** - REST endpoints, operations, maintenance
142- **[patterns.md](patterns.md)** - PyIceberg examples, common use cases
143- **[gotchas.md](gotchas.md)** - Troubleshooting, best practices, limitations
144 
145## See Also
146 
147- [Cloudflare R2 Data Catalog Docs](https://developers.cloudflare.com/r2/data-catalog/)
148- [Apache Iceberg Docs](https://iceberg.apache.org/)
149- [PyIceberg Docs](https://py.iceberg.apache.org/)
150