Modern AI Pipelines on a Lakehouse: Iceberg, Spark, and Why the Stack Matters

Most AI pipelines look great in a notebook. Clean sample data, a handful of features, a model that trains in seconds. Then you point it at production data and everything breaks. Schemas drift. Rows disappear mid-job. Your feature table reflects last Tuesday’s state, not today’s. The model is fine — the data infrastructure underneath it isn’t.

This isn’t a modeling problem. It’s a plumbing problem. And the Lakehouse architecture — specifically the combination of Apache Iceberg and Apache Spark — is the most practical answer the industry has converged on.

The Data Infrastructure Spectrum

To understand why Lakehouse matters, it helps to understand what came before it.

Data warehouses (Snowflake, Redshift, BigQuery) gave you ACID transactions, structured queries, and reliable consistency. The cost was lock-in, rigid schemas, and storage billed at a premium. Getting raw, unstructured, or semi-structured data in was always a fight.

Data lakes (raw files on S3 or GCS) gave you cheap, flexible storage for any data format. The cost was chaos — no schema enforcement, no transaction guarantees, no easy way to handle deletes or updates. Read performance on large datasets was painful.

Lakehouse takes the best of both: open file formats on cheap object storage, with a table format layer that adds the consistency and performance guarantees you’d expect from a warehouse. You keep full control of your data, your compute is decoupled from your storage, and your AI pipelines can work directly on the same tables your analysts query.

Apache Iceberg: The Table Format That Changes Everything

Iceberg is not a storage engine and not a query engine. It’s a table format specification — a way of organizing metadata on top of Parquet (or ORC, Avro) files that gives you capabilities raw file-based storage fundamentally lacks.

ACID transactions. Multiple writers can commit simultaneously without corrupting the table. A Spark job writing new rows and a streaming job appending sensor data can coexist safely. This is the property that makes reliable feature stores possible.

Schema evolution without rewrites. You can add, rename, or drop columns and the change is recorded in metadata. Old files aren’t rewritten. Queries against old snapshots still return the old schema. This matters enormously when your upstream data model changes, which it always does.

Time travel and snapshots. Every commit creates a new snapshot. You can query the table as it existed at any point in time with a simple AS OF clause. For AI pipelines this is critical: you can reconstruct the exact feature set used to train a model three months ago, reproduce a prediction from a specific date, or audit why a model’s behavior changed.

Hidden partitioning. Iceberg manages partition metadata internally rather than exposing it through directory paths. Your queries don’t need to know partition keys — the engine figures it out. You can also change the partitioning strategy without rewriting the table.

Row-level deletes and updates. Native support for DELETE, UPDATE, and MERGE operations via copy-on-write or merge-on-read strategies. This is what makes GDPR deletion requests tractable at scale, and what allows feature tables to be updated incrementally rather than rebuilt from scratch.

Apache Spark: The Compute Layer

Spark is the distributed compute engine that processes data across the Lakehouse. It reads from and writes to Iceberg tables, handles transformations at scale, and serves as the backbone of both batch and streaming pipelines.

A few properties matter specifically for AI workloads:

Unified batch and streaming. Spark Structured Streaming uses the same API as batch jobs. A feature engineering pipeline that runs as a nightly batch job can be converted to a streaming pipeline — reading from Kafka, processing events as they arrive, writing incrementally to Iceberg — with minimal code changes.

Native Iceberg integration. Spark’s Iceberg connector supports all table format capabilities: time travel reads, incremental reads (only rows changed since the last checkpoint), schema evolution, and transactional writes. You’re not fighting the format; the engine is built to use it.

DataFrame and SQL APIs. Data scientists who prefer Python DataFrames and data engineers who prefer SQL can work on the same Spark cluster against the same Iceberg tables. PySpark DataFrames can be passed directly to MLlib or converted to Pandas for scikit-learn and PyTorch training.

Scalable feature engineering. The most expensive part of most ML pipelines isn’t training — it’s computing features at scale. Window functions over billions of rows, joins across multiple large tables, aggregations over rolling time windows. Spark distributes this work across a cluster and writes the results back to Iceberg with full transaction guarantees.

What a Production AI Pipeline Looks Like

Here’s a concrete architecture for a financial data AI pipeline — the kind that computes features from market data and trains a predictive model.

1. Ingestion

Raw data lands in object storage (S3 or GCS) from multiple sources: exchange feeds, SEC EDGAR filings, earnings call transcripts, analyst reports. A Spark Structured Streaming job reads from a Kafka topic and writes to an Iceberg raw table using foreachBatch:

query = (
    spark.readStream
        .format("kafka")
        .option("subscribe", "market-events")
        .load()
    .writeStream
        .format("iceberg")
        .outputMode("append")
        .option("checkpointLocation", "s3://lake/checkpoints/market-events")
        .toTable("lake.raw.market_events")
)

The checkpoint ensures exactly-once delivery. If the job crashes, it resumes from where it left off — no duplicate rows, no gaps.

2. Feature Engineering

A scheduled Spark batch job reads from the raw Iceberg table, applies transformations, and writes to a feature store table. The job uses Iceberg’s incremental reads to process only rows committed since the last run:

from pyiceberg.catalog import load_catalog

catalog = load_catalog("glue")
table = catalog.load_table("lake.raw.market_events")

# Read only new snapshots since last run
df = spark.read.format("iceberg") \
    .option("start-snapshot-id", last_snapshot_id) \
    .load("lake.raw.market_events")

features = (
    df.groupBy("ticker", window("event_time", "1 day"))
      .agg(
          avg("price").alias("avg_price_1d"),
          stddev("price").alias("volatility_1d"),
          sum("volume").alias("total_volume_1d"),
      )
)

features.writeTo("lake.features.market_daily") \
        .option("merge-schema", "true") \
        .overwritePartitions()

Because the write is transactional, a downstream model training job reading the feature table simultaneously will see either the complete new batch or the previous state — never a partial write.

3. Point-in-Time Correct Joins

The classic ML data leakage problem in time-series data: accidentally joining a label (future value) with features that weren’t available at prediction time. Iceberg’s time travel makes point-in-time correct joins straightforward:

# Features as they existed at training cutoff — not today's backfilled values
features_at_cutoff = spark.read.format("iceberg") \
    .option("as-of-timestamp", training_cutoff_ms) \
    .load("lake.features.market_daily")

labels = spark.read.format("iceberg") \
    .load("lake.labels.price_movements")

training_set = features_at_cutoff.join(labels, ["ticker", "date"])

This is the difference between a model that works in backtesting and one that actually works in production.

4. Model Training

The joined training set is a Spark DataFrame. Convert it to Pandas for scikit-learn or use Spark MLlib for distributed training:

# Small-to-medium datasets: collect to Pandas, train with scikit-learn
pdf = training_set.toPandas()
X, y = pdf.drop("label", axis=1), pdf["label"]

from sklearn.ensemble import GradientBoostingClassifier
model = GradientBoostingClassifier(n_estimators=200)
model.fit(X, y)

For datasets too large to fit in memory, Spark MLlib’s distributed gradient boosting or integration with XGBoost on Spark handles the scale.

5. Inference and Lineage

Predictions are written back to an Iceberg table with metadata linking each row to the model version, feature snapshot ID, and training run ID. This creates a complete audit trail: for any prediction, you can trace exactly which features were used, which model version produced it, and what the underlying data looked like at that moment.

predictions.withColumn("model_version", lit(model_version)) \
           .withColumn("feature_snapshot", lit(feature_snapshot_id)) \
           .writeTo("lake.predictions.daily_signals") \
           .append()

Why This Matters for Financial Data Specifically

Financial datasets have properties that make a robust table format non-negotiable.

Corrections and restated data. Reported earnings get restated. Price data gets adjusted for splits. Index constituents change retroactively. Without row-level update support and time travel, you either overwrite history (and invalidate old models) or maintain a rats’ nest of correction tables. Iceberg handles corrections as first-class operations.

Regulatory audit requirements. You need to show what data a model used to produce a specific recommendation, on a specific date, for a specific user. Iceberg’s snapshot history is an immutable audit log. You can reproduce any past state exactly.

Feature freshness. A feature computed on stale data produces stale predictions. Incremental Iceberg reads combined with Spark Structured Streaming let you keep feature tables close to real-time without rebuilding them from scratch on every run.

Schema volatility. SEC filing formats change. Data vendor APIs add fields. Iceberg’s schema evolution means you can add a new field to a feature table without migrating existing data or breaking downstream consumers still reading the old schema.

The Operational Picture

The full stack looks like this:

Each layer is independently replaceable. You can swap Spark for Trino for interactive queries against the same Iceberg tables. You can move from Glue to Nessie for Git-like branching across your entire data lake. The open format is the foundation that makes this possible.

What Usually Goes Wrong

The architecture above is mature and battle-tested. The failure modes are almost always implementation problems, not design problems.

Skipping the catalog. Iceberg tables without a proper catalog (Glue, Nessie, etc.) are hard to discover and manage. Invest in catalog infrastructure early.

Ignoring small file compaction. Streaming writes create many small Parquet files. Without periodic compaction jobs, read performance degrades significantly. Iceberg’s rewriteDataFiles procedure handles this, but you have to schedule it.

Over-partitioning feature tables. Partitioning by ticker and date sounds logical until you have 5,000 tickers and five years of daily data — that’s millions of partition directories. Use Iceberg’s hidden partitioning and let the engine optimize.

Training on the full snapshot. Reading the entire feature table for every training run is expensive. Use incremental reads and checkpoint your training jobs so you’re only processing new data.

The Bigger Picture

AI pipelines fail in production not because the models are bad, but because the data underneath them is unreliable. The Lakehouse pattern — Iceberg for table semantics, Spark for compute — gives you the reliability properties that production AI requires: consistent reads, transactional writes, full audit history, and the ability to reconstruct any past state exactly.

That foundation is what separates an AI feature that works once in a demo from one that works every day in production.

At Spoterra, our data infrastructure is built on these principles. Earnings data, SEC filings, analyst ratings, and market metrics are stored and served with the consistency guarantees that make reliable AI-driven analysis possible. When the model surfaces a signal, the data backing it is accurate, current, and auditable — not a best guess from a pipeline that may or may not have finished running.