carbon_cpu_enriched_sequences

Single Dagster asset · group "cpu" · streaming, bounded-memory execution

Click any block for details

HuggingFaceBio/carbon-pretraining-corpus

config: eukaryote_generator · split: train

ingest.py — create_carbon_stream()

IterableDataset.take(row_limit)

streaming.py — iter_batches()

rows → bounded pa.RecordBatch

ProcessPoolExecutor · fork context · one call per batch

normalize_batch()

normalization.py

ValidationStats.update()

validation.py

enrich_batch()

enrichment.py

streaming.py — as_completed() drain

merge_validation_stats() in parent process

ParquetShardWriter

shard-00000.parquet, shard-00001.parquet, ...

carbon_cpu_enriched_sequences

Dagster MaterializeResult → feeds carbon_pilot_corpus

Detail

Click any block above to see how it works.

Source / output Ingestion Streaming / batching Normalization Validation Enrichment Dagster asset