NetApp will acquire DataPelago, creator of the Nucleus Universal Data Processing Engine (UDPE) that accelerates heterogeneous compute for analytics and generative AI.
Built on open-source Gluten, Velox and Substrait, Nucleus drastically speeds Spark and Trino to deliver standout cost-performance. It seamlessly connects to existing lakehouses, SQL, Python, Airflow, Tableau, Power BI and more without data migration or vendor lock-in. Founded in 2021 by CEO Rajan Goyal and CPO Anand Iyer, DataPelago exited stealth in October 2024 and raised over $75 million total, including a $47 million 2024 financing round.
NetApp CEO George Kurian stated that advancing AI hardware demands equally capable data infrastructure to unlock enterprise data value, and the acquisition strengthens NetApp’s ability to deliver agile data processing for competitive differentiation.
A DataPelago blog noted Nucleus will be natively embedded within NetApp’s data platform, enabling far broader enterprise adoption than the startup could achieve independently. Earlier this year, DataPelago ranked fourth in Data Science on Fast Company’s 2026 World’s Most Innovative Companies list.
Unlike traditional architectures that shift datasets to external CPU/GPU clusters, Nucleus runs accelerated compute directly at the storage tier. Benchmarks against Nvidia cuDF show up to 10.5x faster projections, 10.1x faster filters and 4.3x faster aggregations. Positioned between query engines (Spark, Trino, Flink) and Python frameworks (Ray, Dask), its stack has three modular layers:
1.DataApp: Pluggable integration module to inject acceleration into Spark, Trino and other engines.
2.DataOS: Orchestration layer dynamically mapping data tasks to mixed accelerators for scalable performance tuning.
3.DataVM: Custom virtual machine with domain-specific ISA, offering unified execution abstraction for CPUs, GPUs, FPGAs and custom silicon.
Core Data Execution Flow
1.Data access via framework connectors: Deployed as Spark JAR plugins compatible with standard data connectors, supporting Parquet, ORC, Iceberg, Delta Lake, JSON and storage targets including S3, GCS, ADLS, HDFS and on-prem arrays. Acceleration extends to storage integrations while retaining native query semantics via standard engine I/O layers.
2.Query planning and optimization: The host engine generates a physical plan; DataApp converts it to an intermediate representation via Gluten/Substrait. An intelligent optimizer builds optimal data flow graphs and allocates CPU/GPU resources for DataOS/DataVM execution.
3.Cross-hardware data transfer optimizations: Operator/kernel fusion and streaming execution eliminate full intermediate materialization to cut I/O overhead. Zero-copy shared memory reduces CPU-GPU data duplication, while DataVM’s ISA leverages LLVM, CUDA and ROCm to route tasks to optimal hardware via columnar vectorized processing. These tweaks lift GPU utilization to 80–90% by minimizing cross-domain data shuffling.
Nucleus operates on-prem and across major public clouds. Its Spark accelerator delivers 3–4x throughput gains versus Databricks Photon; it also integrates with Snowflake by accelerating open table format processing and upstream/downstream Spark/Trino pipelines feeding the warehouse. DataPelago states Nucleus cuts infrastructure costs by up to 80% and delivers 10x faster performance than legacy pipelines. By removing mandatory data duplication between operational and AI systems, it eliminates the primary bottleneck slowing enterprise AI rollout, with live deployments at large global enterprises across verticals.
DataPelago CEO Rajan Goyal said the firm’s core mission is resolving data processing bottlenecks holding back AI innovation. Merging with NetApp pairs its breakthrough acceleration tech with NetApp’s extensive data infrastructure portfolio; businesses have invested heavily in GPUs and models yet suffer fragmented data silos that leave hardware underutilized, and the combined stack streamlines large-scale AI deployment.
NetApp frames the acquisition as a landmark portfolio upgrade, bringing GPU-accelerated processing directly into storage workflows to deliver true zero-copy enterprise data activation for AI. Transaction financials remain undisclosed; given Nucleus’ enterprise traction, tight synergy with NetApp AIDE and rising agentic AI demand, the purchase valuation likely lands at a 4–5x multiple of DataPelago’s total $75M funding. DataPelago will operate as a fully owned NetApp subsidiary.
Key Distinction: Nucleus vs KV Cache vs NetApp AIDE
1.Nucleus and KV Cache differentiation: Nucleus optimizes pre-inference data ingestion, transformation and querying before data reaches GPU servers. Nvidia’s KV Cache is an in-GPU memory optimization active only post data arrival on GPU hardware to boost token generation efficiency. Nucleus speeds data delivery to compute; KV Cache optimizes model runtime after data lands on accelerators.
2.NetApp AIDE vs Nucleus UDPE: AIDE is ONTAP/AFX-locked AI preprocessing for LLMs and agents, functioning as proprietary ETL with built-in vector database, metadata cataloging, RAG serving and Nvidia AI Enterprise integration (including NIM vectorization microservices). Nucleus is storage-agnostic and accelerates generic Spark/Trino workloads. Integrating Nucleus with AIDE would strengthen transformation, training, fine-tuning and inference pipelines, enabling accelerated lakehouse workloads directly on NetApp AFX storage; an end-to-end AIDE+Nucleus bundled solution is foreseeable.
NetApp CPO Syam Nair commented Nucleus delivers software-defined acceleration at storage, enabling zero-copy data preparation, governance and AI activation across CPUs and GPUs. NetApp manages more enterprise multi-environment data than any rival, and the next wave of AI competitiveness hinges on processing data at its source—DataPelago’s engineering team accelerates this strategic roadmap.
Beijing Qianxing Jietong Technology Co., Ltd.
Sandy Yang/Global Strategy Director
WhatsApp / WeChat: +86 13426366826
Email: yangyd@qianxingdata.com
Website: www.qianxingdata.com/www.storagesserver.com
Business Focus:
ICT Product Distribution/System Integration & Services/Infrastructure Solutions
With 20+ years of IT distribution experience, we partner with leading global brands to deliver reliable products and professional services.
“Using Technology to Build an Intelligent World”Your Trusted ICT Product Service Provider!
आपका संदेश 20-3,000 अक्षरों के बीच होना चाहिए!
Hindi
