News
Tutorial: How to ship AI/BI Dashboard changes safely at scale with Databricks Asset Bundles
3+ hour, 5+ min ago (1512+ words) by Eason Gao, Noah Sommerfeld and Jen Lim The thought of a board meeting starting with a dashboard full of errors should keep analytics teams up at night. So should discovering, after the fact, that a hiring plan, product launch,…...
Real-Time Mode: Ultra-low latency streaming on Spark APIs without a second engine
1+ day, 9+ hour ago (332+ words) by Navneeth Nair, Jerry Peng and Abhay Bothra Apache Spark Structured Streaming has long powered mission-critical data pipelines at scale, from streaming ETL to analytics and machine learning. But as operational use cases evolved, teams began demanding something more: sub-second…...
Announcing SQL Server connector from Lakeflow Connect, now Generally Available
5+ mon, 1+ week ago (321+ words) by Elise Georis, Peter Pogorski and Giselle Goicochea The SQL Server connector makes it easy to ingest data from various SQL Server environments into the lakehouse, where it can be used for analytics and business intelligence across the organization, including:…...
Run OpenAI Models Directly on Databricks
5+ mon, 1+ week ago (316+ words) by Ahmed Bilal, Ana Nieto, Margaret Amori, Hanlin Tang and The Mosaic AI Engineering Team With Databricks, you can securely tap into OpenAI's GPT-5 model and build agents that reason on your enterprise data.Agent Bricks then helps you iterate…...
From Lag to Agility: Reinventing Freshworks’ Data Ingestion Architecture
5+ mon, 1+ week ago (760+ words) by Anand Chiddarwar, Kiran Kumar, Tamajit Hati, Pavan Parasa and Vasan Vembusrini Achieving this requires a powerful, flexible, and optimized data pipeline'which is exactly what we were set out to build. Rapid growth exposed core challenges: As scale and complexity…...
Enforce consistent, secure tagging across data and AI assets with Governed Tags in Unity Catalog (Public Preview)
5+ mon, 1+ week ago (358+ words) by Kristen Wilder, Tao Feng, Menglei Sun, Kelly Albano and Tianyi Zhang Today, we are excited to announce'Governed Tags in Public Preview across all regions on'AWS,'Azure, and'Google Cloud! Tags in Unity Catalog (UC) are widely used for discovery, cost…...
Introducing Spatial SQL in Databricks: 80+ Functions for High-Performance Geospatial Analytics
5+ mon, 2+ week ago (1037+ words) by Kent Marten and Michael Johns Every day, billions of data points are tied to places on the map. Delivery routes, store visits, road networks, cell towers, and crop fields, all carry important context for making business decisions. The problem…...
Databricks on Databricks: Scaling Database Reliability
5+ mon, 2+ week ago (538+ words) However, even with faster diagnostics, we were still often reacting to issues after they occurred. The next logical step in our journey was to prevent these issues from ever making it to production in the first place. Just as static…...
Customizing Databricks Assistant with Instructions
5+ mon, 2+ week ago (403+ words) by Gal Oshri, Chloe Chan, Neil Palleti and Kyle Gilbreath If you've ever told Databricks Assistant something like "use snake_case," "prefer Plotly," or "pull sales from catalog.schema.gold_sales" you've probably said it more than once. Repeating the same thing every session…...
How Observability in Lakeflow Helps You Build Reliable Data Pipelines
5+ mon, 3+ week ago (556+ words) by Theresa Hammer and Joanna Zouhour In this blog, we will introduce Lakeflow's observability capabilities and show how to build reliable, fresh, and healthy data pipelines. Observability for data engineering is the ability to discover, monitor, and troubleshoot systems to…...