Top 5 Insights from Microsoft Fabric Community Conference You Need to Know
At the end of September, Microsoft Fabric Community Conference took place in Stockholm. During this four-day event, numerous sessions were organized to keep professionals and fabric enthusiasts up to date with all the new features and technological developments of the platform. I was genuinely impressed by the latest advancements from the Fabric development team. With so many new features and improvements, I have chosen to highlight five standout features which I think you should be aware of.
1. Fast Copy in Dataflows Gen 2
Dataflows help us ingesting and transforming our data, making it the ideal tool to use in a Data Pipeline to ingest data into a Lakehouse. With the introduction of fast copy, it’s now also possible to ingest terabytes of data in an efficient way.
For example, loading a 6 GB CSV file into a Lakehouse table in Microsoft Fabric is now 8 times faster and 3 times cheaper.
The key benefits to remember are:
Enhanced performance and reduced costs.
Automatic enablement for data sizes over 100 MB or 5 million rows.
Support for various data sources like ADLS Gen2, Blob storage, Azure SQL DB, and more.
High-performance data transfer from on-premises sources via an on-premises gateway.
2. Dataflow Gen 2 Incremental refresh
Alongside the development of the fast copy feature, a new enhancement for dataflows has been introduced: incremental data refresh. This feature significantly improves data processing efficiency by updating only the data that has changed since the last refresh which ensures faster dataflows and better resource usage.
3. T-SQL notebooks
A third feature that caught my eye is the introduction of T-SQL notebooks which are currently available for public preview, expanding the language support to include T-SQL alongside Spark. This update allows T-SQL developers to use notebooks for creating and organizing their queries, with Markdown cells for better documentation.
Key features include:
Adding Data Warehouses or SQL analytics endpoints to notebooks.
Running T-SQL code directly against connected warehouses or endpoints.
BI Analysts can execute cross-database queries for business insights.
Access to comprehensive charting tools and collaborative development.
4. Unity catalog Databricks
If you already work with Databricks you will be a fan of this feature. You can now integrate Databricks Unity Catalog tables directly into Microsoft Fabric by creating a “Mirrored Azure Databricks Catalog” item. This feature links your Azure Databricks workspace URL and desired catalog, creating shortcuts for each table without copying data. Changes in Unity Catalog sync automatically with Fabric, allowing access through SQL endpoints, Spark notebooks, and Power BI reports.
Key features include:
Mirrored Databases: Fabric creates three items when mirroring an Azure Databricks Unity Catalog: a mirrored Azure Databricks item, a SQL analytics endpoint on a Lakehouse, and a default semantic model.
Access Methods: Users can access mirrored data via autogenerated SQL analytics endpoints or Power BI with Direct Lake mode.
Metadata Sync: Automatic synchronization of catalog changes (schemas and tables) from Azure Databricks to Fabric is enabled by default.
Selection Behavior: Users can select or exclude specific schemas and tables, with certain filtration conditions applied (e.g., materialized views and non-Delta format tables are excluded).
5. Copilot for Data Warehouse
A last feature that I noticed is the introduction of Microsoft Copilot to Synapse Data Warehouse. Copilot for Data warehousing is an AI assistant that helps streamline data warehousing tasks by integrating with your Fabric warehouse. It generates T-SQL code using metadata like table and column names, primary keys, and foreign keys, but not the actual data in tables.
Key features include:
Natural Language to SQL: Generate SQL queries from simple natural language questions.
Code Completion: AI-powered suggestions to enhance coding efficiency.
Quick Actions: Fix and explain SQL queries quickly.
Intelligent Insights: Smart suggestions based on warehouse schema and metadata.
As indicated in the beginning of this blog, these 5 features are only a small part of all the new developments shared at the event. If you would like to see a total overview of all the features, I would like to refer you to Microsoft's ubiquitous blog.
Wrap up
As you can see, Microsoft has put the Microsoft Fabric Community Conference on the market as a not-to-be-missed event if you work with MS Fabric. The numerous new features and developments that were unveiled ensure that the platform continues to develop and evolve into a mature platform within the market. I am looking forward to the developments that will be ready for us at the next events.
Do you have questions about certain features in Fabric or would you like to know more about how you can use Fabric to boost your data platform within your company? Do not hesitate to contact us!
Esli Van Acoleyen
Esli is an experienced Data Analytics Consultant with several years of experience. His expertise lies in data engineering, analytics, and architecture, primarily on the Microsoft platform. He has a strong interest in Microsoft Fabric and continuously seeks to expand his knowledge in this area.
Outside of work, Esli enjoys playing padel and is always open to connecting with new people, whether for a chat or a game on the padel court.
Anxious to know what Plainsight could mean for you?
Read more “About us”
Consider “your career at Plainsight 🚀”
Any questions? “Contact us”