Databricks Serverless: Reducing Costs While Enhancing Productivity

At the recent Databricks Data + AI Summit 2024, Databricks made a significant announcement: they are going serverless. Following up on their promise, Databricks announced general availability of serverless compute for notebooks, jobs and Delta Live Tables (DLT). This is in addition to some features that already existed serverless on the platform like a SQL serverless warehouse for example. This blog will look into these new features but also what was already available on the platform. This will give an overview of the capabilities of the Databricks serverless platform.

What is a serverless platform?

To be able to explain the advantages of using a serverless platform, it is important to define what it is.

A serverless platform is a form of computing where the cloud provider manages the infrastructure. This allows developers to focus solely on writing and deploying code without worrying about infrastructure management. Despite what the name says, servers are still used, but instead of the developers managing the platform, the cloud provider does the job.

Main differences with non-serverless platforms

• Server management: in non-serverless platform, the developers set up and maintain the virtual machines.

• Capacity planning: non-serverless platform require capacity to be allocated beforehand for each task. Serverless platforms manage resources based on the demand in workload. Scaling the servers is not necessary anymore, as this is managed by the platform provider.

• Billing: non-serverless platforms bill based on usage of resources. If a resource is active but unused, this will still be billed. Serverless platform bill based on execution time. When a workload is finished, the underlying resources will be unallocated and no further billing will occur for the unused resource.

To summarize: servers are still used for workloads, but they are billed and allocated in a different way than non-serverless platforms.

The following chart illustrates how this is managed by Databricks:

On the right, we see the cloud account with the classic compute plane where normally you manage your own resources. The serverless compute plane here is located inside the Databricks account. This means that databricks manages all computes instead of having to do it yourself. This is still connected to the same resources used with Databricks so shifting to serverless requires minimal effort and changes.

Why use serverless compute?

To illustrate why a serverless compute instead of using self-managed clusters can be used, there are three main pillars:

• ·Productivity

• Cost efficiency

• Reliability.

Firstly, productivity is improved as Databricks manages the cloud resources, reducing the overhead for users. The serverless feature is also especially useful in notebooks for purposes. Where you would normally have to wait about 5 minutes for a cluster to spin up, this process now just needs a few seconds. This means that data scientists and engineers can access instant compute resources, enabling them to on their core tasks rather than infrastructure management and waiting for clusters to be ready. This also reduces maintenance of the infrastructure, which leads to more productivity and efficiency.

Secondly, serverless computes can be more cost efficient. These offers faster start-up and scaling times compared to normal computes, which minimizes idle time and ensures that you only pay for the resources that are being used. It is important to note that serverless computes can be more expensive than a fully managed and optimized infrastructure. This, however, takes time and needs to be evaluated for every workload and use case. This is why it is always important to factor in the total cost of ownership (TCO). This includes development and maintenance costs. These are often the most expensive parts of developing a data platform. If this cost can be cut by using a serverless platform, this can significantly decrease the overall TCO even if compute costs rise.

Lastly, a serverless compute gives more reliability. In the case of Databricks, a serverless compute provide automatic instance type failover. This feature ensures that if an instance becomes unavailable or fails, Databricks can automatically switch to another type without interrupting or failing your jobs. This helps maintain the continuity and reliability of your data processing jobs. Databricks also provides warm instance pools. These pools refer to a set of pre-initialized instances that are kept ready to be used or “warm”. When there are sudden spikes in demand or shortages, there are other instances available to be used. This makes sure that Databricks can quickly allocate them to your jobs, reducing the time you must wait for new instances to start up.

Serverless compute options

Serverless compute can be used in different ways on the Databricks platform. The following services support serverless computing:

• Serverless Compute for Notebooks (new): This allows you to execute code in notebooks with serverless computes. The benefit of this is that notebooks can be run without having startup time and they are scalable if there is a higher workload necessary. This will mainly be used for development

• Serverless Compute for Workflows (new): Databricks jobs can be run without configuring and managing infrastructure. It can decrease the time a job needs because there is no cluster spin up time. This feature also optimizes and scales compute resources automatically based on your workload demands.

• Serverless Delta Live Tables (DLT) Pipelines (new): Similarly to workflows, using serverless can improve performance and reduced processing time for DLT pipelines. This can also reduce costs.

• Serverless SQL Warehouses: Having a serverless warehouse gives access to instant query processing, without having to wait for a cluster to start up. It also scales the warehouse automatically if a workload demands more resources.

• Model Serving: Databricks provides a highly available and low-latency service for deploying AI models. This serverless feature ensures that your models are served efficiently without the need for manual infrastructure management.

When not to use serverless?

While Serverless may be of added value in most scenarios, sometimes you’re better off with the traditional compute instances. Here are some reasons why preferring the older compute instances:

I need to use my compute in specific networks/VNETs

Serverless compute cannot be deployed in a specific network. Some workloads need to be done in a specific network configuration; e.g. to access peered infrastructure components such as SQL Databases or due to security reasons.

I have steady and reliable workloads in an operational setting

If you’re using Databricks in a reliable and stable environment, then you might be better off with provisioning your own set of clusters. Why? Using serverless comes with a slightly higher base cost. If you know your server load in advance you can plan for this appropriately, reducing cost. In this case, it can be interesting to split between having the old workloads on non-serverless and new workloads on serverless to reduce maintenance and development costs.

I have custom requests for my compute

If you’re using specific features that are not supported by Serverless Compute, then the classic way of working remains an option. Read more about the limitations of serverless computing on Databricks.

How to use the new serverless compute in notebooks & workflows?

To use the new serverless compute in notebooks, a setting needs to be adjusted in the feature enablement page of the Databricks account.

Here, the feature “Serverless compute for Workflows, Notebooks, and Delta Live tables” needs to be enabled.

After this feature is enabled, a serverless compute instance can be used in a notebook by going to the notebook. When choosing a cluster, the serverless compute option will now appear. This can take some time after enabling the option is the settings.

Now when running the notebook, this serverless compute will be used. No more waiting for a cluster to spin up.

For workflows, it is possible to select the serverless compute in the Compute options list. These workflows have the advantage that multiple tasks can be executed on the same compute.

Wrap up

This blog has explained what ‘Serverless’ stands for in Databricks as well as its pros and cons. We hope you found this information helpful and insightful, and that it provides a solid foundation for understanding how serverless computing can benefit your organization. If you have any further questions or need more detailed guidance, please don’t hesitate to get in touch for further information about using serverless computes for your workloads.

Benoît Zwartjes

Hello, I’m Benoît and I’m a data analytics professional with a solid background in engineering and Artificial Intelligence. I have experience in designing, developing, and deploying data pipelines and architectures, as well as providing insights and solutions based on data analysis and visualization. I enjoy applying my skills and knowledge to solve real-world problems and create value from data. Recently released features or tools? I’m always keen to enhance my data engineering and analytics capabilities.

Anxious to know what Plainsight could mean for you?

• Read more “About us”

• Consider “your career at Plainsight 🚀”

• Any questions? “Contact us”