Want to learn more about Databricks? We can help. We've written some handy guides and listed ressources we've found helpful when we started our Databricks journey. We keep all these resources up to date so we hope you find these resources handy.
Data Engineering
This post explains how to create and write to Delta Lake tables using Rust on AWS S3 while addressing the challenge of eventual consistency. It demonstrates what can happen if multiple concurrent writers are not synchronised when writing data to a Delta table and how this challenge can be overcome. By the end of this post, you will also gain a better understanding of how the Delta Transaction Protocol works and how cloud storage services differ.
Data Engineering
Most data engineers correlate delta lake format with Spark and Databricks. That's not true. Delta can be used by so many other tools and most cloud providers have added delta support to their analytics tools. In this post we will see how to use delta lake from a Rust client.
Data Engineering
Do you use Databricks? Do you think it is just a hosted optimised flavour of Apache Spark? In that case you are missing many features and ideas that can take your productivity to the next level. This post has tips and tricks to take you to the next level on Databricks.
General
There is tonnes out of Information out there about Databricks and it can be a bit overwhelming. Where does somebody even start? We have done you a favor and curated a list of learning materials we found useful when we started our Databricks journey and we share with new employees.
Data Engineering
This post explains how to create and write to Delta Lake tables using Rust on AWS S3 while addressing the challenge of eventual consistency. It demonstrates what can happen if multiple concurrent writers are not synchronised when writing data to a Delta table and how this challenge can be overcome. By the end of this post, you will also gain a better understanding of how the Delta Transaction Protocol works and how cloud storage services differ.
Data Engineering
Most data engineers correlate delta lake format with Spark and Databricks. That's not true. Delta can be used by so many other tools and most cloud providers have added delta support to their analytics tools. In this post we will see how to use delta lake from a Rust client.
Data Engineering
Do you use Databricks? Do you think it is just a hosted optimised flavour of Apache Spark? In that case you are missing many features and ideas that can take your productivity to the next level. This post has tips and tricks to take you to the next level on Databricks.
General
There is tonnes out of Information out there about Databricks and it can be a bit overwhelming. Where does somebody even start? We have done you a favor and curated a list of learning materials we found useful when we started our Databricks journey and we share with new employees.