Effective Data Engineering using Composer
A framework for balancing the rapid iteration required for building Composer DAGs and the reliability demands that occur when it is productionised by leveraging SRE fundamentals.
Who does this blog target?
This blog is meant for those who have a basic understanding of cloud (such as what a bucket is, what pipelines are, and more). They may however not have a deep understanding of the common workflows of a data engineer. Or, they might know it without having a deep understanding of the infrastructure that lives behind it - to enable their work to be delivered as tables that are easy to consume, and has good cost and reliability metrics. Someone like a business analyst with strong SQL and some python skills would benefit from this, especially if they are looking to work independently and want to increase their scalability.
What are the "shared" resources in this setup?
Bone Fides
At my time in Nine, I worked on the data team. This was a really great experience for me, and is described in the home page. I owe a lot of what I learned to the people there!