“A data engineer that does’nt evaluate his pipeline is like a chef who does’nt taste the food before serving it.” — Ancient Proverb.
So you have successfully created an end to end data pipeline!
you have done it!
You wrote all the code, adjusted all the integration, set the configurations and connected all the tools.
You even ran a test or two to see the data really flows and there is actually some data on the dashboards.
Your work is done and you can finally move on to the next project in peace, well… not quite.
Before submitting a pipeline to…
In recent years, along with the growth of data being generated, organizations have been investing in order to leverage that data.
Hence, there is a need for data-savvy personnel, who are able to observe, translate and pass on the information about the environments in which the organization operates, should it be a business, technological, financial, or any other environment.
These teams of data professionals are in fact the eyes and optic nerves of the organization.
Since there is no widespread standard for the job titles and descriptions, the titles and functions of professional data workers are often misunderstood.