Articles
How to export data from Google Big Query into AWS S3 + EMR hive or AWS Athena
Perquisites
-
GCP basics - create machine, or use cloud shell.
-
Big query basics - export data
-
a machine @@ GCP
-
gsutil - google cloud storage utility - copy/ls etc.
-
S3 bucket + with user (access key + secret key)
-
AVRO tools
-
JAVA
Why the author of "We’re All Using Airflow Wrong and How to Fix It" needs to attend my meetup!
In this article “We’re All Using Airflow Wrong and How to Fix It“, The Author, Has claimed there 3 cons/reasons for why we are using airflow wrongly:
-
First, because each step of this DAG is a different functional task, each step is created using a different Airflow Operator. Developers must spend time researching, understanding, using, and debugging the Operator they want to use