Articles

How to export data from Google Big Query into AWS S3 + EMR hive or AWS Athena

Perquisites

  1. GCP basics - create machine, or use cloud shell.

  2. Big query basics - export data

  3. a machine @@ GCP

  4. gsutil - google cloud storage utility - copy/ls etc.

  5. S3 bucket + with user (access key + secret key)

  6. AVRO tools

  7. JAVA

Why the author of "We’re All Using Airflow Wrong and How to Fix It" needs to attend my meetup!

In this article “We’re All Using Airflow Wrong and How to Fix It“, The Author, Has claimed there 3 cons/reasons for why we are using airflow wrongly:
 

  1. First, because each step of this DAG is a different functional task, each step is created using a different Airflow Operator. Developers must spend time researching, understanding, using, and debugging the Operator they want to use

Contact Us

Tel. 972-52-3771400