This week I familirized myself with the apache airflow tool
First I went through courses and grasped basic idea and got into way of learnin by doing
1. Created a DAG that can be combined to sqlite server, and made several tasks that are based on create, input and update
-there is a dependency that first a "create table" tasl is run and only after insert and update tasks run, or if table creation fails the other tasks do not run
-the purpouse of the DAG(direct acyclic graph) is to filter or group given csv table based on a given condition
-the dag first gets dataset values and cleanes from null values and performs specific tasks by using branching, based on a given variable
-tasks interact with each other by using xcom values
-Now on the process on the optimization
-Further I am going to implement and perform operations on it with integration to airflow
finished the course “Learning Apache Airflow” by Janani Ravi https://www.linkedin.com/learning/certificates/526bdf263b0d2b5508bac5ef3595bf84f1eac22a138a2f6e665a3a3df4788043?trk=share_certificate #apacheairflow #itautomation.
Learned about cron operators and passing arguments throughtout tasks and using TaskFlow api