Testing dynamically generated dags in Airflow

In my daily life of writing airflow dags, one day I encountered an use case where the dags have to be dynamically generated based on a json config.

I stumbled upon the internet and found the astronomer blog on creating dynamic dags. The first step in the blog was a fit to my uses with little changes. Particularly the config where I need to create dags based on the config rather with a simple range function. I opted to a python file for configuration as it was easier to maintain the parameters like start date and schedule_interval explicitly.

After completing the dag creation now comes the part where the generated dags have to be unit tested. I searched the web for testing dynamically generated dags. But I couldn’t found one. Hence the write up.

The dynamic dag creation with config looked like

Now the difficult part is how do I test these dags 🤔.

pytest to the rescue.

I made use of pytest fixtures to mock that dag_config which can be execute d before a particular test is run. I choose to mock the dag_config because at the time of the writing the dag_config had no entries and were supposed to be populated by someone else. Also, this saves me to no alter the tests after adding/deleting a dag in the config. Hence I mocked the dag_config to run the tests for a dag that remained same no matter how the dag_config changed.

As you see we have mocked the dag_config only for this particular test case and we don’t have to alter the test case whenever someone added/deleted the entries in dag_config. And we also tested a dynamically generated dag based on the mock.

Thanks for reading

Regards,

Santhosh Kumar M