Skip to main content

How to setup a local dev environment in Airflow

The concept is showing as following

Setup process:​

A running Airflow server within docker

The following steps are applied on the local dev computer

  1. install docker and docker-compose
  2. git clone https://github.com/ninja-van/airflow-boilerplate.git && cd airflow-boilerplate
  3. install virtualenv and create a new venv
pip install virtualenv
virtualenv .venv
  1. Activate the venv and install packages as required. This step may take a long time.
source .venv/bin/activate
pip install -r requirements-airflow.txt
pip install -r requirements-dev.txt
  • if you face any conflict of packages, you can comment the lines of pandas in requirements-airflow.txt
  • You may need to remove specified version of snappy in requirements-airflow.txt and leave it to pip to resolve which version to install
  1. Initializing airflow docker-compose -f docker/docker-compose.yml up -d airflow_initdb