This repository contains the code for an ETL Pipeline that extracts data from the National Vulnerability Database, transforms into a format that would be usable for Data Analytics and Data Science purposes and loads it into a CSV file.
The ETL Pipeline is divided into 3 main parts - Extractor, Transformer, and Loader.
Each part is an independent process that spawns from the parent process. To share data between each process, it uses a Queue.
This project uses the bare minimum resources and Data Engineering practices. Based on the requirements, it can be extended to use different technologies and tools but it does gets the work done.
- You may need to install
Python 3.11.0. Previous versions are not tested and may not work. - Install the dependencies using
pip install -r requirements.txt - Run the
main.pyfile usingpython main.py --api XXXX-XXXX-XXXX[Replace XXXX-XXXX-XXXX with your API key. If none provided, the code will still execute but will be limited to 1 request per 6 seconds in accordance with the NVD API Terms of Use.] - If you want to customize the start date for the execution (default is January 1, 2010) from which the data needs to be collected, run the code using
python main.py -y YYYY -m MM -d DD[Replace YYYY, MM, and DD with the year, month, and day respectively.]
-
Setup the monthly execution of the ETL Pipeline (maybe a Scheduled Kaggle Notebook that clones this repository and executes it. Airflow deployment is also an option but it will end up costing money.)
-
Kaggle API Integration to directly update the OSS Vulnerabilities Dataset that I maintain.