RealtimePyConfigSync

Real-time detection and loading of dynamic configuration changes from an etcd database to a Python process.

---

Instead of periodically pooling configurations from a database at fixed intervals, we can utilize etcd database to watch for changes on specific keys (configs). Whenever new changes occur on these keys, the etcd server will push those changes to the client, and then react on them accordingly.

---

Under the hood, it utilizes the python-etcd3 library, which provides a callback method to listen for changes on keys.

Here are some additions made :

• No boilerplate code is required.
• There is no need to manually define which callback function type needs to be bound.
• It provides a JSON deserializer by default, but you can define your own deserialization logic based on your requirements.
• It supports auto-reconnection with etcd in case of a connection failure.
• You can specify the number of connection retries and the retry delay.
• It applies different deserialization methods to keys if you have mixed key-value pairs.
• It allows for easy addition of new observers to perform desired operations on keys.
• You can remove the parse_engine observer in case of simple key-value pairs.

---

etcd

A distributed, reliable key-value store for the most critical data of a distributed system. etcd is a CNCF project.

Read more about etcd

---

Requirements

• Python
• etcd
• Etcd Manager - Optional

---

Installation

Clone the Repository

1. Open your terminal or command prompt.
2. Change to the directory where you want to clone the project.
3. Execute the following command to clone the repository:

git clone https://github.com/faizanfareed/RealtimePyConfigSync.git

Setup

Navigate to the project directory:

cd RealtimePyConfigSync

Create a virtual environment (optional but recommended):

For venv:

python -m venv env

For virtualenv:

virtualenv env

Activate the virtual environment:

For Windows:

.\env\Scripts\activate

For Unix/macOS:

source env/bin/activate

Install the project dependencies:

pip install -r requirements.txt

---

Run etcd server as a Docker

https://hub.docker.com/r/bitnami/etcd/

Once your Docker is up and running, open the app.py module and update the configurations there. After making the necessary changes, run the module.

---

Install etcd manager on ubuntu

https://snapcraft.io/install/etcd-manager/ubuntu

---

License

License under a MIT License

python-etcd3 license Apache License 2.0.

---

Contributing

• Fork, clone or make a pull request to this repository.
• Ask here https://github.com/faizanfareed/RealtimePyConfigSync/discussions