Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
162 changes: 72 additions & 90 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,124 +4,106 @@ Useful template to bootstrap new professional data science and python projects.

[Documentation](https://deepsense-ai.github.io/ds-template/)

![Preview how to use cookiecutter template for ds-template](https://github.com/deepsense-ai/ds-template/blob/main/docs/_static/make_template.gif?raw=true)

##### Table of Contents
* [What is it?](#what-is-it)
* [What are the benefits?](#what-are-the-benefits)
* [Getting started](#getting-started)

# What is it?

Its intended use is to generate basic, most common configuration - however each team and developer is encouraged to modify it for its special needs.
A comprehensive project template generator for data science and Python projects, which generates basic,
most common configuration using proven templates.

It is a result of our experiences with building data science projects and is a part of our internal best practices, however it is not a silver bullet and should be treated as a starting point for your project.
Especially some settings might be less/more restrictive than you needs but we believe it is better to start with a good baseline and modify it later than to start from scratch.
Especially some settings might be less/more restrictive than you need, but we believe it is better to start with a good baseline and modify it later than to start from scratch.

# What are the benefits?

Generated project consists of:

1. Basic python package structure:
* `pyproject.toml` - central configuration for build system, package metadata, dependencies, and tool configurations (including linters and formatters)
* a very minimal python code + example test
1. pre-commit hooks:
* `ruff` - fast linter and formatter that replaces the functionality of:
* `black` - code formatting
* `flake8` - style and linting
* `pycln` - unused imports cleanup
* `isort` - import sorting
* `pylint` - static code analysis
* `pyupgrade` - code modernization for given python version
* `bandit` - security issue detection - required for SOC 2 Certification
* `mypy` - static type checking
* `jupytext` - (optional) syncs jupyter notebooks to plain python files
1. Sphinx documentation:
* basic preconfigured documentation template
* recommended extensions
* page with list of autogenerated third party python packages list with licenses
1. Basic script to create venv
1. Minimal README.md file
1. Preconfigured semantic versioning with bump2version
1. Dockerfile for pre-commit image
1. Gitlab integration (default, optional):
* linter stage (`pre-commit run --all`)
* tests (`pytest`) + code coverage
* license checks of installed packages
* building and hosting documentation on GitLab Pages
* building package and uploading to private GitLab Package registry
* security: `trivy` - required for SOC 2 Certification
* steps to rebuild linter docker image
1. GitHub integration (optional):
* linter stage (`pre-commit run --all`)
* tests (`pytest`) + code coverage
* license checks of installed packages
* building and hosting documentation on GitHub Pages (if enabled)
* testing if package can be build
* security: `trivy` - required for SOC 2 Certification, but is reporting only
1. Other less important files (more configurations, `.gitignore` etc.)
1. **Modern Python Workspace Structure**:
* `pyproject.toml` - modern Python project configuration with uv workspace support
* `mise.toml` - development environment management
* `packages/` directory for multi-package monorepo structure
* Individual package templates (API, CLI, Core, Frontend, Worker, Library)

2. **Code Quality & Linting** (via Ruff):
* **Code formatting** - automatic code formatting
* **Import sorting** - organized import statements
* **Type checking** - mypy integration for type safety
* **Security scanning** - bandit for security vulnerability detection
* **Code style** - comprehensive linting rules (pycodestyle, pyflakes, pylint, etc.)
* **Documentation** - pydocstyle for docstring standards
* **Modernization** - pyupgrade for Python version compatibility

3. **Testing & Coverage**:
* **pytest** - modern testing framework with async support
* **Coverage reporting** - comprehensive test coverage analysis
* **Test discovery** - automatic test detection and execution

4. **Documentation** (MkDocs with Material theme):
* **Modern documentation** - MkDocs with Material Design theme
* **Interactive features** - search, navigation, Mermaid diagrams
* **Auto-generated content** - package documentation and API references
* **Custom styling** - branded documentation appearance

5. **CI/CD Integration** (optional - choose GitHub, GitLab, or None):
* **GitHub Actions** (if selected):
- Automated linting and testing on every push/PR
- Security scanning with Trivy
- Package building and artifact upload
- Code coverage reporting
* **GitLab CI** (if selected):
- Multi-stage pipeline (lint, test, package, pages, security, deploy)
- Docker-based pre-commit image for faster builds
- GitLab Pages documentation hosting
- Package registry integration
- Security vulnerability scanning

6. **Development Tools**:
* **uv** - fast Python package manager and project management
* **License checking** - automated license compliance validation
* **Docker support** - pre-commit Docker image for consistent environments
* **Scripts** - utility scripts for package creation and management

Most up-to date descriptions, tips and explanations are in the [documentation](https://deepsense-ai.github.io/ds-template/).

# Getting started

### Generate project template locally:
## Usage

Install **cookiecutter** (at least **>=2.1.1** version) first and then point it to this repository.
```bash
$ pip install cookiecutter>=2.1.1
# Create a new data science application
uvx ds-template
```
Select from predefined package types and generate a standard project structure

**Cookiecutter** will ask you set of questions so it can generate customized project.
## Development

``` bash
$ cookiecutter ds-template/
client_name [ds]: Client Name
project_name [default]: Sunglass
repo_name [client-name-sunglass]:
...
```
To set up for development:

### How to initialize new repository with the template:
```bash
# Clone the repository
git clone https://github.com/deepsense-ai/ds-template.git
cd ds-template

Firstly, you need to create a new project. The name should be of the following convention:
# Install dependencies
uv sync

`<client_name>-<project_name>`
# Run the CLI
uv run ds-template
```

Execute the following steps then:
## Creating Custom Templates

Approach 1 (clone empty):
Templates are stored in the `templates/` directory. Each template consists of:

```bash
# clone empty repository to repo_name
$ git clone <GIT-SSH>
# install cookiecutter if not yet installed
$ pip install cookiecutter
# generate cookiecutter with --force and ensure the repo_name is set to the same name as directory you cloned git repository to.
$ cookiecutter -f git@github.com:deepsense-ai/ds-template.git
# finally, add all files, commit and push.
$ git add .
$ git commit -m "Initialize repository with default project template"
$ git push origin
```

Approach 2 (initialize git locally and push to remote):
1. A directory with the template name
2. A `template_config.py` file with template metadata and questions
3. Template files, with `.j2` extension for files that should be processed as Jinja2 templates

```bash
# install cookiecutter if not yet installed
$ pip install cookiecutter
# generate project
$ cookiecutter git@github.com:deepsense-ai/ds-template.git
# enter created directory
$ cd <project-name>
# now we need to connect it to repository (assuming empty repository)
$ git init
$ git remote add origin <GIT-SSH>
$ git fetch
$ git checkout -t origin/main
# finally, add all files, commit and push.
$ git add .
$ git commit -m "Initialize repository with default project template"
$ git push --set-upstream origin main
$ git push origin
```
Available variables in templates:
- `project_name`: Name of the project
- `pkg_name`: Name of the python package
- `python_version`: Python version
- Custom variables from template questions
10 changes: 0 additions & 10 deletions build_docs.sh

This file was deleted.

14 changes: 0 additions & 14 deletions cookiecutter.json

This file was deleted.

37 changes: 23 additions & 14 deletions docs/Makefile
Original file line number Diff line number Diff line change
@@ -1,20 +1,29 @@
# Minimal makefile for Sphinx documentation
# Minimal makefile for MkDocs documentation
#

# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS ?=
SPHINXBUILD ?= sphinx-build
SOURCEDIR = .
BUILDDIR = _build
# You can set these variables from the command line
MKDOCS ?= mkdocs
SOURCEDIR = .
BUILDDIR = site

# Put it first so that "make" without argument is like "make help".
# Put it first so that "make" without argument is like "make help"
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
@echo "MkDocs documentation build commands:"
@echo " serve Start development server"
@echo " build Build the documentation"
@echo " clean Clean the build directory"
@echo " deploy Deploy to GitHub Pages"

.PHONY: help Makefile
.PHONY: help serve build clean deploy

# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
serve:
@$(MKDOCS) serve

build:
@$(MKDOCS) build

clean:
@rm -rf "$(BUILDDIR)"

deploy:
@$(MKDOCS) gh-deploy
Binary file removed docs/_static/artifacts.png
Binary file not shown.
Binary file removed docs/_static/coverage.png
Binary file not shown.
Binary file removed docs/_static/docs_lic.png
Binary file not shown.
Binary file removed docs/_static/gh.png
Binary file not shown.
Binary file removed docs/_static/gh_artifacts.png
Binary file not shown.
Binary file removed docs/_static/gh_coverage.png
Binary file not shown.
Binary file removed docs/_static/gh_pages.png
Binary file not shown.
Binary file removed docs/_static/make_template.gif
Binary file not shown.
Binary file removed docs/_static/pip_reg.png
Binary file not shown.
Binary file removed docs/_static/stage_preparation.png
Binary file not shown.
Binary file removed docs/_static/stages_main.png
Binary file not shown.
Binary file removed docs/_static/stages_mr.png
Binary file not shown.
Binary file removed docs/_static/template_files.png
Binary file not shown.
Binary file removed docs/_static/tests.png
Binary file not shown.
18 changes: 18 additions & 0 deletions docs/assets/extra.css
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
/* Additional styling for MkDocs Material theme */
.md-typeset h1 {
color: #2196f3;
}

.md-typeset h2 {
color: #1976d2;
}

/* Code blocks styling */
.md-typeset pre {
border-radius: 4px;
}

/* Navigation improvements */
.md-nav__title {
font-weight: 600;
}
90 changes: 0 additions & 90 deletions docs/conf.py

This file was deleted.

Loading