Collection of Utilities to do Network Analysis and Data Integration. This is made targeting analysis of point data in a river system but it should work for any network analysis that work on directed graph/network.
The main component of the NADI System is the DSL. It can be used for network based data analysis. The repo contains the tools for using nadi through Command Line Interface (CLI), Integrated Development Environment (IDE), mdbook (documentation writing tool), or as a python library.
The overall NADI workflow in case of a river network is shown below. This repository contains the components for the Network Analysis part of the system.
Directed graph is a common way to represent systems with dependencies where each edge connects the node with its input node that it depends on. In terms of rivers the output node gets its water from the input node. In version control (git), each edge is a commit that is applied on top of its input node. NADI provides a way to easily load such networks along with node attributes.
The data associated with points in a river system is loaded as a network, and along side the functions loaded through the plugins, the Domain Specific Programming language (DSL) can be used to run different arithmetic, logical, or functional analysis on the metadata of the points.
This allows us to do network based data analysis quickly, with intuitive syntax compared to using general purpose programming language like Python. But the general purpose languages have more flexibility, so a way to couple with them is provided through the plugin mechanism.
Please refer to the NADI Book for details on the key concepts, the syntax of the language, and other components of the NADI System.
For developers the API documentation is on docs.rs.
Video Demo: https://www.youtube.com/watch?v=qKsrigRrPKo Web User Manual: https://nadi-system.github.io/0.8.0/ PDF User Manual: https://nadi-system.github.io/0.8.0/data/nadi-book.pdf Dev Reference: https://docs.rs/nadi_core/latest/nadi_core/
Prebuilt binaries are available for windows in the releases page. Use the nadi-ide binary for the GUI, refer to Nadi Book for details on other binaries.
If you want to build it from source, which works on Linux, Windows, MacOS and Android (Termux, but without the IDE), clone this repository. And build it with cargo as follows:
git clone https://github.com/Nadi-System/nadi-system
cd nadi-system
cargo build --releaseThe compiled binaries will be in target/release directory. You can run nadi-ide from there for GUI, or nadi for CLI.
Or you can directly do the following:
cargo run --release --bin nadi-ideSome OS might not come with the system dependencies required for the NADI System.
One dependencies of nadi-ide (rfd) suggests the following dependencies:
| Distribution | Installation Command |
|---|---|
| Fedora | dnf install gtk3-devel |
| Arch | pacman -S gtk3 |
| Debian & Ubuntu | apt install libgtk-3-dev |
Besides that you might need the following for Debian:
build-essentiallibatk1.0-0libglib2.0-dev
If you cannot compile the program in your OS, please make an issue, even if you're able to solve it, that will help other people using the same OS.
The developer has only tested compilation on, Arch Linux and Windows OS.
You can run tests using the cargo command, this will run the tests in all the components as well as their documentation.
cargo testNadi System consists of the following components:
Core library in Rust, it has the basic data structures and the logic for the plugin system.
This is a macro library that facilitates writing nadi plugins.
Command Line Interface (CLI) to run nadi tasks file from terminal. Can visualize the parsed tasks, or run them.
Integrated Development Environment (IDE) for Nadi tasks. You can use this for writing your nadi tasks script, browse documentations, visualize network, run tasks, etc. You do not need to install any other program if you want to use Nadi by itself if you install it.
You can install this and run nadi from python, you can use python syntax to define your own functions as well as run the functions loaded from nadi plugins. This gives better flexibility for research purposes, and for prototyping.
There are some plugins that are given by default called internal plugins. And some you can get from nadi-plugins-rust repository.
Geographic Information System (GIS) tool for nadi. It can help download stream lines (NHDPlus), USGS streamgages, basins, etc as well as run network detection algorithm for detecting network that is the backbone of nadi system.
Nadi GIS is available as a command line utility as well as a QGIS plugin.
Nadi kernel is a custom kernel for jupyter. You can install this kernel and use NADI directly from Jupyter notebook.
To install use:
nadi-kernel --installThe screenshot below shows a sample run of NADI notebook. Note that it uses typst and cairo plugins, you can not reproduce it without them.
You can contribute to NADI System even without coding experience in Rust by reporting bugs, suggesting features, and helping with documentation.
Please refer to CONTRIBUTING.md for further details on specific roles and tasks you can do. For developers, refer to the Future Work section below for things I would appreciate help with (but please make an issue and tell me before you start).
And please refer to architecture.md file to read how the components of the NADI are arranged in this repository.
- Write more tests so we don't break anything important in future.
- Write more plugin functions. Depends on users use cases.
- Write example problems and solutions using NADI.
- Convert many node/network functions to env function by taking the attribute/series etc directly now.
- Convert any functions that take Vec or similar that could use
*argssyntax now.
- Write layout algorithm for the directed graph (not just trees), take algorithms from Graphviz and other papers.
- Implement more functionality in the
jupyter kernel. - Write a
nadi-serverCLI tool, that opens a server. It opens up an API where users can send tasks to run, and it can return the output from that. - Server should have
mutableandimmutableoption. In the immutable option users can only run immutable functions. - Write editor modes for nadi, maybe using
lspso users can expand it to their editors with minimal work.- Currently, besides the NADI IDE, there are some syntax highlight available for web (through highlight.js), and sublime syntax files in
extra/syntax-highlightdirectory. - There is no intelligent analysis of scripts, you can only run it to get errors. Better error handling while parsing, and then
- Currently, besides the NADI IDE, there are some syntax highlight available for web (through highlight.js), and sublime syntax files in
- Expose Series and TimeSeries to the nadi-py library
- Write converters for Series <-> numpy and timeseries <-> pandas/polars
- [OPT] Write a way to directly run a nadi repl from python
Advanced:
- Explore ways to make optional python plugin mechanism for nadi
Nadi will run normally except will have a underlying python interpreter, that has the python plugins loaded in it. Anytime we need to call the functions from python plugin, nadi will use the python types to communicate with them and return the values.
This mechanism will allow users to write simple python code in the nadi interface itself for faster prototyping compared to out current executable plugin mechanism. But we do need to consider if it is really worth the effort.
- Implement type check for series/timeseries and user defined functions. The type information is not used in the language itself, but just asserted once before assignment to avoid any errors.
- Make Template and Timeline as other types the functions/expressions can take and return. Maybe add t"temp {x}" type syntax for template string. It gets passed as Template instead of normal strings. That saves the computation required to parse the template all the time from String. Same for Timeline. But the timeline is already messed up, so idk what to do with it.
- Implement Structs, add Date and Time as structs, add structs to the expression result and function input as well. For function Input, the structs made in the same plugin can be used directly, otherwise it will be converted to AttrMap for compatibility. We need to think about how to implement that. Could be just struct with name and AttrMap for fields. Struct definitely needs types and default values.