Phoenix

Table of COntents

• Overview
  • Infrastructure
• Development
  • Running Phoenix
    • With Docker
    • With Hypothesis
• Database
• API
• License

---

Overview

Feel free to skip ahead to Development, if you're more interested in technical implementation.

Phoenix is a web application to submit points to the Terms of Service; Didn't Read (ToS;DR) project. Think of a point as a succinct, easy-to-understand mini-conclusion sourced from the complex text of either Privacy Policy or Terms of Service agreements. For example, You can retrieve an archive of your data, or Your personal data is used for automated decision-making, profiling or AI training.

Points are used by ToS;DR to analyse Privacy Policies and Terms of Service agreements across the web, then grading online services based on their respect for users' privacy and personal data. By services, we mean websites and web applications. Think YouTube, or Amazon.

The heart of Phoenix lies in user-submitted points (thanks to our awesome community), which are reviewed for accuracy and approved by our dedicated team of curators. Users submit points by annotating a service's documents, e.g. Privacy Policy, which Phoenix obtains and stores through web-scraping.

Once a service has been comprehensively analysed in Phoenix, as determined by our curators, the grades appear on tosdr.org.

An A grade indicates that the service treats users with a high level of respect for their privacy and personal data. Grades D and E indicate that the services treat users without much respect for their privacy and personal data. Examples include GitHub and Facebook.

The grades are calculated based on an algorithm, which, in our case is really just a simple list of conditions that factor in the balance of any given service's points, the relationship of the points to each other, and certain thresholds. This algorithm is subject to change. You can read more about it on our forum. We discuss how it has changed here: Updating the algorithm for service grades – thoughts?

Infrastructure

Phoenix is a Ruby on Rails application with some Vue.js mixed in, using a PostgreSQL database.

For annotation, Phoenix relies on a customised instance of Hypothesis, including both forks of the Hypothesis web application repository and the Hypothesis client.

This is a new implementation, merged for use in production on May 17, 2023. That being said, it is still experimental.

When used, Phoenix also connects to two other services: Atlassian, which runs both our widget (helps us to monitor the application's health), and jrbit.de (where our databases live as well as our system for reporting errors).

---

Development

Requirements:

We rely on Docker as an attempt to be OS-agnostic.

• Docker and docker-compose

If you are not able to use Docker, you'll need to have the following dependencies installed:

• Ruby 3.4.9
• Rails (Phoenix uses version 7.1.x)
• Bundler 2.4.14
• Postgres 15
• Node.js (to access Yarn)

Please refer to the Quickstart Guide for more information on manual set-up. If anything is either missing or out of date, please let us know.

There are two ways to run Phoenix in development: with or without Hypothesis. Plenty of hacking around can be done without running Hypothesis, particularly if you just want to get your feet wet or see what the codebase is all about.

If you are not interested in running Hypothesis, please skip ahead to Running Phoenix.

NOTE: The use of Hypothesis within Phoenix is not supported without Docker

Running Phoenix

First of all, clone Phoenix using Git and enter the cloned directory:

git clone --recurse-submodules https://git.tosdr.org/tosdr/phoenix.git
cd phoenix

The following steps should be completed from the root of the cloned repository.

With Docker

If you have installed docker-compose, getting the application running involves five configuration commands, after which it can be started with a single command in the future.

To prepare the application, run the following commands to build the application and initialise the database:

docker-compose build
docker network create elasticsearch
docker network create dbs
docker-compose up
docker exec -it edittosdrorg_web_1 rails db:seed

NOTE: We use docker networks to facilitate development with Hypothesis. Hypothesis and Phoenix share a database, as well as an Elasticsearch instance:

• The dbs network is the shared database.
• The elasticsearch network is the shared Elasticsearch instance.

From then on, you can start the application by running:

docker-compose up

If you add or remove dependencies (e.g. Ruby Gems), add the following arguments:

docker-compose up --build --remove-orphans

Here's an explanation of what each step does:

1. docker-compose up

This will launch the following services:

• The Phoenix web application
• The database (postgres:11.5-alpine)
• Elasticsearch (needed to run Hypothesis)
• adminer (for inspecting the database)

The Phoenix web application runs on port 9090 (http://localhost:9090).

If you wish simply to run the web application and the database, you can launch these services on their own by running:

docker-compose up web db

• To debug application code using byebug, open a new terminal tab and attach to the running web application container:

  docker attach phoenix-web-1 # Or however the web container is named

• While we are in the process of developing our test coverage, our test environment also runs in Docker and relies on rspec-rails; you can launch tests by running:

  docker-compose -f docker-compose.yml -f docker-compose.test.yml run --rm web bundle exec rspec

• To access the Rails console, run:

  docker-compose run web rails c

2. Create your user via the sign-up page, or use one of the seeded users.

This will only work if you have seeded the database.

There are three seeded users: tosdr_admin, tosdr_curator, tosdr_user. You can log in with any one of the following email addresses (password is Justforseed1):

• admin@selfhosted.tosdr.org - Admin account
• curator@selfhosted.tosdr.org - Curator account
• user@selfhosted.tosdr.org - User account

3. If you create your own user, confirm it manually.

First you'll need to access the Rails console:

docker-compose run web rails c

Find, then confirm your user:

> user = User.find_by_username('your_username')
> user.confirm

4. Sign-in.

5. To debug the database, try the following command:

docker exec -it db psql -U postgres

Due to a bug in the seeds you will currently need to run:

insert into document_types values (0, 'terms of service', now(), now(), null, 1, 'approved');

To annotate a service, navigate to the services page from the top-right menu, choose a service, and click View Documents. Begin by highlighting a piece of text from this page (both H and the Hypothesis client must be running).

For a demonstration of how annotations work, feel free to inspect the video attached to this PR.

With Hypothesis

Part 1:

H is the Hypothesis web service and API. To use it with Phoenix, clone our fork of H in the same directory as the Phoenix clone, then change directory into the H repository.

The correct branch to work from is the phoenix-integration branch.

git clone --recurse-submodules https://github.com/tosdr/h.git ./h
cd h

ATTENTION: The official documentation for installing H can be found here. Consult these docs as needed.

Note the prerequisites:

Before installing your development environment you’ll need to install each of these prerequisites:

• Git
• Node.js and npm
  • For Linux:
    • If using Debian/Ubuntu: Follow nodejs.org’s instructions for installing node. The node version provided in the Ubuntu repositories is outdated.
    • For macOS: Use Homebrew to install node.
• Docker: Follow the instructions detailed here to install.
• pyenv:
  • For Linux: Read this document to install.
  • For macOS: Use Homebrew

Your Node version in the shell in which you are developing must be more recent than 12.20.0. To manage Node versions, we suggest using nvm.

With pyenv, you will need to install python version 3.8.12. From the h/ directory:

pyenv install 3.8.12
pyenv init

Then reload your shell:

pyenv shell 3.8.12

If pyenv has trouble finding the python binary, you may need to add configuration to either your .bashrc or your .zshrc as documented here.

After all is set, run:

make services

which launches the docker services needed to run H.

If this is your first time configuring and launching H, run:

make db # This will configure the database schema

If this is your first time, run make dev to install the dependencies (requires both node and yarn). Otherwise, it will start the server on port 5000 (http://localhost:5000).

The make dev command will not start H in debug mode, i.e. you will not be able to run pdb.set_trace(). To run H in debug mode, exit the make dev process and run:

tox -e dev --run-command 'gunicorn --paste conf/development-app.ini  'h:app:create_app' -b 0.0.0.0:5000'

You will have to exit and restart tox whenever changes are made to the code. Additionally, in debug mode, certain functionalities may be restricted. You will not be able to create and persist annotations from the client if H is running in debug mode, for example.

To launch the shell and poke around in the database, run:

make shell

Now you can create an admin user from the shell. You will need an admin user to set up OAuth between H and the Hypothesis client. Follow these instructions.

Finally, log in as admin, and configure OAuth. Instructions can be found here. Ensure that you export the CLIENT_URL and CLIENT_OAUTH_ID environment variables to the h/ directory, i.e. the same shell in which you launch the H dev server.

---

Part two:

First clone into the root of the Phoenix repository our fork of the Hypothesis client (use the phoenix-integration branch), then enter the directory:

git clone https://github.com/tosdr/client --branch=phoenix-integration ./client
cd client

If successful,run:

make dev

You will need a Node version that is more recent than 12.20.0. H has to be running. Instructions can be found here, if needed.

---

Database

This wiki has been deprecated. We are in the process of updating it.

All the details on the database schema can be found on the wiki.

---

API

All the details on the API can be found here

---

License

AGPL-3.0+ (GNU Affero General Public License, version 3 or later)