Development

Overview

NeoDB is a Django project, and it runs side by side with a modified version of Takahe (a separate Django project, code in neodb_takahe as submodule). They communicate mainly thru database and task queue, the diagram in Docker Installation demostrate a typical architecture. Currently the two are loosely coupled, so you may take either one offline without immediate impact on the other, which makes it very easy to conduct maintenance and troubleshooting separately. In the future, they may get combined but it's not decided and will not be decided very soon.

Before writing code

Obviously a working version of local NeoDB instance has to be established first, see install guide. If you are to test anything related with ActivityPub, a setup with externally reachable real domain name might be required, using localhost may cause quite a few issues here and there.

Install development related packages

python3 -m pip install -r requirements-dev.txt

Install pre-commit hooks

$ pre-commit install
pre-commit installed at .git/hooks/pre-commit

Make sure code in submodule is updated

git submodule update --init

then install requirements and pre-commit hook for neodb-takahe project (using a different venv is preferred)

Migration

Always run python3 manage.py migrate for takahe first, before do that for neodb.

If models in takahe/models.py are changed, just regenerate takahe/migrations/0001_initial.py instead of adding incremental migrations, because these migrations will never be applied except for constructing a test database.

Run Test

python3 manage.py test will run the tests

Alternative you may create the test database from freshly created database:

CREATE DATABASE test_neodb WITH TEMPLATE neodb;

and run the test without re-create it every time

$ python3 manage.py test --keepdb

Using existing test database for alias 'default'...
System check identified no issues (2 silenced).
........................................................
----------------------------------------------------------------------
Ran 56 tests in 1.100s

OK
Preserving test database for alias 'default'...

Development in Docker

To run local source code with docker compose, add NEODB_DEBUG=True in .env or host environment, and use --profile dev instead of --profile production in commands, e.g.:

$ NEODB_DEBUG=True docker compose --profile dev up -d

The dev profile is different from production:

code in NEODB_SRC (default: .) and TAKAHE_SRC (default: ./neodb-takahe) will be mounted and used in the container instead of code in the image
runserver with autoreload will be used instead of gunicorn for both neodb and takahe web server
/static/ and /s/ url are not map to pre-generated/collected static file path, NEODB_DEBUG=True is required locate static files from source code
one rqworker container will be started, instead of two
use dev-shell and dev-root to invoke shells, instead of shell and root
there's no automatic migration container, but it can be triggered manually via docker compose run dev-shell neodb-init

Note:

Python virtual environments inside docker image, which are /neodb-venv and /takahe-venv, will be used by default. They can be changed to different locations with TAKAHE_VENV and NEODB_VENV if needed, usually in a case of development code using a package not in docker venv.
Some packages inside python virtual environments are platform dependent, so mount venv built by macOS host into the Linux container will likely not work.
Python servers are launched as app user, who has no write access to anywhere except /tmp and media path, that's by design.
Database/redis used in the container cluster are not accessible from host directly, which is by design. Querying them can be done by either apt update/install client packages in dev-root or root container, or create compose.override.yml to uncomment ports section.

To run local unit tests, use docker compose run dev-shell neodb-manage test

Development with Github Codespace

At the time of writing, docker compose will work in Github Codespace by adding this in .env:

NEODB_SITE_DOMAIN=${CODESPACE_NAME}-8000.${GITHUB_CODESPACES_PORT_FORWARDING_DOMAIN}

Applications

Main django apps for NeoDB:

users manages user in typical django fashion
mastodon this leverages Mastodon API and Twitter API for user login and data sync
catalog manages different types of items user may review, and scrapers to fetch from external resources, see catalog.md for more details
journal manages user created content(review/ratings) and lists(collection/shelf/tag), see journal.md for more details
social present timeline for local users, see social.md for more details
takahe communicate with Takahe (a separate Django server, run side by side with this server, code in neodb_takahe as submodule)
legacy this is only used by instances upgraded from 0.4.x and earlier, to provide a link mapping from old urls to new ones. If your journey starts with 0.5 and later, feel free to ignore it.

5.4 KiB Raw Blame History