diff --git a/docs/development.md b/docs/development.md index b24d9f4a..2c12909b 100644 --- a/docs/development.md +++ b/docs/development.md @@ -109,7 +109,7 @@ NEODB_LANGUAGE=zh-hans neodb-manage test Update translations: ``` -django-admin makemessages --no-wrap --no-obsolete -i neodb-takahe -a +django-admin makemessages --no-wrap --no-obsolete -i .venv neodb-takahe -a # edit .po files, run the following to make sure edits are correct django-admin makemessages --no-wrap --no-obsolete -i .venv -i neodb-takahe -a django-admin compilemessages -i .venv -i neodb-takahe @@ -118,6 +118,7 @@ django-admin compilemessages -i .venv -i neodb-takahe Development in Docker Compose ----------------------------- The `dev` profile is different from `production`: + - code in `NEODB_SRC` (default: .) and `TAKAHE_SRC` (default: ./neodb-takahe) will be mounted and used in the container instead of code in the image - `runserver` with autoreload will be used instead of `gunicorn` for both neodb and takahe web server - /static/ and /s/ url are not map to pre-generated/collected static file path, `NEODB_DEBUG=True` is required locate static files from source code @@ -126,13 +127,14 @@ The `dev` profile is different from `production`: - there's no automatic `migration` container, but it can be triggered manually via `docker compose run dev-shell neodb-init` Note: + - Python virtual environments inside docker image, which are `/neodb-venv` and `/takahe-venv`, will be used by default. They can be changed to different locations with `TAKAHE_VENV` and `NEODB_VENV` if needed, usually in a case of development code using a package not in docker venv. - Some packages inside python virtual environments are platform dependent, so mount venv built by macOS host into the Linux container will likely not work. - Python servers are launched as `app` user, who has no write access to anywhere except /tmp and media path, that's by design. - Database/redis/typesense used in the container cluster are not accessible from host directly, which is by design. Querying them can be done by one of the following: - - `neodb-manage dbshell` - - `neodb-shell redis-cli -h redis` - - create `compose.override.yml` to uncomment `ports` section. + - `neodb-manage dbshell` + - `neodb-shell redis-cli -h redis` + - or create `compose.override.yml` to uncomment `ports` section. - To expose the neodb and takahe web server directly, in the folder for configuration, create `compose.override.yml` with the following content: ``` @@ -159,11 +161,12 @@ NEODB_SITE_DOMAIN=${CODESPACE_NAME}-8000.${GITHUB_CODESPACES_PORT_FORWARDING_DOM Applications ------------ Main Django apps for NeoDB: + - `users` manages user in typical Django fashion - - `mastodon` this leverages [Mastodon API](https://docs.joinmastodon.org/client/intro/) ~and [Twitter API](https://developer.twitter.com/en/docs/twitter-api)~ for user login and data sync + - `mastodon` this leverages [Mastodon API](https://docs.joinmastodon.org/client/intro/), [Threads API](https://developers.facebook.com/docs/threads/) and [ATProto}(https://atproto.com) for user login and data sync - `catalog` manages different types of items user may collect, and scrapers to fetch from external resources, see [catalog.md](internals/catalog.md) for more details - - `journal` manages user created content(review/ratings) and lists(collection/shelf/tag), see [journal.md](internals/journal.md) for more details - - `social` present timeline and notification for local users, see [social.md](internals/social.md) for more details + - `journal` manages user created content(review/ratings) and lists(collection/shelf/tag/note), see [journal.md](internals/journal.md) for more details + - `social` present timeline and notification for local users - `takahe` communicate with Takahe (a separate Django server, run side by side with this server, code in `neodb_takahe` as submodule), see [federation.md](internals/federation.md) for customization of ActivityPub protocol - `legacy` this is only used by instances upgraded from 0.4.x and earlier, to provide a link mapping from old urls to new ones. If your journey starts with 0.5 and later, feel free to ignore it. diff --git a/docs/internals/catalog.md b/docs/internals/catalog.md index a0c0a5a1..6b1982f7 100644 --- a/docs/internals/catalog.md +++ b/docs/internals/catalog.md @@ -88,6 +88,7 @@ classDiagram Add a new site -------------- + - If official API is available for the site, it should be the preferred way to get data. - add a new value to `IdType` and `SiteName` in `catalog/common/models.py` - add a new file in `catalog/sites/`, a new class inherits `AbstractSite`, with: @@ -99,16 +100,14 @@ Add a new site * a classmethod `id_to_url()` * a method `scrape()` returns a `ResourceContent` object * `BasicDownloader` or `ProxiedDownloader` can used to download website content or API data. e.g. `content = BasicDownloader(url).download().html()` - * ... - - see existing files in `catalog/sites/` for more examples + * check out existing files in `catalog/sites/` for more examples - add an import in `catalog/sites/__init__.py` - add some tests to `catalog//tests.py` according to site type + add `DOWNLOADER_SAVEDIR = '/tmp'` to `settings.py` can save all response to /tmp - + run `python3 manage.py cat ` for debugging or saving response file to `/tmp`. Detailed code of `cat` is in `catalog/management/commands/cat.py` - + move captured response file to `test_data/`, except large/images files. Or if have to, use a smallest version (e.g. 1x1 pixel / 1s audio) + + run `neodb-manage cat ` for debugging or saving response file to `/tmp`. Detailed code of `cat` is in `catalog/management/commands/cat.py` + + move captured response file to `test_data/`, except large/images files. Or if have to, replace it with a smallest version (e.g. 1x1 pixel / 1s audio) + add `@use_local_response` decorator to test methods that should pick up these responses (if `BasicDownloader` or `ProxiedDownloader` is used) - run all the tests and make sure they pass - - Command: `python3 manage.py test [--keepdb]`. + - Command: `neodb-manage python3 manage.py test [--keepdb]`. - See [this issue](https://github.com/neodb-social/neodb/issues/5) if `lxml.etree.ParserError` occurs on macOS. - - add a site UI label to `common/static/scss/_sitelabel.scss` + - add a site UI label style to `common/static/scss/_sitelabel.scss`