lib.itmens/boofilsic/settings.py
Henri Dickson 14b003a44a add all NeoDB features to NiceDB (#115)
* fix scraping failure with wepb image (merge upstream/fix-webp-scrape)

* add filetype to requirements

* add proxycrawl.com as fallback for douban scraper

* load 3p js/css from cdn

* add fix-cover task

* fix book/album cover tasks

* scrapestack

* bandcamp scrape and preview ;
manage.py scrape <url> ;
make ^C work when DEBUG

* use scrapestack when fix cover

* add user agent to improve compatibility

* search BandCamp for music albums

* add missing MovieGenre

* fix search 500 when song has no parent album

* adjust timeout

* individual scrapers

* fix tmdb parser

* export marks via rq; pref to send public toot; move import to data page

* fix spotify import

* fix edge cases

* export: fix dupe tags

* use rq to manage doufen import

* add django command to manage rq jobs

* fix export edge case

* tune rq admin

* fix detail page 502 step 1: async pull mastodon follow/block/mute list

* fix detail page 502 step 2: calculate relationship by local cached data

* manual sync mastodon follow info

* domain_blocks parsing fix

* marks by who i follows

* adjust label

* use username in urls

* add page to list a user\'s review

* review widget on user home page

* fix preview 500

* fix typo

* minor fix

* fix google books parsing

* allow mark/review visible to oneself

* fix auto sync masto for new user

* fix search 500

* add command to restart a sync task

* reset visibility

* delete user data

* fix tag search result pagination

* not upgrade to django 4 yet

* basic doc

* wip: collection

* wip

* wip

* collection use htmx

* show in-collection section for entities

* fix typo

* add su for easier debug

* fix some 500s

* fix login using alternative domain

* hide data from disabled user

* add item to list from detail page

* my tags

* collection: inline comment edit

* show number of ratings

* fix collection delete

* more detail in collection view

* use item template in search result

* fix 500

* write index to meilisearch

* fix search

* reindex in batch

* fix 500

* show search result from meilisearch

* more search commands

* index less fields

* index new items only

* search highlights

* fix 500

* auto set search category

* classic search if no meili server

* fix index stats error

* support typesense backend

* workaround typesense bug

* make external search async

* fix 500, typo

* fix cover scripts

* fix minor issue in douban parser

* supports m.douban.com and customized bandcamp domain

* move account

* reword with gender-friendly and instance-neutral language

* Friendica does not have vapid_key in api response

* enable anonymous search

* tweak book result template

* API v0

API v0

* fix meilisearch reindex

* fix search by url error

* login via twitter.com

* login via pixelfed

* minor fix

* no refresh on inactive users

* support refresh access token

* get rid of /users/number-id/

* refresh twitter handler automatically

* paste image when review

* support PixelFed (very long token)

* fix django-markdownx version

* ignore single quote for meilisearch for now

* update logo

* show book review/mark from same isbn

* show movie review/mark from same imdb

* fix login with older mastodon servers

* import Goodreads book list and profile

* add timestamp to Goodreads import

* support new google books api

* import goodreads list

* minor goodreads fix

* click corner action icon to add to wishlist

* clean up duplicated code

* fix anonymous search

* fix 500

* minor fix search 500

* show rating only if votes > 5

* Entity.refresh_rating()

* preference to append text when sharing; clean up duplicated code

* fix missing data for user tagged view

* fix page link for tag view

* fix 500 when language field longer than 10

* fix 500 when sharing mark for song

* fix error when reimport goodread profile

* fix minor typo

* fix a rare 500

* error log dump less

* fix tags in marks export

* fix missing param in pagination

* import douban review

* clarify text

* fix missing sheet in review import

* review: show in progress

* scrape douban: ignore unknown genre

* minor fix

* improve review import by guess entity urls

* clear guide text for review import

* improve review import form text

* workaround some 500

* fix mark import error

* fix img in review import

* load external results earlier

* ignore search server errors

* simplify user register flow to avoid inconsistent state

* Add a learn more link on login page

* Update login.html

* show mark created timestamp as mark time

* no 500 for api error

* redirect for expired tokens

* ensure preference object created.

* mark collections

* tag list

* fix tag display

* fix sorting etc

* fix 500

* fix potential export 500; save shared links

* fix share to twittwe

* fix review url

* fix 500

* fix 500

* add timeline, etc

* missing status change in timeline

* missing id in timeline

* timeline view by default

* workaround bug in markdownx...

* fix typo

* option to create new collection when add from detail page

* add missing announcement and tags in timeline home

* add missing announcement

* add missing announcement

* opensearch

* show fediverse shared link

* public review no longer requires login

* fix markdownx bug

* fix 500

* use cloudflare cdn

* validate jquery load and domain input

* fix 500

* tips for goodreads import

* collaborative collection

* show timeline and profile link on nav bar

* minor tweak

* share collection

* fix Goodreads search

* show wish mark in timeline

* resync failed urls with local proxy

* resync failed urls with local proxy: check proxy first

* scraper minor fix

* resync failed urls

* fix fields limit

* fix douban parsing error

* resync

* scraper minor fix

* scraper minor fix

* scraper minor fix

* local proxy

* local proxy

* sync default config from neodb

* configurable site name

* fix 500

* fix 500 for anonymous user

* add sentry

* add git version in log

* add git version in log

* no longer rely on cdnjs.cloudflare.com

* move jq/cash to _common_libs template partial

* fix rare js error

* fix 500

* avoid double submission error

* import tag in lower case

* catch some js network errors

* catch some js network errors

* support more goodread urls

* fix unaired tv in tmdb

* support more google book urls

* fix related series

* more goodreads urls

* robust googlebooks search

* robust search

* Update settings.py

* Update scraper.py

* Update requirements.txt

* make nicedb work

* doc update

* simplify permission check

* update doc

* update doc for bug report link

* skip spotify tracks

* fix 500

* improve search api

* blind fix import compatibility

* show years for movie in timeline

* show years for movie in timeline; thinner font

* export reviews

* revert user home to use jquery https://github.com/fabiospampinato/cash/issues/246

* IGDB

* use IGDB for Steam

* use TMDB for IMDb

* steam: igdb then fallback to steam

* keep change history

* keep change history: add django settings

* Steam: keep localized title/brief while merging IGDB

* basic Docker support

* rescrape

* Create codeql-analysis.yml

* Create SECURITY.md

* Create pysa.yml

Co-authored-by: doubaniux <goodsir@vivaldi.net>
Co-authored-by: Your Name <you@example.com>
Co-authored-by: Their Name <they@example.com>
Co-authored-by: Mt. Front <mfcndw@gmail.com>
2022-11-09 19:56:50 +01:00

364 lines
10 KiB
Python

"""
Django settings for boofilsic project.
Generated by 'django-admin startproject' using Django 3.0.5.
For more information on this file, see
https://docs.djangoproject.com/en/3.0/topics/settings/
For the full list of settings and their values, see
https://docs.djangoproject.com/en/3.0/ref/settings/
"""
import os
import psycopg2.extensions
from git import Repo
# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
# https://docs.djangoproject.com/en/3.2/releases/3.2/#customizing-type-of-auto-created-primary-keys
DEFAULT_AUTO_FIELD = 'django.db.models.AutoField'
# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/3.0/howto/deployment/checklist/
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = 'nbv58c^&b8-095(^)&_BV98596v)&CX#^$&%*^V5'
# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True
ALLOWED_HOSTS = ['*']
# To allow debug in template context
# https://docs.djangoproject.com/en/3.1/ref/settings/#internal-ips
INTERNAL_IPS = [
"127.0.0.1"
]
# Application definition
INSTALLED_APPS = [
'django.contrib.admin',
'hijack',
'hijack.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'django.contrib.humanize',
'django.contrib.postgres',
'django_sass',
'django_rq',
'simple_history',
'markdownx',
'management.apps.ManagementConfig',
'mastodon.apps.MastodonConfig',
'common.apps.CommonConfig',
'users.apps.UsersConfig',
'books.apps.BooksConfig',
'movies.apps.MoviesConfig',
'music.apps.MusicConfig',
'games.apps.GamesConfig',
'sync.apps.SyncConfig',
'collection.apps.CollectionConfig',
'timeline.apps.TimelineConfig',
'easy_thumbnails',
'user_messages',
'django_slack',
]
MIDDLEWARE = [
'django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'hijack.middleware.HijackUserMiddleware',
'simple_history.middleware.HistoryRequestMiddleware',
]
ROOT_URLCONF = 'boofilsic.urls'
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [],
'APP_DIRS': True,
'OPTIONS': {
'context_processors': [
'django.template.context_processors.debug',
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
# 'django.contrib.messages.context_processors.messages',
"user_messages.context_processors.messages",
'boofilsic.context_processors.site_info',
],
},
},
]
WSGI_APPLICATION = 'boofilsic.wsgi.application'
# Database
# https://docs.djangoproject.com/en/3.0/ref/settings/#databases
if DEBUG:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': os.environ.get('DB_NAME', 'test'),
'USER': os.environ.get('DB_USER', 'donotban'),
'PASSWORD': os.environ.get('DB_PASSWORD', 'donotbansilvousplait'),
'HOST': os.environ.get('DB_HOST', '172.18.116.29'),
'OPTIONS': {
'client_encoding': 'UTF8',
# 'isolation_level': psycopg2.extensions.ISOLATION_LEVEL_DEFAULT,
}
}
}
else:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'boofilsic',
'USER': 'doubaniux',
'PASSWORD': 'password',
'HOST': 'localhost',
'OPTIONS': {
'client_encoding': 'UTF8',
# 'isolation_level': psycopg2.extensions.ISOLATION_LEVEL_DEFAULT,
}
}
}
# Customized auth backend, glue OAuth2 and Django User model together
# https://docs.djangoproject.com/en/3.0/topics/auth/customizing/#authentication-backends
AUTHENTICATION_BACKENDS = [
'mastodon.auth.OAuth2Backend',
]
# Internationalization
# https://docs.djangoproject.com/en/3.0/topics/i18n/
LANGUAGE_CODE = 'zh-hans'
TIME_ZONE = 'Asia/Shanghai'
USE_I18N = True
USE_L10N = True
USE_TZ = True
if not DEBUG:
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True
SECURE_SSL_REDIRECT = True
SECURE_HSTS_PRELOAD = True
SECURE_HSTS_INCLUDE_SUBDOMAINS = True
SECURE_HSTS_SECONDS = 31536000
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'simple': {
'format': '{levelname} {asctime} {name}:{lineno} {message}',
'style': '{',
},
},
'handlers': {
'file': {
'level': 'INFO',
'class': 'logging.FileHandler',
'filename': os.path.join(BASE_DIR, 'log'),
'formatter': 'simple'
},
},
'root': {
'handlers': ['file'],
'level': 'INFO',
'propagate': True,
},
}
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/3.0/howto/static-files/
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'static/')
STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.ManifestStaticFilesStorage'
AUTH_USER_MODEL = 'users.User'
SILENCED_SYSTEM_CHECKS = [
"auth.W004", # User.username is non-unique
"admin.E404" # Required by django-user-messages
]
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'media/')
PROJECT_ROOT = os.path.abspath(os.path.dirname(__name__))
SITE_INFO = {
'site_name': 'NiceDB',
'support_link': 'https://github.com/doubaniux/boofilsic/issues',
'version_hash': None,
'settings_module': os.getenv('DJANGO_SETTINGS_MODULE'),
'sentry_dsn': None,
}
# Mastodon configs
CLIENT_NAME = os.environ.get('APP_NAME', 'NiceDB')
SITE_INFO['site_name'] = os.environ.get('APP_NAME', 'NiceDB')
APP_WEBSITE = os.environ.get('APP_URL', 'https://nicedb.org')
REDIRECT_URIS = APP_WEBSITE + "/users/OAuth2_login/"
# Path to save report related images, ends with slash
REPORT_MEDIA_PATH_ROOT = 'report/'
MARKDOWNX_MEDIA_PATH = 'review/'
BOOK_MEDIA_PATH_ROOT = 'book/'
DEFAULT_BOOK_IMAGE = os.path.join(BOOK_MEDIA_PATH_ROOT, 'default.svg')
MOVIE_MEDIA_PATH_ROOT = 'movie/'
DEFAULT_MOVIE_IMAGE = os.path.join(MOVIE_MEDIA_PATH_ROOT, 'default.svg')
SONG_MEDIA_PATH_ROOT = 'song/'
DEFAULT_SONG_IMAGE = os.path.join(SONG_MEDIA_PATH_ROOT, 'default.svg')
ALBUM_MEDIA_PATH_ROOT = 'album/'
DEFAULT_ALBUM_IMAGE = os.path.join(ALBUM_MEDIA_PATH_ROOT, 'default.svg')
GAME_MEDIA_PATH_ROOT = 'game/'
DEFAULT_GAME_IMAGE = os.path.join(GAME_MEDIA_PATH_ROOT, 'default.svg')
COLLECTION_MEDIA_PATH_ROOT = 'collection/'
DEFAULT_COLLECTION_IMAGE = os.path.join(COLLECTION_MEDIA_PATH_ROOT, 'default.svg')
SYNC_FILE_PATH_ROOT = 'sync/'
EXPORT_FILE_PATH_ROOT = 'export/'
# Allow user to login via any Mastodon/Pleroma sites
MASTODON_ALLOW_ANY_SITE = False
# Timeout of requests to Mastodon, in seconds
MASTODON_TIMEOUT = 30
MASTODON_CLIENT_SCOPE = 'read write follow'
#use the following if it's a new site
#MASTODON_CLIENT_SCOPE = 'read:accounts read:follows read:search read:blocks read:mutes write:statuses write:media'
MASTODON_LEGACY_CLIENT_SCOPE = 'read write follow'
# Tags for toots posted from this site
MASTODON_TAGS = '#NiceDB #NiceDB%(category)s #NiceDB%(category)s%(type)s'
# Emoji code in mastodon
STAR_SOLID = ':star_solid:'
STAR_HALF = ':star_half:'
STAR_EMPTY = ':star_empty:'
# Default password for each user. since password is not used any way,
# any string that is not empty is ok
DEFAULT_PASSWORD = 'ab7nsm8didusbaqPgq'
# Default redirect loaction when access login required view
LOGIN_URL = '/users/login/'
# Admin site root url
ADMIN_URL = 'tertqX7256n7ej8nbv5cwvsegdse6w7ne5rHd'
# Luminati proxy settings
LUMINATI_USERNAME = 'lum-customer-hl_nw4tbv78-zone-static'
LUMINATI_PASSWORD = 'nsb7te9bw0ney'
SCRAPING_TIMEOUT = 90
# ScraperAPI api key
SCRAPERAPI_KEY = 'wnb3794v675b8w475h0e8hr7tyge'
PROXYCRAWL_KEY = None
SCRAPESTACK_KEY = None
# Spotify credentials
SPOTIFY_CREDENTIAL = "NzYzNkYTE6MGQ0ODY0NTY2Y2b3n645sdfgAyY2I1ljYjg3Nzc0MjIwODQ0ZWE="
# IMDb API service https://imdb-api.com/
IMDB_API_KEY = "k23fwewff23"
# The Movie Database (TMDB) API Keys
TMDB_API3_KEY = "deadbeef"
TMDB_API4_KEY = "deadbeef.deadbeef.deadbeef"
# Google Books API Key
GOOGLE_API_KEY = 'deadbeef-deadbeef-deadbeef'
# IGDB
IGDB_CLIENT_ID = 'deadbeef'
IGDB_ACCESS_TOKEN = 'deadbeef'
# Thumbnail setting
# It is possible to optimize the image size even more: https://easy-thumbnails.readthedocs.io/en/latest/ref/optimize/
THUMBNAIL_ALIASES = {
'': {
'normal': {
'size': (200, 200),
'crop': 'scale',
'autocrop': True,
},
},
}
# THUMBNAIL_PRESERVE_EXTENSIONS = ('svg',)
if DEBUG:
THUMBNAIL_DEBUG = True
# https://django-debug-toolbar.readthedocs.io/en/latest/
# maybe benchmarking before deployment
REDIS_HOST = os.environ.get('REDIS_HOST', '127.0.0.1')
RQ_QUEUES = {
'mastodon': {
'HOST': REDIS_HOST,
'PORT': 6379,
'DB': 0,
'DEFAULT_TIMEOUT': -1,
},
'export': {
'HOST': REDIS_HOST,
'PORT': 6379,
'DB': 0,
'DEFAULT_TIMEOUT': -1,
},
'doufen': {
'HOST': REDIS_HOST,
'PORT': 6379,
'DB': 0,
'DEFAULT_TIMEOUT': -1,
}
}
RQ_SHOW_ADMIN_LINK = True
SEARCH_INDEX_NEW_ONLY = False
SEARCH_BACKEND = None
# SEARCH_BACKEND = 'MEILISEARCH'
# MEILISEARCH_SERVER = 'http://127.0.0.1:7700'
# MEILISEARCH_KEY = 'deadbeef'
# SEARCH_BACKEND = 'TYPESENSE'
# TYPESENSE_CONNECTION = {
# 'api_key': 'deadbeef',
# 'nodes': [{
# 'host': 'localhost',
# 'port': '8108',
# 'protocol': 'http'
# }],
# 'connection_timeout_seconds': 2
# }