
* fix scraping failure with wepb image (merge upstream/fix-webp-scrape) * add filetype to requirements * add proxycrawl.com as fallback for douban scraper * load 3p js/css from cdn * add fix-cover task * fix book/album cover tasks * scrapestack * bandcamp scrape and preview ; manage.py scrape <url> ; make ^C work when DEBUG * use scrapestack when fix cover * add user agent to improve compatibility * search BandCamp for music albums * add missing MovieGenre * fix search 500 when song has no parent album * adjust timeout * individual scrapers * fix tmdb parser * export marks via rq; pref to send public toot; move import to data page * fix spotify import * fix edge cases * export: fix dupe tags * use rq to manage doufen import * add django command to manage rq jobs * fix export edge case * tune rq admin * fix detail page 502 step 1: async pull mastodon follow/block/mute list * fix detail page 502 step 2: calculate relationship by local cached data * manual sync mastodon follow info * domain_blocks parsing fix * marks by who i follows * adjust label * use username in urls * add page to list a user\'s review * review widget on user home page * fix preview 500 * fix typo * minor fix * fix google books parsing * allow mark/review visible to oneself * fix auto sync masto for new user * fix search 500 * add command to restart a sync task * reset visibility * delete user data * fix tag search result pagination * not upgrade to django 4 yet * basic doc * wip: collection * wip * wip * collection use htmx * show in-collection section for entities * fix typo * add su for easier debug * fix some 500s * fix login using alternative domain * hide data from disabled user * add item to list from detail page * my tags * collection: inline comment edit * show number of ratings * fix collection delete * more detail in collection view * use item template in search result * fix 500 * write index to meilisearch * fix search * reindex in batch * fix 500 * show search result from meilisearch * more search commands * index less fields * index new items only * search highlights * fix 500 * auto set search category * classic search if no meili server * fix index stats error * support typesense backend * workaround typesense bug * make external search async * fix 500, typo * fix cover scripts * fix minor issue in douban parser * supports m.douban.com and customized bandcamp domain * move account * reword with gender-friendly and instance-neutral language * Friendica does not have vapid_key in api response * enable anonymous search * tweak book result template * API v0 API v0 * fix meilisearch reindex * fix search by url error * login via twitter.com * login via pixelfed * minor fix * no refresh on inactive users * support refresh access token * get rid of /users/number-id/ * refresh twitter handler automatically * paste image when review * support PixelFed (very long token) * fix django-markdownx version * ignore single quote for meilisearch for now * update logo * show book review/mark from same isbn * show movie review/mark from same imdb * fix login with older mastodon servers * import Goodreads book list and profile * add timestamp to Goodreads import * support new google books api * import goodreads list * minor goodreads fix * click corner action icon to add to wishlist * clean up duplicated code * fix anonymous search * fix 500 * minor fix search 500 * show rating only if votes > 5 * Entity.refresh_rating() * preference to append text when sharing; clean up duplicated code * fix missing data for user tagged view * fix page link for tag view * fix 500 when language field longer than 10 * fix 500 when sharing mark for song * fix error when reimport goodread profile * fix minor typo * fix a rare 500 * error log dump less * fix tags in marks export * fix missing param in pagination * import douban review * clarify text * fix missing sheet in review import * review: show in progress * scrape douban: ignore unknown genre * minor fix * improve review import by guess entity urls * clear guide text for review import * improve review import form text * workaround some 500 * fix mark import error * fix img in review import * load external results earlier * ignore search server errors * simplify user register flow to avoid inconsistent state * Add a learn more link on login page * Update login.html * show mark created timestamp as mark time * no 500 for api error * redirect for expired tokens * ensure preference object created. * mark collections * tag list * fix tag display * fix sorting etc * fix 500 * fix potential export 500; save shared links * fix share to twittwe * fix review url * fix 500 * fix 500 * add timeline, etc * missing status change in timeline * missing id in timeline * timeline view by default * workaround bug in markdownx... * fix typo * option to create new collection when add from detail page * add missing announcement and tags in timeline home * add missing announcement * add missing announcement * opensearch * show fediverse shared link * public review no longer requires login * fix markdownx bug * fix 500 * use cloudflare cdn * validate jquery load and domain input * fix 500 * tips for goodreads import * collaborative collection * show timeline and profile link on nav bar * minor tweak * share collection * fix Goodreads search * show wish mark in timeline * resync failed urls with local proxy * resync failed urls with local proxy: check proxy first * scraper minor fix * resync failed urls * fix fields limit * fix douban parsing error * resync * scraper minor fix * scraper minor fix * scraper minor fix * local proxy * local proxy * sync default config from neodb * configurable site name * fix 500 * fix 500 for anonymous user * add sentry * add git version in log * add git version in log * no longer rely on cdnjs.cloudflare.com * move jq/cash to _common_libs template partial * fix rare js error * fix 500 * avoid double submission error * import tag in lower case * catch some js network errors * catch some js network errors * support more goodread urls * fix unaired tv in tmdb * support more google book urls * fix related series * more goodreads urls * robust googlebooks search * robust search * Update settings.py * Update scraper.py * Update requirements.txt * make nicedb work * doc update * simplify permission check * update doc * update doc for bug report link * skip spotify tracks * fix 500 * improve search api * blind fix import compatibility * show years for movie in timeline * show years for movie in timeline; thinner font * export reviews * revert user home to use jquery https://github.com/fabiospampinato/cash/issues/246 * IGDB * use IGDB for Steam * use TMDB for IMDb * steam: igdb then fallback to steam * keep change history * keep change history: add django settings * Steam: keep localized title/brief while merging IGDB * basic Docker support * rescrape * Create codeql-analysis.yml * Create SECURITY.md * Create pysa.yml Co-authored-by: doubaniux <goodsir@vivaldi.net> Co-authored-by: Your Name <you@example.com> Co-authored-by: Their Name <they@example.com> Co-authored-by: Mt. Front <mfcndw@gmail.com>
362 lines
13 KiB
Python
362 lines
13 KiB
Python
import re
|
|
from decimal import *
|
|
from markdown import markdown
|
|
from django.utils.translation import gettext_lazy as _
|
|
from django.db import models, IntegrityError
|
|
from django.core.serializers.json import DjangoJSONEncoder
|
|
from django.db.models import Q, Count, Sum
|
|
from markdownx.models import MarkdownxField
|
|
from users.models import User
|
|
from django.utils import timezone
|
|
from django.conf import settings
|
|
|
|
|
|
RE_HTML_TAG = re.compile(r"<[^>]*>")
|
|
MAX_TOP_TAGS = 5
|
|
|
|
|
|
# abstract base classes
|
|
###################################
|
|
class SourceSiteEnum(models.TextChoices):
|
|
IN_SITE = "in-site", settings.CLIENT_NAME
|
|
DOUBAN = "douban", _("豆瓣")
|
|
SPOTIFY = "spotify", _("Spotify")
|
|
IMDB = "imdb", _("IMDb")
|
|
STEAM = "steam", _("STEAM")
|
|
BANGUMI = 'bangumi', _("bangumi")
|
|
GOODREADS = "goodreads", _("goodreads")
|
|
TMDB = "tmdb", _("The Movie Database")
|
|
GOOGLEBOOKS = "googlebooks", _("Google Books")
|
|
BANDCAMP = "bandcamp", _("BandCamp")
|
|
IGDB = "igdb", _("IGDB")
|
|
|
|
|
|
class Entity(models.Model):
|
|
|
|
rating_total_score = models.PositiveIntegerField(null=True, blank=True)
|
|
rating_number = models.PositiveIntegerField(null=True, blank=True)
|
|
rating = models.DecimalField(
|
|
null=True, blank=True, max_digits=3, decimal_places=1)
|
|
created_time = models.DateTimeField(auto_now_add=True)
|
|
edited_time = models.DateTimeField(auto_now=True)
|
|
last_editor = models.ForeignKey(
|
|
User, on_delete=models.SET_NULL, related_name='%(class)s_last_editor', null=True, blank=False)
|
|
brief = models.TextField(_("简介"), blank=True, default="")
|
|
other_info = models.JSONField(_("其他信息"),
|
|
blank=True, null=True, encoder=DjangoJSONEncoder, default=dict)
|
|
# source_url should include shceme, which is normally https://
|
|
source_url = models.URLField(_("URL"), max_length=500, unique=True)
|
|
source_site = models.CharField(_("源网站"), choices=SourceSiteEnum.choices, max_length=50)
|
|
|
|
class Meta:
|
|
abstract = True
|
|
constraints = [
|
|
models.CheckConstraint(check=models.Q(
|
|
rating__gte=0), name='%(class)s_rating_lowerbound'),
|
|
models.CheckConstraint(check=models.Q(
|
|
rating__lte=10), name='%(class)s_rating_upperbound'),
|
|
]
|
|
|
|
def get_absolute_url(self):
|
|
raise NotImplementedError("Subclass should implement this method")
|
|
|
|
@property
|
|
def url(self):
|
|
return settings.APP_WEBSITE + self.get_absolute_url()
|
|
|
|
def get_json(self):
|
|
return {
|
|
'title': self.title,
|
|
'brief': self.brief,
|
|
'rating': self.rating,
|
|
'url': self.url,
|
|
'cover_url': settings.APP_WEBSITE + self.cover.url,
|
|
'top_tags': self.tags[:5],
|
|
'category_name': self.verbose_category_name,
|
|
'other_info': self.other_info,
|
|
}
|
|
|
|
def save(self, *args, **kwargs):
|
|
""" update rating and strip source url scheme & querystring before save to db """
|
|
if self.rating_number and self.rating_total_score:
|
|
self.rating = Decimal(
|
|
str(round(self.rating_total_score / self.rating_number, 1)))
|
|
elif self.rating_number is None and self.rating_total_score is None:
|
|
self.rating = None
|
|
else:
|
|
raise IntegrityError()
|
|
super().save(*args, **kwargs)
|
|
|
|
def calculate_rating(self, old_rating, new_rating):
|
|
if (not (self.rating and self.rating_total_score and self.rating_number)
|
|
and (self.rating or self.rating_total_score or self.rating_number))\
|
|
or (not (self.rating or self.rating_number or self.rating_total_score) and old_rating is not None):
|
|
raise IntegrityError("Rating integiry error.")
|
|
if old_rating:
|
|
if new_rating:
|
|
# old -> new
|
|
self.rating_total_score += (new_rating - old_rating)
|
|
else:
|
|
# old -> none
|
|
if self.rating_number >= 2:
|
|
self.rating_total_score -= old_rating
|
|
self.rating_number -= 1
|
|
else:
|
|
# only one rating record
|
|
self.rating_number = None
|
|
self.rating_total_score = None
|
|
pass
|
|
else:
|
|
if new_rating:
|
|
# none -> new
|
|
if self.rating_number and self.rating_number >= 1:
|
|
self.rating_total_score += new_rating
|
|
self.rating_number += 1
|
|
else:
|
|
# no rating record before
|
|
self.rating_number = 1
|
|
self.rating_total_score = new_rating
|
|
else:
|
|
# none -> none
|
|
pass
|
|
|
|
def update_rating(self, old_rating, new_rating):
|
|
"""
|
|
@param old_rating: the old mark rating
|
|
@param new_rating: the new mark rating
|
|
"""
|
|
self.calculate_rating(old_rating, new_rating)
|
|
self.save()
|
|
|
|
def refresh_rating(self): # TODO: replace update_rating()
|
|
a = self.marks.filter(rating__gt=0).aggregate(Sum('rating'), Count('rating'))
|
|
if self.rating_total_score != a['rating__sum'] or self.rating_number != a['rating__count']:
|
|
self.rating_total_score = a['rating__sum']
|
|
self.rating_number = a['rating__count']
|
|
self.rating = a['rating__sum'] / a['rating__count'] if a['rating__count'] > 0 else None
|
|
self.save()
|
|
return self.rating
|
|
|
|
def get_tags_manager(self):
|
|
"""
|
|
Since relation between tag and entity is foreign key, and related name has to be unique,
|
|
this method works like interface.
|
|
"""
|
|
raise NotImplementedError("Subclass should implement this method.")
|
|
|
|
@property
|
|
def top_tags(self):
|
|
return self.get_tags_manager().values('content').annotate(tag_frequency=Count('content')).order_by('-tag_frequency')[:MAX_TOP_TAGS]
|
|
|
|
def get_marks_manager(self):
|
|
"""
|
|
Normally this won't be used.
|
|
There is no ocassion where visitor can simply view all the marks.
|
|
"""
|
|
raise NotImplementedError("Subclass should implement this method.")
|
|
|
|
def get_reviews_manager(self):
|
|
"""
|
|
Normally this won't be used.
|
|
There is no ocassion where visitor can simply view all the reviews.
|
|
"""
|
|
raise NotImplementedError("Subclass should implement this method.")
|
|
|
|
@property
|
|
def all_tag_list(self):
|
|
return self.get_tags_manager().values('content').annotate(frequency=Count('content')).order_by('-frequency')
|
|
|
|
@property
|
|
def tags(self):
|
|
return list(map(lambda t: t['content'], self.all_tag_list))
|
|
|
|
@property
|
|
def marks(self):
|
|
params = {self.__class__.__name__.lower() + '_id': self.id}
|
|
return self.mark_class.objects.filter(**params)
|
|
|
|
@classmethod
|
|
def get_category_mapping_dict(cls):
|
|
category_mapping_dict = {}
|
|
for subclass in cls.__subclasses__():
|
|
category_mapping_dict[subclass.__name__.lower()] = subclass
|
|
return category_mapping_dict
|
|
|
|
@property
|
|
def category_name(self):
|
|
return self.__class__.__name__
|
|
|
|
@property
|
|
def verbose_category_name(self):
|
|
raise NotImplementedError("Subclass should implement this.")
|
|
|
|
@property
|
|
def mark_class(self):
|
|
raise NotImplementedError("Subclass should implement this.")
|
|
|
|
@property
|
|
def tag_class(self):
|
|
raise NotImplementedError("Subclass should implement this.")
|
|
|
|
|
|
class UserOwnedEntity(models.Model):
|
|
is_private = models.BooleanField(default=False, null=True) # first set allow null, then migration, finally (in a few days) remove for good
|
|
visibility = models.PositiveSmallIntegerField(default=0) # 0: Public / 1: Follower only / 2: Self only
|
|
owner = models.ForeignKey(User, on_delete=models.CASCADE, related_name='user_%(class)ss')
|
|
created_time = models.DateTimeField(default=timezone.now)
|
|
edited_time = models.DateTimeField(default=timezone.now)
|
|
|
|
class Meta:
|
|
abstract = True
|
|
|
|
def is_visible_to(self, viewer):
|
|
if not viewer.is_authenticated:
|
|
return self.visibility == 0
|
|
owner = self.owner
|
|
if owner == viewer:
|
|
return True
|
|
if not owner.is_active:
|
|
return False
|
|
if self.visibility == 2:
|
|
return False
|
|
if viewer.is_blocking(owner) or owner.is_blocking(viewer) or viewer.is_muting(owner):
|
|
return False
|
|
if self.visibility == 1:
|
|
return viewer.is_following(owner)
|
|
else:
|
|
return True
|
|
|
|
def is_editable_by(self, viewer):
|
|
return True if viewer.is_staff or viewer.is_superuser or viewer == self.owner else False
|
|
|
|
@classmethod
|
|
def get_available(cls, entity, request_user, following_only=False):
|
|
# e.g. SongMark.get_available(song, request.user)
|
|
query_kwargs = {entity.__class__.__name__.lower(): entity}
|
|
all_entities = cls.objects.filter(**query_kwargs).order_by("-created_time") # get all marks for song
|
|
visible_entities = list(filter(lambda _entity: _entity.is_visible_to(request_user) and (_entity.owner.mastodon_username in request_user.mastodon_following if following_only else True), all_entities))
|
|
return visible_entities
|
|
|
|
@classmethod
|
|
def get_available_for_identicals(cls, entity, request_user, following_only=False):
|
|
# e.g. SongMark.get_available(song, request.user)
|
|
query_kwargs = {entity.__class__.__name__.lower() + '__in': entity.get_identicals()}
|
|
all_entities = cls.objects.filter(**query_kwargs).order_by("-created_time") # get all marks for song
|
|
visible_entities = list(filter(lambda _entity: _entity.is_visible_to(request_user) and (_entity.owner.mastodon_username in request_user.mastodon_following if following_only else True), all_entities))
|
|
return visible_entities
|
|
|
|
@classmethod
|
|
def get_available_by_user(cls, owner, is_following): # FIXME
|
|
"""
|
|
Returns all avaliable owner's entities.
|
|
Mute/Block relation is not handled in this method.
|
|
|
|
:param owner: visited user
|
|
:param is_following: if the current user is following the owner
|
|
"""
|
|
user_owned_entities = cls.objects.filter(owner=owner)
|
|
if is_following:
|
|
user_owned_entities = user_owned_entities.exclude(visibility=2)
|
|
else:
|
|
user_owned_entities = user_owned_entities.filter(visibility=0)
|
|
return user_owned_entities
|
|
|
|
@property
|
|
def item(self):
|
|
attr = re.findall(r'[A-Z](?:[a-z]+|[A-Z]*(?=[A-Z]|$))', self.__class__.__name__)[0].lower()
|
|
return getattr(self, attr)
|
|
|
|
|
|
# commonly used entity classes
|
|
###################################
|
|
class MarkStatusEnum(models.TextChoices):
|
|
WISH = 'wish', _('Wish')
|
|
DO = 'do', _('Do')
|
|
COLLECT = 'collect', _('Collect')
|
|
|
|
|
|
class Mark(UserOwnedEntity):
|
|
status = models.CharField(choices=MarkStatusEnum.choices, max_length=20)
|
|
rating = models.PositiveSmallIntegerField(blank=True, null=True)
|
|
text = models.CharField(max_length=5000, blank=True, default='')
|
|
shared_link = models.CharField(max_length=5000, blank=True, default='')
|
|
|
|
def __str__(self):
|
|
return f"Mark({self.id} {self.owner} {self.status.upper()})"
|
|
|
|
@property
|
|
def translated_status(self):
|
|
raise NotImplementedError("Subclass should implement this.")
|
|
|
|
@property
|
|
def tags(self):
|
|
tags = self.item.tag_class.objects.filter(mark_id=self.id)
|
|
return tags
|
|
|
|
class Meta:
|
|
abstract = True
|
|
constraints = [
|
|
models.CheckConstraint(check=models.Q(
|
|
rating__gte=0), name='mark_rating_lowerbound'),
|
|
models.CheckConstraint(check=models.Q(
|
|
rating__lte=10), name='mark_rating_upperbound'),
|
|
]
|
|
|
|
# TODO update entity rating when save
|
|
# TODO update tags
|
|
|
|
|
|
class Review(UserOwnedEntity):
|
|
title = models.CharField(max_length=120)
|
|
content = MarkdownxField()
|
|
shared_link = models.CharField(max_length=5000, blank=True, default='')
|
|
|
|
def __str__(self):
|
|
return self.title
|
|
|
|
def get_plain_content(self):
|
|
"""
|
|
Get plain text format content
|
|
"""
|
|
html = markdown(self.content)
|
|
return RE_HTML_TAG.sub(' ', html)
|
|
|
|
class Meta:
|
|
abstract = True
|
|
|
|
@property
|
|
def translated_status(self):
|
|
return '评论了'
|
|
|
|
|
|
class Tag(models.Model):
|
|
content = models.CharField(max_length=50)
|
|
|
|
def __str__(self):
|
|
return self.content
|
|
|
|
@property
|
|
def edited_time(self):
|
|
return self.mark.edited_time
|
|
|
|
@property
|
|
def created_time(self):
|
|
return self.mark.created_time
|
|
|
|
@property
|
|
def text(self):
|
|
return self.mark.text
|
|
|
|
@classmethod
|
|
def find_by_user(cls, tag, owner, viewer):
|
|
qs = cls.objects.filter(content=tag, mark__owner=owner)
|
|
if owner != viewer:
|
|
qs = qs.filter(mark__visibility__lte=owner.get_max_visibility(viewer))
|
|
return qs
|
|
|
|
@classmethod
|
|
def all_by_user(cls, owner):
|
|
return cls.objects.filter(mark__owner=owner).values('content').annotate(total=Count('content')).order_by('-total')
|
|
|
|
class Meta:
|
|
abstract = True
|