lib.itmens/common/importers/goodreads.py

203 lines
9.4 KiB
Python
Raw Normal View History

add all NeoDB features to NiceDB (#115) * fix scraping failure with wepb image (merge upstream/fix-webp-scrape) * add filetype to requirements * add proxycrawl.com as fallback for douban scraper * load 3p js/css from cdn * add fix-cover task * fix book/album cover tasks * scrapestack * bandcamp scrape and preview ; manage.py scrape <url> ; make ^C work when DEBUG * use scrapestack when fix cover * add user agent to improve compatibility * search BandCamp for music albums * add missing MovieGenre * fix search 500 when song has no parent album * adjust timeout * individual scrapers * fix tmdb parser * export marks via rq; pref to send public toot; move import to data page * fix spotify import * fix edge cases * export: fix dupe tags * use rq to manage doufen import * add django command to manage rq jobs * fix export edge case * tune rq admin * fix detail page 502 step 1: async pull mastodon follow/block/mute list * fix detail page 502 step 2: calculate relationship by local cached data * manual sync mastodon follow info * domain_blocks parsing fix * marks by who i follows * adjust label * use username in urls * add page to list a user\'s review * review widget on user home page * fix preview 500 * fix typo * minor fix * fix google books parsing * allow mark/review visible to oneself * fix auto sync masto for new user * fix search 500 * add command to restart a sync task * reset visibility * delete user data * fix tag search result pagination * not upgrade to django 4 yet * basic doc * wip: collection * wip * wip * collection use htmx * show in-collection section for entities * fix typo * add su for easier debug * fix some 500s * fix login using alternative domain * hide data from disabled user * add item to list from detail page * my tags * collection: inline comment edit * show number of ratings * fix collection delete * more detail in collection view * use item template in search result * fix 500 * write index to meilisearch * fix search * reindex in batch * fix 500 * show search result from meilisearch * more search commands * index less fields * index new items only * search highlights * fix 500 * auto set search category * classic search if no meili server * fix index stats error * support typesense backend * workaround typesense bug * make external search async * fix 500, typo * fix cover scripts * fix minor issue in douban parser * supports m.douban.com and customized bandcamp domain * move account * reword with gender-friendly and instance-neutral language * Friendica does not have vapid_key in api response * enable anonymous search * tweak book result template * API v0 API v0 * fix meilisearch reindex * fix search by url error * login via twitter.com * login via pixelfed * minor fix * no refresh on inactive users * support refresh access token * get rid of /users/number-id/ * refresh twitter handler automatically * paste image when review * support PixelFed (very long token) * fix django-markdownx version * ignore single quote for meilisearch for now * update logo * show book review/mark from same isbn * show movie review/mark from same imdb * fix login with older mastodon servers * import Goodreads book list and profile * add timestamp to Goodreads import * support new google books api * import goodreads list * minor goodreads fix * click corner action icon to add to wishlist * clean up duplicated code * fix anonymous search * fix 500 * minor fix search 500 * show rating only if votes > 5 * Entity.refresh_rating() * preference to append text when sharing; clean up duplicated code * fix missing data for user tagged view * fix page link for tag view * fix 500 when language field longer than 10 * fix 500 when sharing mark for song * fix error when reimport goodread profile * fix minor typo * fix a rare 500 * error log dump less * fix tags in marks export * fix missing param in pagination * import douban review * clarify text * fix missing sheet in review import * review: show in progress * scrape douban: ignore unknown genre * minor fix * improve review import by guess entity urls * clear guide text for review import * improve review import form text * workaround some 500 * fix mark import error * fix img in review import * load external results earlier * ignore search server errors * simplify user register flow to avoid inconsistent state * Add a learn more link on login page * Update login.html * show mark created timestamp as mark time * no 500 for api error * redirect for expired tokens * ensure preference object created. * mark collections * tag list * fix tag display * fix sorting etc * fix 500 * fix potential export 500; save shared links * fix share to twittwe * fix review url * fix 500 * fix 500 * add timeline, etc * missing status change in timeline * missing id in timeline * timeline view by default * workaround bug in markdownx... * fix typo * option to create new collection when add from detail page * add missing announcement and tags in timeline home * add missing announcement * add missing announcement * opensearch * show fediverse shared link * public review no longer requires login * fix markdownx bug * fix 500 * use cloudflare cdn * validate jquery load and domain input * fix 500 * tips for goodreads import * collaborative collection * show timeline and profile link on nav bar * minor tweak * share collection * fix Goodreads search * show wish mark in timeline * resync failed urls with local proxy * resync failed urls with local proxy: check proxy first * scraper minor fix * resync failed urls * fix fields limit * fix douban parsing error * resync * scraper minor fix * scraper minor fix * scraper minor fix * local proxy * local proxy * sync default config from neodb * configurable site name * fix 500 * fix 500 for anonymous user * add sentry * add git version in log * add git version in log * no longer rely on cdnjs.cloudflare.com * move jq/cash to _common_libs template partial * fix rare js error * fix 500 * avoid double submission error * import tag in lower case * catch some js network errors * catch some js network errors * support more goodread urls * fix unaired tv in tmdb * support more google book urls * fix related series * more goodreads urls * robust googlebooks search * robust search * Update settings.py * Update scraper.py * Update requirements.txt * make nicedb work * doc update * simplify permission check * update doc * update doc for bug report link * skip spotify tracks * fix 500 * improve search api * blind fix import compatibility * show years for movie in timeline * show years for movie in timeline; thinner font * export reviews * revert user home to use jquery https://github.com/fabiospampinato/cash/issues/246 * IGDB * use IGDB for Steam * use TMDB for IMDb * steam: igdb then fallback to steam * keep change history * keep change history: add django settings * Steam: keep localized title/brief while merging IGDB * basic Docker support * rescrape * Create codeql-analysis.yml * Create SECURITY.md * Create pysa.yml Co-authored-by: doubaniux <goodsir@vivaldi.net> Co-authored-by: Your Name <you@example.com> Co-authored-by: Their Name <they@example.com> Co-authored-by: Mt. Front <mfcndw@gmail.com>
2022-11-09 13:56:50 -05:00
import re
import requests
from lxml import html
from datetime import datetime
# from common.scrapers.goodreads import GoodreadsScraper
from common.scraper import get_scraper_by_url
from books.models import Book, BookMark
from collection.models import Collection
from common.models import MarkStatusEnum
from django.conf import settings
from user_messages import api as msg
import django_rq
from django.utils.timezone import make_aware
re_list = r'^https://www.goodreads.com/list/show/\d+'
re_shelf = r'^https://www.goodreads.com/review/list/\d+[^?]*\?shelf=[^&]+'
re_profile = r'^https://www.goodreads.com/user/show/(\d+)'
gr_rating = {
'did not like it': 2,
'it was ok': 4,
'liked it': 6,
'really liked it': 8,
'it was amazing': 10
}
class GoodreadsImporter:
@classmethod
def import_from_url(self, raw_url, user):
match_list = re.match(re_list, raw_url)
match_shelf = re.match(re_shelf, raw_url)
match_profile = re.match(re_profile, raw_url)
if match_profile or match_shelf or match_list:
django_rq.get_queue('doufen').enqueue(self.import_from_url_task, raw_url, user)
return True
else:
return False
@classmethod
def import_from_url_task(cls, url, user):
match_list = re.match(re_list, url)
match_shelf = re.match(re_shelf, url)
match_profile = re.match(re_profile, url)
total = 0
if match_list or match_shelf:
shelf = cls.parse_shelf(match_shelf[0], user) if match_shelf else cls.parse_list(match_list[0], user)
if shelf['title'] and shelf['books']:
collection = Collection.objects.create(title=shelf['title'],
description=shelf['description'] + '\n\nImported from [Goodreads](' + url + ')',
owner=user)
for book in shelf['books']:
collection.append_item(book['book'], book['review'])
total += 1
collection.save()
msg.success(user, f'成功从Goodreads导入包含{total}本书的收藏单{shelf["title"]}')
elif match_profile:
uid = match_profile[1]
shelves = {
MarkStatusEnum.WISH: f'https://www.goodreads.com/review/list/{uid}?shelf=to-read',
MarkStatusEnum.DO: f'https://www.goodreads.com/review/list/{uid}?shelf=currently-reading',
MarkStatusEnum.COLLECT: f'https://www.goodreads.com/review/list/{uid}?shelf=read',
}
for status in shelves:
shelf_url = shelves.get(status)
shelf = cls.parse_shelf(shelf_url, user)
for book in shelf['books']:
params = {
'owner': user,
'rating': book['rating'],
'text': book['review'],
'status': status,
'visibility': 0,
'book': book['book'],
}
if book['last_updated']:
params['created_time'] = book['last_updated']
params['edited_time'] = book['last_updated']
try:
mark = BookMark.objects.create(**params)
mark.book.update_rating(None, mark.rating)
except Exception:
print(f'Skip mark for {book["book"]}')
pass
total += 1
msg.success(user, f'成功从Goodreads用户主页导入{total}个标记。')
@classmethod
def parse_shelf(cls, url, user): # return {'title': 'abc', books: [{'book': obj, 'rating': 10, 'review': 'txt'}, ...]}
title = None
books = []
url_shelf = url + '&view=table'
while url_shelf:
print(f'Shelf loading {url_shelf}')
r = requests.get(url_shelf, timeout=settings.SCRAPING_TIMEOUT)
if r.status_code != 200:
print(f'Shelf loading error {url_shelf}')
break
url_shelf = None
content = html.fromstring(r.content.decode('utf-8'))
title_elem = content.xpath("//span[@class='h1Shelf']/text()")
if not title_elem:
print(f'Shelf parsing error {url_shelf}')
break
title = title_elem[0].strip()
print("Shelf title: " + title)
for cell in content.xpath("//tbody[@id='booksBody']/tr"):
url_book = 'https://www.goodreads.com' + \
cell.xpath(
".//td[@class='field title']//a/@href")[0].strip()
# has_review = cell.xpath(
# ".//td[@class='field actions']//a/text()")[0].strip() == 'view (with text)'
rating_elem = cell.xpath(
".//td[@class='field rating']//span/@title")
rating = gr_rating.get(
rating_elem[0].strip()) if rating_elem else None
url_review = 'https://www.goodreads.com' + \
cell.xpath(
".//td[@class='field actions']//a/@href")[0].strip()
review = ''
last_updated = None
try:
r2 = requests.get(
url_review, timeout=settings.SCRAPING_TIMEOUT)
if r2.status_code == 200:
c2 = html.fromstring(r2.content.decode('utf-8'))
review_elem = c2.xpath(
"//div[@itemprop='reviewBody']/text()")
review = '\n'.join(
p.strip() for p in review_elem) if review_elem else ''
date_elem = c2.xpath(
"//div[@class='readingTimeline__text']/text()")
for d in date_elem:
date_matched = re.search(r'(\w+)\s+(\d+),\s+(\d+)', d)
if date_matched:
last_updated = make_aware(datetime.strptime(date_matched[1] + ' ' + date_matched[2] + ' ' + date_matched[3], '%B %d %Y'))
else:
print(f"Error loading review{url_review}, ignored")
scraper = get_scraper_by_url(url_book)
url_book = scraper.get_effective_url(url_book)
book = Book.objects.filter(source_url=url_book).first()
if not book:
print("add new book " + url_book)
scraper.scrape(url_book)
form = scraper.save(request_user=user)
book = form.instance
books.append({
'url': url_book,
'book': book,
'rating': rating,
'review': review,
'last_updated': last_updated
})
except Exception:
print("Error adding " + url_book)
pass # likely just download error
next_elem = content.xpath("//a[@class='next_page']/@href")
url_shelf = ('https://www.goodreads.com' + next_elem[0].strip()) if next_elem else None
return {'title': title, 'description': '', 'books': books}
@classmethod
def parse_list(cls, url, user): # return {'title': 'abc', books: [{'book': obj, 'rating': 10, 'review': 'txt'}, ...]}
title = None
books = []
url_shelf = url
while url_shelf:
print(f'List loading {url_shelf}')
r = requests.get(url_shelf, timeout=settings.SCRAPING_TIMEOUT)
if r.status_code != 200:
print(f'List loading error {url_shelf}')
break
url_shelf = None
content = html.fromstring(r.content.decode('utf-8'))
title_elem = content.xpath('//h1[@class="gr-h1 gr-h1--serif"]/text()')
if not title_elem:
print(f'List parsing error {url_shelf}')
break
title = title_elem[0].strip()
description = content.xpath('//div[@class="mediumText"]/text()')[0].strip()
print("List title: " + title)
for link in content.xpath('//a[@class="bookTitle"]/@href'):
url_book = 'https://www.goodreads.com' + link
try:
scraper = get_scraper_by_url(url_book)
url_book = scraper.get_effective_url(url_book)
book = Book.objects.filter(source_url=url_book).first()
if not book:
print("add new book " + url_book)
scraper.scrape(url_book)
form = scraper.save(request_user=user)
book = form.instance
books.append({
'url': url_book,
'book': book,
'review': '',
})
except Exception:
print("Error adding " + url_book)
pass # likely just download error
next_elem = content.xpath("//a[@class='next_page']/@href")
url_shelf = ('https://www.goodreads.com' + next_elem[0].strip()) if next_elem else None
return {'title': title, 'description': description, 'books': books}