Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,9 @@ uv run main.py --platform xhs --lt qrcode --type detail

# 打开对应APP扫二维码登录

# 开始爬取前,先检查已保存的 cookie / 代理是否还能用(只发一次请求,不会启动爬虫)
uv run main.py --platform xhs --check_session

# 其他平台爬虫使用示例,执行下面的命令查看
uv run main.py --help
```
Expand Down
3 changes: 3 additions & 0 deletions README_en.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,9 @@ uv run main.py --platform xhs --lt qrcode --type detail

# Open corresponding APP to scan QR code for login

# Before crawling, check whether the saved cookies/proxy still work (one request, no crawling)
uv run main.py --platform xhs --check_session

# For other platform crawler usage examples, execute the following command to view
uv run main.py --help
```
Expand Down
3 changes: 3 additions & 0 deletions README_es.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,9 @@ uv run main.py --platform xhs --lt qrcode --type detail

# Abrir la APP correspondiente para escanear código QR para login

# Antes de rastrear, compruebe si las cookies/proxy guardados siguen funcionando (una sola petición, sin rastreo)
uv run main.py --platform xhs --check_session

# Para ejemplos de uso de rastreador de otras plataformas, ejecute el siguiente comando para ver
uv run main.py --help
```
Expand Down
9 changes: 9 additions & 0 deletions cmd_arg/arg.py
Original file line number Diff line number Diff line change
Expand Up @@ -251,6 +251,14 @@ def main(
rich_help_panel="Account Configuration",
),
] = config.COOKIES,
check_session: Annotated[
bool,
typer.Option(
"--check_session",
help="Check whether the saved cookies/proxy still work, then exit without crawling",
rich_help_panel="Runtime Configuration",
),
] = False,
specified_id: Annotated[
str,
typer.Option(
Expand Down Expand Up @@ -412,6 +420,7 @@ def main(
headless=config.HEADLESS,
save_data_option=config.SAVE_DATA_OPTION,
init_db=init_db_value,
check_session=check_session,
cookies=config.COOKIES,
specified_id=specified_id,
creator_id=creator_id,
Expand Down
5 changes: 5 additions & 0 deletions docs/常见问题.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,11 @@ A:在config/base_config.py 中 XHS_SPECIFIED_ID_LIST 参数用于控制需要
Q: 刚开始能爬取数据,过一段时间就是失效了?<br>
A:出现这种情况多半是由于你的账号触发了平台风控机制了,❗️❗️请勿大规模对平台进行爬虫,影响平台。<br>

## 如何在开始爬取前确认登录态还在
Q: 每次都要跑起来才发现 cookie 过期 / 代理不通,有没有更快的办法?<br>
A:执行 `uv run main.py --platform xhs --check_session`。它只发一次登录态请求就退出,并告诉你失败的可能原因(cookie 失效、代理不可用、访问受限等)。全部通过时退出码为 0,有失败时为 1,方便定时任务判断。<br>
注意:抖音(dy)与百度贴吧(tieba)的登录态只能在浏览器里确认,预检对这两个平台只能给出 cookie 层面的线索,结果为“未知”。<br>

## 如何更换另一个账号
Q: 如何更换登录账号?<br>
A:删除项目根目录下的 brower_data/ 文件夹即可 <br>
Expand Down
8 changes: 8 additions & 0 deletions main.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@
from media_platform.xhs import XiaoHongShuCrawler
from media_platform.zhihu import ZhihuCrawler
from tools.async_file_writer import AsyncFileWriter
from tools.session_check import run_session_check
from var import crawler_type_var


Expand Down Expand Up @@ -101,6 +102,13 @@ async def main() -> None:
global crawler

args = await cmd_arg.parse_cmd()

# 预检只做一次登录态请求就退出,不建表、不启动爬虫。
# 放在 --init_db 之前:两个都传时,--init_db 的 return 会让预检被静默跳过。
if args.check_session:
exit_code = await run_session_check([config.PLATFORM])
raise SystemExit(exit_code)

if args.init_db:
await db.init_db(args.init_db)
print(f"Database {args.init_db} initialized successfully.")
Expand Down
Loading