Implement Bitrix24Partners support (issue #70)#109
Open
KarlsonComplete wants to merge 53 commits into
Conversation
- Replace `docker-compose` with `docker compose` commands in `Makefile` - Remove deprecated `setBitrix24PartnerId` method from `Bitrix24Partner` class
…tory` instance and simplify `setPhone` logic in `Bitrix24Partner`.
…improved context.
…andler создания партнера.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request refactors the Bitrix24Partners module by renaming bitrix24PartnerId to bitrix24PartnerNumber and transitioning the repository to a composition-based approach. It also introduces email validation, updates PHP and PHPUnit version requirements, adds a maintainer skill definition, and improves testing infrastructure with a new partner builder. Feedback identifies inconsistent Docker command usage in the Makefile, suggests a more standard PHP version constraint, notes PSR-12 style regressions, recommends better HTML error handling, and identifies a typo in a directory name.
camaxtly
requested changes
Apr 19, 2026
…cked, MarkAsActive и Update.
…logic: - Add `Bitrix24PartnerNotFoundException` handling to UseCase Handlers (`Create`, `Update`, `Delete`, `MarkAsBlocked`, `MarkAsActive`) for better error management and logging. - Enhance field validation in `Update.Command` to ensure stricter checks. - Introduce `logoUrl` field in `Bitrix24Partner` entity and adjust related UseCases and tests. - Refactor tests to include new scenarios and improve data consistency.
- Replace `docker-compose` with `docker compose` in `Makefile`. - Introduce `bitrix24:partners:scrape:v2` command with updated logic. - Register `ScrapePartnersCommand_V2` in the console application.
- Integrate Monolog for detailed logging throughout the scraping process. - Refactor `ScrapePartnersCommand_V2` to enhance parsing logic and improve error handling. - Introduce support for fetching partner logos and storing results in a CSV file. - Streamline HTTP requests and implement configurable delays.
…`Makefile` and `console` setup.
…а, изменил сигнатуры методов сущности и добавил поддержку логотипа партнера.
KarlsonComplete
changed the title
camaxtly
reviewed
Apr 26, 2026
camaxtly
reviewed
Apr 26, 2026
camaxtly
requested changes
Apr 26, 2026
- Add `Uuid::v7()` support in `Bitrix24Partner` constructors and builders. - Improve validation for partner creation with `bitrix24PartnerNumber`. - Update Doctrine mapping for `bitrix24PartnerNumber` to use `unique` attribute.
…erBuilder: - Remove redundant `withBitrix24PartnerNumber` calls from tests. - Adjust `Bitrix24PartnerBuilder` to expand possible number range. - Reorder `clear()` in `Update.HandlerTest` for better test consistency.
…dize database truncation across functional tests.
- Implement stricter non-empty string checks for `site`, `email`, `openLineId`, `externalId`, and `logoUrl` in `Update.Command` and `Bitrix24Partner` entity. - Update associated tests to cover new validation scenarios.
…s, and refactor `Makefile` and `console` setup.
…aper` services and integrated them into `ScrapePartnersCommandV2` for improved modularity and reusability.
…ndling with new helper methods.
…ScrapePartnersCommandV2: добавлена обработка блокировок, настройки задержек, детектирование пустых страниц и обновлён механизм работы с доменом.
…rtnersCsvCommand, улучшил обработку пустых страниц и блокировок в ScrapePartnersCommandV2, добавил загрузку .env.local.
camaxtly
reviewed
May 10, 2026
camaxtly
reviewed
May 10, 2026
camaxtly
reviewed
May 10, 2026
camaxtly
reviewed
May 10, 2026
camaxtly
requested changes
May 10, 2026
…ментацию по сценарию импорта партнёров, переработал ScrapePartnersCommand и UpdatePartnersCommand для синхронизации с новой спецификацией.
- Implement `Command` and `Handler` classes for upsert functionality. - Add validation for required fields in `Upsert\Command`. - Introduce `Upsert\Handler` to handle create, update, or skip logic based on partner data. - Update documentation to include new upsert scenarios and sync modes (`full`, `partial`). - Add unit and functional tests to cover all upsert scenarios, including creation, update, and skip cases. - Refactor partner import logic to align with the new upsert behavior.
…arser dependency: - Move PartnerHtmlParser logic into PartnerPageScraper as a dependency. - Simplify scraper command implementations (`UpdatePartnersCommand`, `ScrapePartnersCommand`) by using the consolidated `fetchPartnerData` method. - Introduce verbosity-based output control in `ScrapePartnersCommand`. - Add `PartnerData` DTO to structure scraped partner data. - Update tests to align with new scraping logic and enhance maintainability.
…ity: - Introduce `ScrapeWorkflow` to centralize scraping flow and state management. - Add contextual verbosity-based output handling to `ScrapePartnersCommand` and `UpdatePartnersCommand`. - Replace raw data arrays with `ScrapeResult` and `PartnerData` DTOs for better type safety. - Streamline `ScrapeStateManager` to improve progress tracking and state resumption. - Refactor utility methods across `PartnerPageScraper` for better extensibility and logging.
…bility: - Move scraper-related DTOs (`PartnerData`, `ScrapeResult`, `ScrapeConfig`) to `UseCase\Scrape` namespace. - Replace `ScrapeWorkflow` and `UpdateWorkflow` in `Infrastructure` with modular workflows under `UseCase\Scrape`. - Refactor `UpdatePartnersCommand` to use `UpdateWorkflow` and streamline options. - Improve type safety and replace raw arrays with DTOs across scraping functionality. - Migrate tests to match namespace changes and cover new workflows. - Remove deprecated scraper classes (`ScrapeWorkflow`, legacy DTOs).
- Standardize variable naming (`$e` to `$throwable`) for consistency. - Simplify conditional branches in `ScrapePartnersCommand`. - Add minor formatting adjustments (empty lines) for better code clarity.
camaxtly
reviewed
May 17, 2026
…raper commands refactoring.
…gional configurations: - Introduce `BanDetector` to identify scraping bans based on empty page patterns. - Replace `base-domain` inputs with `zone` (`ru`, `kz`) to centralize region handling. - Refactor workflows (`ScrapeWorkflow`, `UpdateWorkflow`) to integrate ban detection and handle new `zone` parameter. - Adjust commands (`ScrapePartnersCommand`, `UpdatePartnersCommand`) for streamlined regional scraping and improved verbosity. - Update DTOs (`ScrapeConfig`, `PartnerData`, `ScrapeResult`) to incorporate zone-specific logic and skipped partner tracking. - Migrate CSV file format to include `zone` instead of `base_domain`. - Add tests and documentation updates for new functionality and input changes.
…Bitrix24 PHP library maintainer skill files.
…apeWorkflow` for simplified CSV handling. Update `Bitrix24Zone` base domain URLs.
…ity, and improve progress reporting. Update Bitrix24 zone handling for domain and URL resolution.
…Command`: `partner-delay` и `page-delay` заменены на `partner-detail-delay` и `catalog-page-delay` соответственно, добавил корректную обработку заголовков карточек партнёров в парсере и обновил документацию о новой структуре опций.
…вил документацию по командам скрейпинга и импорта партнёров.
…ацию всех пунктов, включая BanDetector, processPartner, Bitrix24Zone enum, и обновление документации.
camaxtly
requested changes
May 24, 2026
| - Один партнёр однозначно определяется по номеру партнёра в Bitrix24 | ||
| - Команда: `bitrix24:partners:import` | ||
|
|
||
| ## Четыре сценария импорта |
Collaborator
There was a problem hiding this comment.
Если у нас выгрузка в csv может быть полная и не полная, то и сценарии будут разные.
|
|
||
| ## Четыре сценария импорта | ||
|
|
||
| ### Сценарий 1: Создание нового партнёра |
Collaborator
There was a problem hiding this comment.
Собери всю инфу по сценариям в 1-м месте и будет более читаемо.
| ├── ScrapeStateManager.php # Управление state-файлами (resume) | ||
| ``` | ||
|
|
||
| ## Общие принципы скрейпинга |
Collaborator
There was a problem hiding this comment.
Перефразировать и главное объяснить зачем и как это используется
| | Опция | Описание | По умолчанию | | ||
| |-------|----------|--------------| | ||
| | `--base-url` | URL страницы списка партнёров | `https://www.bitrix24.ru/partners/country__19/` | | ||
| | `--output-file` | Путь к выходному CSV файлу | `partners.csv` | |
Collaborator
There was a problem hiding this comment.
больше контекста для дефолта --- язык + таймстэмп в название файла, путь от корня или документ рута
| | `--partner-delay` | Задержка между партнёрами (сек) | `2` | | ||
| | `--insecure` | Отключить проверку SSL | `false` | | ||
| | `--resume` | Продолжить с места обрыва | `false` | | ||
| | `--full-refresh` | Перезаписать существующий файл | `false` | |
Collaborator
There was a problem hiding this comment.
кажется, это приведет только к проблемам
|
|
||
| --- | ||
|
|
||
| ### Сценарий 4: Обновление конкретных партнёров по ID |
|
|
||
| --- | ||
|
|
||
| ### Сценарий 5: Полный пайплайн (скрейп → импорт) |
Collaborator
There was a problem hiding this comment.
Весь импорт в доку по импорту
|
|
||
| --- | ||
|
|
||
| ### Сценарий 6: Точечное обновление (update → partial import) |
Collaborator
There was a problem hiding this comment.
Импорт в доку по импорту
|
|
||
| --- | ||
|
|
||
| ## Запуск через Docker (Makefile) |
| | `--resume` | Продолжить с места обрыва (из state-файла) | `false` | | ||
| | `--full-refresh` | Перезаписать существующий файл | `false` | | ||
|
|
||
| ### Сценарий `partners:update` |
Collaborator
There was a problem hiding this comment.
Объединить с scrape, просто добавить в команду --partner-ids`
…r update logic into `ScrapeWorkflow` and `ScrapePartnersCommand` for improved modularity and maintainability. Update ScrapeConfig to support specific partner updates. Streamline progress tracking and error handling.
…r file organization: - Replace `outputFile` with `outputDir` across configurations, workflows, and commands. - Introduce timestamped filenames for output files. - Update state management to support directory-based operations. - Adjust tests, Makefile commands, and verbosity outputs accordingly.
…scrape configuration to introduce `requestDelay`. Simplify `ScrapeWorkflow` start context resolution and improve state management consistency.
- Remove `.opencode/plans/2026-05-26-fix-output-dir-resume-logic.md` to declutter repository. - Refactor `ScrapeConfig` to use nullable `outputFile` and remove auto-generation. - Update `ScrapeStateManager` to store full `output_file` paths and add output-dir validation on resume. - Split `resolveStartContext` into `resolveResumeContext` and `getPageRange` in `ScrapeWorkflow`. - Refactor `ScrapePartnersCommand` for single-pass output-file resolution and verbosity improvements. - Introduce `ScrapeOptions` for streamlined scraping configurations.
…ortStats`, update method signatures, and streamline CSV file handling and import command default directory.
…leted partners, add option to include deleted in queries, and improve logging for skipped operations.
…enarios in `partner-scenarios.md` - Introduce `partner-common.md` with guidelines for CSV format, ban detection, and partner handling. - Simplify and consolidate `partner-scenarios.md` to focus on key scraping use cases. - Replace per-scenario options with centralized documentation for `partners:scrape` command. - Update examples and workflows to reflect streamlined scraping architecture and functionality.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This commit implements comprehensive support for Bitrix24Partners following the established DDD/CQRS patterns in the codebase.
Features Implemented:
Core Domain
Use Cases (CQRS)
CLI Tools
Testing
Technical Details:
Resolves: #70