ci(accessibility-report): generate a report of full and partial accessibility violations

[sianford]

Proposed behaviour

Generate a report of failed and incomplete accessibility issues in HTML file playwright/test-report/accessibility-report.html when Playwright is run in CI mode. The HTML file should be opened in a browser by the user, providing a UI containing the following:

• Count of incomplete checks and violations
• Search and filtering options, to search by component name, test name, or issue description; filter by type (incomplete/violation) or severity (critical/serious/moderate/minor)
• Information for each issue, including the rule ID, impact level, description, help text, and a link to detailed documentation
• Which DOM elements are affected with their selectors and HTML
• Options to export the data as CSV or JSON to aid with logging bugs

The report is deleted each time the Playwright suite is run.

The Playwright README.md file is updated with details of where to find the report.

HTML opened in the browser:

Data downloaded as CSV:

Current behaviour

Incomplete accessibility checks report out in the console when the Playwright regresson suite is run in CI mode, as below:

…/flat-table.pw.tsx:1650:9 › Accessibility tests › should render in the transparent-base theme for accessibility tests

ACCESSIBILITY SCAN INCOMPLETE. Incomplete rules: 
        - aria-valid-attr-value: this is a critical accessibility issue. Ensure all ARIA attributes have valid values
Please check and ensure that the "flat-table" component meets accessibility criteria manually.

Unlike full accessibility failures, these do not report out in the Playwright run report.

Checklist

• Commits follow our style guide
• Related issues linked in commit messages if required
• Screenshots are included in the PR if useful
• All themes are supported if required
• Unit tests added or updated if required
• Playwright automation tests added or updated if required
• Storybook added or updated if required
• Translations added or updated (including creating or amending translation keys table in storybook) if required
• Typescript d.ts file added or updated if required
• Related docs have been updated if required

QA

• Tested in provided StackBlitz sandbox/Storybook
• Add new Playwright test coverage if required
• Carbon implementation matches Design System/designs
• UI Tests GitHub check reviewed if required

Additional context

A solution was initially provided in the form of JSON attachments (PR: #7647). However, this was not easy for QAs to extract the required information from which to log bugs, so a different solution was sought.

Testing instructions

1. Check out the branch locally and run npm install.
2. Run the entire Playwright regression suite in CI mode with command npm run test:ct.
3. Once the run has completed, verify the accessibility report has been generated and is in playwright/test-report.
4. Open the HTML file accessibility-report.html in a browser, either directly from VS Code or via your carbon directory structure on your machine.
5. Check the search and filtering options work as expected.
6. Test the csv export facility, with filtered and full data, and the JSON download.

[DipperTheDan]

suggestion: You might be better of just disabling the console stuff for the full file rather than line by line.

[paulrobinson88]

Looks good to me Sian, not sure if it's possible, but it would be handy if the search could be filtered to component only. For example, if I want to show only violations for the Menu component, searching for the word 'menu' will bring up any tests or components that include the word menu, like action-popover otherwise I think it will be very useful to us, plus we could filter by component in Excel using the csv I guess.