Skip to content

benchmarks: add perf benchs framework and test suites - #477

Open
GermanAizek wants to merge 1 commit into
ubuntu:mainfrom
GermanAizek:integrate-benchmarks
Open

benchmarks: add perf benchs framework and test suites#477
GermanAizek wants to merge 1 commit into
ubuntu:mainfrom
GermanAizek:integrate-benchmarks

Conversation

@GermanAizek

Copy link
Copy Markdown
Contributor
  • Framework (benchmark-framework.h/c):

    • High-precision timing via clock_gettime(CLOCK_MONOTONIC)
    • Warmup cycles to eliminate cache and branch predictor cold starts
    • Full statistical profile: Mean, Median (p50), Min, Max, p95/p99 tail latencies, relative standard deviation (RSD% jitter), throughput (ops/s), and process RSS memory delta
    • Formatted terminal table output with color coding
    • Machine-readable JSON export (--output) and baseline comparison (--compare) for automated performance regression detection in CI/CD
  • Benchmark Suites:

    • config: single-file parsing (small, standard, large 50 seats), drop-in directory merging (5 and 20 conf.d files), seat pattern globbing resolution, and high-frequency key lookup throughput
    • user_list: algorithmic scaling of sorted user insertions (10, 100, 1000, and 5000 users), user lookup by name, UID and shell filtering, and ~/.dmrc parsing
    • xauthority: MIT-MAGIC-COOKIE-1 generation, atomic record writes, entry matching and updates in 10- and 50-entry files
    • xdmcp: packet serialization/deserialization (Request, Manage, Accept), roundtrip encode-decode throughput, and string formatting
  • Tooling and Build Integration:

    • CLI runner: benchmarks/lightdm-benchmark
    • Automation helper: benchmarks/run-benchmarks.sh (--quick, --full, etc.)
    • Makefile targets: 'make bench' and 'make bench-quick'

How to use benchmarks (fast tutorial)

========================================================================
HOW TO BUILD AND RUN THE BENCHMARKS
========================================================================
1. PREREQUISITES
----------------
The benchmark suite depends on standard GLib, GObject, and GIO development
libraries (already used by LightDM) and links against `common/libcommon.la`
and object files from `src/`:
  # Ensure the main tree is built first
  make -C common
  make -C src
2. QUICK START VIA ROOT MAKEFILE
--------------------------------
You can run the benchmarks directly from the project root directory:
  # Run full benchmark suite (~30 iterations per test)
  make bench
  # Run quick smoke benchmarks (2 warmups, 5 iterations)
  make bench-quick
3. USING THE AUTOMATION SCRIPT (`run-benchmarks.sh`)
---------------------------------------------------
The `benchmarks/run-benchmarks.sh` helper script wraps the runner binary
with convenient presets and handles automated compilation:
  # Run quick smoke test across all suites
  ./benchmarks/run-benchmarks.sh --quick
  # Run full benchmark suite
  ./benchmarks/run-benchmarks.sh --full
  # Run a specific benchmark suite (config, user_list, xauthority, xdmcp)
  ./benchmarks/run-benchmarks.sh --suite config
  ./benchmarks/run-benchmarks.sh --suite user_list
  ./benchmarks/run-benchmarks.sh --suite xauthority
  ./benchmarks/run-benchmarks.sh --suite xdmcp
  # Filter benchmarks by name substring
  ./benchmarks/run-benchmarks.sh --pattern sort_insert
  ./benchmarks/run-benchmarks.sh --pattern encode
  # Save a baseline before making code changes
  ./benchmarks/run-benchmarks.sh --save-baseline benchmarks/baseline.json
  # Compare performance after code changes against the baseline
  ./benchmarks/run-benchmarks.sh --compare benchmarks/baseline.json
  # Dump raw JSON results to stdout for CI/scripting
  ./benchmarks/run-benchmarks.sh --quick --json
  # List all registered benchmarks without executing
  ./benchmarks/run-benchmarks.sh --list
4. DIRECT CLI EXECUTION (`lightdm-benchmark`)
---------------------------------------------
The compiled binary `benchmarks/lightdm-benchmark` supports granular flags:
  ./benchmarks/lightdm-benchmark [OPTIONS]
  Options:
    -f, --filter=PATTERN      Filter by suite or benchmark name substring
    -s, --suite=SUITE         Filter by suite (config, user_list, xauthority, xdmcp)
    -n, --name=NAME           Filter by benchmark name
    -i, --iterations=N        Number of measured runs (default: suite-specific, 20-50)
    -w, --warmup=N            Number of warmup runs (default: suite-specific, 2-5)
    -o, --output=FILE.json    Save structured benchmark report to JSON file
    -c, --compare=BASE.json   Compare current run against baseline JSON
    -j, --json                Output JSON report directly to stdout
    -l, --list                List available benchmarks and exit
    -h, --help                Show help screen
========================================================================
BENCHMARK SUITES OVERVIEW
========================================================================
1. `config` (Configuration Engine):
   - `load_small`: Parse minimal INI config file (~10 lines).
   - `load_standard`: Parse realistic distribution config (~150 lines:
     LightDM core settings, [Seat:*], XDMCPServer, VNCServer).
   - `load_large_50seats`: Parse enterprise multi-seat config (~1000 lines,
     50 seat definitions).
   - `merge_5_files` / `merge_20_files`: Load base config and sequentially
     merge drop-in snippet files from `conf.d/`.
   - `seat_glob_resolve`: Resolution and pattern matching of seat names
     (`seat0`, `seat-usb1`) against globbed sections (`[Seat:*]`, `[Seat:seat*]`)
     via `g_pattern_match_simple()`.
   - `key_lookup_string` / `key_lookup_bool_int` / `key_has_key`: High-frequency
     throughput of configuration value queries.
2. `user_list` (User Management & Enumeration):
   - `sort_insert_10` / `100` / `1000` / `5000`: Evaluates list insertion
     and alphabetical display name sorting, highlighting the O(N^2) scaling
     characteristics of insertion sort under directory/LDAP scale.
   - `lookup_by_name`: Linear search throughput across 1,000 user records.
   - `filter_passwd_entries`: Filtering `/etc/passwd` entries against UID bounds,
     hidden shells (`/bin/false`, `/usr/sbin/nologin`), and hidden system accounts.
   - `dmrc_parse`: Parsing user session preferences from `~/.dmrc`.
3. `xauthority` (X Server Authorization):
   - `generate_cookie`: Generation of 128-bit MIT-MAGIC-COOKIE-1 tokens.
   - `write_single_record`: Atomic creation of authority files on disk.
   - `update_record_10_file` / `update_record_50_file`: Reading, matching by
     (family, address, number), updating, and re-serializing authority files.
4. `xdmcp` (XDMCP Network Protocol):
   - `encode_request` / `decode_request`: Binary serialization and deserialization
     of complex XDMCP Request packets (with connections list and auth data).
   - `encode_manage` / `decode_manage`: Manage packet serialization.
   - `encode_accept` / `decode_accept`: Accept packet serialization.
   - `roundtrip_request`: Full end-to-end cycle (encode -> decode -> free)
     measuring network packet processing bandwidth and ops/sec.
   - `packet_tostring`: String formatting overhead for logging/debugging.
========================================================================
METRICS EXPLANATION
========================================================================
- `Mean`: Average latency per operation.
- `Median (p50)`: 50th percentile latency (less affected by system outliers).
- `Min / Max`: Minimum and maximum observed run times.
- `p95 / p99`: Tail latencies (critical for UI smoothness and responsiveness).
- `RSD%`: Relative Standard Deviation (StdDev / Mean * 100%). Values < 5%
  indicate stable, reliable measurements.
- `Throughput`: Maximum throughput in operations per second (ops/s, K ops/s, M ops/s).
- `RSS Delta`: Change in process resident memory (KB) via `/proc/self/statm`.
========================================================================
WORKFLOW EXAMPLE: DETECTING REGRESSIONS IN A PR
========================================================================
  # Step 1: Check out main branch and save baseline
  git checkout main
  make bench-quick
  ./benchmarks/run-benchmarks.sh --save-baseline baseline.json
  # Step 2: Check out feature/optimization branch
  git checkout my-feature-branch
  make bench-quick
  # Step 3: Run comparison
  ./benchmarks/run-benchmarks.sh --compare baseline.json
The comparison table highlights speedups in green (FASTER) and performance
regressions exceeding 5% in red (SLOWER / REGRESSION).

- Framework (benchmark-framework.h/c):
  - High-precision timing via clock_gettime(CLOCK_MONOTONIC)
  - Warmup cycles to eliminate cache and branch predictor cold starts
  - Full statistical profile: Mean, Median (p50), Min, Max, p95/p99 tail
    latencies, relative standard deviation (RSD% jitter), throughput (ops/s),
    and process RSS memory delta
  - Formatted terminal table output with color coding
  - Machine-readable JSON export (--output) and baseline comparison (--compare)
    for automated performance regression detection in CI/CD

- Benchmark Suites:
  - config: single-file parsing (small, standard, large 50 seats), drop-in
    directory merging (5 and 20 conf.d files), seat pattern globbing resolution,
    and high-frequency key lookup throughput
  - user_list: algorithmic scaling of sorted user insertions (10, 100, 1000,
    and 5000 users), user lookup by name, UID and shell filtering, and
    ~/.dmrc parsing
  - xauthority: MIT-MAGIC-COOKIE-1 generation, atomic record writes, entry
    matching and updates in 10- and 50-entry files
  - xdmcp: packet serialization/deserialization (Request, Manage, Accept),
    roundtrip encode-decode throughput, and string formatting

- Tooling and Build Integration:
  - CLI runner: benchmarks/lightdm-benchmark
  - Automation helper: benchmarks/run-benchmarks.sh (--quick, --full, etc.)
  - Makefile targets: 'make bench' and 'make bench-quick'
  - Comprehensive documentation in benchmarks/README.md
@jpeisach

jpeisach commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

So... I don't think we (Neal and I) intended for LightDM to uh... "expand" as much, more of less just keep maintaining, add features

That being said while this is cool, I'm not sure if as a project LightDM should have this?

I invite you to discuss this on Matrix.

Also, if LLMs were used, please disclose it :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants