Fix Android bugreport parser coverage

[besendorf]

Summary

This PR makes Android bugreport parsing match the data that is actually present in the Honor and Samsung reports used for the audit. It fixes parsers that stopped at the wrong boundary, parsed only one subsection, dropped per-user or per-UID context, or assigned values to the wrong record.

The result contract is intentionally cleaned up instead of preserving the old inconsistent shapes. Bugreport output is therefore marked as schema version 2.

The PR also fixes the repository-wide mypy failures caused by invariant module-registry list types. Those annotations do not change runtime module selection.

Compatibility and review notes

• This is a breaking bugreport output change. Consumers should use output_schema_version: 2 from the command metadata to identify the new contract.
• Resolver results are now flat records with a shared shape instead of activity-specific fields or a receiver dictionary keyed by intent.
• Package data is now typed and user-scoped. Version and SDK fields are integers when Android emitted integers; booleans and null are no longer retained as strings.
• Battery Daily fields were renamed to describe their meaning: from/to become period_start/period_end, and vers becomes version_code.
• Tombstones produce one canonical result per crash ID, even when both plaintext and protobuf files exist. Both representations are still parsed and retained.
• Filesystem archive timestamps retain their local value and also expose timezone provenance and a UTC conversion when the device timezone is known.

Detailed changes

Output contract and section extraction

Files: src/mvt/common/command.py, src/mvt/android/cmd_check_bugreport.py, and src/mvt/android/modules/bugreport/base.py.

• Add optional output_schema_version command metadata and set Android bugreport analysis to version 2.
• Add a shared extract_command_section() helper for non-dumpsys command blocks such as system properties and process tables.
• Stop a command section at any dumpstate separator line beginning with ------, including separators that contain command timing text.

Why: the previous GetProp extraction looked for a line equal to ------. Real reports use decorated separator lines, so the parser could consume the rest of dumpstate. The explicit schema version makes the intentional normalized field changes detectable by downstream consumers.

Activity and receiver resolver tables

Files: src/mvt/android/artifacts/package_resolvers.py, dumpsys_package_activities.py, dumpsys_receivers.py, and the receiver bugreport wrapper.

• Introduce one resolver-table parser shared by activities and receivers.
• Parse all six resolver categories Android emits:
  • Full MIME Types
  • Base MIME Types
  • Wild MIME Types
  • Schemes
  • Non-Data Actions
  • MIME Typed Actions
• Emit one record per component with resolver_type, key, package_name, component, and filter_count.
• Keep the receiver security logging and IOC checks, but apply them to the normalized list records.

Why: both old parsers only entered the Non-Data Actions subsection. MIME and scheme registrations in the reports were silently lost. The receiver parser also returned a different container shape from the activity parser, which made consumers handle equivalent Android structures differently.

Accessibility

Files: src/mvt/android/artifacts/dumpsys_accessibility.py and its bugreport wrapper.

• Parse installed, enabled, binding, bound, and crashed service sets.
• Support both inline modern service sets and older multiline output.
• Track the Android user ID and deduplicate services by (user_id, component).
• Split the component into package_name and service_name.
• Add boolean state fields plus accessibility_tool for Android's (A11yTool) marker.

Why: the old parser handled either installed services or one modern enabled-services line and emitted inconsistent service values. It could not show whether a service was merely installed, actively enabled or bound, crashed, or associated with another Android user.

ADB state and trusted keys

File: src/mvt/android/artifacts/dumpsys_adb.py.

• Recognize Android Binary XML keystores beginning with ABX\x00.
• Recover valid embedded base64 public keys, deduplicate them, and calculate the same fingerprint and user metadata used for plaintext/XML keys.
• Use last_connected: null when ABX numeric metadata cannot be recovered reliably.
• Normalize key material and other byte values to JSON-safe strings and convert textual booleans to booleans.

Why: ADB keystores in the audited reports were ABX rather than plaintext XML. The old code left that data as raw bytes and did not expose the trusted keys. The recovery deliberately retains only fields MVT can parse reliably instead of inventing values from damaged binary tokens.

AppOps

File: src/mvt/android/artifacts/dumpsys_appops.py.

• Parse every Uid ... block rather than depending on a Uid 0 starting point.
• Retain UID-level state, capability, appWidgetVisible, and default operation modes on each package.
• Give every operation a stable name, mode, and entries shape.
• Retain access/reject event type, UID state, absolute timestamp, relative time, duration, attribution tag, and running-state entries.
• Make serialization and risky-permission alerts use the normalized event field and tolerate operations without timestamped entries.

Why: the previous state machine lost UID context and several event attributes, could associate trailing entries with the wrong package or permission, and omitted running operations. These are fields already present in dumpsys and useful for understanding how and when an operation was used.

Battery Daily

File: src/mvt/android/artifacts/dumpsys_battery_daily.py.

• Preserve full daily period timestamps instead of truncating them to dates.
• Rename them to period_start and period_end.
• Parse numeric version codes as integers and rename vers to version_code.
• Retain duplicate update lines through an occurrences count rather than silently discarding them.
• Mark records explicitly as update, uninstall, or downgrade.
• Store previous_version_code as an integer for downgrade records.
• Sort periods chronologically before comparing package versions and clear prior version state after an uninstall.

Why: the report contains time-of-day information and repeated update records that were previously lost. Normalized integer versions prevent string-shaped data from leaking into comparisons and output, while explicit actions make the result self-describing.

Battery History

File: src/mvt/android/artifacts/dumpsys_battery_history.py.

• Limit parsing to the Battery History block.
• Parse all job and top-state tokens found on a line rather than stopping after the first matching event.
• Parse wake-lock events independently.
• Support wall-clock entries and elapsed-time entries anchored by RESET:TIME/TIME markers.
• Derive an absolute timestamp for elapsed events when an anchor is available while retaining the original time_elapsed.
• Normalize WorkManager-decorated job service names and derive package names from job and wake-lock forms where possible.
• Allow events without a derivable package name and skip only package IOC matching for those records.

Why: the old if/elif parser emitted at most one event per line, recognized only narrow job/wake formats, and did not reconstruct timestamps from the anchors already present in the report. Decorated scheduler names also produced incorrect package names.

Database operations

File: src/mvt/android/artifacts/dumpsys_dbinfo.py.

• Continue across every connection and operation list in a database pool.
• Parse connection_number and is_primary.
• Emit typed pid and duration_ms values.
• Retain timestamp, action, status, SQL text, the operation path, and the enclosing pool_path.
• Use the pool path when an operation does not repeat its path.

Why: the previous parser effectively handled only a narrow operation line and cleared pool state too early, dropping later connections. Its regex also discarded duration, status, connection metadata, and explicit per-operation paths that Android had already provided.

Packages and permissions

Files: src/mvt/android/artifacts/dumpsys_packages.py and its bugreport wrapper.

• Parse both Packages: and Hidden system packages:, recording package_type as active or hidden_system.
• Parse app_id, version name/code, min/target SDK, install/update timestamps, and installer.
• Convert integers, booleans, and null to their corresponding Python/JSON types.
• Retain declared, requested, install, and runtime permissions with permission type, grant state, and flags.
• Keep per-user records with user ID, install/enabled/stopped/hidden/suspended state, install/uninstall reasons, data directory, first-install time, and runtime permissions.
• Attach legacy top-level firstInstallTime to user 0 when the report uses the older layout.
• Emit timeline entries for package install, last update, and each available per-user first-install timestamp.
• Include per-user runtime permissions when calculating dangerous-permission counts.
• Deduplicate rooting-package alerts without suppressing IOC checks for the same package.

Why: the old parser stopped before hidden packages, flattened user-specific state, treated typed values as strings, and could place runtime permissions into the wrong scope. That made multi-user reports ambiguous and caused valid package/permission data to disappear.

Platform compatibility overrides

Files: src/mvt/android/artifacts/dumpsys_platform_compat.py and its bugreport wrapper.

• Parse every ChangeId(... rawOverrides={...}) record instead of only change ID 168419799.
• Emit change_id, optional change_name, enabled/disabled change_state, overridable, package_name, and typed override_value.
• Rename module messaging so the records are described as compatibility overrides, not uninstalled applications.

Why: raw compatibility overrides are not an uninstall list. The old hard-coded change ID both mischaracterized the source and discarded overrides for every other Android compatibility change.

System properties

Files: src/mvt/android/artifacts/getprop.py and its bugreport wrapper.

• Match a complete [name]: [value] property line.
• Preserve empty values and values containing bracket-like content.
• Use the shared command-boundary extractor so only the SYSTEM PROPERTIES block reaches the parser.

Why: the previous non-greedy expression required a non-empty value, and the wrapper's separator check could feed later dumpstate sections into GetProp.

Filesystem timestamps

File: src/mvt/android/modules/bugreport/fs_timestamps.py.

• Read persist.sys.timezone from the report's system properties.
• Keep the original local modified_time.
• Add modified_time_utc when the timezone can be resolved.
• Add timezone and timestamp_source (zip_metadata or filesystem_metadata).
• Leave UTC as null and log the unknown zone instead of guessing when the timezone cannot be resolved.

Why: ZIP timestamps do not carry a timezone offset. Treating a naive archive timestamp as UTC changes the represented instant. The new metadata makes the source and conversion explicit.

New Processes bugreport module

Files: src/mvt/android/artifacts/processes.py, src/mvt/android/modules/bugreport/processes.py, and module registration.

• Extract the PROCESSES AND THREADS command section.
• Drive parsing from the actual header row rather than fixed column positions.
• Support Android's optional LABEL, TID, scheduler, policy, priority, and either CMD or NAME columns.
• Convert numeric fields to integers and retain the final command as one field.
• Continue package/process IOC matching using the normalized command basename.

Why: process-table columns vary by Android/vendor build. The fixed-position parser shifted values into the wrong fields whenever optional columns appeared, and the parser was not registered for normal bugreport analysis.

New Settings bugreport module

Files: src/mvt/android/artifacts/settings.py, src/mvt/android/modules/bugreport/settings.py, and module registration.

• Extract the SettingsProvider dumpsys section.
• Parse CONFIG, GLOBAL, SECURE, and SYSTEM namespaces separately for every user.
• Store namespaces as keys such as secure:user_0 and retain each setting's emitted value.
• Reuse the existing dangerous-setting checks on the parsed bugreport data.

Why: bugreports already contain SettingsProvider data, but only AndroidQF inputs had a settings path. Without a bugreport parser, security-relevant settings present in the supplied files were ignored.

New Mounts bugreport module

Files: src/mvt/android/artifacts/mounts.py, src/mvt/android/modules/bugreport/mounts.py, and module registration.

• Parse Linux /proc/<pid>/mountinfo records from extracted bugreport files.
• Retain mount/parent IDs, major/minor device, root, mount point, device, filesystem type, optional fields, and combined mount/superblock options.
• Decode escaped spaces and derive is_system_partition and is_read_write.
• Deduplicate identical mount namespace records and aggregate the process IDs that observe each mount.
• Reuse the existing suspicious mount and file-path IOC checks.

Why: the bugreports contain per-process mount namespaces, while the existing artifact only understood the simpler mount command form and was not wired into bugreport analysis. Deduplication keeps shared namespaces useful without emitting the same mount once per process.

Tombstones: plaintext and protobuf together

Files: src/mvt/android/artifacts/tombstone_crashes.py and src/mvt/android/modules/bugreport/tombstones.py.

• Group tombstone_NN and tombstone_NN.pb under one crash_id.
• Always attempt both plaintext and protobuf parsing when both files exist.
• Produce one canonical crash record:
  • prefer a successfully parsed protobuf record;
  • fill only empty canonical fields from successfully parsed plaintext;
  • fall back completely to plaintext if protobuf parsing fails or is absent.
• Retain a sources entry for each available representation with file name, file timestamp, parse status, parse error, and its parsed record.
• Record disagreements between successfully parsed representations in differences, excluding source file metadata.
• Skip a crash only when neither representation can be parsed.
• Refactor the artifact parsers to return records so the module can merge them without mutating results twice.
• Reject empty inputs explicitly and decode damaged plaintext with replacement characters so one invalid byte does not discard an otherwise useful crash.
• Remove only a matching pair of outer quotes from text values rather than stripping meaningful apostrophes.
• Trim protobuf NUL padding from SELinux labels.
• Preserve the timezone semantics supplied by the tombstone timestamp instead of forcibly relabeling it as UTC.

Why: Android bugreports commonly contain both representations of the same crash. The old module emitted duplicates and treated the choice as either/or. It also lost the usable representation if one parser failed. The canonical result gives consumers one crash while preserving provenance, parse failures, and source disagreements for forensic review.

Module registry typing and mypy

Files: the AndroidQF, Android backup, intrusion-log, iOS backup, iOS filesystem, and iOS mixed module package registries.

• Annotate each registry as list[type[MVTModule]].
• Keep the same classes in the same runtime order.

Why: Python lists are invariant. Mypy inferred each constant as a list of one specialized module base and then rejected assignment or concatenation into Command.modules: list[type[MVTModule]]. Declaring the intended shared base type resolves all five repository-wide errors without casts and without changing behavior.

Strict parser result typing

Files: Accessibility, AppOps, Battery History, Packages, Processes, and Settings artifacts.

• Add concrete element types to empty list/dictionary result initializers so mypy does not have to infer the shape from an untyped empty container.
• Narrow the Processes command field to str before applying string operations; malformed non-string commands are ignored for IOC matching.

Why: the CI environment uses mypy 2.3, which no longer accepts these empty result initializers without an explicit type and correctly treats dynamically built process fields as str | int. These annotations describe the parser output already produced at runtime; the guard also prevents a malformed process record from reaching rsplit.

[github-actions]

Tests	Skipped	Failures	Errors	Time
245	1 💤	0 ❌	0 🔥	12.268s ⏱️