Selection and Inspection

Selection & Inspection

Tier: Beginner Commands covered: headers, count, sample, slice, select, search, searchset, flatten, table, color, lens, clipboard, sniff

Note

Per-command flag reference lives in /docs/help/. This page is the workflow layer — when to reach for each command and how they compose.

These are the daily drivers. Once you've finished Getting Started, this is the page to bookmark.

Quick decision table

If you want to…	Use	Notes
See what columns a file has	headers	Use --union across many files to spot schema drift
Know how many rows are in a file	count	Instantaneous with an index; fast with the Polars feature otherwise
Pick out specific columns	select	Powerful syntax: ranges, regex, exclusion, randomization
Filter rows by a pattern	search	Regex by default; --literal and --exact for plain matching
Find PII / multi-pattern matches in one pass	searchset	Reads N regexes from a file, runs them all at once
Grab a random sample	sample	Ten methods — incl. stratified, weighted, varopt, mergeable-reservoir, cluster, timeseries
Look at a specific row range	slice	Use --index N for a single row; --invert for "everything but"
Render a single record vertically	flatten	Pair with slice -i N
Make a CSV human-readable in the terminal	table	Elastic tabstops; can also output aligned-TSV or Fixed-Width
The same, but colorized	color	Color-codes types (string/number/date); fits terminal width; light/dark theme auto
Browse a CSV/Parquet/JSONL interactively	lens	TUI viewer with search and filter
Move data through the OS clipboard	clipboard	qsv clipboard reads; qsv clipboard --save writes
Probe an unknown file (or URL)	sniff	Delimiter, schema, MIME type — even without downloading

headers

Show the field names of a CSV. With multiple inputs and --union, you get the deduplicated union of headers — useful for spotting schema drift across monthly exports.

Example: schema drift check across 12 monthly NYC 311 exports

# Each file is supposed to have the same schema. Are they actually identical?
qsv headers --union nyc311-2024-*.csv

If new columns appear in later months, --union surfaces them; if you expected only the intersection, your downstream tooling needs adjustment.

Example: column count without column names

qsv headers --just-count wcp.csv
# 7

See also: /docs/help/headers.md, select, safenames.

count

The fastest way to know "how big is this thing." With an index it returns instantaneously regardless of file size. Without an index, the Polars-backed multithreaded reader is used (when the polars feature is enabled).

Example: instant counts on a 15 GB NYC 311 export

qsv index nyc311-full.csv        # 14 seconds, one time
qsv count nyc311-full.csv        # instantaneous, every time after
# 27851234

Example: row count and width statistics for capacity planning

qsv count --width --json wcp.csv
# {"count":2699354,"width":{"max":113,"avg":34,"median":34,...}}

The width breakdown (max/avg/median/min/variance/stddev/MAD) helps you predict memory needs before running an in-memory command like sort or dedup.

See also: /docs/help/count.md, index, Performance Tuning.

sample

Ten sampling methods, each with different memory and probabilistic guarantees. The default (reservoir / indexed) is what you want 90 % of the time. The other eight are for specific statistical needs.

Example: stratified sample for representative coverage

You're auditing 1 M NYC 311 service requests but want 20 examples per Borough — not 20 from the population at large:

qsv sample --stratified Borough 20 NYC_311_SR_2010-2020-sample-1M.csv

Example: weighted sample skewed toward big cities

Build a 5,000-row sample of wcp.csv where larger cities are more likely to be picked (proportional to Population):

qsv sample --weighted Population 5000 wcp.csv > weighted_sample.csv

Example: Bernoulli sample from a remote CSV without downloading the whole file

qsv sample --bernoulli 0.001 https://example.com/huge.csv > sample.csv

--bernoulli streams the file and selects each row independently with the given probability, so you don't pay the full download cost.

Example: variance-bounded weighted sampling without a stats cache (--varopt)

--weighted needs the stats cache to learn the maximum weight up front. --varopt doesn't — it's a true single-pass weighted reservoir sampler (A-ExpJ keying, Efraimidis & Spirakis 2006) with bounded variance, well suited to heavy-tailed weight distributions:

qsv sample --varopt Population 5000 wcp.csv > varopt_sample.csv

Example: sample sharded inputs and merge them without re-reading the data (sketch I/O)

--varopt and --mergeable-reservoir (a mergeable uniform reservoir, Vitter's Algorithm R) can serialize their sampler state to a binary sketch. Sketches from separate runs are then merged into one sample — ideal for sharded or incremental pipelines:

# sample each shard, writing a mergeable sketch of the sampler state
qsv sample --mergeable-reservoir --sketch-out shard1.sk 10000 shard1.csv
qsv sample --mergeable-reservoir --sketch-out shard2.sk 10000 shard2.csv

# merge the sketches into one uniform 10k-row sample of the combined stream
qsv sample --sketch-in shard1.sk,shard2.sk 10000 -o combined_sample.csv

The merge step reads only the sketches — not the shard CSVs. Sketches embed the source header, so --sketch-in still emits a schema-bearing CSV. The on-disk sketch format is qsv-specific (not interoperable with other tools' sketches).

See also: /docs/help/sample.md, stats, frequency.

slice

Slice by row range. With an index, only the requested rows are parsed — invaluable for spot-checking offsets deep in a huge file.

Example: instantly inspect row 9,000,000 of a 28M-row file

qsv index nyc311-full.csv     # one-time, ~14s
qsv slice --index 9000000 nyc311-full.csv | qsv flatten

Example: get the last 100 rows

qsv slice --start -100 wcp.csv

Example: every row except the last 100 (for sanity-trimming export artifacts)

qsv slice --start -100 --invert wcp.csv > trimmed.csv

Example: slice as JSON

qsv slice --start 0 --len 5 --json wcp.csv

See also: /docs/help/slice.md, flatten, sample.

select

qsv's column-selector mini-language. It's used by select itself and by the --select flag on sort, search, searchset, apply, replace, frequency, stats, schema, validate, sqlp, and many more (any command marked 👆 in the Command Reference).

Example: drop PII columns from a public release

qsv select '!/SSN|account_no|password|phone|email/' raw.csv > public.csv

The ! prefix inverts the selection; the slashes mark a regex.

Example: reorder + dedupe columns for a downstream consumer

qsv select 'AccentCity,Country,Region,Population,Latitude,Longitude' wcp.csv > reordered.csv

Example: keep the first column and append a reversed copy of all columns

qsv select '1,_-1' data.csv
# 1 = first column; _ = special "last column" marker; _-1 = last-down-to-first range

Example: shuffle column order deterministically with a seed

qsv select 1- --random --seed 42 wcp.csv

Example: move a "big" column to the start or end without listing every other column

select has no --move-to-start / --move-to-end flag yet (#3906). And qsv select '!Big,Big' won't work — the ! prefix excludes Big from the entire selection, even if it's listed again later. Two idiomatic workarounds:

Option A — duplicate, then drop the original (single pipe, recommended):

Prepend (or append) the column to the full 1- range, then drop the original occurrence using bracket indexing. [0] is the first occurrence in document order, [1] is the second, etc.

# Move "Big" to the start: select Big first, then everything (Big appears twice),
# then drop the second occurrence (the original).
qsv select 'Big,1-' wcp.csv | qsv select '!Big[1]'

# Move "Big" to the end: everything first, then Big again, then drop the original.
qsv select '1-,Big' wcp.csv | qsv select '!Big[0]'

Multi-column moves work the same way — count the duplicates left-to-right:

# Headers: A,Big,C,X,X,X,D — move Big and the 3rd X (X[2]) to the start
qsv select 'Big,X[2],1-' input.csv | qsv select '!Big[1],X[3]'
# → Big,X,A,C,X,X,D

Option B — cat columns with process substitution (two streams, more visual):

QSV_SKIP_FORMAT_CHECK=1 qsv cat columns \
  <(qsv select '!Big' wcp.csv) \
  <(qsv select  'Big' wcp.csv) > reordered.csv     # Big to end

QSV_SKIP_FORMAT_CHECK=1 qsv cat columns \
  <(qsv select  'Big' wcp.csv) \
  <(qsv select '!Big' wcp.csv) > reordered.csv     # Big to start

QSV_SKIP_FORMAT_CHECK=1 is needed because process substitution exposes each stream as /dev/fd/N, which has no extension for qsv to sniff. Works in bash and zsh.

See also: /docs/help/select.md — the column-selector syntax reference for every 👆 command.

search

Regex filter on rows. Multithreaded when an index exists. --literal and --exact modes for plain-string matching.

Example: case-insensitive search for "noise" complaints

qsv search -i --select 'Complaint Type' 'noise' NYC_311_SR_2010-2020-sample-1M.csv > noise.csv

Example: PII scan that flags rows (doesn't filter) and writes a marker column

qsv search --flag pii_flag '\b\d{3}-\d{2}-\d{4}\b' raw.csv > flagged.csv

The --flag option keeps every row but adds a pii_flag column set to the row number for matches and 0 otherwise.

Example: preview the first 10 matches without scanning the whole file

qsv search --preview-match 10 'urgent' tickets.csv 2>&1 >/dev/null | head

Example: count rows matching a literal string without writing the matched rows

qsv search --literal --count --quiet 'N/A' raw.csv

See also: /docs/help/search.md, searchset, replace.

searchset

Run many regexes in a single streaming pass. The killer feature is PII / compliance scanning: drop 30 regexes in a file, get one report.

Example: PII scan with qsv's bundled regex set

# qsv ships a starter PII regex file
curl -LO https://raw.githubusercontent.com/dathere/qsv/master/resources/examples/searchset/pii_regexes.txt
qsv searchset pii_regexes.txt customer_export.csv > rows_with_pii.csv

The regex file uses one pattern per line; # starts a comment. SSNs, credit cards, emails, bank account numbers, and phone numbers are all in the bundled file.

Example: which complaint types contain profanity?

echo -e "(?i)\\b(damn|hell|crap)\\b\n(?i)\\bwtf\\b" > swear_regex.txt
qsv searchset --select 'Complaint Type,Descriptor' swear_regex.txt 311.csv | qsv frequency

See also: /docs/help/searchset.md, apply --operations censor for one-shot profanity censoring.

flatten

One column per line. Best paired with slice -i N for "show me row N as something I can actually read."

Example: read a single NYC 311 service request

qsv slice -i 42 NYC_311_SR_2010-2020-sample-1M.csv | qsv flatten

Unique Key       30281872
Created Date     12/31/2014 11:59:45 PM
Closed Date      01/05/2015 12:01:00 AM
Agency           NYPD
Agency Name      New York City Police Department
Complaint Type   Noise - Residential
Descriptor       Loud Music/Party
…

Example: condense long fields to 40 characters for a tldr view

qsv slice -i 0 nyc311.csv | qsv flatten -c 40

See also: /docs/help/flatten.md, slice, table.

table

Elastic-tabstop aligned columns for the terminal. Also emits aligned-TSV (a valid TSV that's also visually aligned) and Fixed-Width Format (FWF) with a column-position comment line.

Example: pretty top-5 rows of a stats result

qsv stats --everything wcp.csv | qsv slice --len 5 | qsv table

Example: produce an aligned TSV for a downstream tool that wants TSV

qsv table --align leftendtab wcp.csv > wcp-aligned.tsv

Example: Fixed-Width Format export with position-comment header

qsv table --align leftfwf wcp.csv > wcp-fwf.txt
# First line is "#1,9,42,..." enumerating column start positions

table loads the entire CSV into memory — pair with slice or sample to trim huge files first.

See also: /docs/help/table.md, color (colorized cousin), lens (interactive), flatten.

color

table with colors. Same elastic-tab alignment, but with color-coded data types (string vs number vs date), terminal-fit truncation, and theme auto-detection (light vs dark). Loads the entire CSV into memory — pair with slice or sample for large files.

The polars feature lets color also display Arrow, Avro, Parquet, JSON arrays, and JSONL.

Example: colorized top-10 cities by population

qsv search --select Country '^us$' wcp.csv \
  | qsv sort --select Population --numeric --reverse \
  | qsv slice --len 10 \
  | qsv color

Example: force colors when piping (or running in CI)

QSV_FORCE_COLOR=1 qsv stats wcp.csv | qsv color | less -R

Example: override terminal theme detection

QSV_THEME=DARK qsv color wcp.csv

Example: browse a Parquet file (polars feature)

qsv to parquet outdir/ wcp.csv
qsv color outdir/wcp.parquet

See also: /docs/help/color.md, table — uncolored alternative, lens — interactive viewer, Environment Variables — QSV_FORCE_COLOR, QSV_THEME, QSV_TERMWIDTH.

lens

Interactive TUI viewer. Powered by csvlens. With the polars feature it also opens Parquet, Arrow, JSONL, and Avro — and decompresses .gz, .zlib, .zst transparently.

Example: browse a 1M-row NYC 311 sample with search

qsv lens NYC_311_SR_2010-2020-sample-1M.csv
# Inside lens: / starts a regex search; q quits; ? shows full key bindings

Example: browse a Parquet file produced by qsv to parquet

qsv to parquet wcp.parquet wcp.csv
qsv lens wcp.parquet

Example: browse a snappy-compressed CSV without an explicit decompression step

qsv lens nyc311.csv.sz

See also: /docs/help/lens.md, table, Conversion & I/O.

clipboard

Read from the OS clipboard (qsv clipboard) or write to it (qsv clipboard --save / -s). Closes the gap between qsv and apps like Excel, Sheets, and Numbers.

Example: paste a CSV from your clipboard into a pipeline

# Copy a CSV table from a spreadsheet, then:
qsv clipboard | qsv stats | qsv table

Example: send a stats table to your clipboard for pasting into Slack/email

qsv stats wcp.csv | qsv table | qsv clipboard --save

See also: /docs/help/clipboard.md, prompt — OS file dialog.

sniff

Sniff the delimiter, header row, preamble lines, quote character, encoding, average record length, row count, and per-column data types — by sampling the first 1,000 rows by default. Works on local files and remote URLs.

sniff is fast and dialect-tolerant: unlike schema and stats (which scan the whole file and need well-formed CSV), it samples and copes with remote files, unusual delimiters and quoting. The trade-off is that its inferences are probabilistic — for a guaranteed schema, reach for schema or stats. If sniff guesses a dialect wrong, pin it with --delimiter and/or --quote.

With the magika feature, it identifies 200+ file types via Google's AI-based content detection.

Choosing how much to sample (--sample, default 1000)

• 1000 (default) — inspect that many rows
• 0 — inspect the entire file
• a fraction 0 < n < 1 — inspect that percentage of the file (e.g. 0.20 = 20%)
• for a URL, it is how many lines to pull without downloading the whole file

Indexed local files draw a whole-file distributed sample

When sniffing a local file that has a CSV index (qsv index data.csv, or auto-created via QSV_AUTOINDEX_SIZE), sniff stops reading only the first N rows. Within the --sample budget it samples across the whole file: the first & last 5 rows, 5 rows each around the 25th/50th/75th percentiles, and the rest random. This catches column types and dates that only appear late in a file (e.g. a code column that is all-numeric for the first 1,000 rows but alphanumeric near the end). It is automatic whenever a fresh .idx exists — no extra flag — and falls back to the first-N sample when the file is not indexed. Delimiter, header and preamble detection still come from the contiguous head of the file, so they are unaffected.

The same distributed sample sharpens stats date inference: stats --infer-dates uses its default --dates-whitelist sniff to choose which columns to treat as Date/DateTime, so on an indexed file those columns are picked from the whole-file sample too. See Performance Tuning → indexing.

Example: probe a remote CSV without downloading it

qsv sniff https://opendata.cityofnewyork.us/data.csv

Output reports delimiter, quote, header presence, content length, estimated row count, and a per-field type/name list. Add --json / --pretty-json for machine-readable output, or --stats-types to report the same type names as stats (e.g. Integer/String).

Example: more accurate inference on a large local file via an index

qsv index data.csv          # one-time; writes data.csv.idx
qsv sniff --json data.csv   # samples first/last + quartile windows + random

Example: detect MIME type only (no schema inference) — useful for CKAN harvesting

qsv sniff --no-infer https://example.com/data.xlsx

Example: stale-URL audit for a list of CKAN resources

# resource_urls.csv has a column "url"
qsv select url resource_urls.csv \
  | qsv behead \
  | xargs -I {} qsv sniff --no-infer --json {}

See also: /docs/help/sniff.md, Validation & Schema, schema for a complete (non-sampled) schema inference, stats for guaranteed type inference, Indexing, Compression & Diff → index, Performance Tuning.

See also

• Command Reference (index) — every command grouped by category
• Transform & Reshape — clean and reshape after you've found what you want
• Aggregation & Statistics — stats, frequency, pragmastat
• Cookbook → Inspect an Unknown CSV
• Performance Tuning — when to index and why
• docs/whirlwind_tour.md