Our Methodology
How we turn scattered government data into a trustworthy, searchable catalog.
Quality by the numbers
Core principles
Never trust metadata alone
We download and inspect every actual data file. Titles and descriptions from government portals are often wrong, missing, or outdated. Our metadata comes from what the data actually contains, not what a portal claims.
Freshness is a fact, not an opinion
Every dataset gets a freshness score computed from its actual date range, claimed update frequency, and source liveness. If data hasn't been updated since 2018, we say so — prominently.
Four layers of quality checks
Every dataset passes through four independent verification layers: deterministic parser extraction, AI enrichment cross-check, an independent AI audit of the final stored metadata, and a QA Loop that tests live chat responses against pre-written ground truth. Parser values always win over AI opinion. Each layer catches errors the others miss.
Self-healing over human maintenance
When a government agency changes a file format, the system detects the change and adapts automatically. If automatic recovery fails, the previous known-good state is preserved. The system never silently serves wrong data.
Why this matters: what we actually found
These aren't unusual. This is typical of government data portals nationwide. It's why every dataset in our catalog goes through the pipeline described below.
Every dataset goes through this pipeline
Source research and verification
Before adding any data source, we research the publishing agency, verify data formats, check update cadence, and look for overlaps with existing sources. We investigate whether apparent data gaps are real problems or normal publication delays. Each source gets a detailed research document.
Download and inspect actual files
We never catalog from titles and descriptions alone. Every downloadable file is retrieved, parsed, and inspected. For spreadsheets, this means detecting header rows, handling merged cells, stripping footnotes, and converting date formats. For APIs, we fetch actual records and validate the schema.
Deterministic metadata extraction
Year ranges, temporal granularity (daily, monthly, annual), geographic granularity (state, county, island, zip code), and data type are extracted directly from the data using deterministic rules. These values are never guessed or inferred by AI — they come from the actual cell values, column headers, and data patterns.
Automated verifiers then check the extracted output against the source file: every column name in a PDF-sourced dataset must match a header that's actually present in the PDF, and every measure mentioned in a source's description must correspond to a real dataset. These guardrails catch fabricated columns and over-broad scope claims before anything ships.
AI enrichment with real data context
An AI model analyzes each dataset using the actual file structure, sample rows, and field names — not just the title. It generates descriptions, topic tags, research use cases, potential join keys for combining datasets, and known limitations. Every AI-generated value is labeled as such.
Cross-verification
After enrichment, the AI-inferred metadata is compared against the parser-extracted values. Year ranges, granularity classifications, and data types are checked independently. Mismatches are flagged as quality issues. Over 130 cross-check discrepancies have been detected and resolved this way.
Format change detection
On every re-sync, the system compares the current data structure against the previously stored structure. Minor changes (a column added or renamed) are accepted automatically. Major changes trigger an investigation. If the system can't determine what happened, it preserves the last known-good state and flags the issue for review.
Independent metadata audit
After extraction, enrichment, and cross-verification, a separate AI model reviews the final stored metadata for plausibility — acting as an independent auditor that sees the complete picture. It checks whether year ranges match the title, whether granularity classifications make sense for the data described, and whether any values are contradictory.
This third layer catches errors that both the parser and the enrichment cross-check missed — like a dataset titled “2021 to 2025” with a parsed end year of 2078, or an employer directory incorrectly classified as census-tract-level data. Across our catalog, this audit has identified and corrected over 1,700 metadata fields.
QA Loop
Before a new source ships, we run a fourth check: an iterated test loop that calls our live chat API with pre-written questions and verifies every response against ground truth. Each round uses ten orthogonal prompts covering different failure modes — retrieval, field accuracy, year coverage, source surfacing, cross-source composition. A source is only considered ready when three consecutive rounds pass with no structural bugs.
This layer catches cross-cutting errors that the three static layers can't: a source description that overclaims scope, a catalog field list that looks reasonable in isolation but conflicts with the actual source file, an AI description that confidently repeats a bad value. On a single source, this loop has caught fabricated scope claims and hundreds of records with invented column names — errors that had passed all three static checks.
Cross-source linking
When multiple sources cover the same topic, we identify the overlap and annotate the relationship. Stale copies are flagged so the AI can steer users to the fresher authoritative source. Complementary sources are linked so the AI can explain what each adds — for example, one source may have monthly granularity while another has longer historical coverage. Hundreds of stale copies have been identified and hundreds of datasets carry cross-source annotations.
Continuous monitoring
The entire pipeline runs on every re-sync. Format changes are detected, new data is enriched, cross-checks are re-run, and quality issues are tracked over time. Thousands of quality issues have been identified and resolved across the catalog to date.
Not blind trust in AI
We didn't throw everything at AI and hope for the best. Government data is messy in ways that are unique to each source — merged cells in one spreadsheet, date formats in another, undocumented API quirks in a third. Every data source was individually researched, inspected, and validated by a human before being added to the catalog.
AI handles the scale — enriching thousands of datasets with descriptions and tags, running cross-checks, detecting format changes. But humans make the judgment calls: Is this data source trustworthy? Is this 10-month gap a publication delay or a real problem? Should we recommend this curated version over the raw original?
We also don't cut corners on the AI we use. When we tested a smaller, cheaper model for our metadata audit, it missed every error we asked it to find — including year ranges of 2098 and obviously wrong geographic classifications. We chose the more capable model because quality matters more than cost savings.
The result is a system where AI does what it's good at (processing at scale, finding patterns, generating descriptions) and humans do what they're good at (questioning assumptions, catching subtle errors, making context-dependent decisions). Neither could produce this catalog alone.
One more guardrail worth naming. The AI assistant occasionally mangles a link in its answer — copying the wrong page number into a URL, or swapping characters between two similar-looking addresses. We verify every link the assistant puts in front of you against the catalog before it reaches your screen, and silently correct mismatches it didn't quite get right. When the right answer can't be determined with high confidence, we leave the link alone rather than guess. The pattern repeats throughout the system: deterministic checks around the AI's output, because the right behavior for a research tool is to be quietly correct, not loudly approximate.
Source quality tiers
Not all data sources are equal. We classify each source so the AI can recommend the best starting point when multiple sources cover the same topic.
The original government agency that produced the data. Freshest, most granular, but sometimes in messy formats.
A high-quality aggregator that cleans, extends, and adds value to original data — longer time series, geographic breakdowns, or consistent formatting. We track the original upstream source.
A copy with no added value. Often stale compared to the original. We flag these and point users to the fresher authoritative source.
What we're honest about
We don't host data. We host enriched metadata that makes data discoverable. You always go to the original source to access the actual data.
Some sources are stale. About 20% of datasets in our catalog have freshness scores below 40, mostly from the state's open data portal. We flag this prominently rather than hiding it.
AI descriptions are labeled. When a dataset description was generated by AI rather than written by the publishing agency, we track that provenance. AI-generated content is useful but should be verified against the actual data.
Government data has gaps. Publication delays, website upgrades, and funding changes can cause data to stop flowing temporarily. We investigate each case and document what we find rather than assuming the worst.
We catch and fix our own errors too. Our independent metadata audit found over 1,700 fields in our own catalog that needed correction — year ranges that contradicted titles, granularity classifications that didn't match the data. We fixed every one. The difference between us and most data portals isn't that we never make mistakes — it's that we have a process to find and correct them. Most portals don't.
How AI search works
When you ask a question, the AI doesn't just run a keyword search. It reasons through your question in multiple steps:
Decompose — breaks your question into data needs: topics, geography, time period, granularity.
Search iteratively — runs multiple searches with different queries and filters. A question about Maui wildfires triggers searches for tourism, employment, housing, and environmental data separately.
Inspect — reads the full metadata of the most promising datasets to understand their fields, limitations, and how they connect.
Combine — suggests how datasets from different sources work together, identifying shared keys like county, year, or island.
Flag gaps — honestly reports what data is missing, what's stale, and suggests alternatives.