Keeping Independent Evidence in Play
Evidence integration and defect-matched information in AI-supported consumer-credit determinations.

Kerry Thornhill
Proposed studies · No results reported
Research source: 2026-09-09
Public summary: 2026-09-19, v1



RESEARCH QUESTION

Under what information-order and source-validity conditions do people preserve independently diagnostic evidence when evaluating an AI-supported institutional determination? Can information matched to a specific defect improve targeted inquiry while preserving warranted reliance on valid advice?
The current plan examines two functions in synthetic credit cases: data source and quality, and governing-standard–operation fit. It does not attempt to validate the entire six-function register in one study.


AIM 1 · MEASURE EVIDENCE INTEGRATION

An Epistemic Posture Task will ask adults to make repeated probability judgments using an independent diagnostic cue and advice labeled as AI or as a performance-matched human expert. Source reliability, cue strength, agreement, and valid defeaters will vary.
Three presentation conditions separate evidence order from initial commitment: evidence first with an initial judgment, evidence first without that judgment, and adviser first. Evidence-only and adviser-only trials provide benchmarks.
Primary outcomes are independent-evidence and adviser weights, selective response to valid defeaters, and prediction loss relative to the task’s integrated posterior. Agreement with advice or an order effect alone will not count as epistemic preemption.


AIM 2 · TEST DEFECT-MATCHED INFORMATION

Public materials, cognitive interviews, and compensated consumer and accessibility input will inform synthetic credit cases. Cases will contain a stipulated record defect, a stipulated criterion or proxy defect, or no designated defect.
Planned information conditionsConditionWhat it testsFull task-matched routeWhether the needed inquiry can be performed adequately.Focal omissionWhether removing a function causes a selective task loss when length and salience are matched.Faithful substituteWhether a different route can perform the same epistemic task.Conventional reasonHow a matched reason compares with task-specific grounds.
Measures include recognition of the material question, request fidelity, diagnostic accuracy, responsiveness to actual defects, and restraint when no defect is present.


WHAT WOULD CHANGE THE FRAMEWORK?

If AI and human advice produce equivalent effects, the mechanism may concern epistemic authority generally.If data-quality and criterion-fit tasks do not separate, the local partition should be merged.If omission does not produce selective loss, the bounded necessity claim fails.If additional information increases indiscriminate suspicion or false challenges, that is a backfire finding.If categorical behavioral profiles cannot be recovered reliably, continuous evidence weights and prediction loss remain the measures.
Usefulness is not necessityA full information bundle improving performance would show usefulness. A bounded necessity claim also requires selective omission loss, an adequate faithful substitute, and no material increase in false challenges.


PLANNED SEQUENCE

Year 1. Obtain human-subjects approval; develop and pilot the task; assemble the materials corpus; conduct interviews; preregister Aim 1.Year 2. Complete Aim 1; freeze the credit constructs and rival representation; pretest Aim 2.Year 3. Conduct Aim 2; apply revision rules; release synthetic materials and analyses.
These are planning stages, not completed milestones. Sample sizes will be determined through simulation-based power analysis before preregistration. No participant’s live credit decision will be affected by the proposed experiments.


INTENDED PUBLIC OUTPUTS

The plan includes an accessible task generator, synthetic credit cases, a coding guide, preregistrations, power simulations, analysis code, and teaching materials. Accessibility and supported use are part of the research design. Null, backfire, burden, and unequal-effect findings are intended outputs as well.
Explore the public worked example →

https://contestability.ai/research-plan/