Criteria for Assessing DEIA Efficacy
- Jordan Burris
- Peter Davis
Workspace for creating consensus driven methodology and tools for measuring identity outcomes, impacts across different demographics, and overall return on investment (ROI) for adopting more inclusive practices.
Category Definitions:
Diversity - Organizations should seek to ensure a range of people with various background (e.g., racial, socioeconomic, gender, cultural, age, experience) are represented in the organization
Equity - Organizations should seek to deliver solutions that promote fair treatment, and remove systemic barriers for underserved communities participating in the system. Impacts should be measured and made transparent.
Inclusion - Organizations should seek to actively participate and engage with the broader community to understand where identity systems are falling short and how they should be improved. Feedback should inform product development.
Accessibility - Organizations should seek to design solutions and services in a manner that all people including those with disabilities can fully and independently use them.
Category | Criteria | Description | Measurement Approach | Comments |
|---|---|---|---|---|
| Auto Decision Rate | How often the solution returned a conclusive decision (“accept” or “reject”) for a submission. |
| Consider hierarchy w/ things like resubmit rate. |
| True Reject Rate | How often the solution accurately rejected a fraudulent submission |
| Should we consider for different dimensions of verification (bio vs pii vs other) |
| True Accept Rate | How often the solution accurately accepted a legitimate submission |
| Should we consider for different dimensions of verification (bio vs pii vs other) |
| False Reject Rate | How often the solution falsely rejected a legitimate submission |
| Should we consider for different dimensions of verification (bio vs pii vs other) |
| False Accept Rate | How often the solution falsely accepted a fraudulent submission |
| Should we consider for different dimensions of verification (bio vs pii vs other) |
| Resubmit Rate / # of attempts required | How often a person had to resubmit or engage in the process to achieve an outcome |
|
|
| Abandonment Rate | % of people that began but never completed the process |
| Sub-categories (evidence, technology, process burden |
| Time to Complete / Time taken to conduct a task. | How long the identity proofing process took to complete |
|
|
| Coverage Accuracy For Demographics | How often an identity can be uniquely resolved for a given population via authoritative data sources
|
| Scrutinize - may be a bridge too far. |
| Manual Review Rate | How often an identity has to be reviewed through an in-person or out of band method to resolve to a unique individual. |
|
|
| % of channel referral | How often a user to shifted from one channel to another to complete the identity proofing process |
| Consideration on making this another lens for all the metrics - focusing on what the macro metrics look like across all channels. |
| Disparity between initial adjudication and final adjudication | % difference between the initial determination on an identity vs the ultimate determination for the identity |
| Perhaps shifting to appeals / redress rate or other. What is the percentage of reversal following appeal / redress |
|
|
|
|
|
|
|
|
|
|