Key Takeaways
- A vendor selects its own reference customers. Research on industrial purchasing found buyers already discount this somewhat, yet most AI vendor RFPs in financial services still treat a positive reference call as close to a pass/fail gate rather than a curated, biased data point (Hada, Grewal and Lilien, 2013).
- IDC research cited in Lenovo's 2025 CIO Playbook found roughly 88% of AI proofs of concept never reach production deployment - a conversion rate of about four in every thirty-three pilots started. A reference customer's pilot-stage success says little about what happens at your scale, in your risk tier (IDC, cited in Schuman, 2025).
- A September 2026 Australian life insurance sanction shows the mechanism directly: 46% of 10,105 recorded customer-notice breaches traced to a single outsourced distribution partner, undetected by the insurer's own monitoring for roughly eight months until a customer complaint surfaced it. The regulator ruled that outsourcing a function never transfers the liability that comes with it (Life Insurance Code Compliance Committee, cited in Insurance Business Magazine, 2026).
- US banking regulators' newly proposed third-party risk management guidance, released 11 September 2026, explicitly abandons one-size-fits-all vendor checklists in favour of testing whether a specific relationship's risk profile matches the institution's own oversight capacity - not whether the vendor has worked well for someone else (Board of Governors of the Federal Reserve System, FDIC, OCC and NCUA, 2026).
- The fix is not abandoning reference checks. It is replacing "are you happy with this vendor" with a structured test of whether the reference customer's operating-discipline maturity - not merely its satisfaction - resembles your own.
Introduction
Every AI vendor shortlist in banking and insurance eventually reaches the same procurement ritual: three reference calls, a satisfaction score, a box ticked. It feels like diligence. It is treated as diligence in board papers and investment committee memos. And it tells a buyer almost nothing about the one thing that actually determines whether an AI deployment survives contact with a regulated production environment - whether the institution on the other end of that call runs AI the way yours will have to.
The reason is structural, not a matter of any single vendor being dishonest. A vendor chooses which customers to put forward as references. That customer is, almost by definition, the vendor's best-case deployment: the one with the cleanest data, the most forgiving risk appetite, the most mature internal monitoring, or simply the best relationship with the account team. A satisfied reference proves the vendor's product can work somewhere. It does not prove the product will work the way your institution needs it to - under your regulatory regime, at your transaction volume, inside your escalation and audit requirements.
This matters more in 2026 than it did two years ago, for two converging reasons. First, AI vendor evaluation has become the dominant procurement workload across BFSI technology budgets, and reference checks remain almost the only qualitative signal procurement teams use that a purely quantitative RFP scorecard cannot capture - which means the industry is leaning harder on exactly the signal most structurally biased toward a positive answer. Second, regulators in multiple jurisdictions have spent 2026 making explicit what was previously implicit: that outsourcing a capability does not outsource accountability for how it performs, and that a generic vendor track record is not evidence of fit for a specific institution's risk profile. Both threads converge on the same practical question this article answers: what should actually replace "they gave us a good reference" as a selection criterion.
What a reference call actually measures
A reference call, however well conducted, measures one thing reliably: whether a specific customer, selected by the vendor, is currently satisfied enough to say so on the record. It does not measure whether that customer's institution is comparable to yours in the dimensions that determine whether an AI deployment survives its second year in production - transaction volume, regulatory regime, risk appetite, or the operating discipline around monitoring, escalation and audit trail.
This is a documented finding in procurement research, not a cynical assumption. A study of industrial purchasing managers published in the Journal of Supply Chain Management examined exactly this dynamic and found that "reference customers are selected by suppliers, not by purchasing managers" - the bias is built into who gets asked to speak, before the call ever happens. The same research found two things worth building into any evaluation process: more experienced purchasing managers already discount referral information for this reason, and - counterintuitively - a reference that volunteers some negative information alongside the positive is perceived as more credible, not less, because it reads as less curated (Hada, Grewal and Lilien, 2013). Most AI vendor RFPs in BFSI procurement do neither. They treat the reference call as confirmatory rather than diagnostic, and they rarely ask the one question - what went wrong, and how did you catch it - that the research says actually produces a trustworthy signal.
The scale of the underlying deployment-failure problem compounds this. IDC research conducted with Lenovo and published in its 2025 CIO Playbook found that roughly 88% of AI proofs of concept never reach full-scale production deployment - a conversion ratio of about four successful deployments for every thirty-three pilots launched (IDC, cited in Schuman, 2025). Set against that base rate, a reference customer who reached production and stayed there is already a statistical outlier among the vendor's full customer base, not a representative data point. A buyer hearing from that one in four tells them the deployment is possible. It tells them almost nothing about the odds of it going well inside a different institution with a different risk tier and a different internal capability to run it.
Two 2026 developments that make this concrete
Two developments since the middle of this year move this argument from a procurement-theory observation to a live regulatory and commercial fact pattern.
The first is a sanction, not a hypothetical. On 1 September 2026, Australia's Life Insurance Code Compliance Committee published findings against a life insurer whose annual policy notices - covering bonus benefit disclosures, premium structure presentation, and anniversary-date timing - breached the Life Insurance Code of Practice 10,105 times between August 2024 and January 2026. Of those breaches, 4,686 - 46% - were directly attributable to delays caused by a third-party distribution provider the insurer had engaged to issue the notices. The insurer's own monitoring did not catch the pattern; a customer complaint did, roughly eight months after the failures began. The Committee's finding was unambiguous: outsourcing a function does not transfer the liability for meeting the Code's obligations, and the insurer was required to demonstrate that its own governance, controls and assurance arrangements - not the vendor's - could prevent recurrence (Life Insurance Code Compliance Committee, cited in Insurance Business Magazine, 2026). Nothing in the public record suggests this vendor lacked satisfied reference customers elsewhere. What it lacked, relative to this particular insurer's oversight needs, was a demonstrated capability to flag its own delays before they accumulated into ten thousand breaches.
The second is regulatory, and it points the same direction from a different angle. On 11 September 2026, the Federal Reserve, FDIC, OCC and NCUA jointly proposed an update to US interagency third-party risk management guidance that would replace the existing framework entirely. The proposal is explicitly principles-based rather than checklist-based, instructing institutions to weigh the "magnitude and harm" of each specific third-party relationship rather than applying uniform due-diligence depth across every vendor regardless of risk profile (Board of Governors of the Federal Reserve System, FDIC, OCC and NCUA, 2026). Read alongside the European Banking Authority's parallel move toward lifecycle-based third-party oversight - covering due diligence, ongoing monitoring and exit planning for arrangements supporting critical functions - the direction across jurisdictions is the same: supervisors are moving away from treating third-party diligence as a generic checkbox exercise and toward requiring institutions to test whether a specific vendor relationship's risk profile is one their own organisation is actually equipped to oversee. A generic vendor track record, reference customers included, is precisely the kind of checkbox evidence this regulatory direction is moving away from accepting at face value.
This sits alongside, rather than repeats, the pattern this publication documented in September's post-mortem on three AI operating failures at a major Australian bank, a global fintech lender and a US insurer (sourceCode, 17 September 2026). None of those three institutions is documented as having failed a vendor reference check - each very plausibly could have produced satisfied reference customers elsewhere, since the underlying technology worked broadly as designed. What was missing in each case was the operating discipline - production monitoring, escalation design, audit trail - to catch divergence between the demo and the deployment before it became a public incident. A reference check, as currently practised, is not built to surface that gap, because it asks the wrong question of the wrong source.
The Reference Transfer Test
The practical fix is not to discard reference checks - a vendor with no customers willing to speak is a legitimate red flag in its own right. It is to stop treating a satisfied reference as sufficient evidence of fit, and replace it with a structured test of whether the capability that worked at the reference customer will actually transfer to your institution. We call this the Reference Transfer Test, a four-part structure procurement and risk teams can apply to every reference call on an AI vendor shortlist.
- Operating-discipline parity. Does the reference customer's production monitoring, escalation design and audit-trail maturity - the same five criteria this publication's Operating Discipline Scorecard weights in full vendor evaluation - resemble what your institution can sustain, or does the reference customer simply have a larger, better-resourced risk function than you do?
- Risk-tier and scale correspondence. Does the reference operate at comparable transaction volume, under a comparable regulatory regime, and against a comparable customer risk profile? A reference deployment serving a narrow, low-stakes use case at a smaller institution is weak evidence for a high-volume, high-stakes deployment at a systemically important one, even where both are nominally "successful."
- The negative-disclosure test. Ask directly what went wrong during the deployment, how it was caught, and how long detection took - and treat a reference who cannot or will not answer as a materially weaker signal than one who can. The research behind this is explicit: a reference that volunteers negative information alongside the positive reads as more credible, not less, precisely because it signals the conversation wasn't scripted by the vendor's account team (Hada, Grewal and Lilien, 2013).
- Artefact verification, not conversation only. Where possible, ask the reference customer - not the vendor - to describe or show the actual monitoring dashboard, escalation log, or incident post-mortem from their deployment, rather than relying on a verbal assurance that "it's been great." A reference willing to walk through a real operational artefact is a fundamentally different data point than one offering a general endorsement.
A reference call that clears all four is meaningfully stronger evidence of fit than a vendor-curated satisfaction score. A reference call that clears none of them should be treated as marketing collateral, not diligence.
Business and technology implications
For procurement teams, this means restructuring the reference-check stage of the RFP from a single satisfaction question into a short structured interview built around the four tests above - adding perhaps twenty minutes to a call that otherwise produces a far weaker signal. For risk and compliance functions, it means the reference-check record itself should become part of the documentation a supervisor would expect to see under the direction both the proposed US guidance and the EBA's lifecycle approach are taking - evidence that the institution tested relationship-specific fit, not merely collected a vendor's preferred testimonials. For CTOs and CDOs, it means weighting a reference customer's technical environment and scale explicitly in the evaluation, rather than treating "reference confirmed" as a undifferentiated checkbox alongside benchmark accuracy scores. For CFOs, the implication is more direct still: the Australian case above shows that the cost of a mismatched reference-to-reality gap is not a line item in the original business case - it surfaces later, as remediation cost, regulatory exposure and reputational repair, often years after the vendor selection decision that could have tested for it.
A fair counterargument
It would be overstating the case to suggest reference checks have no diagnostic value. A vendor unable to produce any customer willing to speak, under any conditions, remains a legitimate and serious red flag - that signal is real and should not be discarded. Nor is operating-discipline mismatch the only reason an AI deployment fails; genuine capability gaps, unclear business value and organisational change resistance are well-documented, separate failure modes that a reference-check redesign will not fix on its own. The argument here is narrower and more specific: a reference check conducted as a simple satisfaction question, answered by a vendor-selected customer, is weak evidence of fit for a different institution at a different scale and risk tier - and BFSI procurement teams currently treat it as far stronger evidence than that structure can actually support.
What leaders should do next
The immediate step is not building a new vendor-governance programme. It is taking the next scheduled reference call on an active AI vendor evaluation and running it against the four-part test above, rather than the standard satisfaction script. Ask the reference what went wrong and how it was caught. Compare their regulatory regime, transaction volume and risk appetite against your own, explicitly, before weighting the call's results. Where a vendor can only offer references who cannot or will not discuss a failure, treat that as information about the evaluation, not an awkward gap to work around.
The sourceCode’s perspective
In our delivery work with banks, insurers and insurtechs across Australia, Southeast Asia and the Gulf, the vendor-selection conversations that go well are rarely the ones with the most polished reference list. They are the ones where procurement and risk teams ask a prospective delivery partner to walk through what has actually gone wrong on a comparable past engagement, and how that was caught before it became a production incident - the same negative-disclosure test this article recommends applying to any AI vendor.
A delivery partner who can answer that question candidly, with artefacts rather than assurances, is telling a buyer something a satisfaction score cannot. Connect us here for further discussion!
Conclusion
A reference customer's satisfaction is evidence that a vendor's best-case deployment is going well. It was never designed to answer the question BFSI procurement teams actually need answered: will this capability work the way we specifically need it to, given our regulatory regime, our risk appetite, and our own capacity to monitor it once it's live.
The evidence from supply-chain research, deployment-failure statistics, a 2026 regulatory sanction and converging international supervisory guidance all point the same way - operating-discipline fit, tested deliberately, is the signal procurement teams should be weighting.
A reference call is a good place to start testing for it. It stops being good enough the moment it becomes the only test applied.
Frequently Asked Questions
Why isn't a positive vendor reference check reliable evidence for AI vendor selection? Because the vendor selects which customers are offered as references, the sample is structurally biased toward the vendor's best-case deployments. Research on industrial purchasing found purchasing managers already discount referral information for this reason, and found that a reference's willingness to disclose negative information - rare in a vendor-curated process - is actually what makes a reference more credible (Hada, Grewal and Lilien, 2013).
What does the 88% AI pilot failure statistic have to do with vendor reference checks? IDC research found roughly 88% of AI proofs of concept never reach full production deployment. A reference customer who did reach production is therefore already an outlier within the vendor's broader customer base, not a representative result - which weakens the inference that a similar outcome is likely for a different buyer (IDC, cited in Schuman, 2025).
What happened in the Australian life insurance case, and why does it matter for AI vendor selection? A life insurer breached Code of Practice obligations 10,105 times over roughly eighteen months, with 46% of breaches traced to a third-party distribution partner's delays and detected only via a customer complaint eight months after the failures began. The regulator ruled outsourcing does not transfer liability. It is a concrete illustration of what an operating-discipline mismatch between an institution and a vendor looks like once it surfaces in production (Life Insurance Code Compliance Committee, cited in Insurance Business Magazine, 2026).
What is the Reference Transfer Test? A four-part structure for evaluating an AI vendor's reference customers: operating-discipline parity, risk-tier and scale correspondence, a negative-disclosure test, and verification of real operational artefacts rather than verbal assurance - designed to test whether a reference customer's success will actually transfer to a different institution's circumstances.
Does this mean BFSI procurement teams should stop doing reference checks? No. A vendor unable to produce any willing reference remains a legitimate red flag. The recommendation is to restructure how reference calls are conducted and weighted, not to remove them from the evaluation process.
Reference List
Board of Governors of the Federal Reserve System, Federal Deposit Insurance Corporation, Office of the Comptroller of the Currency and National Credit Union Administration (2026) Agencies propose updated guidance on third-party risk management. 11 September. Available at: https://www.federalreserve.gov/newsevents/pressreleases/bcreg20260911a.htm (Accessed: 26 October 2026).
Hada, M., Grewal, R. and Lilien, G.L. (2013) 'Purchasing managers' perceived bias in supplier-selected referrals', Journal of Supply Chain Management, 49(4), pp. 81-95.
Insurance Business Magazine (2026) Outsourced function, owned liability: what 10,000 life insurance breaches mean for your clients. 3 September. Available at: https://www.insurancebusinessmag.com/au/news/life-insurance/outsourced-function-owned-liability-what-10000-life-insurance-breaches-mean-for-your-clients-588306.aspx (Accessed: 26 October 2026).
Schuman, E. (2025) '88% of AI pilots fail to reach production, but that's not all on IT', CIO, 25 March. Available at: https://www.cio.com/article/3850763/88-of-ai-pilots-fail-to-reach-production-but-thats-not-all-on-it.html (Accessed: 26 October 2026).
sourceCode (2026) What Three Failed AI Vendor Selections Have in Common (A Procurement Post-Mortem). 17 September. https://www.sourcecode.com.au/blog/ai-vendor-selection-criteria-insurance-banking (Accessed: 26 October 2026).