Graded on a Curve They Wrote: The Conflict of Interest Hiding in Every Vendor Health Check
Photo: Richter Frank-Jurgen, CC BY-SA 2.0, via Wikimedia Commons
Somewhere in your organization right now, an infrastructure assessment is being drafted. It will arrive as a polished PDF: a maturity score on a five-point scale, a heat map of risk domains, a gap analysis comparing your environment to an "industry benchmark." It will look authoritative. It may well be accurate.
It will also have been produced by the company most likely to win the contract to fix whatever it finds.
This is the room rater's dilemma, and it is more common than most procurement teams care to admit. The health check, the readiness assessment, the optimization review — these artifacts carry the visual weight of an independent audit while serving a fundamentally different function. Understanding that difference is not cynicism. It is due diligence.
The Grade Is the Sales Funnel
Consider the incentives at play. A systems integrator performs a "complimentary" infrastructure assessment and assigns your data center a maturity score of 2.4 out of 5. That number does real work: it creates urgency, justifies budget requests, and establishes a baseline against which future remediation — conveniently, remediation they can provide — will be measured.
None of this requires bad faith. A competent assessor may genuinely believe the score. But the structural conflict remains: when the same firm defines the criteria, administers the test, and sells the tutoring, the grade is less a measurement than a proposal. A 2.4 from a vendor with remediation services to sell and a 2.4 from a party with nothing at stake are two very different numbers, even if the underlying infrastructure is identical.
The pattern is familiar to anyone who has watched the US financial industry evolve. Credit rating agencies being paid by the issuers they rated was a central fault line of the 2008 crisis. The lesson was not that raters were dishonest — it was that incentive structures shape outputs regardless of individual integrity.
Where Vendor Assessments Systematically Drift
Drift shows up in predictable places, and knowing them helps you read between the lines of any assessment report.
The benchmark is proprietary and unfalsifiable. "Industry standard" is invoked without citation. You cannot verify what the 3.0 baseline represents, who else scored there, or whether the scale has ever been calibrated against anything external. A score you cannot interrogate is an assertion, not a measurement.
Gaps map suspiciously well to the services catalog. Run the experiment: overlay the assessment's recommendations against the vendor's service offerings. When every identified gap has a corresponding line item in the remediation catalog, you are not reading findings. You are reading a quote in disguise.
Complexity is scored as risk. Vendor assessments routinely penalize heterogeneity — multiple hypervisors, mixed cloud providers, a legacy platform still doing real work. But heterogeneity is sometimes the rational residue of deliberate decisions. A firm that profits from consolidation will always find sprawl.
The remediation timeline is conveniently tight. Urgency is the assessment's most valuable byproduct. "Critical gaps" with 90-day remediation windows compress procurement review and discourage competitive bidding. Genuine risk rarely keeps a vendor's sales calendar.
A Framework for Stress-Testing the Grade
You do not need to reject vendor assessments outright. You need to treat them as one input among several, weighted accordingly. A practical discipline:
1. Demand the methodology before the score. Ask for the rubric, the weighting, and the benchmark population before any findings are presented. A vendor that resists disclosing how the grade was computed has told you something important before the engagement begins.
2. Split assessment from remediation — contractually. If a vendor performs the assessment, bar them from bidding on the remediation without a fresh competitive process, or vice versa. Even the appearance of separation changes how findings are framed.
3. Anchor to an independently validated baseline. Before any vendor walks your floor, establish your own reference points: internal incident history, capacity utilization data, and at minimum one assessment from a party with no services to sell. Frameworks from NIST and the CIS Controls provide externally auditable anchors that a proprietary maturity scale cannot match.
4. Require findings to be reproducible. Every gap should cite evidence — telemetry, configuration data, ticket histories — that your own team can verify. An assessment you cannot re-derive is an opinion with formatting.
5. Test the counterfactual. For each recommendation, ask: who benefits if we do this, and who benefits if we don't? Then ask what a firm with no stake in the answer would say. The distance between those two answers is your conflict-of-interest premium.
Independent Baselines Are Cheaper Than Wrong Bets
The objection is always cost. Independent assessment takes time; the vendor's health check is free. But free is doing a lot of work in that sentence. An inflated maturity gap can steer millions toward remediation you didn't need, while an inflated score can lull you into deferring remediation you did. Either error costs more than the independent baseline that would have caught it.
The strongest enterprises we work with don't treat vendor assessments as verdicts. They treat them as opening bids — useful signal, clearly marked as coming from an interested party, and always tested against a baseline the vendor didn't write. Your infrastructure deserves a grade. Just make sure someone other than the grader holds the answer key.
