Robustness

How much do B-index results depend on scoring subjectivity and on the choice of aggregation function? Two internal experiments on the full Q3-2026 dataset, published here so the numbers can be checked rather than trusted.

1 · Scoring-noise sensitivity

The rubrics anchor scores at 0, 3, 5, 7 and 10; intermediate values involve evaluator judgment. To bound the effect of a one-point disagreement, each of the 11 parameters was raised by one point in isolation (2,003 perturbations across 184 platforms): the median shift in B is 4.8%, with a 90th percentile of 10.9%.

A harsher test adds independent uniform noise from {−1, 0, +1} to every parameter of every platform simultaneously, clipped to [0, 10] — 200 Monte Carlo draws for each of four fixed seeds. Rank order survives: the mean Spearman correlation with the unperturbed ranking is 0.96 in every run. Zone assignments do not: on an average draw, roughly 9% of platforms cross into a different interpretation zone.

Reading: rank-level claims are robust to ±1 scoring subjectivity; zone-level claims are sensitive near the thresholds. The V4 multi-rater protocol is designed around exactly this asymmetry.

2 · Aggregation-form ablation

Does the modified-CES aggregation change the cross-sectional ranking relative to naive alternatives computed from the same 11 raw scores?

Reading: the ranking is nearly invariant to these functional-form choices. That is reassuring for rank-level claims — and a constraint on what rank correlations can prove. Absolute B values, zone thresholds, and the CES-specific structure cannot be defended by rank agreement; they require direct testing, which is part of the V4 validation agenda. This ablation covers three naive alternatives and does not establish robustness to functional form in general.

Reproduce

The script below reproduces every number on this page from the public API. No dependencies beyond the Python standard library. Scores in the API are the live dataset; results drift only if the underlying scores change between quarterly releases.

# Reproduces every number on this page against the live API.
# python3 robustness.py
import json, math, random, statistics, urllib.request

data = json.load(urllib.request.urlopen('https://blackholeindex.com/api/bhi'))
P = data['platforms']; K = ['d','m','a','p','n','c','x','s','h','o','t']

def V3(v):
    d,m,a,p,n,c,x,s,h,o,t = [v[k]/10 for k in K]
    c4 = ((math.sqrt(d)+math.sqrt(m)+math.sqrt(a)+math.sqrt(p))/4)**2
    syn = 1 + 0.3*(math.sqrt(d*m)+math.sqrt(a*p))/2
    cap = c*(1+0.5*n)*c4*syn*(1+0.5*o)*(1+0.3*t)
    esc = 0.1 + ((math.sqrt(x)+math.sqrt(s)+math.sqrt(h))/3)**2*max(0.3, 1-0.35*c4)
    return cap/esc

def srank(v):
    idx = sorted(range(len(v)), key=lambda i: v[i]); r = [0.0]*len(v); i = 0
    while i < len(idx):
        j = i
        while j+1 < len(idx) and v[idx[j+1]] == v[idx[i]]: j += 1
        for k2 in range(i, j+1): r[idx[k2]] = (i+j)/2 + 1
        i = j + 1
    return r

def spearman(a, b):
    ra, rb = srank(a), srank(b); n = len(a)
    ma, mb = sum(ra)/n, sum(rb)/n
    num = sum((x-ma)*(y-mb) for x, y in zip(ra, rb))
    den = math.sqrt(sum((x-ma)**2 for x in ra)*sum((y-mb)**2 for y in rb))
    return num/den

def zone(b): return 0 if b < 0.4 else 1 if b < 0.7 else 2 if b < 1.3 else 3 if b < 2.5 else 4

BV = [{k: float(p[k]) for k in K} for p in P]
B = [V3(v) for v in BV]

# 1. Single-parameter sensitivity: +1 on one parameter, all others fixed
deltas = []
for v, b in zip(BV, B):
    for k in K:
        if v[k] < 10:
            pv = dict(v); pv[k] += 1
            deltas.append(abs(V3(pv)-b)/b)
print('single +1: median', statistics.median(deltas), 'p90', sorted(deltas)[int(len(deltas)*0.9)])

# 2. Scoring-noise Monte Carlo: uniform {-1,0,+1} on every parameter of every
#    platform, clipped to [0,10]; 200 draws per seed
for seed in (42, 7, 123, 2026):
    random.seed(seed); rhos = []; flips = []
    for _ in range(200):
        nb = [V3({k: max(0, min(10, v[k]+random.choice((-1, 0, 1)))) for k in K}) for v in BV]
        rhos.append(spearman(B, nb))
        flips.append(sum(1 for a, b in zip(B, nb) if zone(a) != zone(b))/len(P))
    print('seed', seed, 'mean rho', statistics.mean(rhos), 'mean zone flips', statistics.mean(flips))

# 3. Aggregation-form ablation: naive alternatives vs full V3.1
n1 = [sum(v[k] for k in ['d','m','a','p','n','c','o','t']) - sum(v[k] for k in ['x','s','h']) for v in BV]
n2 = [(sum(v[k] for k in ['d','m','a','p','n','c','o','t'])/8)/(0.1+sum(v[k] for k in ['x','s','h'])/3) for v in BV]
n3 = [(sum(v[k] for k in ['d','m','a','p','n','c'])/6)/(0.1+sum(v[k] for k in ['x','s','h'])/3) for v in BV]
print('linear diff', spearman(B, n1), 'ratio-8/3', spearman(B, n2), 'ratio-6/3', spearman(B, n3))

Limitations

1. Preliminary and internal. These experiments were designed and run by the same single evaluator who produced the scores. They bound the mechanical effect of scoring noise; they say nothing about systematic bias, which only the independent multi-rater study can address.
2. Noise model. Uniform ±1 noise is a modeling choice. Real inter-rater disagreement may be larger on some parameters, correlated across parameters, or asymmetric.
3. Rank invariance is expected, not remarkable. Monotone aggregations of the same inputs correlate highly in ranks almost by construction. The substantive conclusion is the constraint this places on validation design, not the high ρ itself.

Companion page: Validation report · Published 2026-07-20