We Shuffled Our Own Indicator's Scores. The Composite Lost. Two Factors Won.

We sell an indicator that grades order blocks and fair value gaps from 0 to 100.

We wanted to know whether that grade means anything, so we used the one test that leaves little room to argue. Shuffle the scores across the zones. If graded zones only look different because of where they sit, shuffled grades will separate outcomes just as well.

The composite failed on our data. Two of its five ingredients passed decisively.

We're publishing all of it, because the same transparency that lets us write this is the product's actual pitch. A tool that prints its own track record has to survive its own audit — this is the audit.

What we tested

The SMC AI-Scored Toolkit detects order blocks and fair value gaps, then stamps each zone at birth with a 0–100 score. Five transparent factors feed it: volume conviction, displacement, liquidity sweep, higher-timeframe alignment, and time of day. The weights shift with market regime (trend, range, volatile); the score freezes at creation and never repaints.

The product's own tooltip says the score is "neither … a prediction." Its on-chart track record labels itself "small sample, NOT an edge." The pitch has always been context, not a signal.

Fine. But a 0–100 number on a zone implies, at minimum, that 80 should behave differently than 20. That is testable, so we tested it.

The design

Data. The same tape as study #1 in this series — GC=F (COMEX gold futures), 5-minute bars with real exchange volume, 50 full CME sessions from 2026-04-29 to 2026-07-10, 13,640 bars, two holiday partials excluded — re-exported at bar level, because the VP study froze aggregated profiles and this one needs raw OHLCV. Both the export script and the frozen bar file are committed; the analysis reads only the committed file.

The rig. We ported the shipped Pine rules to standalone TypeScript: detection, all five factors, regime classification, scoring, and the resolution state machine. It runs at the toolkit's default settings, with each rule annotated by the Pine line it mirrors. Every number below reproduces with one command:

node --experimental-strip-types scripts/research/smc-zone-study.ts

Outcomes. A zone counts as tested when price returns to its proximal edge. From there, the toolkit's defaults decide the race: respected if price moves 1.0 ATR in the zone's favor from that edge; failed if a bar closes beyond the far edge. Failure wins ties.

An undecided zone expires after 50 bars and is excluded, exactly as the shipped code excludes expiries from training.

The rig detected 1,729 zones (241 order blocks, 1,488 FVGs). 1,693 resolved: 982 respected, 711 failed, for a 58.0% base respect rate. Treat that number carefully.

It comes from a mechanical race between a 1-ATR bounce and a close-through. It is not a win rate, a point we return to below.

The null. For the headline test, we shuffled scores across the resolved zones 2,000 times (fixed seed, committed code). Each time, we measured the respect-rate gap between the top and bottom score terciles. Real information should beat that shuffled deck; decoration should not.

Shuffling breaks any score→outcome link while leaving everything else untouched. It needs no synthetic levels or model of "random zones," and avoids the null-design traps that bit us in study #1.

One more guard. Zones cluster: one impulse can print several FVGs onto the same subsequent price path. Those neighboring outcomes aren't independent, which makes raw p-values look better than they are.

So we re-ran everything on a thinned pool, keeping a zone only if no kept zone of the same kind and side was born within the previous 12 bars. The filter leaves 1,109 zones; we report both readings.

Finding 1 — the composite score fails the shuffle test

The toolkit's default "act" threshold is a birth score of 70. On this tape, zones at or above it respected 59.8% of the time.

Zones below 40 respected 58.1%. The middle respected 57.4%.

Those three numbers are the finding. The rest is confirmation:

On 50 sessions of gold, the composite 0–100 score — at its shipped default weights — separated zone outcomes no better than a shuffled deck of the same scores.

One sensitivity reading did flash significant. Demand a 1.5-ATR reaction, and the raw pool shows +7.9 points at p=0.003. Thin the clustered zones, and it falls to +4.3 at p=0.12.

That is a clustering artifact, not a result to sell. We report it so you can see the effect disappear.

Finding 2 — two of the five factors are real

We then ran the same shuffle test on each ingredient alone. The picture inverted.

Displacement measures how hard the impulse that created the zone moved relative to ATR. It was the strongest factor in the study and strengthened under thinning (+13.5 raw → +18.0 thinned). Volume conviction held too.

Liquidity sweep, higher-timeframe alignment, and time of day showed nothing detectable here. They were not negative; they were flat. They may earn their keep on another market or timeframe, but on this tape they were passengers.

Finding 3 — the composite dilutes its own signal

Why does a score built partly from two genuinely informative factors fail the test its ingredients pass? The arithmetic answers it.

Under the default weight tables, volume and displacement together receive 40% of the weight in a trend regime, 65% in a volatile one, and just 25% in a range. The other 35–75% goes to the three factors that measured flat here.

Range was also the most common regime in this window: 788 of the 1,693 resolved zones. So the biggest slice of the pool was scored under weights that lean hardest on the dead factors.

The composite isn't wrong because its inputs are noise. It's diluted because the defaults spread weight democratically across factors that aren't equally informative — at least not on this tape.

That is a fixable kind of wrong. Now for the useful part.

What survives, honestly

The 58% base rate is not an edge. The product already says so.

The respect definition is a mechanical race (1-ATR bounce vs close-through, failure-first). The toolkit's own dashboard labels its track record "small sample, NOT an edge."

That honesty holds. Nothing in this study contradicts the product's stated claims, because the product never claimed prediction.

What we tested is whether the grade adds information. At default weights, here: no. Two ingredients: yes.

The study points straight at a feature the toolkit already ships. Adaptive mode is off by default. Its entire job is to learn factor weights from how zones actually resolve on your chart, using online logistic regression on resolved zones only.

We built it as an option. This result gives it something real to learn toward. On gold 5m, it should, in principle, discover what the rig did: load displacement and volume, fade the rest.

The transparency loop closed. We could run this audit because the toolkit ships readable source for its zones and their factor and resolution logic. Owners can port those rules to another market and see whether the factors behave differently.

That is the product working as designed, even when the finding stings.

Limits — read before quoting

One symbol (gold futures), one timeframe (5m), one 50-session window. Factor behavior is plausibly market-specific; killzone timing on an FX pair may act nothing like it does on gold.

Our rig is a faithful port of the Pine at defaults, not the TradingView runtime itself. Every rule is line-annotated against the shipped source and the port is committed for inspection, but a port is a port.

Two shipped housekeeping rules are deliberately not applied, because they exist for TradingView's object budget rather than for the statistics: the 500-bar zone age-out and the 400-zone display pool. Adversarial review flagged the first, so the script now prints the age-pruned reading too — 46 late-resolving zones drop, the base rate stays 58.0%, the composite stays null (+1.8 pts, p=0.27 raw; −1.4, p=0.64 thinned), and displacement stays real (+13.7, p < 0.0005). No conclusion moves.

The killzone factor applies the Pine's default session strings in New York wall clock — a deliberate mapping choice, stated so it can be argued with. Chart-timezone conventions vary (COMEX itself keeps Central time), any choice shifts the windows, and the factor measured null regardless.

Zones share price paths even after thinning; our 12-bar rule reduces dependence but does not eliminate it. We also ran multiple tests.

The sensitivity grid exists to expose fragility, not to shop for significance. That is why we reported the one flashy p-value and dismissed it as an artifact.

Expired zones (tested, undecided after 50 bars) are excluded exactly as the shipped code excludes them from training: 2 zones here, immaterial.

What we're changing

Three changes follow. Otherwise, this would just be content marketing:

1. The toolkit's docs will link this study from the scoring section, so buyers see the default-weight caveat next to the feature it caveats.
2. We're opening a tune ticket: evaluate shipping regime defaults that weight displacement and volume harder, with this rig as the benchmark.
3. The adaptive mode graduates from "experimental option" to "recommended after warm-up" in the how-to, with the honest note that it learns your market's weights; those may not be gold's.

If a future window or another market overturns any number above, the committed script will say so before we do.

The shuffle test

Take any indicator score. Rank what it grades, split the top third from the bottom third, then shuffle the grades a couple of thousand times and see whether reality beats the shuffle. The test takes twenty lines of code and has no respect for marketing.

Our composite lost this round. Two of its ingredients beat it cold. Both facts are now public beside the code that produced them.

An honest score isn't one that always passes. It's one that was actually tested — and prints the result either way.