looptimum Black-box optimization for expensive evaluations Discuss a pilot

Anonymized ML campaign

Guided GPT training-recipe search in ten bounded evaluations.

Four deterministic initialization evaluations established coverage, followed by six guided evaluations through a client-owned training workflow.

Discuss a pilot Read the public evidence

Observed campaign results

10/10

Successful evaluations

4 initialization + 6 guided

1.02%

Lower held-out loss

Selected versus fixed baseline

~25%

Fewer parameters

Selected versus fixed baseline

Result boundary

A useful observed result, presented at its actual scale.

The selected candidate recorded a held-out loss index of 98.98 against the fixed baseline at 100.00—1.02% lower—while its normalized parameter index was 75.4 against 100.0. This was a single-seed, small-budget campaign, not a claim of statistical significance or global optimality.

Campaign method

Bound the recipe, then learn from every completed evaluation.

The search exposed four generic controls: learning rate, microbatch size, gradient accumulation, and model depth. Four deterministic initialization evaluations were followed by six guided evaluations. All ten completed successfully.

Learning rate Microbatch size Gradient accumulation Model depth

Evaluation boundary

The evaluator stayed external and client-owned.

Looptimum proposed a bounded candidate, the training workflow returned one finite locked held-out loss index, and Looptimum ingested the terminal observation. The controller did not need to own or embed the application-specific training environment.

Normalized campaign figures

Normalized held-out loss across ten evaluations in an anonymized GPT training campaign

Figure

Normalized objective progression

The guided phase produced the two strongest observed candidates; lower indexed loss is better.

Zero-based indexed comparisons of held-out loss and parameter count for a fixed baseline and selected candidate

Figure

Baseline versus selected candidate

The selected candidate combined a modest loss improvement with a separate reduction in model parameters.

Observed trajectory

Guidance found the two strongest observed candidates.

  • Lower held-out loss index is better; the fixed baseline is 100.00.
  • The best observed index improved to 99.10, then 98.98.
  • Later evaluations explored the bounded space without improving that result.
  • This sequence is evidence from this campaign, not a general guided-versus-random claim.

Looptimum lifecycle

Suggest, evaluate externally, ingest, repeat.

  • Suggest one candidate from the current bounded campaign state.
  • Evaluate it in the existing training workflow.
  • Ingest one terminal scalar observation into a file-backed history.
  • Resume from that auditable state for the next decision.

Limitations

What this campaign does not establish.

  • It used one seed and a small evaluation budget.
  • The selected candidate is the best observed candidate, not a proven global optimum.
  • It does not establish statistical significance or a causal effect from any one control.
  • It does not claim transfer to another workflow, broader quality, or production readiness.
  • Approximately 25% fewer parameters is not presented as 25% faster or cheaper.

Continue

Bring a bounded, expensive evaluation loop.

We can assess whether a client-owned evaluator and a small auditable campaign are a good fit for Looptimum.

Assess a similar pilot Read the public evidence