NLASmith · CONAIISI 2026
Neuronpedia makes Natural Language Activations easy to inspect one at a time. NLASmith is the missing loop: datasets, token policies, configurable judges, and aggregated metrics. This lets hypotheses about internal representations be run, reproduced, and compared.
The gap · 01
Natural Language Autoencoders turn a hidden activation into a sentence and can reconstruct the activation from that text. Those verbalizations are interpretable, but they are not a literal or infallible readout of the model’s state.
02
Neuronpedia already exposes NLA through a web UI and an API. Coordinating that across many examples while keeping the decisions that produced each result is still left to ad hoc scripts.
The artifact under test is not only the visible reply. It is the NLA at positions fixed before the run begins.
03 · Method
Datasets, repeated runs, automatic evaluators, comparative views. NLASmith applies that methodological shape to the NLA at positions fixed by the experiment, rather than only the visible reply.
04 · Scope
This is not a new interpretability method and it does not train a new autoencoder. It is so the same flow can test different hypotheses without being locked to one phenomenon or domain.
Conceptual pipeline
Each dataset example runs under a fixed configuration. Selected positions go to the NLA service. Verbalizations are scored with criteria defined up front.
A fixed list of prompts is the unit of comparison. Two experiments on the same set can be contrasted per example and as aggregates.
The observed position is part of the definition: last user token, first assistant token, or both. It is not something decided after reading each reply.
Neuronpedia runs the completion and returns the verbalization at those positions. NLASmith does not train or host the autoencoder.
A rubric, a judge model, and an output schema: boolean, score, or category, with optional reasoning. The judge is a measuring instrument, not ground truth.
Per example artifacts stay inspectable. Presence rate, mean score, and category mix let you see patterns instead of isolated screenshots.
Product
The first version keeps only what turns a manual inspection into a reproducible process.
Create a prompt list, identify each example, and optionally attach a reference. The dataset is what makes two runs comparable.
Keep the position rule constant so last user versus first assistant is an experimental factor, not an afterthought.
Neuronpedia hosts the language model and the NLA. The lab wraps completion and explain into one internal schema.
Natural language rubric, judge model, and feedback keys. The same loop can detect a theme, score a criterion, or classify verbalizations.
The orchestrator walks the dataset, records progress per example, and separates API failures from negative judgments.
Tables and charts on the same prompts. Inspect a single verbalization without losing the global view of the experiment.
An experiment
In some open-ended questions, verbalizations seemed tied to Reddit or forum style discourse even when the generated answer never named that source. We ran that hunch as an experiment in the prototype: dataset, token position, NLA source, rubric, and aggregated metrics.
How to read the numbers
An NLA can simplify or invent detail. Fraser-Taliente et al. already warn against treating the sentence as a faithful dump of the activation.
An LLM judge moves with the rubric, the judge model, and what you put in context. Those choices are stored with the run so the measurement can be audited.
Comparing models may mix layers or autoencoders that are not strictly equivalent. Cost grows with prompts, positions, and evaluators.
Datasets, evaluators, and experiment rows persist. Open the prototype, attach a judge, and turn a one time inspection into a configuration you can repeat.
