When a problem lands on my desk, I first work out what it actually requires and picture what a successful output would look like. If I cannot picture that output, I take it as a sign that I have not understood the problem yet; work before that point is likely to be noise.
- My next steps depend on what I know already:
I do not stop at the first working solution. I experiment with new methods and different angles, and probe its edges. Experiments can improve a solution or break it; either way, I learn something and keep the lesson for the next problem of that shape. I would rather spend twenty minutes understanding why something works than one minute copying a solution that happens to work.
For a first reinforcement-learning project, I recommend trying MiniGrid with a memory variant: it is small enough to run on a laptop while still being difficult enough to be useful. This is a practical starting point alongside my work on problem solving and adaptive computation.
When I need to diagnose failure on a hard task, I compare a smaller version of the same model on a smaller version of the same task. If the smaller setup works but the real setup fails, I treat capacity or data as the likely problem; if the smaller setup also fails, I suspect something in the pipeline is broken.
Before committing to a new method, I look for the smallest experiment that would tell me whether it is worth the full effort. If I can describe that experiment in two sentences, I run it; if I cannot, I do not yet understand the method well enough.
I stop tuning and ship when my next proposed change is a guess rather than a hypothesis—when I have no clear reason to expect it to help.
In ADT-DQN, I explored whether a value-based agent could spend compute only on states where it is uncertain. Transformer Q-networks such as DTQN perform well on partially observable tasks, but run their full depth even on trivial states. Adaptive-computation methods such as ACT and PonderNet had been used in supervised learning, but were regarded as unstable with replay buffers in value-based reinforcement learning; I saw this combination as an open problem.
I put a Q-value head at every Transformer layer and supervised each head against the Bellman target from the deepest head. I unrolled all layers during training so the replay buffer always received a stable signal, and halted only at action time: the agent stopped at the first layer where the variance of Q-values across actions fell below a threshold. I added a compute-cost penalty to the loss to create pressure to stop early.
I compared five halting signals: TD-error, uncertainty, action agreement, Q-entropy, and representation change. I considered TD-error the most principled signal because it is what the method ultimately tries to minimise, and used uncertainty as a target-free practical stand-in. I also checked a depth histogram for collapse to a single depth, a common and silent failure mode in this setup.
On MiniGrid-MemoryS9, a sparse-reward memory task, average depth fell to 3.6 from 6.0, about 40% less compute per decision.
On MiniGrid-MemoryS9, success rate fell to about 0.38 from about 0.45. I regarded this as a hit, but small relative to the compute saved—a tradeoff worth tuning rather than evidence that the approach is broken.
Baseline DQN and DRQN stayed near zero on the same task. I concluded that ADT-DQN learned to use less compute without collapsing, addressing the research question while giving up some success rate. The available results do not establish which of the five halting signals works best, what threshold or compute-cost penalty settings to use, or whether the result generalises beyond MiniGrid-MemoryS9.
In retrospect, I would strengthen the evaluation before drawing broader conclusions. I want to run multiple seeds and report confidence intervals, and test in more than one environment. The current result is only one point on the compute-versus-success curve, so it does not show how the tradeoff changes across settings.
- I want to implement true early exit rather than only measure where the agent would have stopped, and measure the actual wall-clock speedup.
My goal is to map the full compute-versus-success tradeoff: the current result is one point on a curve, and I want to see the curve. The number of seeds, confidence-interval method, environments, and sweep settings remain to be chosen.
- I want to build and test TD-error halting, the signal I argued for but did not test.
- I want to sweep both the halting threshold and the compute-cost penalty, rather than report only one setting.
In value-based reinforcement learning, I think treating network depth as fixed wastes compute: the same number of layers is used for every state and decision. I see adaptive computation as a more interesting research direction than simply making the model bigger, with preserving the replay buffer as a constraint.
My example is a transformer in a Q-network running six layers of inference to decide “turn left” in a hallway with one door.
The research question I want to investigate is how to teach a model to stop when it already knows the answer, without breaking the replay buffer. Many people in the area think adaptive computation and value-based RL do not mix; I disagree, and think the problem may be that people have been halting at the wrong time. I do not yet specify a method or evaluation, so how to identify the right stopping time remains open.
- If I have solved a similar problem, I start from what worked.
I am sharpest when people ask me how to do something, rather than what they should do. I draw on experience when it applies; otherwise, I take the problem apart until I find a way forward. I want to understand why a solution works, not just that it works. I enjoy how questions because they use what I already know and push me to learn when a problem is new. For an unfamiliar question, I may give a better answer than someone who has seen it a hundred times: I reason from first principles instead of matching the problem to a template that only half-fits.
- If I have not, I look at how others solved problems with a similar shape: what they tried, found, and learned.
- If neither gives me a way forward, I break the problem into the smallest steps I can and repeat the same process on each piece.