Demo

CP-Agent is a simple agent loop from Stefan Szeider’s paper on agentic constraint programming. The model is asked what to do next. If it replies with Python code, the code runs, the output is added to the conversation, and the model is asked again. When the model replies that it’s done, the loop stops and returns its answer. I wrote this loop in µNorman and ran it against Claude Sonnet 5 and a real Python process. Below are the traces from three of those runs.

How to read a trace

Each line is one time the program reached outside of itself. An ask line is a call to the model, and a call line is a call to the Python kernel. The columns are the input and output tokens, the cost, when the call started and ended (counted from the start of the run), and how it turned out. µNorman records these lines for every run automatically, so the trace is a complete record of what the agent did and what it cost.

1. A one-step problem

Task: What is the sum of all multiples of 3 or 5 below 1000?

┌─ sum of multiples of 3 or 5 below 1000
│  ask   step     469 in    50 out   $0.001438    182ms → 2845ms   ok
│  call  py/exec     0 in     0 out       $0.00   2845ms → 2916ms   ok
│  ask   step     524 in    24 out   $0.001288   3037ms → 4695ms   ok
│
│  2 model calls, 1 kernel calls
│  spent $0.002726 of the $0.50 budget in 4695ms
└─ answer: "233168"

The model wrote one piece of Python, the kernel ran it, and the output was added to the conversation. That’s why the second ask sent 524 input tokens instead of 469: the conversation now included the code and its output. The model then replied that it was done, with the answer 233168, which is correct. The whole run cost about a quarter of a cent and took under 5 seconds.

2. A word problem

Task: A farmer has chickens and cows, 30 heads and 74 legs. How many of each? Work it out with code and check your answer.

┌─ chickens and cows
│  ask   step     489 in   130 out   $0.002278    150ms → 3336ms   ok
│  call  py/exec     0 in     0 out       $0.00   3336ms → 3425ms   ok
│  ask   step     598 in    65 out   $0.001846   3525ms → 4929ms   ok
│
│  2 model calls, 1 kernel calls
│  spent $0.004124 of the $0.50 budget in 4929ms
└─ answer: "7 cows and 23 chickens (7*4+23*2=74 legs, 7+23=30 heads)."

I asked the model to check its answer, so I expected more rounds than in the first problem. Instead, it wrote one piece of code that solved the problem and checked the result at the same time, then gave its answer. The answer is right: 23 chickens and 7 cows.

3. The same problem with too little money

The same program and the same first problem, but with a budget of $0.002 instead of $0.50.

┌─ the same task, capped at $0.002
│
│  0 model calls, 0 kernel calls
│  spent $0.00 of the $0.50 budget in 188ms
└─ failed: OverBudget

   the cap held: spent $0.00 of $0.002

(Ignore the “$0.50 budget” in the summary line. That was a labeling bug in my test report, which I’ve since fixed. The real limit for this run was $0.002.)

Nothing was spent, and the model was never called. Before every model call, µNorman sets aside the most that call could possibly cost: the tokens it’s about to send, plus the longest answer the Step type allows, plus room for the model to think. For one CP-Agent step, that worst case is about $0.04. That doesn’t fit in $0.002, so µNorman refused the call before sending anything, and the run failed with OverBudget.

This run shows the agent refusing to start, not stopping partway through. The last test in the program below shows the partway case: with a $0.041 limit, two rounds run, and the third model call is refused before it’s sent. By then the run has only spent about $0.0005. That’s the tradeoff of checking against the worst case. The budget can never be overspent, but a run can stop long before its actual spending gets close to the limit.

You could add a budget check like this to an agent written in Python. The difference is that in µNorman the check can’t be forgotten or skipped. A program can only reach the outside world through ask (the model) and call (a tool), and both check the budget before anything happens.

Try the same program here

This is the program that produced the traces above, with some scripted tests added at the end. In the live runs, it talked to a real model and a real Python process. Here, a script plays both parts and the clock is simulated, so it’s free and runs offline in your browser. Press Run to try it.

cp-agent.nrm

(grant claude (model))                       ; defaults: Claude Sonnet 5
(grant py     (kernel [exec (Text 4000)]))   ; a real Python process in live mode

(datatype Step
  [Exec (code (Text 2000))]      ; "run this and show me the output"
  [Done (answer (Text 500))])    ; "this is my final answer"

(val cp-prompt
  "You solve problems by writing Python. Reply with Exec to run code, and you
will be shown its output. Reply with Done when you have the answer. Write
small steps, and print what you want to see.")

;; If the Python code fails, the error goes back to the model so it can fix
;; its code. Running out of money or time doesn't go back to the model: it
;; stops the program.
(define observe (k code)
  (catch (call k exec code) e
    (case e [(ToolError m) m] [_ (fail e)])))

(define react (model k ctx fuel)
  (if (<= fuel 0)
      (fail 'out-of-fuel)
      (case (ask model Step ctx 'step)
        [(Exec code)
           (react model k
                  (append ctx (list (Message [role Assistant] [content (show (Exec code))])
                                    (Message [role Tool]      [content (observe k code)])))
                  (- fuel 1))]
        [(Done answer) answer])))

(define cp-agent (model k task)
  (budget ([cost $0.50] [time 5min])
    (react model k
           (list (Message [role System] [content cp-prompt])
                 (Message [role User]   [content task]))
           8)))

;; Scripted, the model and kernel are canned, so this is exact and free.
(script solving
  [step    (reply "{\"tag\":\"Exec\",\"code\":\"print(sum(range(101)))\"}") (out 20) (latency 3s)
           (reply "{\"tag\":\"Done\",\"answer\":\"5050\"}")                 (out 12) (latency 2s)]
  [py/exec (result "5050") (latency 1s)])

(under ([script solving] [cost $1.00] [time 5min])
  (check-expect (cp-agent claude py "What is the sum of 0 to 100?") "5050")
  (check-within (cp-agent claude py "What is the sum of 0 to 100?") ([time 6s])))

;; The first piece of code fails, the model sees the error, and the second works.
(script recovering
  [step    (reply "{\"tag\":\"Exec\",\"code\":\"print(undefined_name)\"}") (out 20) (latency 2s)
           (reply "{\"tag\":\"Exec\",\"code\":\"print(42)\"}")             (out 18) (latency 2s)
           (reply "{\"tag\":\"Done\",\"answer\":\"42\"}")                  (out 12) (latency 2s)]
  [py/exec (error "NameError: name 'undefined_name' is not defined") (latency 1s)
           (result "42") (latency 1s)])

(under ([script recovering] [cost $1.00] [time 5min])
  (check-expect (cp-agent claude py "Print 42.") "42"))

;; Stopping partway: each model call sets aside about $0.04 for its worst case,
;; so with a $0.041 limit two rounds run and the third call is refused unsent.
(script grinding
  [step    (reply "{\"tag\":\"Exec\",\"code\":\"x = 1\"}") (out 15) (latency 1s)
           (reply "{\"tag\":\"Exec\",\"code\":\"x = 1\"}") (out 15) (latency 1s)
           (reply "{\"tag\":\"Exec\",\"code\":\"x = 1\"}") (out 15) (latency 1s)]
  [py/exec (result "") (latency 0s) (result "") (latency 0s) (result "") (latency 0s)])

(define capped (cap)
  (budget ([cost cap])
    (react claude py
           (list (Message [role System] [content cp-prompt])
                 (Message [role User]   [content "Keep going."]))
           8)))

(under ([script grinding] [cost $1.00] [time 5min])
  (check-fail   (capped $0.041) OverBudget)
  (check-within (capped $0.041) ([cost $0.001] [time 2s])))

What these runs show (and what they don’t)

µNorman doesn’t make the model any smarter. The quality of the answers comes from the model and the prompt. The CP-Agent paper even found that a 50-line prompt worked better than an 800-line one, and that adding more structure to the loop made the results worse.
Nothing here is cheaper or faster than writing the same agent in Python. What µNorman changes is that the cost is capped before the run starts and every call is recorded.
These runs don’t test concurrency. µNorman can run independent steps at the same time, and in a simulated example four calls finish in 40 seconds instead of 60. But CP-Agent is a sequential loop, so none of that shows up here.

To really measure performance, I would run my µNorman version of CP-Agent side by side with the original Python version on part of the CP-Bench benchmark, and compare how many problems each solves, the cost per solved problem, and how long each takes. I haven’t done that yet, so I can’t claim that µNorman makes agents perform better.

What these runs do show is narrower: with µNorman, you know exactly what an agent did and what it cost, and you know it can’t spend more than you allowed.

Running it yourself

To run the scripted tests on your own machine (free, no API key needed):

cargo run -- examples/cp-agent.nrm

To run against the real model and a real Python process, set ANTHROPIC_API_KEY and make sure Python is on your PATH. Each run costs a few cents:

cargo test --test live_cp_agent -- --ignored --nocapture
cargo run -- --live --python python3 --trace examples/cp-agent.nrm

Warning: there is no sandbox. The Python kernel runs whatever code the model writes, with your permissions, on your machine. Run it inside a container or virtual machine you don’t mind losing.

LIVE-SETUP.md explains how to get an API key and how to cap your spending.