Who needle is for#
IoT developers building always-on device agents
Needle fits edge devices that need a persistent tool-calling agent without cloud connectivity. Declare your device's actuators as tools, pass the user query, and the model returns a grammar-guaranteed call. No API key, no network round-trip, no per-token billing.
Skip if:
Your device has a reliable network connection and latency is not a constraint. Cloud function-calling APIs offer larger context and more capable reasoning for connected scenarios.
Python developers doing structured extraction in pipelines
The needle.extract() helper takes a Pydantic model and a text string and returns a typed object. Because extraction is tool calling with one declared schema, schema conformance is guaranteed. Fits document parsing, receipt extraction, and entity recognition steps in ETL pipelines.
Skip if:
Your source text exceeds roughly 200 tokens. The 256-token context limit makes Needle unsuitable for full-document extraction; chunk or preprocess the text first.
ML engineers fine-tuning on proprietary tool schemas
The LoRA fine-tuning workflow synthesizes training data from your tool schemas, fine-tunes on the frozen base, and exports a self-contained .cact that runs on the same engine. The tuned model is a drop-in weight replacement with no recompilation needed.
Skip if:
Your tool schemas are general-purpose and similar to standard function-calling patterns. The base model already handles common tool-calling tasks without fine-tuning.
The problem it solves#
Running capable AI agents on tiny devices has a hardware problem: most language models need gigabytes of RAM and depend on cloud APIs for every inference call. A phone, robot, or smart home hub running an agentic workflow cannot afford the latency, bandwidth, or per-token cost of routing every tool call to a cloud provider.
The output reliability problem compounds the hardware constraint. Cloud function-calling APIs return probabilistic text that application code must parse and validate before acting on. Schema mismatches, hallucinated arguments, and malformed JSON are recurring failure modes, each requiring error-handling code that adds latency and complexity. On a device where a failed call cannot cheaply retry against a larger model, that failure handling is the difference between a working agent and a broken one.
How it solves it#
Grammar-constrained tool calling
Every token the model emits is constrained by a byte-level grammar compiled from your tool schemas. The result is JSON that is guaranteed to conform to your schema, not just likely to. The model cannot produce a malformed argument or invent a field not declared in the schema.
14MB single-binary engine
Weights and inference engine are baked into a single 14MB binary. A full conversation session runs in approximately 28MB of RAM. There are no separate model files to manage and inference makes no network calls after the one-time engine download.
Calibrated confidence scoring
Every response carries a confidence score computed from two signals: a calibrated post-hoc scoring head and the decoding probability of the call tokens. A call is accepted only when both agree. Set a threshold, act above it, and route to a larger model below it.
Tool retrieval for large catalogues
Declare more than five tools and a built-in contrastive retrieval head embeds each schema once, then selects the five highest-scoring tools per turn. The grammar rebuilds over only that subset. An unselected tool is unreachable, not just unlikely, so the model focuses correctly.
LoRA fine-tuning and single-file export
Fine-tune on your domain with LoRA on the frozen base. At export the adapter merges into the base and quantizes to a single .cact file that runs on the same engine with no recompilation. Training data can be synthesized from your tool schemas with the needle generate-data command.
Bounded memory across long sessions
A 256-token sliding window keeps tools pinned as KV sinks, so total session memory stays near 28MB regardless of conversation length. Long agentic loops do not accumulate unbounded memory over time.
Strengths and trade-offs#
Strengths
- No cloud dependency at inference timeAfter the one-time engine download, inference runs fully offline. No API keys, no per-token billing, no network round-trip latency. This matters for robotics, offline assistants, and edge devices where network connectivity is intermittent or restricted.
- Smallest model in its benchmark classAt 14MB in CQ2-bit quantization, Needle 2 is 5x to 70x smaller than FunctionGemma 270M, LFM2.5 230M, and Apple FM while trading benchmark wins on tool-calling tasks. The size difference is what makes it the only option when device RAM is below 100MB.
- Schema-guaranteed structured outputDecoding is constrained by a compiled byte-level grammar, so JSON output is always schema-conformant. Application code does not need to validate the model's response before acting on it. This removes an entire class of error handling from on-device agent code.
- MIT license with no usage restrictionsMIT license means you can run it commercially, fork it, and integrate it into proprietary products without restriction. No AGPL copyleft, no commercial use clause, no per-seat fee.
Trade-offs
- -No free-text generationNeedle solves every request as a function call. A query that no declared tool can serve returns an empty call [] with no text output. There is no free-text fallback. If your use case requires a mix of tool calls and conversational responses, a general-purpose LLM with optional tool use is the right fit instead.
- -256-token context windowThe sliding context window caps usable context at 256 tokens. Long or complex queries that require reasoning over extended input will exceed the window. Needle is designed for short, task-focused exchanges, not document-length reasoning or multi-turn planning over large histories.
- -Python-only packageThe pip package is Python-only. Integrating Needle into non-Python runtimes requires calling the engine via subprocess or waiting for additional language bindings to be released by the Cactus Compute team.
needle vs alternatives#
Needle vs Cloud Function-Calling APIs
Needle targets the same tool-calling task as OpenAI function calling, Anthropic tool use, and Google Gemini function calling, but the deployment model is different. Cloud APIs require a network round-trip, an API key, and per-token billing on every call. Needle runs the model on-device; once the engine is cached, inference makes no network calls.
| Feature | Needle 2 | Cloud APIs (OpenAI, Gemini) |
|---|---|---|
| License | MIT | Proprietary |
| Inference location | On-device | Cloud |
| Network required at inference | No | Yes |
| Per-token billing | No | Yes |
| Model binary size | 14MB | N/A (cloud-hosted) |
| RAM required | ~28MB | N/A (cloud-hosted) |
| Context window | 256 tokens | 8k-128k+ tokens |
Needle is the better choice when the device is intermittently connected, round-trip latency is unacceptable, or per-token costs are a constraint at scale. Cloud APIs are the better choice when the task requires a long context window, complex multi-step reasoning over large inputs, or general-purpose language generation alongside structured output. Needle has no free-text fallback.
Needle vs Other Small On-Device Models
The README benchmarks Needle 2 against FunctionGemma 270M, LFM2.5 230M, and Apple FM on tool-calling tasks. At 14MB (CQ2-bit), Needle is 5x to 70x smaller than those models while trading benchmark wins.
| Model | Parameters | Approximate size |
|---|---|---|
| Needle 2 | 45M | 14MB (CQ2-bit) |
| FunctionGemma | 270M | ~540MB (f16) |
| LFM2.5 | 230M | ~460MB (f16) |
The size difference is the primary reason to choose Needle: it fits hardware where 200M+ parameter models cannot run. The trade-off is model capacity. The larger models have more parameters available for varied tool descriptions and edge cases, and may handle out-of-distribution queries better without fine-tuning. Needle's grammar-constrained decoding gives it a structural output guarantee those models do not explicitly claim.
Install and self-host#
Install via pip; the inference engine downloads from Hugging Face on first use and caches locally.
```bash
pip install cactus-needle
```What it's built on#
- Languages
- JavaScriptPython
FAQ#
Is Needle free to use commercially?
Yes. Needle is MIT licensed. You can use it commercially, fork it, and integrate it into proprietary products without restriction. The only cost is the compute on your own device or server.
How does Needle compare to OpenAI function calling?
OpenAI's function calling requires a network call to OpenAI's servers; there is no on-device option. Needle runs entirely on-device with no network dependency at inference time, at the cost of a smaller context window (256 tokens vs thousands) and no free-text generation. For tool-calling tasks on edge hardware, Needle trades benchmark wins with models 5x to 70x its size.
Can Needle generate conversational text responses?
No. Needle solves every request as a function call. A query that no declared tool can serve returns an empty call [] with no text output. If you need a mix of tool calls and conversational language generation, you need a different model. Needle is purpose-built for structured output only.
How do I install Needle?
Install with pip install cactus-needle. The inference engine downloads once from Hugging Face on first use and caches locally. Subsequent runs are fully offline. Describe your tools with the @needle.tool decorator and call agent.run() to query the model.
What devices can Needle run on?
Any device with a Python runtime and approximately 28MB of available RAM. The README lists phones, wearables, smart home hubs, robots, and microcontrollers as target hardware. The model weights and engine together are 14MB.
Similar open-source tools#
r3
Minimalist Forth language for games and systems programming
agency-agents
Expert AI agent personalities for every workflow
t3code
Control your coding agents from one interface
open-seo
Open source alternative to Semrush and Ahrefs
PyScrappy
AI-native web scraping toolkit + MCP server
transformers
Model-definition framework for state-of-the-art ML