About This Project

The Agent Task Break Detector is a fine-tuned ModernBERT-base cross-encoder that answers a single question on each user turn of an AI agent session: whether the new message depends on the existing conversation or is a new task that a fresh session could handle. The model outputs a probability that the message is a new task. A harness hook can use this score to start a new session automatically when an unrelated task is introduced into a long-running session. Because it runs locally on every turn, it is designed as a small encoder rather than a remote API call. The project is at version 0.1 and is labelled experimental; it was trained only on synthetic data and has not yet been evaluated on real, hand-labelled sessions.

The model takes the new message as one input segment and the four most recent agent and user exchanges, newest first, as a second segment, with a maximum length of 512 tokens. It was fine-tuned from answerdotai/ModernBERT-base on the Agent-Task-Break-Detector dataset using the Hugging Face transformers library, and is distributed in both fp32 safetensors and dynamically quantised int8 ONNX formats. On a synthetic test split of 608 rows, the fp32 model reached 0.951 accuracy and 0.986 AUC-PR, compared with 0.760 to 0.793 accuracy for TF-IDF baselines. The int8 ONNX version reached 0.929 test accuracy, and the documentation recommends a separate threshold of 0.76 for that format because its probabilities are lower than fp32 output.

The model is most relevant to developers building coding agents or other long-running LLM tool sessions who need to manage context windows and decide when to clear conversation history. The documentation recommends operating at high precision for the new-task class, since a wrong split discards useful working context. It is English-only and has not been measured on real session data, which the authors note is longer and messier than the training set.

Project Details

Tags

transformersonnxsafetensorsmodernberttext-classificationagentscontext-managementtopic-shiftcross-encoderendataset:danielrosehill/agent-task-break-detectorbase_model:answerdotai/modernbert-basebase_model:quantized:answerdotai/modernbert-basedoi:10.57967/hf/11383license:apache-2.0text-embeddings-inferenceendpoints_compatibleregion:us

Explore More Projects