Hugging Face
Search models, datasets, users...
Ctrl K
Models
Datasets
Spaces
Buckets
Docs
Enterprise
Pricing
Website
Tasks
HuggingChat
Collections
Languages
Organizations
Community
Blog
Posts
Daily Papers
Hardware
Learn
Discord
Forum
GitHub
Solutions
Team & Enterprise
Hugging Face PRO
Enterprise Support
Inference Providers
Inference Endpoints
Storage Buckets
Log In
Sign Up
Sajib Acharjee Dip
Sajib-006
12
10
3
Follow
webbrain-one-432451's profile picture
1 follower
ยท
18 following
https://www.linkedin.com/in/sajib006/
Sajib-006
sajib006
AI & ML interests
Multimodal Learning, Generative AI, Bioinformatics
Recent Activity
commented
on
a paper
about 21 hours ago
Invariant-Aware Harness Enforcement to Protect LLM Agents from Silent API Failures
commented
on
a paper
about 21 hours ago
Invariant-Aware Harness Enforcement to Protect LLM Agents from Silent API Failures
posted
an
update
1 day ago
๐ Your agent's API said HTTP 200. The answer was still wrong. APIs fail silently. A misnamed filter gets ignored, a total gets capped at the page size, or an AND is read as OR, and the API still returns a normal-looking number. LLM agents trust it. We found this in 11 of 15 public research APIs, with counts inflated up to 691 times. Asking the model to double-check doesn't fix it. Checklists, reviewer models and LLM judges still leave over 70% of answers wrong. Even when a judge flags the bad count, agents report it anyway 77% of the time. So we moved the decision out of the model. SilentGuard is a harness that tests every tool result against simple invariants that any correct API must satisfy: count(A and B) + count(A and not B) == count(A) If a result breaks the rule twice, the harness withholds it, and the model never sees the wrong number. ๐ Results on 7 open models: wrong answers fall from 74.0% to 1.7% on the faults the checks target, with no loss on clean tasks. On three frontier models they fall from 70.2% to 0.5%. ๐ค AgentOrchBench is a new benchmark of silent API failures with exact answers. It has count, list, retail and multi-step workflow tasks, plus 8 fault types you can switch on. from datasets import load_dataset tasks = load_dataset("Sajib-006/AgentOrchBench", "count", split="test") Dataset: https://huggingface.co/datasets/Sajib-006/AgentOrchBench Code (pip-installable, no dependencies): https://github.com/Sajib-006/silentguard Paper: https://arxiv.org/abs/2611.20595 Try the 30-second demo with no model and no API key: python examples/01_quickstart.py Joint work with Brian Ouyang and Lala Motlhabi (Genentech). Feedback welcome! ๐
View all activity
Organizations
Sajib-006
's Spaces
1
Sort:ย Recently updated
Running
BenSyc Project
๐
Bengali-context conversational sycophancy benchmark