COMMUNITY RESEARCH · April 2026
Framework Compliance
Empirical tracking of which AI agent frameworks honor SOM Directives when encountered in robots.txt. Based on the Discovery Gap research by Hurley (2026).
As of April 2026, no major framework has reached Level 1 compliance. This page tracks progress.
01
Compliance Levels
The SOM Directives specification defines three compliance levels. See Directives for the full spec.
Level 0
Aware — Parses robots.txt for SOM-* directives, does not change behavior.
Level 1
Respectful — Honors SOM-Endpoint and fetches SOM instead of raw HTML when present.
Level 2
Full — Honors all directives including rate limits, attribution, paywall paths.
02
Current Status
| Framework | Maintained by | Level | Notes |
|---|---|---|---|
| LangChain | LangChain Inc. | Level 0 | No robots.txt SOM parsing. HTML fetching via WebBaseLoader. |
| LlamaIndex | LlamaIndex Inc. | Level 0 | SimpleWebPageReader fetches raw HTML. No SOM directive support. |
| Browser Use | browser-use | Level 0 | Chromium-based. Does not check robots.txt directives. |
| CrewAI | CrewAI Inc. | Level 0 | Uses LangChain tools underneath; inherits same gap. |
| AutoGen | Microsoft | Level 0 | WebSurferAgent fetches HTML. No SOM awareness. |
| Smolagents | Hugging Face | Level 0 | VisitWebpageTool fetches raw HTML. |
| Plasmate | Plasmate Labs | Level 2 | Reference implementation. Checks robots.txt before fetching. |
| OpenClaw | OpenClaw | Level 1 | Checks robots.txt. SOM endpoint discovery in progress. |
03
What Level 1 Compliance Requires
For framework authors implementing SOM directive support:
- On each fetch, retrieve robots.txt for the target domain (if not already cached).
- Parse all SOM-* directives using the spec at somspec.org/directives.
- If SOM-Endpoint is present and SOM-Format is "SOM/1.0" (or preferred), fetch from endpoint instead of raw HTML.
- Pass the SOM JSON to the model context instead of the raw HTML response.
- Respect SOM-Freshness by caching SOM responses appropriately.
04
Open Source Tracking
This page is manually maintained. If your framework has implemented compliance, open an issue or PR at github.com/dbhurley/somspec.