TDMRep
TDMRep — the TDM Reservation Protocol — is a web protocol for declaring whether you reserve your rights over text and data mining of your content, published as a JSON file at a fixed address on your domain.
It answers a different question from robots.txt. Robots.txt says who may fetch a page; TDMRep says what may legally be done with it once fetched. The two are often conflated, and they are not the same permission.
What does TDMRep actually declare?
A file at «/.well-known/tdmrep.json» carrying two properties, which is the whole protocol. Its own abstract calls it «a simple and practical Web protocol, capable of expressing the reservation of rights relative to text & data mining (TDM) applied to lawfully accessible Web content».
- `tdm-reservation` — a boolean. Whether you reserve your rights over mining this content.
- `tdm-policy` — a URL pointing to the conditions under which mining is licensed, if it is.
- The address follows RFC 8615, the standard that reserves `/.well-known/` for declarations of this kind.
Is TDMRep a W3C standard?
No, and the specification says so about itself in plain words: «This specification was published by the Text and Data Mining Reservation Protocol Community Group. It is not a W3C Standard nor is it on the W3C Standards Track».
It is a Final Community Group Report, published on 10 May 2024. That is a real and finished document produced by an open working group, and it is not the same thing as a W3C Recommendation. Almost everyone who cites it gets this wrong.
Why would a site publish one?
The motivation is legal rather than technical. Article 4 of the European copyright directive on the digital single market lets rightsholders reserve text and data mining, and reserving it requires expressing that reservation in a machine-readable way. TDMRep is an attempt at the format.
Whether it is honoured is a separate matter from whether it is expressed, and this is where honesty matters: declaring a reservation is not the same as enforcing one. Treat it as stating a position, not as a technical block.
Note the lag: the European directive it answers is from 2019 and this format arrived in 2024, 5 years later. A right you cannot express in a form machines read is a right that is hard to exercise at web scale.
Where does this come from?
The specification itself, the standard that governs its address, and the documentation that shows why permission to fetch and permission to use are two questions.
- TDM Reservation Protocol, W3C Community Group Final Report
The primary source, published 10 May 2024. Both quotations above come from it, including the one where it states it is not a W3C Standard.
- RFC 8615, Well-Known Uniform Resource Identifiers
The standard that reserves `/.well-known/` and gives an address like this one its legitimacy.
- OpenAI crawler documentation
Shows the other half of the picture: which bot fetches for training and which for answering. Access and use are governed separately, and TDMRep only speaks to use.
Frequently asked questions about TDMRep
Is TDMRep the same as blocking AI crawlers?
- No. Blocking a crawler in robots.txt stops it fetching the page. TDMRep declares what may lawfully be done with content that was fetched legitimately. A site can allow every crawler and still reserve mining rights, and that combination is coherent rather than contradictory.
Does publishing TDMRep stop my content being used for training?
- TDMRep states a reservation; it does not enforce one. Whether a given operator honours it depends on that operator and, ultimately, on jurisdiction. Publishing one records your position in a machine-readable form, which is what the European provision it responds to asks for.