An archive that belongs to me proves nothing

I run an archive that records what websites say. There is a problem with that sentence, and it is the word “I”.

An archive that belongs to me proves nothing. I could open the database tonight, change one row, reseal it, and nobody outside would notice. Anyone who has argued about a document in front of someone who decides things knows the reflex: whoever produced the record has an interest in what it says. So the design question was never how to store pages. It was how to make my own word irrelevant.

That question has a boring answer and an interesting one. The boring answer is cryptography. The interesting one is two machines in two countries that never talk to each other.

What is actually being fought about is the date

A website has no memory. It only has a present tense. That is fine until two parties disagree about the past, and then it is the whole problem.

Take the machine readable side of it, which is where this started. Four files carry most of what a site tells crawlers and AI systems: robots.txt, ai.txt, /.well-known/tdmrep.json for the text and data mining reservation, and llms.txt. In a dispute, nobody argues about whether a reservation exists. Go look, it is right there. The argument is about when it appeared. One side says the opt-out has been up since March. The other says it was not there at crawl time and went up afterwards. Both are certain. Neither has anything to show.

What people reach for at that point is their own server logs, their own git history, their own CMS revisions. All produced by the party that benefits from them. I am not saying that is worthless. I am saying I would not want it to be the only thing on the table.

And the same shape shows up far away from AI. A competitor changes the disputed page the day after your warning letter. A domain’s nameservers move twice during a trademark case. A platform quietly edits its terms and nobody kept the old version. Same structure every time: the thing that mattered was public on a Tuesday, and by the time it matters, Tuesday is gone.

Two witnesses, and why they must not agree

So: two machines. One in Germany, one in France. Different providers, different networks, different buildings, different jurisdictions inside the EU. Same software, installed by the same script, because identical configuration is a condition here and not a convenience. Every night at 03:00 UTC both wake up and fetch the same things from the same domains.

They do not coordinate. They do not compare notes. Each stores what it saw, builds a Merkle tree over its own day, and seals that day at 23:50 with its own root hash.

Here is the part that surprises people, and it is my favourite thing about the whole system: the two daily roots never match. Not once. They are not supposed to.

Each witness crawls on its own schedule and ends up with its own set of leaves. If the two roots were ever identical, that would not be reassuring, it would be a warning sign, because it would mean the two machines are less separate than I claim. Agreement happens one level down, at the individual observation: this domain, this file, this content hash, recorded by both, hours apart. Two parties who never spoke saying the same thing about the same object is worth something. Two parties producing a byte identical summary is worth an audit.

It inverts the usual instinct. Redundancy wants mirrors. Evidence wants independence, and independence is messier by definition.

Sealing, and what leaves my control

A daily root on its own is just a number I wrote down. It gets interesting when it leaves my hands, so every root is anchored three ways the same night.

One anchor goes into OpenTimestamps and ends up in the Bitcoin blockchain. Cheap, public, awkward to explain in a hearing, impossible to quietly rewrite.

The second is a qualified electronic timestamp from a trust service provider on the EU trusted list. That is the one lawyers care about, because Article 41(2) of the eIDAS Regulation attaches a presumption to it: the accuracy of the date and time it indicates, and the integrity of the data it is bound to, are presumed. I am quoting, not interpreting. Whether that presumption gets you anywhere against a particular opponent is a question for someone with a bar admission, and I do not have one.

The third is an ordinary public trail, so the value demonstrably existed outside my machines that day, readable without special tooling.

The verification material is published, on purpose. Until last August I had anchors nobody could check with standard tools, which made the whole claim hollow. Now a third party can take the sealed root, take the timestamp token, and verify it with openssl ts -verify against the published chain. One detail worth knowing in advance: the signing certificate of the timestamping authority expires in December 2027, while the evidence keeps running. Verification after that date needs the -attime flag. That is a verification detail, not an expiry of anything.

What gets recorded, and the fight it belongs to

The archive records five classes of thing. Each one exists because a specific argument keeps happening. This is the part where it stops being abstract.

The machine readable signals. robots.txt, ai.txt, the TDM reservation, llms.txt, fetched daily. The dispute: an AI vendor says the reservation was not there when they crawled. You say it was. Recorded on both sides of the date in question, by two machines that do not know each other, and sealed the same night.

One page, today, before it changes. A lawyer sees an infringing claim, a missing mandatory disclosure, a price that violates a rule. The single most predictable thing in that situation is that the page will be different by the time the other side answers. The usual tool is a screenshot, which is a party’s own record of a party’s own screen. Here the URL goes into that night’s run on both machines and the extract for that day comes back with the bytes, the response headers, the TLS chain and the UTC time of the fetch. Not pixels. Bytes. If what is disputed is how something looked, this is the wrong tool and I say so on the page.

Named pages, daily, for as long as it runs. Terms, a price list, a disclosure, a licence page. The dispute is usually about the version in force on a given day, and the honest answer requires a series, not a snapshot.

Domain, DNS and registry. The RDAP response from the registry, the answers from the authoritative nameservers, and the homepage. This is domain dispute territory: who held it on day Y, where did it point, what was on it. Bad faith under the UDRP is shown through the history, and today a complainant assembles that history themselves from whatever they can find. A brand owner can also have lookalike domains recorded before anything happens, which is the only way the history exists at the moment it turns out to be needed. The same material answers a different question for a cyber insurer, who wants to know what the mail and TLS configuration looked like the day before the incident rather than after the cleanup.

Agent endpoints. A public MCP endpoint, asked every day what tools it declares, plus the agent card next to it. Two opposite buyers, which is how I know the fight is real. The operator who has to show that his endpoint did not declare the tool an agent supposedly used. And the company whose agents call somebody else’s endpoint, worried about the quiet redefinition of a tool after approval. Asking an endpoint what it can do means sending a POST, not fetching a file, which is a different kind of crawl and needed its own build.

Platform terms. Under the P2B Regulation a platform has to give business users advance notice before changing its terms. Proving that a clause appeared without that notice requires a daily outside record of a page nobody thought to keep.

Two of these are running today. Some of the others only get built when somebody actually orders them, and I would rather say that here than let a price list imply a warehouse. The recording is the part that cannot be produced retroactively, which is why it runs first and everything else follows it.

Being recorded and asking for the record are two different things

This is the distinction people get wrong most often, so here it is on its own.

Being recorded means a domain is fetched every night, and what it said goes into that night’s seal. It is a standing act. It produces nothing you can read. It creates no document, sends no report, and on any normal day there is nothing to look at. Its entire purpose is that the material exists later, whether or not anyone ever asks.

Asking for the record is the opposite motion. Something has gone wrong, a date is in dispute, and someone needs a document: the stored bytes for a given domain and a given period, with the hashes, the Merkle path, the daily roots, the anchors and the instructions to verify all of it without me. That is an extract, it is produced per case, and it costs separately every time, including the second time about the same matter years later.

Two consequences that follow from the split, and both surprise people:

• You can order an extract about a domain you do not control. Roughly 128,000 EU domains are already in the broad ring, recorded nightly, because reach put them there and nobody asked me. If the domain you are arguing with is one of them, the material exists and predates the argument. Nobody had to buy anything for that to be true.
• Having your own domain recorded does not get you free extracts. That sounds ungenerous until you turn it around: the price cannot depend on who is asking, or the other side would be paying more for the same document than my customer does. At that moment the archive is no longer a witness, it is a party. The separation is not a billing trick, it is the neutrality condition doing its job.

The lies of omission a good archive has to avoid

Here is where an archive can cheat without ever writing a false sentence.

Not every domain is recorded the same way. Around 128,000 EU domains sit in a broad ring: their signal files are fetched daily, but the homepage only weekly, and capped in size. A much smaller core ring gets everything, daily, complete. That is the honest shape of a system paid for out of one person’s pocket. The broad ring is cheap precisely because it does less.

Which sets a trap. Picture an extract, ordered for a real dispute, that hands over a page captured six days before the date that matters, or a file cut off at a size limit, and mentions neither. It would look complete. It would read as authoritative. It would be the worst thing this archive could do, worse than an outright gap, because a gap you can work around and a silent gap you cannot.

So as of this week every extract states, on its first page, before any verification step: which ring the domain was in, what the cadence was, whether a cap applied, how many bytes were stored against how many the server sent. Plus two rules that cost me a rewrite:

• If an observation does not carry the truncation flag in its sealed provenance, the extract calls it unknown. Never “complete”. Unknown is an honest word. Complete is a claim.
• If the live database no longer reaches across the whole window, the extract says outright that a missing day is not evidence of non-observation. It only means this document cannot speak about that day.

The same discipline applies to the system’s own history. The second witness has only been running since 4 August 2026. Everything from 22 July to 3 August is singly witnessed, permanently, and the public log marks it day by day. Double witnessing cannot be retrofitted, because it is a property of the past and not of the software. One day, 3 August, was killed mid seal by the kernel’s out of memory killer and was sealed two days late. That day carries a permanent marker saying so and always will. I could have quietly resealed it. That is exactly why I did not.

What it deliberately does not do

It does not notify anyone when a page changes. Not even when customers ask, and they do. The moment I tell one side that something moved, I have picked a side.

It does not score, rank or compare anybody. There is no public full text search over the collected material, which is not a product decision but a consequence of my own data protection assessment: recording in the public interest is one thing, republishing the raw material is another.

And it does not tell anyone what any of this means for them. I write software that records what a machine saw at a point in time and lets anyone recompute it. Whether that carries the day in front of a court, an authority or an opponent is a question for a lawyer, and I am glad it is, because that is a different profession with a different exam.

Premium and claim

The commercial logic took me longer to accept than it should have. Having a domain recorded is the premium. Ordering an extract years later, when somebody is claiming your site said something it did not, is the claim.

Most people who pay will never order anything. That is not a flaw in the model, that is the model.

Which lands back at the first paragraph. I built two machines in two countries so that my own honesty would stop being load bearing. The right amount of trust to place in me here is none, and that is the feature.

If you want to know whether a domain is already being recorded, the coverage check on machinewitness.eu answers that for free, and the daily root log is public. If you are the sort of person who reads the verification instructions before believing anything: good. That is who this is built for.

Topics: evidencerobots-txttdm-opt-outeidas-timestampdomain-disputesai-crawlers