When your cloud provider deletes your server: a cancel date nobody remembered

What happened

The server was a Contabo VPS we had created on 30 August as a test server. Its last heartbeat reached us at 01:25:50 UTC on 30 September, and an alert that every VM had stopped reporting fired at 03:40 CEST. Our platform had taken no action on the server and had recorded nothing about why it went quiet.

The answer was in the provider’s own records all along.

What the provider’s audit log showed

Contabo keeps a per-instance audit trail you can read through the same API credentials as everything else. Each entry has a timestamp, an action (created, updated, deleted), who did it, and a before-and-after diff of what changed.

So the story was a deliberate end-of-first-month cancellation, set on day one by one of our own account users and then forgotten, honoured by the provider the day after the date. The cancellation date appears in the audit entry’s diff, which means it was visible for the whole month. A date that nobody on our side was tracking.

What this means if you rent servers

• A cancellation date is honoured. Deletion followed the date by a day, which is expected behaviour; nothing on the machine itself announces it, so track the date yourself.
• Anything only on that server went with it. Keep backups somewhere that is not the same provider account. See secure your VPS for the basics.
• When a server goes silent, read the audit log first, before debugging your own software. Check what your provider records; this is a single incident on a single provider and we cannot say how others behave.
• If you manage servers for other people, a provider-side date or change is yours to track, because the machine will not tell you about it.

What we are building: a reconciliation job

Status: rolling out. The job is built and is going through supervised runs; we are not describing it as live.

Every 15 minutes it asks each provider about every server we manage there, compares the answers with our own records, and files one of four findings:

How it behaves, in general terms:

• It never writes to the provider. It can only read: its provider clients are limited to look-up calls, and anything that is not a read is refused before it leaves the machine. It cannot cancel, revoke or recreate anything. The one exception is signing in.
• Two sightings, at least ten minutes apart, before it marks a server as gone. One “missing” answer is recorded but changes nothing, because a provider can return a wrong answer once.
• Unknown is not deleted. A timeout, a failed login or an unreadable list is counted and shown, never turned into a verdict.
• A guard against mass false alarms. If a whole batch of servers at one provider suddenly looks missing, it believes none of them, sends one page, and waits for a human, since that pattern usually means a credentials or outage problem.
• A scheduled cancellation is caught before the date. It pages the operator the first time it sees one, and emails the customer once.
• A confirmed removal pages the operator urgently and emails the customer once, with the date, what the provider’s audit trail says and a plain statement that the data on it is gone. The email carries no internal identifiers.
• Billing and DNS stay with a person. A removed server may still have a paid subscription; whether to cancel, refund or rebuild is a human decision, so the job never touches billing, and its page says so.
• It writes down what it found before it tells anyone, so a crash means a missed message, never a duplicate email.

Had it existed, it would have flagged the test server’s cancellation date at its first run after it was set on 30 August, a month before the deletion.

Limits, said plainly

• This is one incident on one provider. Some providers have no scheduled cancellation, so for them the job can only detect a server that has vanished.
• Part of how the job reads the provider’s answers is taken from the provider’s documentation and this one incident; the supervised runs are what confirm it against the real API.
• It tells us and the customer. It cannot bring a deleted server back, which is why backups outside the provider matter.

Related

• Managed vs self-hosted AI agents — who watches what.
• Secure your VPS — including backups.
• Pin your agent versions — the other kind of silent change.