Telemetry
Papyr reports traces through an ActivitySource and metrics through a Meter, both named Papyr, and log events
through ILogger. It does not depend on the OpenTelemetry SDK: an exporter, dotnet-counters or dotnet-trace can
listen. Without a listener a span or a measurement costs a null check. With the admin API on, GET /api/metrics
answers the metrics without a collector (“Reading the metrics without a collector”). The client library
(Papyr.Client, used by the command line) has its own ActivitySource and Meter, named Papyr.Client; its spans
and metric are the rows of the tables below marked as the client’s.
What is never in it. No user content, room id, user name, colour, byte of a message, token or client address.
Events name a room by its tag, the first 8 characters of its id: enough to find its files
({data directory}/rooms/{tag}*) and to tell its lines apart. Any room id in an exception’s text is cut to its tag
the same way. Still, a guessed name of an open room can be tested against the tags: keep the log as private as the
data directory.
ASP.NET Core’s own telemetry is not Papyr’s. A relay request is GET /relay, with no room in its path or query.
ASP.NET Core’s request span (Microsoft.AspNetCore.Hosting.HttpRequestIn) carries url.path, client.address and
network.peer.address, and every log line written during a request (Papyr’s events 19 and 33, and 11 to 15 when a
join or a write meets them) carries its scope with RequestPath. The admin API’s GET /api/rooms?after={id} and
DELETE /api/rooms/{id} do carry a room id, which ASP.NET Core’s span and request logs may record (url.query,
url.path, QueryString, RequestPath): treat such traces and logs as private as the data directory, or drop those
fields in the exporter. The blob routes’ /blob/{id} carries a blob’s id (the hash of its stored bytes, which opens
nothing and names no room), which the same fields may record; the room’s access key is in a request header, which
ASP.NET Core’s span does not record, but a proxy or a host that logs request headers would hold it as a frame log
would: do not log Papyr-Room. Papyr’s own span holds the route template only.
Spans
| Name | Kind | When | Attributes |
|---|---|---|---|
papyr.room.load |
Internal | A room is loaded from the store. A store that fails sets the status Error; a damaged room is an answer, not an error. |
papyr.room.load.outcome (loaded, damaged, failed), papyr.room.sequence (the last sequence) and papyr.room.update.count (stored updates after the snapshot) when loaded, error.type on failure |
papyr.room.snapshot |
Internal | A client’s snapshot is stored and the updates it covers are dropped. A failed save has the status Error. |
papyr.room.sequence (the snapshot’s upTo), papyr.room.update.count (the updates kept), error.type on failure |
GET /api/status, GET /api/rooms, DELETE /api/rooms/{id}, GET /api/backup, GET /api/metrics, GET /api/openapi.json |
Internal | A request to a route of the admin API, the token check included, named as the OpenTelemetry conventions name an HTTP server span; a child of ASP.NET Core’s request span. Error only for a 500 (event 28, or a route that threw, which ASP.NET Core logs with the exception: a storage failure with every room id in it cut to its tag) or a backup cut off by unreadable files (event 32); a 401, 400, 404, 409 or 503 is an answer, and so is a backup the client went away from. A request under /api that no route takes has no span; its 401 is counted all the same. |
http.request.method, http.route, http.response.status_code, error.type on failure; for GET /api/backup also papyr.backup.outcome (completed, failed or cancelled), papyr.backup.room.count and papyr.backup.bytes, what the room files it read held |
HEAD /blob, PUT /blob/{id}, /blob/{**path} |
Internal | A request to a blob route (Papyr:Blobs:Enabled; relay-protocol.md, “Blobs”), the admission included (the Origin, then the Papyr-Room header), named as the OpenTelemetry conventions name an HTTP server span; a child of ASP.NET Core’s request span. /blob/{**path} is every other path under /blob, whatever its method: its span has no method attribute, since the client chooses the method. Error only when the route failed (a 500, or the 507 of a full disk: a storage failure, whose exception type is set and whose message is not; or a bug, which ASP.NET Core logs with the exception); a 403, 401, 400, 410 or 404 is an answer. Never the blob’s id, the header or the room. |
http.request.method (not on /blob/{**path}), http.route, http.response.status_code, error.type on failure |
GET /blob/{id}, HEAD /blob/{id} |
Internal | A download (relay-protocol.md, “Blobs”): the admission as HEAD /blob, then the lookup, the slot and the send; the span ends with the last byte handed to the connection. Error only when the route threw (a 500: a blob file whose length is not its name’s, a failed read, or a response cut off because the file ended early); a 400, 404, 304, 503 or an aborted download is an answer. Never the blob’s id, the header or the room. |
http.request.method, http.route, http.response.status_code, error.type on failure |
papyr.backup.scheduled |
Internal | A turn of the scheduled backup (operations.md, “Scheduled backups”), a root span. Error when it failed (event 36); skipped, cancelled, or whole with a failed retention (event 37) is an answer. |
papyr.backup.outcome (completed, skipped, failed, cancelled), papyr.backup.room.count and papyr.backup.bytes when completed, error.type on failure |
papyr.client.connection (source Papyr.Client) |
Client | A connection of a client session to a relay, from the attempt to its end; one per attempt, a lost or refused one included. The status is not set: how it ended is an answer (websocket.close.code), not an error. Nothing of the link, the keys, the room or its text. |
websocket.close.code (the server’s close status; 1006 for a connection lost, dropped or never opened), absent when the client ended it |
papyr.client.blob.upload (source Papyr.Client) |
Client | A PUT /blob/{id} of a client, from the request to the end of the answer (relay-protocol.md, “Blobs”); no retry is part of it, a retry is another span. A refusal (409, 413, 429, 507, 503 and the rest) is an answer: the status is not set. Error, without a message, when the request threw, timed out or was cancelled. Never the key, the header, the id or the size. |
http.response.status_code, absent when no answer came |
papyr.client.blob.download (source Papyr.Client) |
Client | A GET /blob/{id} of a client, from the request to the end of the body it read (at most the declared length). A 404, a 503 or a body that is not the declared length is an answer: the status is not set. Error, without a message, when the request threw, timed out or was cancelled. Never the key, the header, the id or the size. |
http.response.status_code, absent when no answer came |
papyr.room.load and papyr.room.snapshot are children of the request span whose connection found the room idle
(usually the joining or sending connection’s), or roots when there is none.
Metrics
| Name | Instrument | Unit | Attributes | Meaning |
|---|---|---|---|---|
papyr.relay.updates |
Counter | {update} |
Updates a room numbered and stored. | |
papyr.relay.ephemeral |
Counter | {message} |
Ephemeral messages (presence, carets) a room passed on. | |
papyr.relay.ephemeral.dropped |
Counter | {message} |
Ephemeral messages dropped because their connection sent more than 16 in a second. A room page sends at most 8 a second, a race page 11; a steady rise is a client that is not a page of this server, or misbehaves. | |
papyr.relay.snapshots |
Counter | {snapshot} |
Snapshots a room stored. | |
papyr.relay.bytes.sent |
Counter | By |
Bytes of messages sent to relay connections, each record’s 4-byte length included; each copy counts. | |
papyr.relay.bytes.stored |
Counter | By |
Body bytes of updates and snapshots written to the room files. | |
papyr.relay.connection.closed |
Counter | {connection} |
papyr.relay.close.reason |
Relay connections closed, by reason (below). |
papyr.room.store.failures |
Counter | {failure} |
papyr.room.store.operation |
Failed operations on a room’s files: append, flush, snapshot, rewrite, close, read (the stored snapshot for a joining connection), blobs (removing a removed room’s blobs, or the expiry’s sweep of the blobs of rooms nobody stores; event 40 says more). Event 14 says more. |
papyr.room.count |
UpDownCounter | {room} |
Rooms in memory. | |
papyr.room.expired |
Counter | {room} |
papyr.room.expiry.reason |
Rooms the expiry’s pass removed: idle, stored rooms nobody wrote to for Papyr:RoomExpiryDays (event 17); expired, rooms past the expiry they were created with, loaded or stored (event 39). |
papyr.room.removed |
Counter | {room} |
Rooms the operator removed (DELETE /api/rooms/{id}, answered 204). |
|
papyr.room.stored.count |
ObservableUpDownCounter | {room} |
Rooms with a file: counted at the start and at each pass of the expiry, kept in between. New rooms are refused (4007) at Papyr:Relay:StoredRooms. |
|
papyr.static.decompressed |
Counter | {response} |
Files of the web root sent decompressed to a request that does not accept Brotli (HEAD, requests accepting br and 304 answers are not counted). |
|
papyr.static.decompression.count |
UpDownCounter | {response} |
Decompressions running now, at most 4. At 4 for long, plain-HTTP clients are waiting or refused. | |
papyr.static.decompression.waiting |
UpDownCounter | {request} |
Requests waiting for a decompression slot, each at most 5 seconds. | |
papyr.static.decompression.refused |
Counter | {request} |
Requests answered 503 (Retry-After: 5) because no slot came free in 5 seconds. Rising: clients hold the slots (see .aborted), or more plain-HTTP clients than 4 slots serve; put the server behind HTTPS, where browsers accept Brotli. |
|
papyr.static.decompression.aborted |
Counter | {response} |
Decompressed answers cut off because the client did not keep pace (no write completed within 30 seconds, or slower than 64 KiB/s after the first 30). The slot is free again. | |
papyr.blob.downloads |
Counter | {download} |
papyr.blob.download.outcome |
GET and HEAD /blob/{id} that passed the admission and named a valid id, by outcome: sent (200; for a GET the whole file was handed to the connection), not_modified (304), not_found (404: no such blob in the room, or its file went before it was opened), busy (503: no download slot within 5 seconds), aborted (the client left, or did not take the file within 10 seconds plus one per 256 KiB; the connection is aborted and the slot is free again) or failed (a 500, or a response cut off, because the file is not as long as its name says; the exception is logged by ASP.NET Core with its room ids cut to tags). |
papyr.blob.bytes.sent |
Counter | By |
Bytes of blobs handed to the connection by downloads that were sent whole (an aborted download counts none). | |
papyr.blob.download.count |
UpDownCounter | {download} |
Blob downloads holding a download slot now, at most 4 (uploads have slots of their own). At 4 for long, slow readers hold the slots. | |
papyr.blob.download.waiting |
UpDownCounter | {download} |
Blob downloads waiting for a download slot, each at most 5 seconds. | |
papyr.blob.uploads |
Counter | {upload} |
papyr.blob.upload.outcome |
Blob uploads (PUT /blob/{id} past the Papyr-Room admission), by outcome: stored (201), exists (204: the room held the blob), refused (any other answer that is a refusal: a status 4xx, the quota’s 507, no slot in 5 seconds), failed (a 500, or the 507 total-bytes of a full disk: events 40) or aborted (the body was not whole within the transfer time, or the client left). |
papyr.blob.bytes.stored |
Counter | By |
Bytes of blobs stored by uploads (their stored length, the 29 bytes of overhead included). | |
papyr.blob.upload.count |
UpDownCounter | {upload} |
Uploads holding one of the 4 upload slots now: copying their body. | |
papyr.blob.upload.waiting |
UpDownCounter | {upload} |
Uploads waiting for an upload slot, each at most 5 seconds. Often above 0: slow clients hold the slots (the 503 is not counted apart: it is refused). |
|
papyr.blob.removed |
Counter | {blob} |
papyr.blob.removal.reason |
Blobs deleted, by reason: room_removed (their room was removed, by the operator or by the expiry) or orphaned (the expiry’s sweep of the blobs of rooms nobody stores, the newest file older than an hour). A removal that fails halfway counts what it deleted. Temporary files of uploads are not counted. |
papyr.blob.bytes.removed |
Counter | By |
papyr.blob.removal.reason |
Bytes of the blobs papyr.blob.removed counts (their stored length, as papyr.blob.bytes.stored), by the same reason. |
papyr.blob.stored.bytes |
ObservableGauge | By |
Bytes of blobs the server holds, from the lengths of the files (Papyr:Blobs:TotalBytes limits it). |
|
papyr.relay.connection.count |
UpDownCounter | {connection} |
Open relay connections, those waiting for their hello included: at most 10 seconds, and up to 10 more for one closed before its hello (4003, 4005, 4009) whose client does not answer the close. | |
papyr.relay.connection.refused |
Counter | {connection} |
papyr.relay.refusal.reason |
Relay requests answered before the upgrade: origin (403: the Origin named neither the request’s own host nor an origin of Papyr:Relay:AllowedOrigins; event 33 at the first), server (503: Papyr:Relay:ServerConnections reached, or Kestrel’s limit of upgraded connections, which gives a connection back a moment after the relay) or source (429: the source held one more than Papyr:Relay:ConnectionsPerSource, the one told with 4005). A blob route refused for its Origin counts as origin here too (same rule, same 403; the relay’s event 33 is written for the relay only). |
papyr.backup.scheduled.runs |
Counter | {run} |
papyr.backup.outcome |
Turns of the scheduled backup: completed (event 34, or 37 when removing an older one failed), skipped (a backup through the admin API was running, event 35), failed (event 36) or cancelled (the server stopped). |
papyr.backup.scheduled.last_success |
ObservableGauge | s |
When the newest whole scheduled backup started, in seconds since 1970-01-01 UTC; at the start, the newest in the backup directory not dated after now. Nothing before there is one. Alert when older than twice Papyr:Backup:IntervalHours. |
|
papyr.admin.unauthorized |
Counter | {request} |
papyr.admin.unauthorized.reason |
Requests under /api answered 401: missing (no bearer token) or wrong. |
papyr.client.connection.closed (meter Papyr.Client) |
Counter | {connection} |
websocket.close.code |
Connections of a client session that ended, by the server’s close status (1006: lost, dropped or never opened). |
papyr.relay.close.reason is the close the connection got:
| Reason | Status | When |
|---|---|---|
client |
1000 | The client closed, or the connection was lost. |
stopping |
1001 | The server stops. |
unavailable |
1011 | The room’s files cannot be read or written, the room is damaged or failed, or no sequence was left (2^53 - 1). A joining client whose snapshot cannot be read within 10 seconds gets 1011 before the first piece; after it, it is aborted without a status. |
slow |
none (1013 in the code) | Its unsent messages passed Papyr:Relay:QueuedBytes while a send to it was pending: aborted. |
busy |
1013 | Papyr:Relay:ReceiveBytes had no room for a large message, or the source held its share of it (configuration.md, “The relay’s limits”). |
timeout |
none | Aborted: the hello was not whole within 10 seconds, a message did not start within 10 seconds or did not arrive whole, or the client did not take one within 10 seconds plus one per 256 KiB. |
room_full, too_large, stored_limit, malformed, too_many_rooms |
4000, 4001, 4002, 4003, 4004 | As spec/relay-protocol.md, “Close statuses”. |
source_limit |
4005 | The source held Papyr:Relay:ConnectionsPerSource connections already, or was in Papyr:Relay:RoomsPerSource other rooms. |
new_room_rate, store_full |
4006, 4007 | The first write of a new room was dropped: the source created Papyr:Relay:NewRoomsPerSourcePerHour rooms within the hour, or the store holds Papyr:Relay:StoredRooms. |
removed |
4008 | The operator removed the room (DELETE /api/rooms/{id}). |
version |
4009 | The hello named a version of the contract the server does not speak (a page of another version). |
version1 |
4009 | A page of version 1 connected to its old address, /relay/{room id}: it is told to reload and closed. |
rejoin |
4010 | Another connection of the room stored a snapshot with content; this one joins again. |
expired |
4011 | The room expired: the hello’s expiry had passed (the room is not loaded, created or counted), or the expiry’s pass removed the loaded room (event 39). |
Reading the metrics without a collector
With the admin API on, GET /api/metrics answers the current values of the Papyr meter and the runtime’s
System.Runtime meter (dotnet.process.cpu.time, dotnet.process.memory.working_set, dotnet.gc.*,
dotnet.thread_pool.*, …) as JSON, or as the Prometheus text format to a scrape (admin-api.md, “Metrics”). A
Grafana dashboard and Prometheus alert rules are in deploy/grafana/. The server listens to its own meter only while
the API is on. Counters are summed since the start and begin again at 0 with the server; there is no history (for a
rate, read twice). Each instrument has at most 1000 series; further sets of attributes go to one series with
otel.metric.overflow = true, as the OpenTelemetry SDKs do (Papyr’s attributes are few and fixed, so this would be
a bug). The meters of ASP.NET Core (Microsoft.AspNetCore.Hosting, Microsoft.AspNetCore.Server.Kestrel, …) and of
System.Net.* are not answered: read them with an OpenTelemetry collector or dotnet-counters.
Log events
No event names a client’s address, quotes a request header or carries a token.
Rooms and the relay: category Papyr.AspNetCore.RoomEvents; Error except 13, 19 and 33 (Warning), 17, 20
and 39 (Information).
| Id | Name | When | What to do |
|---|---|---|---|
| 11 | RoomDamaged |
A room’s files cannot be read as a room (an unknown header, missing updates, a snapshot that is not whole, an upTo or an update numbered above 2^53 - 1, a snapshot body larger than a join can carry). Names the tag and what is wrong. The room is not opened (its connections get 1011); its files are left as they are. |
Restore the files from a backup, or remove them (the room starts empty; a client sends its copy again). |
| 12 | RoomLoadFailed |
Reading a room’s files failed (permissions, I/O error, a snapshot that changed while read). Carries the tag and the exception. Its connections get 1011; the next connection loads it again. | Fix the permissions or the disk. Repeated with every reconnect: the store keeps failing. |
| 13 | RoomTornEndDropped |
The end of a room’s update log was cut off by a crash and was dropped. Carries the bytes and the tag. | Nothing: the clients send those updates again. Repeatedly without a crash, check the disk. |
| 14 | RoomStoreFailed |
Writing or reading a room’s files failed: append (the update was not stored; its sender gets 1011), flush (not retried, since a failed fsync drops what it could not write: the room ends, its connections get 1011, the next connection loads what the files hold), snapshot (not taken), rewrite (the snapshot was taken; the old log stays), close, read (the snapshot for a joining connection: it gets 1011, or is aborted when part was sent). Carries the operation, the tag and the exception, whose message says what is still stored. |
Free disk space or fix the permissions of {data directory}/rooms. |
| 15 | RoomOperationFailed |
An operation of a room threw unexpectedly: a bug. Carries the kind, the tag and the exception. A queued operation (Join, Leave, Update, Ephemeral, Snapshot, Flush, Stop, Remove, Expire): the room goes on, its connection gets 1011. Drain: the room has ended, its connections get 1011, the next connection loads it again. End: closing the room’s files threw; the room has ended anyway. |
Report it with the exception. |
| 16 | RoomFlushRoundFailed |
A round of flushes threw unexpectedly. The room it was at is flushed in the next round (Papyr:FlushIntervalSeconds), the round’s other rooms in this one. Once per round. |
Report it with the exception; repeated every round, check the process’s memory. |
| 17 | RoomExpired |
The expiry removed a stored room nobody wrote to for Papyr:RoomExpiryDays. Carries the tag and the day (UTC) of its last write. |
Nothing. |
| 18 | RoomExpiryFailed |
A pass of the expiry failed, at most once per pass. Carries how many expired rooms it could not remove (each stays whole, or its snapshot alone; either loads), whether the pass went on or ended early, and the first exception. The next pass, an hour later, tries again. | Fix the permissions of {data directory}/rooms or check the disk; when the pass ended unexpectedly, report it. |
| 19 | ForwardedHeaderIgnored |
A relay request carried X-Forwarded-For from an address not in Papyr:TrustedProxies. Every client through that address counts as one source. Once per process. |
Behind a proxy: set Papyr:TrustedProxies (operations.md, “Behind a proxy”). Without one: a client sent the header itself; nothing to do. |
| 20 | StoredRoomsCounted |
At the start: how many rooms the store holds, of Papyr:Relay:StoredRooms. |
Nothing. At or above the limit no new room is stored (4007) until rooms expire or files are removed. |
| 33 | RelayOriginRefused |
A relay request was answered 403 for its Origin: it named neither the request’s own host nor an origin of Papyr:Relay:AllowedOrigins, or was null, empty, doubled or malformed. Once per start; papyr.relay.connection.refused with origin counts every one. The page sees a socket that closed (1006) and retries. |
Every page refused: the proxy rewrites Host (nginx: pass $http_host), or a page of another origin connects (app://...): make the proxy pass the browser’s Host, or list the origin (configuration.md). Otherwise a page of another site tried; nothing to do. |
| 39 | RoomExpiryReached |
The expiry’s pass removed a room past the expiry it was created with (part of its link and its id): a loaded one’s connections got 4011, then its files were removed. Carries the tag. Within the hour after the expiry; no join is taken from that moment (4011). | Nothing. |
Blobs: category Papyr.AspNetCore.BlobEvents (Papyr:Blobs:*, configuration.md); 40 Error, 41 Information. They
name a room by its tag, never a blob’s id, a header or a path.
| Id | Name | When | What to do |
|---|---|---|---|
| 40 | BlobStoreFailed |
Storing, reading or removing a room’s blobs failed (permissions, I/O error, a full disk). Carries the operation: remove (a removed room’s blobs) or sweep (the expiry’s sweep, whose tag is -: the exception names the room), or for an upload find, reserve, open, write, complete, discard (the last is a temporary file that could not be deleted, which keeps its id from being uploaded until the next start; the others end an upload); the tag and the exception, whose room ids are cut to tags. The request got 507 total-bytes for a full disk, else 500; a blob that was not stored is not stored, and the client uploads it again. A removal that failed is written at each pass of the expiry until it works. |
Free disk space or fix the permissions of {data directory}/blobs. A foreign subdirectory in a room’s blob directory fails its removal at each pass: remove it. |
| 41 | BlobsCounted |
At the start, when the blob directory holds blobs, on or off: how many and their bytes, of Papyr:Blobs:TotalBytes. |
Nothing. With the blobs off, the files are kept (and backed up) and served by no route. |
TLS: category Papyr.Server.TlsCertificate, only with Papyr:Tls:CertificatePath and KeyPath set
(configuration.md, “TLS”). They carry the configured paths, never key material.
| Id | Name | When | What to do |
|---|---|---|---|
| 21 | TlsCertificateLoaded |
Information. At the start, and when a changed pair of files was loaded. Carries the path, the expiry (UTC) and the last 8 hex digits of the SHA-256 fingerprint (openssl x509 -noout -fingerprint -sha256 -in <file>). |
Nothing. |
| 22 | TlsCertificateLoadFailed |
Error. A changed pair did not load (a key not the certificate’s, unreadable, not PEM, missing), or is expired or not yet valid while the one in use is valid. Carries both paths, the expiry in use and the exception, which says which. The certificate in use stays; the files are tried again every minute, the event written once per state. |
Usually a renewal half done: the next poll loads them (event 21). Otherwise fix the files or their permissions. |
| 29 | TlsCertificateValidityWarning |
Warning. The certificate in use expires soon (the last tenth of its validity: 9 days of 90), has expired, or is not valid yet. At the start, when it enters such a state, then once a day. Carries the path and both ends of the validity. |
Renew the files or find why the renewal failed; the server loads them within a minute (event 21). |
The admin API: category Papyr.AspNetCore.AdminEvents; 23 and 25 Information, 24, 26 and 27 Warning,
28 Error.
| Id | Name | When | What to do |
|---|---|---|---|
| 23 | AdminApiEnabled |
At the start: the API is on, with the token of the setting named (Papyr:Admin:Token, or Papyr:Admin:TokenFile for a token file, the default one included). |
Keep /api inside at the proxy (admin-api.md). Not expected: remove the setting or the file. |
| 24 | AdminTokenWrong |
A request carried a wrong token and got 401. Names the route, or to no route under /api. At most once a minute per source, for at most 1024 sources a minute; papyr.admin.unauthorized counts every one. |
Once, after a change of the token: a client with the old one. Often, or from outside: someone guesses; check that the proxy keeps /api inside, and change the token if it may have leaked. |
| 25 | RoomRemoved |
The operator removed a room (DELETE /api/rooms/{id}, 204): its connections were closed with 4008 and its files removed. Carries the tag. |
Nothing. |
| 26 | AdminTokenFileMissing |
At the start: Papyr:Admin:TokenFile names a file that does not exist; the API is off. Carries the path. |
Run Papyr.Server token (or put the file there) and restart, or remove the setting. |
| 27 | AdminTokenFileIgnored |
At the start: Papyr:Admin:Token is set and a token file is at the default path; the setting’s token is taken. Carries the path. |
Remove the file or the setting. |
| 28 | RoomRemovalFailed |
A removal could not remove the room’s files whole and got 500: the room is unloaded (its connections got 4008), and what is left loads. Carries the tag and the exception. | Fix the cause (permissions, the disk), then remove the room again. |
Backups: category Papyr.AspNetCore.BackupEvents. GET /api/backup writes 30 and 31 (Information) and 32
(Error); the scheduled backup (operations.md, “Scheduled backups”) 34 (Information), 35, 37 and 38 (Warning)
and 36 (Error). They name the backup’s file, never a room; exceptions have room ids replaced by tags.
| Id | Name | When | What to do |
|---|---|---|---|
| 30 | BackupStarted |
A backup began streaming. | Nothing. |
| 31 | BackupEnded |
A backup ended: completed (the whole tar went out to a connection still open; only the tar’s end, which the restore checks, proves the client got it all), cancelled (the client went away, took nothing for a minute, or the server stops: what it got is cut off) or failed (event 32). Carries the outcome, the rooms and the bytes sent. |
cancelled or failed: back up again. |
| 32 | BackupFailed |
A room’s files could not be read during a backup. The download is aborted without the tar’s end, so curl -f fails and the restore refuses it. Carries the exception. |
Fix the files or the disk, then back up again. |
| 34 | ScheduledBackupWritten |
A scheduled backup was written whole. Carries the file’s name, the rooms and bytes in it, and how many older backups were removed. | Nothing. |
| 35 | ScheduledBackupSkipped |
The turn came while a backup through the admin API was running; nothing was written. The next interval tries again. | Nothing, unless it repeats: a client asks for a backup at the same times. |
| 36 | ScheduledBackupFailed |
A room’s files could not be read, or the backup directory could not be written (disk full, permissions). No backup was written; the unfinished file is removed and every whole backup stays. Carries the exception. The next interval tries again. | Fix what the exception names. |
| 37 | ScheduledBackupRetentionFailed |
The backup was written whole, but removing older backups beyond Papyr:Backup:Keep failed. Carries the file’s name, its rooms and bytes, how many were removed, and the exception. The turn counts as completed; the older backups stay until a later turn. |
Fix the directory’s permissions or the disk; watch its free space. |
| 38 | ScheduledBackupsDatedAhead |
Backups in the directory are dated after this backup’s start: written while the clock was ahead. Carries how many and the start (UTC). The retention removes them first; until then the newest name is not the newest backup. At each backup while such files exist. | Restore by the files’ write times (ls -lt) until they are gone; keep the clock right (NTP). |
ASP.NET Core’s categories (Microsoft.AspNetCore.*) are lowered to Warning by default, so a request writes no
line; Logging:LogLevel:Microsoft.AspNetCore.Hosting=Information brings them back.
Microsoft.AspNetCore.Server.Kestrel.Connections is lowered to Error: Kestrel writes a Warning (24,
ConnectionRejected) for every connection it closes at Papyr:HttpConnections, and a flood would write a line each.
Kestrel’s meter Microsoft.AspNetCore.Server.Kestrel counts them (kestrel.rejected_connections);
kestrel.active_connections and kestrel.upgraded_connections show how full the listeners are (configuration.md,
“Connections”).