Papyr Write together in Markdown. The server can't read it. Try it GitHub

Operations

Running the server: from the source or the container, behind a proxy, the data directory, expiry, backup and restore. Installing and exposing it: self-hosting.md.

From the source

git clone https://github.com/msallin/papyr.git && cd Papyr
dotnet run --project src/Papyr.Server           # http://localhost:5000

A Native AOT binary: dotnet publish src/Papyr.Server -c Release -r linux-x64 -o out (linux-arm64 on an arm64 machine; to cross-compile from x64 Ubuntu, install binutils-aarch64-linux-gnu gcc-aarch64-linux-gnu libc6-dev-arm64-cross and add -p:ObjCopyName=aarch64-linux-gnu-objcopy, as CI does). The publish minifies the web client’s scripts and style sheets with Node.js (version 24, as CI) and esbuild: once, npm ci in build/minify. Without them the publish fails and says so; -p:MinifyWebRoot=false publishes the scripts as they are (larger, the same behaviour). dotnet run needs neither, and the image build brings its own Node.js.

The container

Building and running the image, Compose, a reverse proxy, TLS and what to set on a public instance: self-hosting.md. The commands below say papyr for the image, built from the Dockerfile or ghcr.io/msallin/papyr:<version>.

The image is one Native AOT binary on a base without a shell. It runs as uid 1654, listens on 8080 (8443 for TLS when configured) and keeps the rooms in /data, the only directory it writes besides /tmp. It sets no limit, no expiry and no trusted proxy: the defaults of appsettings.json apply (configuration.md). Every setting is an environment variable with __ for each :: -e Papyr__RoomExpiryDays=30 -e Papyr__Relay__StoredRooms=100. GET /health answers ok.

/data is a VOLUME owned by the app’s user, so a new named volume is writable. A bind mount keeps the host directory’s owner: make it the app’s, or the server cannot create rooms/ and stops.

mkdir -p /srv/papyr && sudo chown 1654 /srv/papyr
docker run -d -p 8080:8080 -v /srv/papyr:/data papyr

Mount a disk-backed directory or volume at /data, never a tmpfs (configuration.md, “Disk”). The container can run with a read-only root filesystem:

docker run -d --read-only --tmpfs /tmp -v papyr-data:/data -p 8080:8080 --memory 256m papyr

The image has a HEALTHCHECK (every 30 s, 5 s timeout, 10 s start period): docker ps shows healthy. It uses the verb Papyr.Server health, which makes one GET to /health of the server on this machine and exits 0 for a 200 (and for a connection closed unanswered at Papyr:HttpConnections, configuration.md, “Connections”), 1 for anything else, with one line on stderr. It reads the address from the server’s settings (--urls, ASPNETCORE_URLS, ASPNETCORE_HTTP_PORTS, else port 5000) or takes a URL: Papyr.Server health http://127.0.0.1:8080/health. In a Dockerfile:

HEALTHCHECK --interval=30s --timeout=5s --start-period=10s CMD ["/app/Papyr.Server", "health"]

The verb token turns the admin API on, and bakes nothing in: admin-api.md, “Turning it on”.

A host whose deploy takes an image and no environment needs the settings baked in with deploy/site.Dockerfile (self-hosting.md, “What to set on a public instance”); pin a version of the published image as its BASE, not latest. The first build takes --build-arg INSTRUCTION_SET=x86-64-v3 to target AVX2 hosts only: the binary stops at the start on a CPU without it. Leave it out for any other x64 host.

Limits per source

The server counts, per source (an IPv4 address, or the first 64 bits of an IPv6 address), the connections, the rooms they are in and the rooms created in the last hour, in memory only. A source over a limit is refused with a close status the page explains (4005, 4006) or with 429 before the upgrade; a full store refuses new rooms with 4007 while the stored rooms go on (configuration.md, “The relay’s limits”). Users behind one address share one source. A tool that runs many clients from one machine (the bench, the browser checks, a developer creating more than 10 rooms an hour) sets the four limits high.

While the connection one beyond ConnectionsPerSource is open (up to 10 seconds, so that the page can say why with 4005), every further connection of the source gets 429 before the upgrade, which a browser shows as a closed socket (1006) without a notice.

When the limits are held full

The limits bound the server; they do not stop a few sources from holding it full (configuration.md, “Disk”, “What the limits do not stop”). No address is logged and no request path names a room, so nothing ties an address to a room; a proxy’s access log does show which addresses hold many connections to /relay. From there:

  • Block the address or network at the proxy. A limit at the proxy per IPv6 /48 or /56 binds what the server’s /64 does not.
  • Remove the rooms’ files ({data directory}/rooms/{roomId}.log and .snapshot) while nobody is in them: a loaded room keeps its files open. The freed places are counted at the next pass of the expiry, within the hour.
  • A shorter Papyr:RoomExpiryDays shortens how long a filler must keep writing.

Behind a proxy

Complete configurations for YARP, Caddy, nginx and Traefik, and what each must do: self-hosting.md, “Behind a reverse proxy”. This section is what to watch for, the rule for the client’s address and the check after a deploy.

Pass the Host the browser sent, port included, or the relay refuses every page with 403 (its Origin is not the request’s own host). The first refusal after a start writes event 33; papyr.relay.connection.refused with the reason origin counts them all.

Set Papyr__TrustedProxies to the address or network the proxy connects from. Without it every connection comes from the proxy’s address, so all users are one source and the limits per source refuse them. With it, the server takes the client’s address from the last entry of X-Forwarded-For, from these peers only; what a client wrote in front is ignored. One hop only: a chain of proxies gives the address the last one saw. A request with X-Forwarded-For from an untrusted peer writes event 19 once per process.

Trusting whole private ranges is sound only when nothing but the proxy can reach the container: a network of its own, the port published on loopback only. On a host whose port is published to a network clients use (-p 8080:8080 in an office or a cloud network), a client could connect from a private address and write any X-Forwarded-For, dodging the limits or using up another user’s. Name the proxy’s own address there, or nothing.

Before the first deploy with expiry, back up /data (“Backup and restore”): the first pass, a minute after the start, removes every room nobody wrote to for Papyr:RoomExpiryDays.

Check after a deploy. Open a room through the proxy: it syncs (“Saved”), and no line of event 19 or 33 follows in the log. Then from two networks (a phone off Wi-Fi and a computer) each opens RoomsPerSource + 1 rooms: only each one’s own last room is refused (4005). A forged header is counted under its real address: node tests/web/limits.mjs <site> sources 203.0.113.1 198.51.100.1 <n> (with <n> the site’s RoomsPerSource, 4 when left out) forges two addresses from one machine and must answer “one source”; node tests/web/limits.mjs <site> connections shows that the proxy passes the 429 through. The line of event 20 at the start names as many rooms as /data/rooms holds; at StoredRooms or above every new room gets 4007: raise it (configuration.md, “Disk”) or remove rooms.

Addresses

Address What
/ The start page: a new private room (options: a password, an expiry), a new public room, a public room by name.
/r#<name>, /r#s.<secret> The room page: the editor with Markdown tools, a preview and the others’ carets. /r alone makes a new open room with a random name, /r#s. a new sealed room. No room opens by its id. Needs HTTPS (or localhost) and a browser with constructed style sheets (Chrome 73, Firefox 101, Safari 16.4 or later); it says so otherwise.
/relay The WebSocket the room page connects to. The room is named in the first message, not in the path.
/health ok while the server runs.
/cli How to get and start the command line, with this server’s address in the example (cli.md). Static, never framed.
/privacy What is encrypted, what the server still sees and what Papyr cannot protect, in plain words (privacy.md has the details). Static, never framed.
/admin The operator’s page (admin-api.md, “The admin page”). Static and public; shows nothing without the token. Keep it off the public site with /api.

What the server sees

What is encrypted and what the operator can read: privacy.md. What it means for the operator:

  • A room’s id opens nothing by itself, but an open room’s id confirms a guess of its name, and whoever knows the name is a member: the admin API’s listing, the events’ tags and a backup show the text of every open room whose name can be guessed. Keep them as private as the data directory.
  • Its spans, metrics and log lines hold no content, no address and no room id (telemetry.md): the log names a room by its tag, the first 8 characters of its id.
  • The server writes no IP address, but a reverse proxy, a TLS terminator or the host in front of it sees every one and may log it (privacy.md, “Your IP address”).
  • No URL holds a room id or a secret, so a reverse proxy’s access log ties no request to a room. The admin API’s paths do hold ids (DELETE /api/rooms/{id}, the listing’s after), so ASP.NET Core’s request logs and spans of them do too.

The data directory

{data directory}/rooms/{roomId}.log        the updates after the snapshot, appended
{data directory}/rooms/{roomId}.snapshot   the latest snapshot
{data directory}/rooms/*.tmp               a snapshot or a rewrite in progress
{data directory}/rooms/.format-5/          an empty directory: no room file of an earlier version was found here
{data directory}/blobs/{roomId}/           the room's images, encrypted (`self-hosting.md`, "Images")
{data directory}/papyr.lock                held by the running server ("One process at a time")
{data directory}/admin-token               the admin API's token, when `Papyr.Server token` made it

The directory is created with mode 0700 and the files with 0600. It must be on a case-sensitive filesystem (room ids differ in case only). A room nobody wrote to has no files. At the start the server deletes the *.tmp files (a crash’s leftovers) and counts the stored rooms (event 20). Until it has once found none, it also reads the first bytes of every room file, and a room file of an earlier version stops the start, with nothing changed. The message names the files and what to do: self-hosting.md, “From room files of an earlier version”. The first line of each file names its version (papyr-room-log 5, papyr-room-snapshot 5 in this version); the format is in spec/room-files.md.

Once a start finds no file of an earlier version, it makes rooms/.format-5, and later starts only list the names: reading every file’s first bytes takes about 2 ms per room on a cold disk (40 s for 10,000 rooms after a reboot). So a file of an earlier version put in after that (copied in by hand, or written by an older image run on the directory) no longer stops the start: the server answers its room as damaged (event 11) and counts, lists, expires and backs it up like any other, and a restore refuses a backup that holds it. Remove rooms/.format-5 to have the next start check every file again: after copying room files in by hand, and after running on the directory an image older than the mark, which does not know it. An image that knows the mark removes the marks of other versions (.format-4, .format-6) at its start, so after a rollback to it the newer version checks again by itself.

Finding a room’s files from the log. A line says room Ab3dEf9x: list {data directory}/rooms/Ab3dEf9x*, and search the log for the tag to see every line of that room. Two rooms share a tag with a chance of about 2 in 10^9 among a thousand rooms; the listing shows it when they do. The tag cannot be used to join the room.

A room whose files cannot be read is not opened (its connections get 1011) and its files are left as they are. Restore them from a backup, or remove both: the room then starts empty, and a page that still holds the text sends it again.

Expiry

Two rules, whichever comes first. The pass runs a minute after the start and then every hour.

  • A room’s own expiry. A sealed room may be made with an expiry (the start page offers 1 hour, 1 day, 1 week, 30 days or never). It is part of the room’s link and id and cannot be changed. From that moment the server joins nobody to the room (4011), and within the hour after it the pass closes the room’s connections with 4011 and removes its files, loaded or not, whatever its last write (event 39, papyr.room.expired with the reason expired). The admin API lists it (expires). It applies whatever Papyr:RoomExpiryDays says.
  • Idle rooms. With Papyr:RoomExpiryDays set, a stored room whose files were last written more than that many days ago is removed (event 17, the reason idle). Only a write keeps a room: an update or a snapshot. A room that is only read is removed Papyr:RoomExpiryDays days after its last edit; a loaded room is never removed for this. A join of a removed room finds it empty; a page that still holds its text stores it again as a new room.

A room’s blobs (the images of its pages) go with it, whichever rule removed it or the operator did. The pass also removes, at its end, the blob directories of rooms that have no file and whose newest blob is more than an hour old (blobs of rooms that were removed while the removal of their blobs failed, or whose files were removed by hand). A removal that fails is written as event 40 (BlobStoreFailed) and tried again at each pass; a directory under {data directory}/blobs/{room id} that holds something the server did not write (a subdirectory, a file you put there) fails its removal at every pass, so event 40 repeats until you remove it by hand.

Keep the clock right: a file time in the future counts as fresh (a clock set back deletes nothing), but a clock that jumps ahead by a month expires every room. A restore must keep the files’ times (cp -p, rsync -t, tar), or the restored rooms count as written at the restore. Files removed by hand free their places of StoredRooms at the next pass. A clock that jumps ahead also passes the rooms’ own expiries: their rooms are refused and removed. A clock set back lets a join create a removed expired room again, empty, under its old id.

Removing a room

Use the admin API (admin-api.md, “Removing a room”), not rm: it closes the room’s connections with 4008, so the open pages stop and do not store the room again, and frees its place of StoredRooms at once. A page that was offline during the removal can still store the room again when it connects.

One process at a time

The server holds {data directory}/papyr.lock (an exclusive lock of the operating system) while it runs; the file itself stays and means nothing. A second server, or a restore, on the same directory stops with a message. The lock does not reach across machines on a network filesystem whose locks are not shared (some NFS and SMB setups), nor between containers on different hosts mounting one volume: never run two servers on one directory.

Backup and restore

A plain copy of the data directory while the server runs is not safe: a copier that reads a room’s .snapshot before a new snapshot and its .log after gets a gap, and the room loads as damaged (1011). What is safe:

  • stop the server, copy the data directory keeping the files’ times (cp -a, rsync -a, tar), start it again;
  • an atomic snapshot of the volume (LVM, btrfs, ZFS, a cloud disk’s snapshot): it holds what a crash would leave, which loads;
  • GET /api/backup (admin-api.md) while the server runs;
  • the scheduled backup (“Scheduled backups”): the same tar, written by the server into a directory of its own.

The backup

Fetch it with GET /api/backup (admin-api.md, “Backup”). The tar holds every stored room as rooms/{id}.snapshot and rooms/{id}.log, and after the rooms their blobs (the images of the rooms) as blobs/{room id}/{blob id}.{length}, nothing else. Each room is taken at one moment between two of its writes and loads as it was then; nothing is stopped. The blobs never change, so each is copied as it is when its turn comes; one that was removed meanwhile is left out and the backup goes on. A backup grows by the blobs: up to Papyr:Blobs:TotalBytes (1 GiB by default) more than the rooms’ files, and the scheduled backups of Keep files keep that many times over. Updates accepted but not yet flushed are in it. A backup cut off (a room’s files could not be read, the server stopped, the client went away or took nothing for a minute) ends without the tar’s end: curl -f fails and the restore refuses it; tar tf may not notice. Ask again.

Keep the tar as private as the data directory (“What the server sees”). A restore keeps each room’s epoch, so a page still open on a room repairs it by sending its text again; what only closed pages held is lost.

The restore

Into a data directory without rooms, with the server stopped, by the binary itself (the image has no shell and no tar):

Papyr.Server restore backup.tar                      # or: Papyr.Server restore < backup.tar
Papyr.Server restore backup.tar --Papyr:DataDirectory /srv/papyr
Papyr.Server restore backup.tar --keep-expired       # rooms past Papyr:RoomExpiryDays too (below)
docker run --rm -i -v papyr-data:/data papyr restore < backup.tar

The volume at /data must be writable for uid 1654; do not run the restore as root (--user 0), or the server could not open its files. Give the restore the settings the server runs with (appsettings.json, Papyr__DataDirectory, --Papyr:...): it reads the data directory, the expiry and the limits (Papyr:Relay:*, Papyr:Blobs:BlobBytes) from them. A blob whose room is not in the backup (the room expired before its blobs did) is restored all the same.

The restore unpacks into rooms.restore and blobs.restore beside rooms/ and blobs/ and renames them only at the end (the blobs first, rooms/ last), so a refusal leaves nothing. It writes the files as the server does (0700, 0600, the files’ times kept, fsync before it reports success; run sync before you rely on it against a power cut), loads every room as a check, prints how many rooms and bytes it restored and exits 0. It refuses, with exit code 1 and a message:

Refused Why
A data directory whose rooms/ holds anything but the empty directories .format-* a server leaves, or that a server or another restore holds Restore into an empty one.
An entry not named rooms/<room id>.log or .snapshot or blobs/<room id>/<blob id>.<length>, not a regular file, or named twice; a tar that is no tar The entry is named.
A blob whose size is not the length in its name, or above Papyr:Blobs:BlobBytes; a data directory whose blobs/ holds files Use the backup as the server wrote it; restore with the BlobBytes of the server the backup came from (--Papyr:Blobs:BlobBytes=...), which the server then needs too; remove the blobs/ directory (a restore that stopped between the moves of blobs/ and rooms/ left it with its blobs and no rooms) or restore into an empty directory.
A tar without its end (two blocks of zeros), or with anything but zeros after it “The backup is cut off: … Take the backup again.”
An entry larger than the limits allow (Papyr:Relay:StoredBytes for a .log, SnapshotBytes for a .snapshot), or more rooms than StoredRooms Restore with the higher limits given (--Papyr:Relay:StoredBytes=...), which the server then needs too.
With Papyr:RoomExpiryDays set, rooms last written longer ago than that The server would remove them in its first pass. --keep-expired restores them dated to the restore; or run with Papyr:RoomExpiryDays=0.
A room file of an earlier version Nothing is restored; export the texts with the image that wrote them (self-hosting.md, “From room files of an earlier version”).

A room past its own expiry is not refused but left out: the restore lists its tag (restore: 1 rooms are past their expiry and were left out: Ab3dEf9x*.) and restores the others. Nothing brings it back, --keep-expired neither: its expiry is part of its id.

A room that does not load is named by its tag (restore: room Ab3dEf9x does not load: ...) and makes the exit code 1, but it is restored as it was: it was damaged in the server it came from too. Remove its files (rooms/Ab3dEf9x*) to start it empty.

Scheduled backups

The server writes the same tar, blobs included, itself every Papyr:Backup:IntervalHours hours into Papyr:Backup:Directory and keeps the newest Papyr:Backup:Keep (configuration.md, “Scheduled backups”). Off by default. It needs no admin API.

docker run -d --name papyr -p 8080:8080 -v papyr-data:/data -v /srv/papyr-backups:/backups \
  -e Papyr__Backup__Directory=/backups -e Papyr__Backup__IntervalHours=6 -e Papyr__Backup__Keep=28 papyr
  • The directory should be on another volume than the data directory: a backup on the disk that fails with the rooms is none. The server creates it (0700) and must be able to write it; in the image, as uid 1654. Each server needs a directory of its own: it holds papyr-backup.lock there while it runs. Keep it as private as the data directory.
  • The files are papyr-{yyyyMMdd}T{HHmmss}Z.tar, named after the UTC start time, mode 0600. A backup is written as .tar.part, synced and then renamed, so a .tar is always whole; a .part is in progress or a leftover, which the next backup and the next start remove.
  • When. The first backup runs a minute after the start, or one interval after the newest backup in the directory when that is later; then every interval. A backup through the admin API that is running makes the scheduled one skip its turn (event 35); a scheduled one makes GET /api/backup answer 409.
  • Retention. After each backup the files of its name pattern beyond the newest Keep are removed, oldest first by name. Nothing else in the directory is touched.
  • The clock. When the clock was set back, backups written while it was ahead are dated after the new ones. The server says so at each backup (event 38), ignores them when it decides when the next is due, and removes them first. Until then restore by the files’ write times (ls -lt), not by name. Keep the clock right (NTP).
  • Failures (a room file that cannot be read, the disk full, the directory gone) remove the unfinished file, keep every whole backup and are logged (event 36) and counted (papyr.backup.scheduled.runs, failed); the next interval tries again. When only the removal of older backups fails, the backup counts as completed with event 37.
  • Watch it with papyr.backup.scheduled.last_success (telemetry.md): when the newest whole backup started. Alert on now - last_success > 2 x interval.

Restoring one is the restore above, with its file:

Papyr.Server restore /srv/papyr-backups/papyr-20261116T030000Z.tar
docker run --rm -v papyr-data:/data -v /srv/papyr-backups:/backups:ro papyr restore /backups/papyr-20261116T030000Z.tar

Off the machine. Papyr does not upload; let your tool take the directory, scheduled a little after the interval’s turn, without the .part files. With restic (it deduplicates, so many tars cost little more than one) or rclone (sync mirrors the retention, copy keeps every backup):

restic -r sftp:backup@vault.example:/srv/restic/papyr backup /srv/papyr-backups --exclude '*.part'
rclone sync /srv/papyr-backups offsite:papyr-backups --exclude '*.part'

High availability

Papyr runs as one instance on one data directory; there is no cluster and no replication. A warm standby: ship the scheduled backups to the other host, and when the first fails, restore the newest there, start the server and move the DNS (or the proxy’s upstream). What can be lost is at most one interval, of which every page still open sends its text again when it reconnects; the switch takes a restore and a start. Sharding rooms over several instances is not possible behind a plain proxy: the room is named inside the WebSocket, not in the URL.

Releasing

A release is a git tag vMAJOR.MINOR.PATCH, with -prerelease for a candidate (v0.3.0-rc.1); nothing else makes a version. The person releasing picks the number: before 1.0 a feat!: or fix!: since the last release raises the minor.

  1. Wait for a green ci run on the commit of main to release.
  2. Optionally a dry run: Actions, release, Run workflow, on main, with the version and dry_run on. It builds everything and pushes nothing.
  3. Tag and push: git tag v0.3.0 <commit> && git push origin v0.3.0.

The workflow release.yml publishes both Native AOT binaries, assembles the image for x64 and arm64, starts both and checks their health, then pushes ghcr.io/msallin/papyr:0.3.0, :0.3 and :latest (a prerelease gets only its own tag). It also creates the GitHub release with the notes as its body: build/release-notes.cs lists the commits since the previous tag, breaking changes first with the “What to change” of their bodies, then features, fixes and performance. To read them before tagging: dotnet run build/release-notes.cs -- v0.1.0 HEAD. To withdraw a release, delete the package version on GitHub; a tag is not moved: release the next patch. The notes written by hand for each minor version live in docs/releases/.

Continuous deployment

The project’s own sites, https://papyr.ch and https://docs.papyr.ch, follow main. The workflow cd.yml builds after every green ci run on main the general image and the site’s image over it (deploy/site.Dockerfile), checks that the site’s image starts and answers, and pushes ghcr.io/msallin/papyr:main and ghcr.io/msallin/papyr-site:main (each also as :sha-<commit>). A push to main that changes the documentation, which ci ignores, builds and pushes ghcr.io/msallin/papyr-docs:main (deploy/docs.Dockerfile: Jekyll, then a small Kestrel site). The hosting VM polls those tags and restarts a site when its tag moves, so a commit is online a few minutes after its run; to go back, run cd by hand on a branch or tag at an earlier commit. The tags main and sha-* are for these two sites only: a self-hosted server takes a release (:0.3, :latest).