Your browser sucks, JavaScript disabled! Gunnar's console needs JavaScript and WebAssembly. Git over SSH on port 2222 does not.

nordisk/monetize

docs/alerts.md

Operator alerts

Every line monetize writes for its operator starts ALERT <code>. The ONE box hears it (monetize_admin::alerts), keeps it by its code and mails it to the operator through the box's own relay (secret.alert_to): the first mail at once, then at most one per ten minutes per code, so nothing is lost and a storm stays one mail. code-red never waits. ALERT: with a colon inside an order's history is a note on that order, not a log line.

crates/monetize-admin/tests/the_alert_table.rs holds this table to the code: every ALERT <code> monetize can write is a row, every row is a code it writes, and an ALERT with no code fails the build.

Severity:

Who: operator is the ONE box's operator (the mail). operator, at once is a CODE RED: it is mailed with no pacing and stays raised until the step that clears it succeeds.

Codes monetize writes

code severity who what it means what to do
code-red critical operator, at once A step that must not fail failed (a custody take-back, an unsaved run): the line names the site, what broke, the error and the next step. Do what its next: says; the next success clears it.
attach.bearer_expiring warning operator An estate's ONE bearer stops opening Signup, Purchase and EstateOrder in a few days. Enrol the estate again (an attach token, then the kit's attach) before the date.
attach.bearer_expired critical operator An estate's ONE bearer has expired: its signups and purchases are refused. Enrol the estate again now.
joined.transient warning operator A joined estate's record could not be read or taken this tick (the guardian store refused a write, the record was unreadable): it stays and every tick tries again (A32-1). Look at the ONE box's disk and its guardian store; restart monetize if the store stays refused.
console.fleet-job-failed warning operator The console's fleet job (the ONE Guardian.Cases read its pages show, RA3-1) failed: the pages show the last fleet it read, with its age. Look at the ONE box's monetize: is it up, does the console's bearer still open Guardian? The job asks again every 30 s.
cloud-guest.state-unwritten warning operator A cloud guest's state was not written after an op; the answer went back as it was. It is retried on the next call; if it repeats, check the guest state store's disk.
entitlement.push_stuck warning operator An entitlement has not reached its product after many pushes. Check the product's door and its bearer; it is still pushed every tick.
entitlement.site_not_holding warning operator A paid fact (a paid plan, its term) is on the ledger, but the site's main does not hold it, so the buyer's plan reads wrong there (ALICA-4). The ONE box queues it to the site again on every tick. Nothing, if it stops after the site's next pull; if it repeats, check that the site's main pulls from this box (its pull slot and bearer).
estate-orders.account-not-made critical operator An order was paid and its account could not be made. Make the account right by hand; nothing is laid for another's account.
estate-orders.account-row notice operator An order's row on its account was not written. The next tick tries again.
estate-orders.custody critical operator An estate's secrets were not forgotten after the hand-off. Remove them from the custody by hand.
estate-orders.dns-by-hand warning operator A DNS record must be set by hand (no writer for the zone). Set the record the line names.
estate-orders.dns-not-found critical operator A give-back did not find the DNS records its lay wrote where they were written, or the writer that wrote them is no longer held. Remove the records the line names by hand, in whichever zone they stand.
estate-orders.dns-takeover warning operator A lay took over existing DNS names of the operator's site. Check that the names should point at the new lay.
estate-orders.dns-writer critical operator The shop has no usable DNS writer: nothing is sold where it writes. Feed or fix the zone's writer.
estate-orders.guardian notice operator The guardian could not be asked about an order's estate. It is asked again on the next tick.
estate-orders.install-abandoned critical operator An install was given up. Settle the order (retry or refund).
estate-orders.install-deadline warning operator An install passed its deadline. Look at the install's log; settle it if it does not finish.
estate-orders.queued-deadline critical operator A paid order waited in the queue for three days with no room to set it up: it is owed now (A32-22). Set it up again or refund it (site settle).
estate-orders.queued-long warning operator A paid order has waited in the queue for six hours (told again every six while it waits); its buyer reads the day it is set up or refunded by (A32-22). Raise the cap or the slots, or settle it.
estate-orders.new-estate warning operator A site the box takes late (OVW-5) was not ordered: the queued job that orders it once the box took it gave up after 120 s, and the line names the site and why. Look whether the box took the site (monetize estates); order it again with Estates.NewEstate when it did.
estate-orders.material warning operator An estate never took its installer material and it was forgotten. The estate cannot be upgraded, recharged or renewed by itself; re-lay or settle it.
estate-orders.owed critical operator A paid order failed to be set up and waits for the operator. Retry it, or cancel it with a refund.
estate-orders.stopped warning operator An order's terms are refused for good (its site left the feed, sells another product, or names none): the run is stopped by name, neither booked nor driven, looked at again every 10 min. Nothing of it is given back or billed. Declare its site in the feed again (it resumes by itself), or terminate it.
estate-orders.page critical operator A payment page was not closed at the vendor, or a payment was captured for nothing. Close the page, or refund the capture, at the vendor.
estate-orders.queue warning operator An order waits in the cloud's queue longer than it should. Check the cloud's capacity and the order's words.
estate-orders.ready warning operator The first-login invite could not be read, so the ready mail waits. Check the estate's invite; the mail goes when it can be read.
estate-orders.refund-forget warning operator A refunded order's custody could not forget its sealed pairing material. Remove the material from the custody by hand.
estate-orders.reversal critical operator A vendor reversed a payment (a refund or a dispute) on an order. Check the order and its estate; take back what the line says.
estate-orders.terminate critical operator A termination step failed or could not take the run. Look at the order; the next tick asks again.
estate-orders.tick critical operator An estate-order tick panicked. Read the panic; report it as a bug.
estate-orders.unwritten warning operator An order's run changed and could not be written. The step runs again; check the order store's disk if it repeats.
estate-orders.update warning operator An estate's update step could not go on. Read the words; run the update again when they are fixed.
front.bearer-missing critical operator The front has no bearer and refuses to serve. Write the bearer file the line names.
grow.failed warning operator A grow failed. Read why; the room stays as it was.
grow.not_told critical operator A sale went past the room and the estate could not be told to grow. Grow the estate by hand or fix its door.
grow.twin-alone critical operator The main grew and its twin did not catch up. Grow or recover the twin.
guardian.looks-refused warning operator jera refused the guardian's provider-looks job as stale (RA3-2): no look at the provider was asked this tick. It should not happen (one key, one generation); if it repeats, restart monetize and tell the lead lane.
guardian.case-unsaved warning operator A guardian case could not be saved. The next tick moves it again from the store.
guardian.started warning operator A box monetize laid (front, main or twin) is STOPPED at the provider, not dead (BOX-START: UpCloud stopped the account's servers on 2026-11-01): the guardian starts it, its disks kept, nothing rebuilt; the front first, then the main, its twin only once the main is heard again. The line names the estate, each box and the case. Nothing, unless the case fails: then find why the provider stopped the box (billing, a hold) before anything is rebuilt.
guardian.operator_rebuild critical operator A guardian case waits for the operator to rebuild a box in the estate's own cloud. Rebuild the box with the product's own ops kit.
box-watcher.account critical operator The ONE-box watcher on dwarf (monetize-box-watcher, outside UpCloud) cannot read an account's UpCloud token from its secret file (the key is named, never the value): it cannot see or start that account's ONE box. Logged to its journal, not mailed. Fix the secret file or the watcher.toml account row.
box-watcher.not_found warning operator The watcher found a watched ONE box in no account (by uuid, or by the address its DNS name resolves to). Logged to its journal. Correct the box row in watcher.toml.
box-watcher.error critical operator A watched ONE box is in a state the watcher does not start (error, or unknown): nothing done. Logged to its journal. Look at the box in the provider's console.
box-watcher.failed critical operator The watcher asked UpCloud to start a stopped ONE box and was refused (or the API could not be read). Logged to its journal. Look at the provider's answer in the journal (billing hold, out of stock) and start the box by hand.
money-log.unwritten warning operator PID 1 could not write the money lines' own home (<data>/money/lines.log, ALICA-3); said once. The lines are still in the boot log, where noise can push them out. Check the data disk (full, read-only); the next money line is tried again.
jobs.failed warning operator A background job of the server (a jera job: a wallet or plan reading, a grow drive, a callback's settle or grant, a keeper, the guest watch, the off-box copy) failed or panicked; the line names its kind and subject. Fix what it names; a duty's next look runs as due, a one-shot's subject is asked again by its caller.
jobs.unstarted warning operator A keeper duty of the server could not start (no runtime). Restart the server; until then the readings are made when a customer asks.
jobs.unrecorded warning operator The server's (or PID 1's) jobs' ledger under <data>/jobs did not open: every job still runs, but none is a row in nornir jobs (RA4-18). Read the error (a full or read-only disk, a second process holding the ledger); restart once it is fixed.
guests.watch-unstarted warning operator The guest watcher did not start. Restart the server after a new or rebuilt guest.
joined.unstamped critical operator A refused joined record could not be marked as refused in itself; it was moved aside as it is (RA1-6). Check the data disk; write the reason into <estate>.json.refused.why by hand before the next restart, or a later app may read it as an older app's and put it back.
joined.why-lost warning operator The reason a joined record was refused could not be written beside it; the record is kept aside with its reason in itself (RA1-6). Check the data disk (full, read-only); nothing is put back.
narrate.no-receiver notice operator This build links no dev-narrate receiver. Nothing, unless you meant to build narrate.
nonpay.deleting notice operator A product is purging a non-paying account. Nothing; it is the nonpay engine's step.
nonpay.delete_failed critical operator A non-paying account's deletion did not complete. It is retried in an hour; check the product if it repeats.
offsite.failed warning operator The off-site backup failed. Fix what it names; the next run tries again.
orders-waiting warning operator Grow orders wait for room in the cloud. Check the cloud's capacity.
plugin.refused warning operator A plugin update or look was refused. Read the plugin row; fix the plugin or its pin.
product-guest.facts-unread critical operator A product guest's facts could not be read: its tickets, test buttons and guardian are off. Fix the guest; it is asked again on the next look.
self-update.confirm warning operator A new release serves, but its slot could not be confirmed. Confirm the slot, or it is rolled back.
self-update.reexec critical operator A self-update's re-exec failed and its children are stopped. Start the box again; read the error.
self-update.refused warning operator A self-update was refused. Read why; the box runs the release it had.
self-update.rollback critical operator A new release did not serve in time and was rolled back. Read the release's log before trying again.
self-update.rolled-back warning operator The box runs a rolled-back release. Read why before the next update.
server.bearer-missing critical operator monetize-server has no bearer and refuses to serve. Write a bearer to the token file (0600).
server.bearer-refused critical operator monetize-server was asked to serve open while a bearer is configured. Drop the open flag.
sold-estate.grow warning operator A grow's go-ahead was not queued for the estate's boxes. Read why; ask again.
sold-estate.keep-boot-unsaved warning operator A box was keep-booted to a release and the run was not saved. Check the run; the step is recorded when it is done.
sold-relay.unsuffixed warning operator At boot, this box still holds the old unsuffixed sold-relay secrets (secret.sold_smtp_url / …sold_smtp_from, before the relay went per product): nothing reads them (RA5-42). Feed sold_smtp_url.<product> and sold_smtp_from.<product> for each product the box sells; then the old ones can go.
spend-cap warning operator A grow was refused by the spend ceiling. Raise the ceiling, or leave the product at its size.
twin-grow.journal warning operator A twin grow's order could not be journaled. A restart may order it again; check the journal's disk.
users.refunded critical operator A purchase was paid and could not be done, so it was refunded. Tell the buyer; check why.

Kinds the box makes of other lines

code severity who what it means what to do
critical critical operator A line that says CRITICAL. Read the line.
grow.room_queued notice operator A sale past the room was queued for its estate. Nothing; the grow follows.
payment.refused warning operator A callback, purchase, checkout, capture, refund or vendor call was refused. Read the refusal.
attach.refused warning operator An estate's enrolment or install notice was refused. Read the refusal; enrol again.
case.<state> notice operator A guardian case moved (a recharge, a rebuild, a fence). Follow the case.