Troubleshoot a watcher

Fix watcher problems by symptom: will not start, files not detected, uploads stuck, auto-update quiet, and where to find logs.

ForLab operators and admins

Find your symptom below. When you are stuck, the logs are the fastest way in. Admins can also inspect heartbeats and events under Watchers in the web app.

Setup and registration

“Instrument is still pending activation”

The instrument was registered but an admin has not confirmed it. Ask an admin to open Instruments and click Confirm. See Manage instruments.

“Instrument already has an active watcher”

Each instrument can have at most one active watcher. An earlier install (often on another PC, or before a reimage) is still registered. The CLI prints the existing watcher id; deregister it before re-running init:

Web app: Watchers, open the existing watcher, click Deregister
API: DELETE /api/v1/watchers/<existing_watcher_id> with a bearer token that has watchers:admin

Deregistration is a soft-delete: history stays under Watchers → Deregistered.

`init` rejects my API key

The wizard rejects values that do not start with dhub_, and strips zero-width and non-breaking-space characters that Windows clipboards inject. If a paste keeps failing, pass --show-key so you can see what the prompt received.

The watcher will not start

Run data-hub-watcher config show to confirm config and environment
Run data-hub-watcher watch --dry-run: it prints the first failing check
Confirm the instrument is active, not pending
On Windows, if the service will not start, see Windows service

The watcher shows “Waiting: folder not found”

Watcher 1.4.0 and later start even when the watch folder is missing, for example when a network drive isn’t connected yet or the folder was renamed. The watcher keeps sending heartbeats but watches nothing, and the Watchers page shows Waiting: folder not found. Hover over the badge to see which folder it’s looking for.

The watcher checks for the folder again once a minute and starts watching as soon as it appears. To fix it:

Create or reconnect the folder on the PC, or
Point the watcher at the right folder from the web app; see Change a watcher’s config

Watcher 1.4.3 and later also go back to this state when the folder disappears while they run, for example when a network drive drops or the folder is deleted. The watcher notices within a minute, looks again a few seconds later so that a drop of a moment doesn’t count, stops watching, and shows Waiting: folder not found. It gives a file that was uploading at that moment up to 10 seconds to finish, cuts it off if it isn’t done, and uploads it again once the folder is back. When the folder returns, the watcher starts watching it again without a restart, and queues a watcher_started event that says it resumed.

Older watchers exit at startup instead, with an error that the watch directory doesn’t exist.

A config change shows “Change rejected”

The watcher checked a change made in the web app and refused it, so it kept its previous settings. The reason shows on the watcher’s Configuration tab and as an error event with details.kind set to config_revision_rejected. The most common reason is a folder that doesn’t exist or can’t be read on the PC. Fix the cause, then save the change again or Restore an earlier version from the history.

“Data Hub can’t read the watcher’s config file”

The watcher’s Configuration tab shows this when the config.yaml the watcher sent breaks the rules the web app checks, for example a relative watch folder, more than 50 file patterns, or a run detection pattern with Python-only syntax such as (?P<name>…). The notice names each field that needs fixing, by its key in the file.

Until it’s fixed, edits made on the PC aren’t recorded in the history, and if Data Hub has no earlier revision of the settings, there are none to edit there. Fix the file on the instrument PC, for example with data-hub-watcher config edit.

A config change stays on “Change pending”

The watcher picks up a change after its next heartbeat, within about a minute. If it stays pending:

Check that the watcher is Online. An offline watcher applies the change when it next starts.
Check that the watcher runs 1.4.0 or later. The web app won’t save changes for older watchers.

The watcher shows Offline right after a restart

The watcher checks in once a minute, so a watcher that was just restarted can take up to a minute to show as Online. Refresh the page after a minute. If it still shows Offline or Unresponsive, check that the process is running and see “Connection error” or “Request timed out”.

Heartbeats fail with `426 Upgrade Required`

The watcher is older than the Minimum supported version an admin set under Settings → Watchers, so the server rejects its heartbeats and it shows as unresponsive. It keeps checking for updates every hour and upgrades itself if Latest version is newer. To upgrade now, run data-hub-watcher self-update. See Roll out watcher releases.

“Connection error” or “Request timed out”

The watcher cannot reach the Data Hub API. Check:

Network connectivity from the instrument PC
Environment in config (staging, production, or preview): data-hub-watcher config show
That the API base URL for that environment is reachable

A run is missing after a connection problem

Watcher 1.2.2 and later retry a run report that failed because the watcher couldn’t reach Data Hub, or because Data Hub returned a server error. The first retry comes after 5 seconds. The wait then doubles up to 5 minutes between tries, and the watcher keeps trying every 5 minutes until the report goes through. It needs no new file and no restart, so the run appears on its own once the problem clears.

The watcher records one run_report_failed event when a run’s report first fails. It doesn’t record another for the same run while it keeps retrying, even if more files in the run finish in the meantime.

A failure that isn’t a connection problem or a server error, such as 401 Unauthorized, is not retried, because it would fail the same way. Each one is recorded. Fix the cause, then restart the watcher.

Older watchers try again only when another file in the run finishes or the watcher restarts. Restart the watcher to report the run.

A run was deleted on Data Hub

A run report that fails with 409 Conflict and a message that starts “This run was deleted on Data Hub” means someone deleted the run. Data Hub refuses a watcher’s report of a deleted run, and the files in that report aren’t added to it.

Watcher 1.4.3 and later record one run_report_failed event for the run and stop reporting it until the watcher restarts. A restart tries once more, and records one more event if the run is still deleted. After you restore the run, restart the watcher to send its files.

Earlier watchers record one event for each file, and do it again each time they start. After you restore the run, they send its files when another file of the run finishes or the watcher restarts.

To stop the errors without restoring the run, move the run’s files out of the watch folder.

Files are not being detected

Confirm the watch directory and file patterns. The instrument’s page in the web app shows both, or run data-hub-watcher config show on the PC. Admins can change them from the web app; see Change a watcher’s config
Check that the instrument saves its files into that folder, not a shared network drive or another folder
Check file patterns. Globs are case-sensitive (*.csv does not match data.CSV)
If files live in subfolders, set run detection to recursive; see Configure run detection and uploads
Run data-hub-watcher watch --dry-run to preview matches
Look for pattern_mismatch events on the dashboard

Files are detected but not uploading

In manual mode, files wait for approval in the upload queue; see Manage the watcher fleet
Check the log for upload_failed
A state_write_failed event in the watcher’s event log means the PC’s disk is full or the watcher’s state file can’t be written. Free up space; the watcher retries on its own and uploads the files once it can write again
A revoked or expired token surfaces as 401 Unauthorized. Create a new token and re-run init; see Issue and revoke tokens

A requested upload stays on Uploading

On an instrument in auto mode, Request upload needs watcher 1.2.1 or later. Older watchers in auto mode never read the upload queue, so the file shows Uploading and never arrives. Update the watcher; see Roll out watcher releases.

Files larger than 5 GB don’t upload

Watchers older than 1.2.0 can’t upload files larger than 5 GB, and Data Hub refuses the upload instead of letting it fail partway. Update the watcher to 1.2.0 or later, which uploads large files in parts.

An upload was cut off by a restart or an update

Watcher 1.2.1 and later retry an automatic upload that didn’t finish the next time the watcher starts, on up to three starts. Two events tell you when it stops trying:

upload_retries_exhausted: the upload still failed after three starts. Choose Upload on the file’s run page to try again.
pending_upload_missing: the file was no longer on disk when the watcher started, so there was nothing to retry.

Older watchers don’t retry; use Upload on the run page instead.

A file stays Pending after the watcher restarted

Watchers before 1.4.1 could lose track of a file in an automatic upload when the watcher stopped without shutting down cleanly, such as after a reboot, a power cut, or a crash, just after it reported the run. The run is in Data Hub, but the file shows Pending and nothing uploads it, because after the restart the watcher treats the file as already handled. Watcher 1.4.1 and later record a run and its files on the PC together, so a restart like that doesn’t leave a file behind.

Updating doesn’t pick up files that an older watcher already lost this way. Ask for them from the run page:

If the watcher is older than 1.2.1, update it first. Older watchers in auto mode ignore upload requests; see A requested upload stays on Uploading.

Open the run, select the files that show Pending, and choose Upload. To ask for the files of several runs at once, select the runs in the run list and choose Upload. See Request an upload.

The watcher picks up the request within a minute and uploads the files that are still on the instrument PC.

Files with the same name were skipped

When a run gathers files from several folders, watchers older than 1.1.0 uploaded only the first file with a given name and skipped the rest. Watcher 1.1.0 and later keep every copy, adding a short tag to later names. To upload files an older watcher skipped:

Update the watcher to 1.1.0 or later.

Stop the watcher (data-hub-watcher service stop, or Ctrl+C in the foreground).

Run data-hub-watcher state forget --prefix your_folder_name/ on the instrument PC, where your_folder_name/ is the folder under the watch directory that held the skipped files.

Start the watcher again. It reports the files in that folder again and uploads the ones Data Hub doesn’t have. A run that was deleted on Data Hub refuses its files instead. See A run was deleted on Data Hub.

Auto-update never fires

Check, in order:

Prefer a long-running service (or scheduled) install; a one-off console session may not stay up for the hourly tick
Environment is not preview: auto-update is hard-disabled there
Activity-window guard is not holding it up: run data-hub-watcher self-update, or ask an admin to flag the release mandatory
Last Heartbeat on the dashboard is recent; a stale watcher is not ticking

Upgrade failure modes (update_started → update_failed, editable installs, missing Scheduled Task) are covered in Install and update the watcher and Roll out watcher releases.

Windows service

The service exits immediately

A config file that doesn't load (missing, unreadable, not saved as UTF-8, malformed, or with a value the watcher doesn't accept) stops the service. watcher.log has an error that starts with Cannot load config, followed by the reason. The error also gives the command that checks the file, in the form data-hub-watcher --config "C:\path\to\config.yaml" config validate. Run it as written, because without --config it checks the config file of whoever runs it, not the one the service uses. Fix what it reports and start the service again.

An env file that isn't saved as UTF-8 stops the service the same way, with an error that starts with Cannot load env file. Windows PowerShell 5.1 writes UTF-16 for > and >>, so echo X=Y >> .env.production breaks a UTF-8 env file. Watchers before 1.4.2 wrote only a traceback to the Application event log for either problem.

If it stops before writing to watcher.log:

Check C:\ProgramData\DataHubWatcher\service-bootstrap.log for a pre-dispatcher crash (missing pywin32, moved venv, corrupt install)
Query the Windows Application event log:

Get-WinEvent -FilterHashtable @{LogName='Application'; StartTime=(Get-Date).AddHours(-1)} |
  Where-Object { $_.ProviderName -match 'Python|DataHubWatcher' -or $_.Message -match 'DataHubWatcher' } |
  Format-List TimeCreated, ProviderName, Id, LevelDisplayName, Message

Run the service in the foreground to see the traceback:

& "C:\path\to\venv\Scripts\python.exe" -m win32serviceutil debug DataHubWatcher

`self-update` says “Upgrade dispatched” but nothing happens

On Windows uv-tool installs, the upgrade runs out of process under SYSTEM and takes 30–60s. Track it:

Get-Service data-hub-watcher
Get-Content ~/.data-hub/upgrade-worker.log -Tail 20 -Wait

Turn on debug logging for the service

Add DATA_HUB_WATCHER_LOG_LEVEL=DEBUG to the env file the service loads (typically ~/.data-hub/.env.<environment>) and restart the service. No reinstall needed. This works with watcher 1.4.2 and later; earlier versions ignore the setting in the env file.

Where are the logs?

Rotating logs (10 MB each, 5 backups). CLI watch and the Windows service share one file:

Platform	Log file
Windows	`C:\ProgramData\DataHubWatcher\watcher.log`
macOS / Linux	`~/.data-hub/watcher.log`

Add --verbose for debug console output: data-hub-watcher --verbose watch. Windows uv-tool upgrade output lands in ~/.data-hub/upgrade-worker.log.