A server-side feed consumer spends much of its life discovering that nothing changed. The interesting engineering work is making that ordinary outcome efficient, distinguishing it from failure, and recovering cleanly when a source or worker becomes unavailable. A dependable design does not need an aggressive polling loop. It needs clear state, bounded retries, and a definition of freshness that matches the application.
This guide proposes a small polling worker with a durable record for each source. It is an architecture pattern, not a promise of a hosted endpoint. Start with the Server Feeds API overview if you are mapping the broader problem. Then decide how long your users can reasonably wait for an update and what they should see when a source falls behind that expectation.
Give each source its own state
Track the source address, the last attempt, the last successful retrieval, the last accepted content change, and the next planned attempt separately. These values answer different operational questions. A source can be reachable while returning an unchanged feed, or reachable while returning a document your parser cannot accept.
Keep transport validators, failure counters, and a concise outcome code with the source record. Avoid treating every source as one shared health state. One slow publisher should not force every other source into the same retry schedule. If your worker uses concurrent requests, define a global limit and a per-origin limit so adding sources does not accidentally turn a polite consumer into a burst of requests against one server.
Use conditional requests when supported
IETF HTTP Semantics, RFC 9110, defines validators and conditional requests. A client can send a previously received ETag in If-None-Match when requesting a representation again. For a conditional GET where the representation is not modified under the relevant condition, a server can return 304 rather than transferring the representation body. Last-Modified and If-Modified-Since provide a time-based conditional mechanism. When both relevant request conditions are present, If-None-Match takes precedence over If-Modified-Since.
In your worker, retain validators from successful responses and associate them with the corresponding resource. Do not parse a 304 response as an empty feed. Keep the last accepted items and record that the source was successfully checked without new content. Test this path explicitly; an unchanged response should be an ordinary successful state rather than an exceptional branch nobody has exercised.
Distinguish transport success from usable content
A successful HTTP status does not guarantee a valid feed document. A source may return an HTML error page, an unexpected media type, or a truncated payload. Record transport and parsing outcomes separately. Keep the last known usable data when a new response cannot be accepted, while marking the source’s content freshness accurately.
Set a maximum response size and a total request deadline appropriate to the application. Do not allow a large or slowly delivered document to occupy a worker indefinitely. When a response is rejected, retain only the diagnostics needed for investigation and apply a retention policy. Copying every failed payload into permanent logs can create unnecessary storage and privacy problems without improving the recovery process.
Schedule for a defined delay budget
Choose a polling interval based on the application’s tolerance for delay and the source’s expected behavior. A team reading queue may not need the same cadence as an operational notification pipeline. Start conservatively, measure actual update patterns, and change the schedule only when the benefit is visible.
Use jitter so many sources do not become due at the same instant after a restart. Keep an upper bound on catch-up work. If the worker was offline for a day, it usually should not replay every missed polling slot as separate immediate requests. The goal is to recover current useful state, not to reenact the schedule mechanically. Where complete event history is required, a short rolling feed may not be a sufficient source contract.
Retry failures without creating a storm
Classify failures before deciding to retry. A temporary connection problem, an overloaded source, an authorization failure, and an invalid document are different situations. A bounded backoff policy can reduce repeated pressure after transient failures, but endlessly retrying an unchanged configuration error does not make it transient.
Record the next attempt time durably rather than making every worker calculate a fresh immediate retry. Respect server-provided retry guidance where applicable to the response and your policy. Add a maximum delay and a route for operator attention after repeated failures. A source that needs review should remain visible as such instead of quietly disappearing from the collection or producing an unending sequence of identical alerts.
Make persistence safe to repeat
Assume a worker can stop after storing an item but before recording that the batch finished. On restart, the same item may arrive again. Use stable item identity and a deliberate update rule so repeated processing does not create duplicate records. Keep a source identifier alongside the item identifier to avoid collisions across publishers.
For a notification workflow, storing the item and sending a notification are separate effects. A useful design can write a pending notification record as part of the durable processing step, then let a separate sender handle delivery. That sender needs its own retry and completion rules. Do not claim exactly-once user-visible behavior merely because one database write is deduplicated; inspect the full path from acquisition to external action.
Advance checkpoints after completed work
A checkpoint should describe work the system can safely consider handled. Advancing it after a download but before successful normalization can skip records during recovery. Advancing it after a partial batch without recording which items succeeded can make replay ambiguous.
Define the unit of completion: a source response, a page, or an individual event. Where one transaction cannot cover all required effects, write down the recovery sequence. Test a restart at each boundary. A small state machine with named outcomes is easier to reason about than a long function whose progress exists only in memory. The feed architecture guide explains the identity contract that makes this recovery behavior possible.
Measure freshness honestly
The last successful request time is not the same as the publication time of the newest item. A healthy source may publish nothing for weeks. Conversely, a worker may fetch successfully while the upstream publisher is delayed. Show the relevant clocks rather than collapsing them into a single green status indicator.
For operations, track request success, accepted-content success, and age of the last observed change separately. For readers, distinguish an item’s stated publication time from when your system found it. If the application promises a particular update delay, define exactly what starts and stops that measurement. Without that definition, a freshness target can look precise while leaving both users and operators unsure what it actually means.
Secure the source boundary
A consumer that accepts arbitrary feed addresses needs a deliberate network policy. Restrict supported schemes, apply redirect limits, and avoid allowing untrusted addresses to reach internal services or local resources. A source allowlist may be the simplest starting point for a small internal project. Keep credentials scoped to the intended source and avoid forwarding them across unexpected redirects.
Treat retrieved documents as untrusted input regardless of their transport status. Apply parser limits and safe rendering rules downstream. A successful TLS connection does not make the content semantically valid or safe to execute. Keep network acquisition separate from any model-assisted processing so a document cannot grant itself additional fetch permissions through embedded instructions.
Rehearse the ordinary failures
Build a local test source that can return unchanged content, a delayed response, a temporary error, invalid XML, and a changed item with the same identity. Stop the worker after persistence and before checkpoint completion. Verify that recovery produces the expected state without a burst of duplicate notifications.
Finish with a readable operations view: which sources are current, which are unchanged, which are delayed, and which need review. The strongest server feeds API design is not the busiest polling loop. It is the one that performs useful work when content changes, stays quiet when nothing changes, and tells the truth when delivery or interpretation needs attention.
