A data feed becomes useful when two systems agree on what a record means, not merely on how to parse it. A number can arrive as valid JSON while still being ambiguous: is it a price, a quantity, or a percentage? Does a missing value mean unknown, unavailable, or deliberately removed? A schema is the beginning of that conversation, but the complete contract also needs provenance, timing, and change semantics.
This guide proposes a practical design process for a small structured feed. The examples use an imaginary catalog, not a live product dataset. You can apply the same questions to operational observations, release metadata, or internal reference data. Start with the Data Feeds API topic page and write a short description of the decision your consumer must make from each record.
Decide whether you publish state or change
A snapshot describes state at a particular point. A change feed describes events that alter state. Suppose a catalog record contains product A with quantity 7. In a snapshot, that may mean the currently available quantity is seven. In a change event, a value of seven could instead describe an addition. The field alone cannot resolve that ambiguity.
Choose one interpretation and name it explicitly. For snapshots, define the boundary of completeness: all products, one supplier, or one page of a larger collection. For change events, define the operation, target identity, event identity, and ordering assumptions. Do not ask consumers to infer a deletion from an omitted record unless the snapshot contract explicitly makes that absence meaningful. A partial export should never masquerade as a complete replacement.
Give records and events different identities
The thing being described and the event describing it are not always the same object. A product may keep one identity across hundreds of updates, while each update has its own event identifier. Storing both allows the consumer to recognize a repeated delivery without losing the relationship to the underlying product.
For an illustrative change record, consider fields named source_id, entity_id, event_id, operation, and values. Keep the source identity even when entity identifiers look globally distinctive. It prevents accidental collisions when another source adopts the same naming convention. Define whether an event can be corrected in place or must be replaced by a later event. Your replay procedure depends on that decision, so it should not be left to whichever database operation is easiest.
Document units beside the fields
A schema that says a field is numeric does not explain its unit. For a price, your contract might require an amount representation and an explicit currency identifier. For a measurement, it might require a unit and the circumstances under which the observation was taken. Choose representations based on the precision and range the application needs, then test the boundaries.
Avoid an overloaded field that sometimes contains a formatted display string and sometimes contains a machine value. A consumer should not need to strip symbols or interpret local punctuation to perform a calculation. Keep display formatting downstream from the underlying record. When a source’s precision is limited, preserve that limitation rather than padding the value in a way that implies a more precise observation.
Make missingness a first-class decision
Absent, null, empty, zero, and false are different values in many application contracts. Decide which are accepted for every important field. In a patch-style event, an absent field might mean “leave the existing value unchanged,” while an explicit null might mean “clear it.” In a full replacement record, absence may instead mean that the source did not supply the information.
Test these cases independently. A consumer that fills every missing number with zero can create a misleading business state. A consumer that ignores every null can fail to remove outdated information. Document the rule in plain language and in fixtures. When the source itself is ambiguous, quarantine the record or preserve an uncertainty state rather than pretending a default value is a verified observation.
Carry provenance through transformations
The W3C Data on the Web Best Practices recommends providing provenance information about data origins and changes. It also addresses metadata, licensing information, identifiers, and access. The useful principle for a feed designer is that a downstream reader should be able to understand where data came from and what happened to it, rather than seeing only a detached final value.
For your implementation, record the source identifier, observation time, transformation version, and relevant source reference. When combining two records, preserve the relationship to both inputs where the application needs it. A derived category should not overwrite the source’s original category without a trace. Separating original and derived fields helps reviewers compare the transformation against the evidence and makes later corrections more manageable.
Define time with the consumer in mind
A record may need several timestamps. An observation time describes when a measurement was made. An effective time describes when a change applies. An ingestion time describes when your system received it. These can differ, especially when sources publish delayed batches. Decide which clock controls each application behavior instead of using one generic timestamp everywhere.
For an inventory projection, the effective order of changes may matter more than arrival order. For a monitoring dashboard, the age of the latest observation may matter more than the last successful download. Store the source’s original time value when it helps diagnose interpretation problems. If a timestamp is missing or invalid, keep that state visible; do not quietly substitute the ingestion time and call it an observation.
Separate validation from acceptance
Structural validation checks whether a record follows the expected shape. Business acceptance asks whether the values are suitable for the intended workflow. A record can pass the first check and fail the second. For example, the correct field types do not prove that a measurement belongs to the expected location or that a source is authorized for a particular collection.
Define a small number of named outcomes: accepted, rejected, and held for review may be enough for a first version. Record a reason code that helps operators understand the result without reading every payload. Avoid letting one invalid record invalidate an entire independent batch unless atomic completeness is part of the contract. Preserve counts so a partial success is not reported as a fully successful import.
Plan compatibility before adding consumers
An optional field can be a modest extension, but changing a unit or operation meaning is a semantic migration. A schema version alone does not make that migration safe. State what each version guarantees, which consumers can read it, and how long previous versions remain usable. Keep the version associated with stored records so historical processing can be explained.
When introducing a new enumeration value, test a consumer that knows only the old values. It should have a deliberate fallback or reject the record clearly, not silently map it to an unrelated category. When renaming a field, decide whether to provide a transition period or a new version boundary. Compatibility is a behavior you verify with old fixtures, not just a label attached to a release.
Rehearse corrections and replay
Create an initial record, apply an update, repeat the update, then deliver an older event late. Your expected final state should be written down before the test. The sequence exposes hidden assumptions about ordering and duplication. Next, correct an earlier event and decide how that correction should affect any derived state. Some applications can rebuild from history; others need an explicit compensating operation.
Your recovery plan should identify the last completed checkpoint and the records needed to resume. Keep retention aligned with that plan. A replay design is not useful if the source history disappears before recovery can occur. For delivery mechanics, the server feed reliability article explores the separate responsibilities of transport, persistence, and checkpoint advancement.
Start with a readable contract
Before scaling the feed, ask another developer to interpret three representative records using only the documentation. Include a missing value, a correction, and a repeated event. Any disagreement identifies a contract gap worth fixing early. A strong data feeds API design makes meaning portable: identity stays stable, units remain explicit, provenance survives transformations, and change handling does not depend on undocumented guesswork.
