Data Release Process

Overview

The publisher source of the underlying data (e.g. government agencies, banks, give-to-get SEO applications, data aggregators) makes data available on a given publication date. There is typically a lag between the last day the source measures data for an upcoming release and the day the source publishes the data. Cybersyn consistently monitors the source and updates the data product on Snowflake Marketplace. To go deeper, see how this process works for public and propriety data below.

Public Data

For our public data sources, Cybersyn schedules the data updates based on a sensor. This sensor reads when new data is pushed to the source page (eg. FRED, BLS, Data Commons, StatCan, etc.).

Please note: Cybersyn’s data releases are dependent on our public sources own release schedules; therefore, if there is a delay by the public source release, Cybersyn data tables will also have a delay in updates.

Once this sensor goes off, Cybersyn automates the data formatting and data cleansing processing.

Upon job completion, the data is updated in our Snowflake instance then replicated (shared) to every cloud region our clients host their instance of Cybersyn Data Product tables.

Proprietary Data

Proprietary data sources send data to Cybersyn based on an established Service Level Agreement (SLA). Cybersyn automates the data formatting and data cleansing processing.

Upon job completion, the data is updated in our Snowflake instance then replicated (shared) to every cloud region our clients host their instance of Cybersyn Data Product tables.

Data Replication Downtime

A 10 minute downtime occurs when there is a schema migration or a new table is published. This is due to the time it takes to replicate the data across every cloud region that our clients host their instance of the Cybersyn Data Product tables.