HomeAccess Our Data NowDemos
Search
⌃K
Links
Comment on page

Changelog

​Subscribe to receive a bi-weekly summary of release notes in your inbox.

December 2023

12/5/23: Financial & Economic Essentials - Added interest rates posted by major chartered banks in Canada for selected products; Added secured overnight financing rates (SOFR) and US Treasury bill rates
Added interest rates for selected products posted by the six major chartered banks in Canada to the FINANCIAL_FRED_ATTRIBUTES, FINANCIAL_FRED_TIMESERIES, and FINANCIAL_FRED_VARIABLE_SERIES_ID_CROSSWALK tables. The typical rate is calculated based on the statistical mode of the rates published by the six banks.
The posted rates cover:
  • Prime lending rate
  • Conventional mortgages
  • Guaranteed investment certificates
  • Personal daily interest savings
  • Non-checkable savings deposits
Added secured over night financing rates (SOFR) and US Treasury bill rates to the FINANCIAL_FRED_ATTRIBUTES, FINANCIAL_FRED_TIMESERIES, and FINANCIAL_FRED_VARIABLE_SERIES_ID_CROSSWALK tables.
Series added (FRED ID):
  • Secured Overnight Financing Rate (SOFR)
  • 30-Day Average Secured Overnight Financing Rate (SOFR30DAYAVG)
  • 90-Day Average Secured Overnight Financing Rate (SOFR90DAYAVG)
  • 180-Day Average Secured Overnight Financing Rate (SOFR180DAYAVG)
  • Secured Overnight Financing Rate Index (SOFRINDEX)
  • 3-Month Treasury Bill Minus Federal Funds Rate (TB3SMFFM)
  • 6-Month Treasury Bill Minus Federal Funds Rate (TB6SMFFM)
12/5/23: Canada Government Essentials - Added interest rates posted by major charted banks in Canada for selected products
Added interest rates for selected products posted by the six major chartered banks in Canada to the CANADA_STATCAN_ATTRIBUTES and CANADA_STATCAN_TIMESERIES tables. The typical rate is calculated based on the statistical mode of the rates published by the six banks.
The posted rates cover:
  • Prime lending rate
  • Conventional mortgages
  • Guaranteed investment certificates
  • Personal daily interest savings
  • Non-checkable savings deposits
12/4/23: Financial & Economic Essentials - Added Monthly State Retail Sales (MSRS) from the US Census Bureau
Added monthly state-level retail sales by NAICS codes to the FINANCIAL_FRED_ATTRIBUTES, FINANCIAL_FRED_TIMESERIES, and FINANCIAL_FRED_VARIABLE_SERIES_ID_CROSSWALK tables. The US Census Bureau publishes year-over-year percent changes for each state using a composite model of Monthly Retail Trade Survey (MRTS) data, administrative data, and third-party data beginning in January, 2019.
Year-over-year percent change estimates for the following industries are now available at the state level:
  • Total Retail Sales Excluding Nonstore Retailers
  • Building Material and Garden Equipment and Supplies Dealers
  • Clothing and Clothing Accessories Stores
  • Electronics and Appliance Stores
  • Food and Beverage Stores
  • Furniture and Home Furnishings Stores
  • Gasoline Stations
  • Health and Personal Care Stores
  • Motor Vehicle and Parts Dealers
  • Sporting Good, Hobby, Musical Instrument and Book Stores
  • Miscellaneous Stores

November 2023

11/30/23: Government Essentials - Added global trade, tariff, and import relationship data from the World Trade Organization (WTO)
Added global trade flows, imposed tariffs, and trade interactions between countries from the World Trade Organization (WTO) . The data details export and import figures for goods and services across different countries and regions, tariff rates and structures that WTO member countries apply to imports from other nations, trade dependencies between countries, and the balance of trade between specific pairs of nations.
  • The WORLD_TRADE_ORGANIZATION_ATTRIBUTES table details the global trade, tariff, and import relationship statistics tracked by the World Trade Organization (WTO).
  • The WORLD_TRADE_ORGANIZATION_TIMESERIES table provides timeseries values by date for the reported trade statistics by country, country group, and global region (as defined by the World Trade Organization).
11/30/23: Government Essentials - Added global health indicators from the World Health Organization (WHO)
Added 1,100+ health-related indicators for 194 members of the World Health Organization (WHO) and their associated country groups and global regions. Example metrics include alcohol consumption among adolescents and adults, tobacco control policies, abortion rates, accessibility of dementia care services, and adolescent fertility rates. Environmental health indicators including air pollution's impact on mortality rates and disability-adjusted life years (DALYs) as well as deaths attributable to the environment are also included.
  • The WORLD_HEALTH_ORGANIZATION_ATTRIBUTES table details the health statistics tracked by the World Health Organization (WHO).
  • The WORLD_HEALTH_ORGANIZATION_TIMESERIES table provides timeseries values by date for the reported health indicators by country, country group, or global region (as defined by global organizations like UNICEF, the United Nations, the World Bank, and the World Health Organization).
11/30/23: Government Essentials, Financial & Economic Essentials - Added country groups and regions from the WHO, WTO, and UN to geography tables
Added additional country groups and geography types to the GEOGRAPHY_INDEX from the World Health Organization (WHO), World Trade Organization (WTO), and United Nations (UN). The member countries of the added geographies are mapped in the GEOGRAPHY_RELATIONSHIPS table. Select new geographic regions include:
  • BRICS members
  • World Trade Organization (WTO) members
  • Association of Southeast Asian Nations (ASEAN)
  • UNICEF regions
  • United Nations regions
  • United Nations Sustainable Development Goal (SDG) regions
  • World Bank regions
  • World Health Organization (WHO) regions and income regions
  • World Bank regions and income groups
11/28/23: Government Essentials - Added US agricultural export sales data from the USDA
Added US Export Sales Reporting (ESR) data on weekly export sales activity for 40+ US agricultural commodities sold abroad from the US Department of Agriculture's (USDA) Foreign Agricultural Service (FAS).
  • The US_DEPARTMENT_OF_AGRICULTURE_COMMODITIES_ATTRIBUTES now includes the export of commodities in addition to the existing production, supply, and distribution variables.
  • The US_DEPARTMENT_OF_AGRICULTURE_COMMODITIES_TIMESERIES table provides the reported metrics for each commodity by GEO_ID.
11/22/23: Financial & Economic Essentials - Added data from the Bank for International Settlements (BIS) on global banking conditions, property prices, and consumer price indicators
Added data from the Bank for International Settlements (BIS) on global banking conditions, property prices, and consumer price indicators
The Bank for International Settlements (BIS) is an international financial institution owned by 60+ central banks that represent countries accounting for ~95% of global GDP. As part of its mission is to support international monetary and financial cooperation, the BIS acts as a bank for central banks across the world.
Added the below BIS data to two new tables, BANK_FOR_INTERNATIONAL_SETTLEMENTS_ATTRIBUTES, which describes the metrics tracked by BIS, and BANK_FOR_INTERNATIONAL_SETTLEMENTS_TIMESERIES, which provides the values of metrics:
  • Residential property prices
  • Central bank policy rates
  • Consumer price indicators
  • Assets and liabilities of internationally active banks, including credit to the non-financial sector as well as the geographical and currency composition of a bank’s balance sheet
11/22/23: Consumer Spending Foundation - Added spend by geography for states and top 100 CBSAs. Added online/offline spend breakdown
Expanded the granularity of values in the CONSUMER_SPENDING_TIMESERIES table to include breakdown by geography. Geographies are grouped both by purchaser primary location (i.e. where the spender lives) and purchase location (i.e. where the physical store is located). The data covers US states and the top 100 core-based statistical areas (“CBSAs”) as measured by population.
  • Purchaser geographies are represented in the PURCHASER_PRIMARY_GEO_ID and PURCHASER_PRIMARY_GEO_NAME fields
  • Purchase location geographies are represented in PURCHASE_LOCATION_GEO_ID and PURCHASE_LOCATION_GEO_NAME fields
  • PURCHASER_PRIMARY_GEO_ID and PURCHASER_PRIMARY_GEO_ID can be used to join the data with Cybersyn’s geography tables such as the GEOGRAPHY_INDEX based on the GEO_ID field
Added purchase channels covering online and offline spend.
  • Purchase channels are reflected in the CHANNEL field in the CONSUMER_SPENDING_ATTRIBUTES table and included in the VARIABLE_NAME values in the CONSUMER_SPENDING_TIMESERIES table
Note that data for “purchase location” geographies only includes offline spend.
11/20/23: Consumer Spending Foundation - Added aggregations by NAICS codes and by 4-5-4 retail months
Expanded the CONSUMER_SPENDING_TIMESERIES and the CONSUMER_SPENDING_ATTRIBUTES tables to include aggregations by NAICS (North American Industry Classification System) and 4-5-4 retail calendar months.
Deprecated MCC_CODE, MCC_CODE_DESCRIPTION and MART_VARIABLE fields in the the CONSUMER_SPENDING_ATTRIBUTES table and replaced them with AGGREGATION_TYPE and AGGREGATION_VALUE.The newly added fields can be used to filter to a desired level of aggregation including to the NAICS, MARTS Segment, and MCC levels.
Additionally, the following fields were added in anticipation of upcoming dataset expansions. Note that these fields only contain total values for now. Future iterations of the product will include more granular cuts of data.
  • Added COMPANY_NAME, PURCHASER_PRIMARY_GEO_ID, PURCHASER_PRIMARY_GEO_NAME, PURCHASE_LOCATION_GEO_ID, and PURCHASE_LOCATION_GEO_NAME to CONSUMER_SPENDING_TIMESERIES
  • Added CHANNEL to CONSUMER_SPENDING_ATTRIBUTES
11/20/23: Financial & Economic Essentials - Added 10 timeseries from FRED covering mortgage rates and additional CPI measures
Added 10 additional timeseries from FRED to the financial_fred_timeseries and financial_fred_attributes tables:
  • MORTGAGE15US: 15-Year Fixed Rate Mortgage Average in the United States
  • MORTGAGE30US: 30-Year Fixed Rate Mortgage Average in the United States
  • CUSR0000SEFV: Consumer Price Index for All Urban Consumers: Food Away from Home in U.S. City Average, Seasonally Adjusted
  • CUUR0000SEFV: Consumer Price Index for All Urban Consumers: Food Away from Home in U.S. City Average, Not Seasonally Adjusted
  • CUSR0000SETG01: Consumer Price Index for All Urban Consumers: Airline Fares in U.S. City Average, Seasonally Adjusted
  • CUUR0000SETG01: Consumer Price Index for All Urban Consumers: Airline Fares in U.S. City Average, Not Seasonally Adjusted
  • CUSR0000SS62031: Consumer Price Index for All Urban Consumers: Admission to Movies, Theaters, and Concerts in U.S. City Average, Seasonally Adjusted
  • CUUR0000SS62031: Consumer Price Index for All Urban Consumers: Admission to Movies, Theaters, and Concerts in U.S. City Average, Not Seasonally Adjusted
  • CUSR0000SEHB: Consumer Price Index for All Urban Consumers: Lodging Away from Home in U.S. City Average, Seasonally Adjusted
  • CUUR0000SEHB: Consumer Price Index for All Urban Consumers: Lodging Away from Home in U.S. City Average, Not Seasonally Adjusted
11/20/23: Government Essentials - Expanded American Community Survey (ACS) history for 1,400+ population variables since 2005 for ~500K geographies
Added historical data from the American Community Survey (ACS) to the AMERICAN_COMMUNITY_SURVEY_ATTRIBUTES and AMERICAN_COMMUNITY_SURVEY_TIMESERIES tables for over 1,400 population variables dating back to 2005 at the following geographic entity levels: country, states, counties, cities, zip codes, core-based statistical areas (CBSAs), census tracts, and census block groups. Example population variable additions include age, race, income, employment status, immigration status, and household status.
Data is as up to date as the latest ACS publication.
11/16/23: Weather & Environmental Essentials - Added 6 tables covering disaster declaration and National Flood Insurance Program (NFIP) insurance data from FEMA
Added Federal Emergency Management Agency (FEMA) data on federally-declared disasters in the United States, disaster recovery public programs, and the National Flood Insurance Program (NFIP) insurance policies and claims. Six new tables were included in the release:
  • FEMA_DISASTER_DECLARATION_INDEX - Details for each federally-declared disaster (e.g. name, type, date, public assistance funding amounts). Table is unique by DISASTER_ID.
  • FEMA_DISASTER_DECLARATION_AREAS_INDEX - Geographic entities (e.g. counties, cities) impacted by each federally-declared disaster. A disaster declaration can include multiple geographic locations.
  • FEMA_MISSION_ASSIGNMENT_INDEX - Work orders issued by FEMA since 2012 to other government agencies, supporting emergency response activation across the US (e.g. requesting transportation support from the US Department of Transportation during a hurricane).
  • FEMA_NATIONAL_FLOOD_INSURANCE_PROGRAM_CLAIM_INDEX - Details on National Flood Insurance Program (NFIP) claims including features of the insured property, information on the flood event precipitating the claim, the cost of the damage, and subsequent insurance payout amounts.
  • FEMA_NATIONAL_FLOOD_INSURANCE_PROGRAM_POLICY_INDEX - Details on National Flood Insurance Program (NFIP) policies including features of the insured property, building and contents insurance coverage, deductibles, rates, and policy durations.
  • FEMA_REGION_INDEX - Human readable names and details for each FEMA Region
11/13/23: Canada Government Essentials - Added 6 timeseries from Statistics Canada covering Canadian debt securities and real estate development
Added 6 new timeseries to the CANADA_STATCAN_TIMESERIES and CANADA_STATCAN_ATTRIBUTES tables from 3 of Statistics Canada’s underlying sources:
  • Bank of Canada: Government of Canada debt gross new issues, retirements and net new issues, and par values
  • Annual Survey of Service Industries: Real estate agents, brokers, and appraisers summary statistics (e.g. salaries, wages, operating revenue)
  • Canada Mortgage and Housing Corporation:
    • Absorptions and unabsorbed inventory, newly completed dwellings, by type of dwelling unit in select census metropolitan areas
    • Absorptions and unabsorbed inventory, newly completed dwellings, by type of dwelling unit in census agglomerations of 50K+
    • Housing starts, under construction and completions in select census metropolitan areas
    • Housing starts, under construction and completions in census agglomerations of 50K+
11/7/23: Cybersyn Public Domain Pro - Added point-in-time history for 45 tables
Added point-in-time history tables for the following Cybersyn datasets.
  • American Community Survey
  • Canada Statcan
  • Carbon Credit Purchases
  • Carbon Intensity
  • Company Index and Characteristics
  • Data Commons
  • Domain Index
  • FHFA
  • Geography Characteristics
  • Home Mortgage Disclosures
  • IMEI
  • IRS
  • NOAA Weather Stations and Metrics
  • OpenFIGI Security Index
  • PermID Security Index
  • Points of Interest (POIs)
  • Urban Crime
  • US Addresses
  • USDA
  • US Treasury
  • USPS Address Changes
11/6/23: Consumer Spending Foundation - Added nominal value projections (dollar-level figures)
Expanded values in the CONSUMER_SPENDING_TIMESERIES table to include nominal estimates. New measures include estimates for revenue ($), transactions (#), and average order value ($). These newly added variables build on the existing year-over-year (%) estimates for those measures already in the table.
Nominal values can be used to measure market share or compare average transaction amounts across retailers. Because these estimates are based on a panel of consumer spending, they are not meant to be projections of the entirety of US spending but they are accurate as a measure of relative spend. For example, the sum of all Chipotle spend will not equal the company's actual spend but Chipotle's market share relative to McDonald's should be correct.
The newly-added variables in the timeseries table include matching variables in the CONSUMER_SPENDING_ATTRIBUTES table with the new variables being MEASURE values of Revenue, Transactions, and AOV.
11/3/23: Consumer Spending Foundation, Government Essentials - Added CALENDAR_INDEX table
Added the CALENDAR_INDEX table which compiles common calendars into a single table. Each calendar type has a unique CALENDAR_ID, which allows users to select which calendar type they want to use. Individual periods within each calendar type include period start and end dates.
The CALENDAR_INDEX currently includes regular calendar periods (days, weeks, months, quarters, and years) and 4-5-4 retail calendar periods (4-5-4 retail months, quarters, and years).
The 4-5-4 retail calendar is a standardized accounting and reporting calendar system used by many retailers, where each fiscal year is divided into 13 weeks, aiming to align with seasonal variations and facilitate more accurate financial comparisons.
11/3/23: Government Essentials - Added global agricultural commodity production and distribution data from the USDA
Added two tables sourced from the US Department of Agriculture's (USDA) Foreign Agricultural Service (FAS) which provides production, supply, and distribution data on agricultural commodities for both the United States and other producing and consuming countries since 1960.
  • us_department_of_agriculture_commodities_attributes describes the production, supply, and distribution metrics tracked for each commodity by the USDA.
  • us_department_of_agriculture_commodities_timeseries table provides the reported metrics for each commodity and country.

October 2023

10/25/23 - Financial & Economic Essentials - Added detailed branch-level data and Summary of Deposits (SOD) data from the FDIC
The Summary of Deposits (SOD) data from the FDIC is an annual survey, capturing branch-level deposits as of June 30 for all FDIC-insured institutions, including U.S. branches of foreign banks.
The fdic_branch_locations_indextable provides details on FDIC-insured bank branches, including branch-specific location information as well as institution-level regulatory and insurance data.
The fdic_summary_of_deposits_attributes table describes the deposit types tracked by the Summary of Deposits (SOD) survey.
The fdic_summary_of_deposits_timeseries table provides the results of the annual Summary of Deposits (SOD) survey going back to 1994 for banks’ branch-level deposits.
10/19/2023: Government Essentials - Added US Federal Government Revenue Collections from the US Treasury Fiscal Data
The US Treasury provides a daily overview of net federal revenue collections from income tax deposits, customs duties, fees for government services, fines, and loan repayments. These collections undergo electronic and/or non-electronic processing, involving various channels such as mail, internet, banking, and over-the-counter transactions, all of which are comprehensively incorporated within this dataset.
The us_treasury_revenue_collections_timeseries table provides daily net collections amounts broken down by tax category and processing channel. The us_treasury_revenue_collections_attributes table details each collection method reported by the US Treasury.
10/18/23: US Insurance & Healthcare Provider Foundation - Added Form 5500 Schedule A Part 1 insurance data from the US Department of Labor.
Expanded the US Department of Labor data to include information found on Form 5500 Schedule A Part 1. The new table, us_department_of_labor_form_5500_broker_index, provides commission and fee amounts received by a broker for an insurance policy. Additional information about the brokers in the table includes their address, classification as an insurance broker, as well as notes pertaining to the compensation disbursed to them.
The us_department_of_labor_form_5500_broker_index can be joined to insurance carrier and policy information to individual Form 5500 filings, using INSURANCE_POLICY_ID and ACK_ID.
10/16/23: Financial & Economic Essentials - Added exchange rates from the Bank for International Settlements (BIS)
Due to the discontinuation of certain currency conversion pairs by the European Central Bank (ECB), our primary source for daily FX rates, we have sourced a number of these pairs from an alternative source, the Bank for International Settlements (BIS), for ongoing history. The Bank for International Settlements will be used to get data for the following currency conversion pairs after September 27, 2023: USD:AED, USD:ARS, USD:CLP, USD:COP, USD:DZD, USD:MAD, USD:PEN, USD:QAR, USD:SAR, USD:TWD, USD:UAH.
Full history from the Bank for International Settlements was added for the following currency pairs: USD:ALL, USD:AUD, USD:BAM, USD:BHD, USD:BND, USD:EUR, USD:GBP, USD:IRR, USD:ISK, USD:KWD, USD:KZT, USD:LKR, USD:MKD, USD:MUR, USD:NPR, USD:NZD, USD:OMR, USD:RSD, USD:RUB, USD:TND, USD:TTD, USD:UYU, USD:VEF, USD:XDR.
10/11/23: Government Essentials - Added population variables to the American Community Survey tables.
Expanded the american_community_survey_attributes and american_community_survey_timeseries tables to include additional population variables related to income, age, and educational attainment.
New series include Household Income in the Past 12 Months (Inflation-Adjusted), Educational Attainment for the Population 25 Years and Over, and Age of Householder By Household Income in the Past 12 Months (Inflation-Adjusted). These series are available by multiple breakdowns (ex. income, age, gender, etc.).
10/11/23: Consumer Spending Foundation - Added company-level estimates; Added COMPANY_INDEX table
Added year-over-year estimates for company-level spend, transaction, and AOV to the CONSUMER_SPENDING_TIMESERIES. Company-level information is identified with the company_id field.
The company_id is a unique identifier assigned by Cybersyn to each company and is joinable to the COMPANY_INDEX, which provides company names and other helpful identifiers such as CIK, LEI, PermID, and more. Note that when the company_id is null, then the row represents data for all companies.
10/09/23: US Housing & Real Estate Essentials - Added U.S. Census Regions & Divisions
Expanded the geography_index table to include U.S. Census Regions and Divisions.
Expanded the geography_hierarchy table to include the relationships between U.S. Census regions and U.S. Census divisions; U.S. Census regions and U.S. states; and U.S. Census divisions and U.S. states.
Census regions include the United States Northeast, Midwest, etc. and census divisions include the United States Middle Atlantic, East North Central, etc.
10/8/23: SEC Filings - Added PERMID_SECURITY_INDEX table to expand security coverage
  • Added one new table to our dataset, PERMID_SECURITY_INDEX, which includes security identifiers from Refinitiv’s PermID database. These are persistent identifiers for active and inactive securities across global asset classes. The table includes over 15K PermIDs for various securities.
  • This data can be merged to the 13F filings data (SEC_HOLDING_FILING_ATTRIBUTES) and can be mapped back to companies using the COMPANY_SECURITY_RELATIONSHIPS table.
10/06/23: Tech & Innovation Essentials - Added in repository of web domains plus included GitHub Archive & US Patents tables
Cleaned and aggregated over 300M domains in a single source to track the list of websites globally into new domain_index table.
Added GitHub Archive and US Patents Grants tables to the product, rebranded product from "IMEI Type Allocation Codes" to "Tech & Innovation Essentials."
10/2/23: Government Essentials - Added population data from the American Community Survey
Expanded our population dataset to include annual estimates from the American Community Survey (ACS) for 2021 and 2022 at multiple geographic levels in the United States.

September 2023

9/18/23: Financial & Economic Essentials - Added 6 timeseries from FRED covering core CPI and industry and commodity-specific PPI data
Added 6 new timeseries from FRED to the financial_fred_timeseries and financial_fred_attributes tables:
  • PPIFIS: Producer Price Index by Commodity: Final Demand
  • PCU4841214841212: Producer Price Index by Industry: General Freight Trucking, Long-Distance Truckload
  • PCU4841224841221: Producer Price Index by Industry: General Freight Trucking, Long-Distance Less Than Truckload
  • PCU3313133131: Producer Price Index by Industry: Alumina and Aluminum Production and Processing
  • WPU101707: Producer Price Index by Commodity: Metals & Metal Products: Cold Rolled Steel Sheet and Strip
  • CPILFESL: (CORE CPI) Consumer Price Index for All Urban Consumers: All Items Less Food & Energy in U.S. City Average
9/17/23: SEC Filings, LLM Training Essentials - Added COMPANY_INDEX, COMPANY_CHARACTERISTICS, and COMPANY_SECURITY_RELATIONSHIP tables; added PermIds
Added three new tables:
  • The COMPANY_INDEX table aggregates commonly used company identifiers (i.e. CIKs, EINs, and LEIs) into a single a single company_id, which can be used across Cybersyn’s datasets as a unique identifier for corporate entities.
  • The COMPANY_SECURITY_RELATIONSHIP table maps OpenFIGI and PermID securities (i.e. securities with multiple "levels" such as OpenFIGI FIGI ID and OpenFIGI Share Class ID) to the Company.
  • TheCOMPANY_CHARACTERISTICS table includes categorical characteristics of a Company (e.g. industry, address, previous names). A characteristic may be temporal with start and end dates indicating the range for which the data is valid.
Added PermId securities published by Refinitiv.
9/15/23: Weather & Environmental Essentials - Added daily weather data from 80K+ weather stations across 180 countries; updated product name
New tables added:
  • NOAA_WEATHER_STATION_INDEX contains metadata on the weather stations from the Global Historical Climatology Network daily (GHCNd) database, including mappings to Cybersyn’s GEO_ID at the country-, state- and zip-level (where applicable).
  • NOAA_WEATHER_METRICS_ATTRIBUTES includes the daily weather variables tracked globally and their measurement details.
  • NOAA_WEATHER_METRICS_TIMESERIES provides the details of the daily global weather variables recorded at each weather station.
Updated product name from "Emissions & Environment Essentials" to "Weather & Environmental Essentials"
9/15/23: US Insurance & Healthcare Provider Foundation - Added healthcare provider emails; combined TELEPHONE and TELEPHONE_EXTENSION into one field; changed TELEPHONE to array to accommodate numerous values
  • Added EMAIL field to the nppes_provider_addresses table with provider emails per address.
  • Combined TELEPHONE and TELEPHONE_EXTENSION from the nppes_provider_addresses table into a single field, TELEPHONE, and removed the TELEPHONE_EXTENSION field.
  • Aggregated all values for TELEPHONE, FAX, and EMAIL that are associated with the same NPI and address into arrays in one row. Rows in the nppes_provider_addresses table are now uniquely defined by NPI and full address.
9/15/23: US Points of Interest & Addresses, Government Essentials - Added FIPS 10-4 country codes and state abbreviations
Expanded the geography_characteristics table to include mappings of FIPS 10-4 country codes and U.S. state abbreviations to country and state-level GEO_IDs, respectively.
9/10/23: US Insurance & Healthcare Provider Foundation - Added table to relate taxonomy classifications to practitioners’ license numbers
Added the NPPES_PROVIDER_TAXONOMY_AND_LICENSE_NUMBERS table that relates taxonomy classifications to practitioners’ license numbers. This table provides users the ability to filter for license numbers based on practitioners’ primary taxonomy.
9/7/23: SEC Filings - Added 13F filings to include data on quarterly investment fund managers’ holdings; added a securities index table based on OpenFIGI data
Expanded our dataset to include individual filings from 13F fund holding reports, which disclose the equity holdings of institutional investment managers. Added three new tables, SEC_HOLDING_FILING_INDEX, SEC_HOLDING_FILING_ATTRIBUTES and OPENFIGI_SECURITY_INDEX.
SEC_HOLDING_FILING_INDEX table contains metadata from individual 13F filings including filing date and filing organization. SEC_HOLDING_FILING_ATTRIBUTES table includes securities' names, market value, number of shares held, and OpenFIGI IDs, which facilitate easier mapping and analysis to outside data sources. Table OPENFIGI_SECURITY_INDEX contains an index of over 2M securities listed on OpenFIGI and can be joined with table SEC_HOLDING_FILING_ATTRIBUTES using TOP_LEVEL_OPENFIGI_ID - the unique identifier for each security in the two tables.
9/6/23: US Insurance & Healthcare Provider Foundation - Added contact information and broker payments to Form 5500 data
  • Added new fields to us_department_of_labor_form_5500_filing_index with Form 5500 contact information including name and phone numbers: ADMIN_SIGNED_NAME, SPONSOR_SIGNED_NAME, DIRECT_FILING_ENTITY_SIGNED_NAME, ADMIN_PHONE_NUM and SPONSOR_DIRECT_FILING_ENTITY_PHONE_NUM.
  • Added new fields to us_department_of_labor_form_5500_policy_index with payments to agents and brokers: COMMISSIONS_PAID_TO_BROKER and FEES_PAID_TO_BROKER.
  • Removed ~10k rows from us_department_of_labor_form_5500_policy_index that had NULL values for each field as they filed no data around insurance policies from Form 5500 Schedule A.

August 2023

8/31/23: US Insurance & Healthcare Provider Foundation - Added deactivated NPI numbers to NPPES data
Added a new table, nppes_npi_index, that contains information on when NPIs were first issued, deactivated, or reactivated - dating back to 2005. This table also includes a boolean flag to indicate if an NPI is currently active.
Note that while all NPIs appear in the nppes_npi_index table, only actively registered NPIs as well as NPIs deactivated after August 1, 2023 appear in the nppes_practitioner_attributes and nppes_organization_attributes tables. This means the dataset does not include attribute-level data (names, type of providers, specialization) on providers with NPIs deactivated before August 2023.
8/27/23: US Points of Interest & Addresses, US Housing & Real Estate Essentials - Added points of interest data from Overture Maps Foundation
Added the point_of_interest_index table, which includes names and categories for points of interest in the US. Each POI is uniquely identified by a POI_ID.
To tie POIs to addresses, we added a new column, ADDRESS_ID, to the us_addresses table to uniquely identify each individual address. This column allows users to join addresses to POIs using the new point_of_interest_addresses_relationships table with POI_ID and ADDRESS_ID as the join keys for the point_of_interest_index table and us_addresses table, respectively.
8/27/23: US Points of Interest & Addresses, US Housing & Real Estate Essentials - Added 7.2M new addresses, removed 49.8M duplicate addresses, deleted 1.2M addresses with Null STREET value
Added 7.2M new addresses covering points of interest from Overture Maps Foundation to the us_addresses table.
Removed 49.8M addresses that were duplicative aside from minor variability in coordinates. Removed 1.2M rows from rows from the us_addresses table where the STREET value contained a string with value Null.
8/27/23: US Points of Interest & Addresses, US Housing & Real Estate Essentials - Added country-level geospatial boundaries to the geography_characteristics table
Added country-level geospatial boundaries to the geography_characteristics table with data from Overture Maps Foundation.
8/13/23: SEC Filings - Added 8-K filings and exhibits for 10-Qs and 10-Ks
Expanded our coverage of SEC documents to include the full text of 8-K filings and associated exhibits. 8-K filings include company press releases, earnings releases, and other major corporate events.
Added the full text of exhibits for 10-K and 10-Q filings. Exhibit types include lists of subsidiaries, merger agreements, and material changes in financial conditions. Exhibits are denoted in the variable and variable_name columns (e.g. 10-K EX-21 Filing Text).
Added the sec_document_id column. This field is a combination of the ADSH (accession number) and the document type (e.g. 10-K). This serves as a unique identifier for each individual component that makes up a filing in cases when one or more exhibits are included in a filing.
8/11/23: Government Essentials, US Housing & Real Estate Essentials, US Addresses & Geographic Areas - Added geospatial boundaries data for territories in the US and Canada
The Census Bureau and Statistics Canada publish geospatial boundaries data for their territories at multiple geographic levels. We added a table geography_characteristics with the boundary coordinates from the most recent releases in both WKT and GeoJSON formats. The table is joinable at different levels using Cybersyn's GEO_ID. This GEO_ID is compatible with all Cybersyn listings that have geographic identifiers. Currently, the geographic levels covered include:
  • State (US and Canada)
  • County (US only)
  • Census Tract (US only)
  • ZIP Code (US only)
  • Dissemination Area and Aggregate Dissemination Area (Canada only)
  • Census Division and Census Subdivision (Canada only)
  • Census Agglomeration and Census Agglomeration Part (Canada only)
  • Census Metropolitan Division and Census Metropolitan Division Part (Canada only)
8/10/23: Financial & Economic Essentials - Added crosswalk to FRED series IDs & 107 new series from GDP, Employment Situation, Housing Starts, and Residential Construction reports
Added a new table, financial_fred_variable_series_id_crosswalk, that enables a join between Cybersyn’s variable and FRED’s unique series ID
Added new series from the following four reports:
  • Gross Domestic Product (data produced by the US Bureau of Economic Analysis)
  • Employment Situation (US Bureau of Labor Statistics)
  • New Residential Construction (US Department of Housing and Urban Development)
  • Quarterly Starts and Completions by Purpose and Design (US Department of Housing and Urban Development)
8/7/23: Government Essentials - Added text-based US government contracts data from SAM.gov
The US government publishes contract opportunities and proposals to do business with the federal government via the System for Award Management (sam.gov) for contracts and awards with a value of at least $25,000. The data goes back to January 2002 and includes metadata providing descriptions of government contracts and the corresponding awards granted for those contracts.

July 2023

7/31/23: Financial & Economic Essentials - Added release_name and release_source for better discoverability
  • release_name: The collection, group of data, or report from which a time series originates. This column can be used as a filter to find related series.
  • release_source: The organization (e.g. FDIC, Federal Reserve) that FRED collects the data from.
Two columns were added to financial_fred_attributes to provide better categorization and discoverability:
7/28/23: Canada Government Essentials - Removed the age_group measure and labour_force_statisticcolumns
Legacy columns from the July 16, 2023 release were removed.
  • age_group
  • measure
  • labour_force_statistic
7/25/23: Financial & Economic Essentials - Added 124 time series from FRED covering a variety of economic data
124 additional time series added from FRED to the financial_fred_timeseries and financial_fred_attributes.
7/16/23: Canada Government Essentials - Added prices, output, and household income; schema changes
  • New datasets:
    • Household income and finances: Household income and consumption, household credit liabilities, and household savings rates
    • Prices and output: Core consumer price index, gross domestic product by industry group, and new housing price indices
  • Updated datasets:
    • StatCan archived a number of retail trade series and published new series. Old series are marked “Archived…” in the report field. The latest series have been included to replace these.
  • Schema changes:
    • The following columns in the canada_statcan_attributes view were updated. The deprecated columns will be removed on 7/28/2023.
      • age_group will be folded into demographic_group, which applies more broadly to age ranges, income groups, and household makeups (e.g., 35 to 44 years, lowest income quintile, elderly persons not in an economic family)
      • measure will be renamed report. The report column displays the StatCan dataset from which the data originates (e.g. Consumer Price Index, New Housing Price Index)
      • labour_force_statistic will be renamed statistic. The new statistic column will provide the label for the specific economic metric that is being reported (e.g. Median After Tax Income, Number of families)
7/3/23: US Insurance & Healthcare Provider Foundation - Added Form 5500 insurer information
Added information from US Department of Labor Form 5500 filings about company benefit providers and the insurance/benefit plans they offer. New tables include us_department_of_labor_form_5500_filing_index and us_department_of_labor_form_5500_policy_index.

June 2023

6/14/23: SEC Filings - Added full text 10-Qs and 10-Ks
Added the full text of 10-K/Q filings. These are contained in the sec_report_text_attributes table.
6/13/23: Financial & Economic Essentials - Added Bureau of Labor Statistics datasets
Added Consumer Price Index (CPI), Average Prices (AP), Job Openings and Labor Turnover Survey (JOLTS), State and Metro Area Employment , Hours, & Earnings (SAE), Local Area Unemployment Statistics (LAUS) from the Bureau of Labor Statistics.
6/1/23: Government Essentials - Added 3,000 new US zip codes from USPS and US Census
  • Using USPS address change data, we added 3,000 zip codes (mostly PO Box) to the dc_geo_index.
  • Using both the USPS address change and US Census Bureau data, we increased the coverage in geography_relationships table with 6,500 new zip and city relationships. We now map 86% of zip codes to a city.

May 2023

5/26/23: Financial & Economic Essentials - Added 5 global central bank policy rates
Central bank policy rates from Brazil, Canada, England, Mexico, and Japan added to fred_timeseries and fred_attributes
5/19/23: Government Essentials - Updated product name from Cybersyn Data Commons to Cybersyn Government Essentials
Rebranded Cybersyn Data Commons as Cybersyn Government Essentials. We updated the naming conventions for schemas, tables, and column names to make them consistent across all of Cybersyn’s existing and future data products.
Cybersyn will continue to support and update your older version of the Data Commons tables.
5/19/23: US Addresses & Geographic Areas, GitHub Events - Added source data from National Address Database (NAD)
Added the National Address Database (NAD) as a source to increase our US address coverage:
  • Increased the coverage from 140 million addresses to more than 188 million.
  • There is now at least one address in more than 85% of zip codes, up from 74% previously.
  • Increased the portion of cities that are mapped to distinct IDs joinable to our other data sets from 24% to over 77%
​
​
Last modified 3d ago