Skip to main content

Overview: From Spreadsheet to Structured Table

The core function of the Data Page is to get your raw insurance data, typically arriving in spreadsheets like bordereaux, into your well-defined Decode Tables. This section details the entire workflow, from initiating an upload to reviewing the processed data and getting it into its final, structured form.

The Goal: Automated Cleaning and Mapping​

Manually cleaning spreadsheets and mapping their columns to a standard format is tedious and error-prone. Decode aims to automate as much of this as possible using AI, while still giving you the crucial final review step.

The High-Level Process​

  1. Upload: You select the target Table and provide the spreadsheet file(s), along with relevant Metadata Tags (often via Data Profiles).
  2. Extraction: Decode reads the spreadsheet, identifies individual sheets within it, and analyzes the structure (schema) of each sheet, including header information and data columns.
  3. Automated Mapping: Using Artificial Intelligence (AI) and a library of "possible values" it learns over time, Decode attempts to automatically match the columns found in your spreadsheet sheet to the columns defined in your target Table's schema. It also extracts header data if mappings are defined.
  4. Status Update: Decode assigns a processing status to each sheet based on the initial mapping attempt (e.g., Pending Review, Missing Row Identifier).
  5. User Review (Crucial Step): You are prompted to review Decode's work for each sheet. You'll see the extracted schema, the proposed mapping, and sample data. You can correct any incorrect mappings, define mappings for unmatched columns (teaching Decode for the future), and handle essential columns like the Row Identifier. You can also define formula-based mappings.
  6. Approval: Once you are satisfied with the mapping and preview, you approve the sheet.
  7. Ingestion: Upon approval, Decode processes the data according to the confirmed mapping and inserts or updates the relevant rows in your BigQuery Table, potentially adding new dynamic columns for new reporting periods.

Why the Review Step is Important​

While Decode's AI is powerful, spreadsheet formats vary wildly. The review step ensures:

  • Accuracy: You confirm that data is going into the correct columns in your Table.
  • Completeness: You handle any columns the AI couldn't automatically map.
  • Control: You have the final say before data enters your structured tables, maintaining data integrity.
  • Learning: Your corrections help Decode improve its automatic mapping for future uploads.

This combination of automation and user oversight provides both efficiency and accuracy.

Let's start by looking at The Upload Workflow in detail.