Ingest

You load a file by dropping it on the Ingest pane, and Totsum turns it into a table you can ask questions about. Totsum can work on a computer with no network connection: loading a file and describing it happen on your machine.

Loading files

Drop one or more files on the Ingest pane, where it says “Drop CSV / JSON / Excel / Parquet files or CSV / Parquet folders here”. Totsum reads these formats:

• CSV: the delimiter is detected, whether it is a comma, a semicolon, a tab or a pipe.
• Excel: .xlsx, .xls, .xlsm and .xlsb. A workbook with several sheets becomes one table per sheet. The import dialog, described below, lets you choose what each sheet contributes.
• JSON: a file whose records hold nested lists is split into a main table and linked child tables, one per nested list, so that each stays a flat table. Deleting the main table deletes its child tables with it.
• Parquet: .parquet and .pq.
• A folder of CSV files, or of Parquet files, dropped as one item.

While a file loads, the pane shows its progress. When it is done, the table appears in the list, and the AI Assistant can use it in the next question.

What happens on arrival

Totsum reads the column types and looks for relationships between the tables you loaded, such as a column in one table that matches the identifier column of another. When a new table has the same structure as one already loaded, the status bar says so, and the AI Assistant knows it can combine them when a question adds them together. Data-quality checks run on the new table in the background, and what they find is kept with the table for the AI Assistant.

Totsum also writes short notes of its own where a column holds a trap that its name does not show: a value such as -1 or -99 that probably stands for “missing”, a total that repeats on every row once two tables are joined, a number column whose name suggests a label but which holds identifiers. Most columns get no note, because their name and type already say enough. On a column that already has a description, the note is added after your text rather than in its place. A table gets an automatic note only when it has no description, for example when two tables share the same columns. The notes appear as ordinary text in the Description column of the Schema tab, and you can edit them like any other description.

Totsum also looks, by name, for documentation that came with the file you dropped: a file in the same folder whose name suggests documentation (a README, a data dictionary, a legend, notes, a codebook, definitions), a sheet of the workbook that you did not import and whose name suggests the same, or notes written above or below the table on a sheet you did import. When it finds one, the AI Assistant offers it: “Found DataDocumentation.txt — use it to describe your data?” with a Read it link. Only if you accept does Totsum open that file and read the descriptions from it. The reading runs in the background, pauses while a question of yours is being answered, and resumes after; when it has added descriptions, the assistant says how many. What it reads fills only the tables and columns that have no description yet.

The Excel import dialog

An Excel workbook rarely starts at cell A1 with clean headers. When you drop one, Totsum opens a dialog with the sheets on the left and a grid of the active sheet on the right. For each sheet it proposes a region: the header rows in one color, an optional description row in another, the data in a third, and the cells it will leave out in grey.

To correct the proposal, select cells in the grid and click a button under Mark as or Exclude:

• Header: use the selected row or rows as the column names.
• Description: treat the selected row as a description of the columns, not as data. Each cell of that row becomes the description of its column.
• Data: add the selected cells to the data that is imported.
• Rows and Columns: leave the selected rows or columns out of the import, which is how you drop totals, notes and blank separators.

Auto-detect proposes the region again from scratch and discards your marks. Untick a sheet in the list to leave it out of the import. Each sheet keeps its own region. The dialog imports the highlighted cells.

Viewing a table

Click a table in the Ingest pane and it opens in the center, with three tabs.

Overview opens first. It shows what stood out on a first look at the table, not everything that is in it.

Data shows the whole table, with every row, and sortable columns.

Schema shows one line per column: its name, how many cells are filled and how many are not, how many distinct values it holds, sample values, and the smallest and largest value. The last column is the Description: double-click a cell in it to write what the column means. Enter saves the text, and Shift+Enter starts a new line. The description of the table itself is in the Table Description field at the bottom of the tab. There is no Save button: a description is kept as soon as you finish typing it. Descriptions are read by the AI Assistant when it answers, and they are saved with the workspace.

Describing your data

The AI Assistant answers better when it knows what the columns mean. Three things give it that context.

Descriptions on the Schema tab, written by hand, as above.

Describe My Data from a Document, in the Edit menu, and also behind the Describe My Data… button that the AI Assistant shows when fewer than half of your tables, or fewer than half of your columns, have a description. You give it a document that lists your column names next to what they mean: a data dictionary, an export specification, a README, as a PDF, a Word document (.docx), a text file or a Markdown file. A scanned PDF, which holds pictures of the pages rather than text, cannot be read. The AI model reads the document on your machine, matches the names it finds to the tables and columns you loaded, and shows the result in a preview: one line per match, each with a checkbox, all ticked at first. A line on a yellow background, marked “[overwrites existing]”, replaces a description that is already there. You untick what is wrong, click Apply Selected, and the descriptions land on the Schema tab. One document can cover several tables. A second run can Update only new or incomplete tables (faster) or Re-extract everything.

Dataset Notes, in the Edit menu: free notes about the dataset as a whole, such as what the data covers, which period, which units, which rows to distrust. The AI Assistant reads them when it answers, and they are saved with the workspace. Double-click a note to edit it; right-click for Edit and Delete.

The blog post How to add descriptions to a dataset shows what a useful description says, with examples.

Refresh, rename, delete

Right-click a table in the Ingest pane, or click the ··· button that appears when you hover it:

• Refresh reads the file again from disk and rebuilds the table. Refresh All in the Edit menu, or F5, does this for every table and recomputes the results.
• Rename changes the table’s name; F2 renames in place.
• Delete removes the table. Before it does, a confirmation lists everything that goes with it: the JSON child tables, and every result built on it. Select several tables with Ctrl-click or Shift-click (Cmd-click on macOS) to delete them behind one confirmation.

Example datasets

File ▸ Try an Example Dataset… opens a picker of datasets that ship with the app, cleared for this use. The welcome page offers the same picker, and on the free trial the Ingest pane shows a Try an example button. Loading an example starts a fresh session and the AI Assistant shows a few ready-made starter questions. On the free trial, questions asked while only example data is loaded do not count against the trial quota; the licensing page gives the numbers.

Before the first file

The first time you load a file, Totsum shows the notice Before you load data: “Before working with sensitive data, your computer and its connections to the outside must be secured by a professional.” Continue loads the file; Continue and don’t ask again loads it and records that you have read the notice.