Scan

The Scan process reads the database schema and creates a list of tables/queries/views and their columns, in preparation for assigning converters.

When the Scan form opens, the tables and queries/views are enumerated and put in the list. The tables are selected and the queries are not.

When you click Scan, a deep review of the selected objects is performed. Alongside the primary key and each column's Min and Max values, the Scan records the table's row count, how many of a column's rows are empty or hold a default value, whether the column carries a unique index, its input mask, and whether it allows zero-length strings. These are not idle statistics. The Pattern converter's capacity check, for instance, subtracts the empty rows from the row count to work out how many values it actually has to invent, and refuses the column if the mask cannot produce that many distinct codes.

Part of the scanning process is checking the tables for single-column numeric or GUID primary keys. Only those tables can be anonymized. While it is possible in principle to anonymize a table with a composite or alphanumeric primary key, it would significantly complicate the anonymization process, and is not supported in this version of Anonymize. A reason of "Unprocessable: no primary key" will be emitted. Technically not correct. We opted for brevity, and mean that no suitable PK was found.

For date columns, the Scan process runs two more checks:

1. Sentinel values, which are frequency spikes in the data. For example, PriceValidUntil may have 1/1/2050 in many records, indicating 'forever'. We are using a statistically valid method called "One-sided binomial test of a proportion" which unifies the two requirements: a spike must be both Material, meaning it accounts for at least p₀ = 10% of the column's non-NULL values, and Statistically confident, meaning there are enough occurrences to be confident the value's true share genuinely exceeds 10%, rather than reaching 10% by small-sample coincidence.
2. Outlier values, which are extreme values on either side of the frequency distribution that deviate significantly from the rest of the data. We are using a statistically valid method called "Hampel fence". See the comments in frmScan.ComputeRobustDateBounds for details.

Both checks exist for the same reason: they decide the Min and Max arguments that the Random Date converter is given. Consider a column whose real dates sit between 2019 and 2024, but which also holds a 1/1/2050 "forever" sentinel and a single year-1900 typo. Taken at face value its range spans a century and a half, and the anonymized dates would be scattered across all of it — data that resembles the original in no useful way. Excluding the sentinels and fencing off the detached extremes leaves a range that reflects what people actually entered. A clean column, or one that is legitimately wide, keeps its true Min and Max: the fence only trims values that are both extreme and sparse.

The raw Min and Max are left as they are; the cleaned pair is stored separately, and only when something was in fact excluded. Sentinels earn a second privilege at Apply time — those rows are preserved unchanged rather than randomized, exactly as NULLs are, so a PriceValidUntil of 1/1/2050 still reads "forever" afterwards. And when the fence trims a genuine outlier you are notified, since that is a judgement you may want to overrule.

For Long Text columns, the Scan process checks the actual lengths of the values, and if all are in the Short Text range, the column is marked as such and it will be treated as Short Text for anonymization purposes. This is important because sometimes text columns are oversized. For example some .NET tools will use nvarchar(256) for many text columns, perhaps to store an EmailAddress. Access treats that as Long Text (which is not comparable), and only Long Text converters (such as Lorem) would be available. With Short Text, many more converters are available, including Email converter.

Queries are in the list as well, supporting the ability to anonymize a subset of the rows. As an example, think of a table storing patient blood analyses:

AnalysisID PK
PatientID FK
MeasurementID FK
DateSampled
MeasuredValue

Different measurements will have very different value ranges. If you create simple queries select * from table where MeasurementID = 1;, etc. then you can apply a different converter for each such query.

The Scan process looks for these very simple queries, and only those can be used to anonymize. Specifically, only queries with a single FROM keyword and no JOIN keyword and a numeric or GUID primary key are allowed. For SQL Server a very complicated query in procedure ImportSchemaSqlServerBE is used.

When the Scan finishes

Two more things happen as the Scan closes, and both are easy to miss because they happen on the way out.

First, column names are normalized to their canonical form — you will see "Normalizing columns..." on the progress dialog. This is what turns Phone, PhoneNumber, BusinessPhone and Telephone into the single canonical name phone number, which is the name Auto-Assign actually matches against. The steps are described under Normalization.

The consequence is worth spelling out: normalization runs here and nowhere else. When you add a record to Column Names, Abbrev Expansion or Filler Words, columns scanned earlier keep the canonical names they were given at the time. You must re-scan before the new configuration can reach them — re-running Auto-Assign on its own is not enough.

Second, if you have scanned the same database before under another record, Anonymize offers to copy the converter assignments across. Pick the source record, and every column whose table and column name match is given the same converter and arguments, marked as a manual assignment. For converters whose arguments are derived from the data — the date range of Random Date, the numeric range of Random Number — the bounds are recomputed from the current scan rather than copied, so anonymizing a fresher extract reflects its newer values instead of the source record's older ones.

A column you deliberately left unassigned is not carried over, and neither is any "No Converter" reason you typed — with one exception: a follower column's "Follower of ..." note comes across, because that relationship rides along in the lead column's copied arguments and the Coverage Report should still explain those columns.