Operations · 5 min read

BulkMerge

This method merges all rows from the client application into the database in bulk — inserting new rows and updating existing ones based on the defined qualifiers. It is supported for DuckDB.

Call Flow Diagram

The diagram below shows the flow when calling this operation.

flowchart TD
    Client["Client<br/>(RepoDB)"] -->|BulkMerge| Source["Entities /<br/>DataTable /<br/>DbDataReader"]
    Source --> Pseudo["Create Pseudo Table<br/>(TEMP by default)"]
    Pseudo --> Appender["DuckDbBulkAppender"]
    Appender -->|Write| PseudoTable[("Pseudo Table")]
    PseudoTable --> Decision{"identityBehavior ==<br/>ReturnIdentity?"}
    Decision -->|NO| Merge["MERGE INTO ... USING ...<br/>WHEN MATCHED THEN UPDATE<br/>WHEN NOT MATCHED THEN INSERT"]
    Merge --> Table[("Target Table")]
    Decision -->|YES| Snapshot["LEFT JOIN snapshot<br/>ORDER BY rowid:<br/>classify matched vs. unmatched rows"]
    Snapshot --> UpdateStep["UPDATE ... FROM ...<br/>(matched rows only)"]
    Snapshot --> InsertStep["INSERT ... WHERE NOT EXISTS ...<br/>RETURNING identity<br/>(unmatched rows only)"]
    UpdateStep --> Table
    InsertStep --> Table
    InsertStep -->|"assign identities<br/>back to entities"| Client
    PseudoTable -->|Drop| Cleanup(["Pseudo Table Dropped"])

Use Case

Use this method to merge rows at high speed. It leverages DuckDB.NET’s native DuckDBAppender via the DuckDbBulkAppender wrapper.

For merging 1,000 or more rows, prefer this method over MergeAll.

A pseudo (staging) table is always created first. The library writes to it via DuckDbBulkAppender internally, then cascades the changes to the target table via a MERGE statement — see Operations (DuckDB) for the underlying mechanics.

Special Arguments

The qualifiers, mappings, bulkCopyTimeout, batchSize, identityBehavior and pseudoTableType arguments are available for this operation.

qualifiers defines the fields used to match existing rows, corresponding to the ON clause. Defaults to the primary (or identity) column if not specified.

mappings (via DuckDbBulkInsertMapItem) defines explicit column mappings between the source properties and the destination columns. When omitted, columns are auto-mapped by name (case-insensitive).

bulkCopyTimeout overrides the command timeout, in seconds.

batchSize overrides the number of rows sent to the server per batch. When not set, all items are sent at once.

identityBehavior (via DuckDbBulkImportIdentityBehavior) controls whether newly generated identity values are set back on the data entities. Disabled (KeepIdentity) by default.

pseudoTableType (via DuckDbBulkImportPseudoTableType) controls the kind of staging table used internally.

Auto currently behaves exactly like Memory (a TEMP table) — see Operations (DuckDB) for details.

The identity column, if any, is always left out of the INSERT column list generated for unmatched rows — a brand-new row’s identity property is typically an unset default (e.g. 0), not a real value meant to be inserted as-is.

Identity Setting Alignment

When identityBehavior is ReturnIdentity, a single atomic MERGE cannot both branch on match and read back generated identities in DuckDB. The library instead runs three separate statements:

A LEFT JOIN snapshot between the staging and target table (ordered by the pseudo table’s rowid) to classify each staged row as matched (returning the target’s existing identity) or unmatched.
A plain UPDATE ... FROM for the matched rows only (skipped entirely if nothing matched).
An insert-only INSERT ... WHERE NOT EXISTS ... RETURNING statement for the unmatched rows, reading back their new identities.

This is not atomic — three round-trips instead of one — and the matched/unmatched classification from step 1 can go stale if another connection modifies the target table before steps 2 and 3 run.

Usability

Given a list of Person models containing both existing and new rows, the following example bulk-merges them into the Person table.

using (var connection = new DuckDBConnection(ConnectionString))
{
    var mergedRows = connection.BulkMerge(people);
}

To specify a batch size:

using (var connection = new DuckDBConnection(ConnectionString))
{
    var mergedRows = connection.BulkMerge(people, batchSize: 100);
}

When batchSize is not set, all rows are sent to the server in a single batch.

DataTable

using (var connection = new DuckDBConnection(ConnectionString))
{
    var table = ConvertToDataTable(people);
    var mergedRows = connection.BulkMerge("Person", table);
}

Dictionary/ExpandoObject

using (var sourceConnection = new DuckDBConnection(sourceConnectionString))
{
    var result = sourceConnection.QueryAll("Person");
    using (var destinationConnection = new DuckDBConnection(destinationConnectionString))
    {
        var mergedRows = destinationConnection.BulkMerge("Person", result,
            qualifiers: Field.From("LastName", "DateOfBirth"));
    }
}

DataReader

using (var sourceConnection = new DuckDBConnection(sourceConnectionString))
{
    using (var reader = sourceConnection.ExecuteReader("SELECT * FROM \"Person\" WHERE (\"IsActive\" = true);"))
    {
        using (var destinationConnection = new DuckDBConnection(destinationConnectionString))
        {
            var rows = destinationConnection.BulkMerge("Person", reader);
        }
    }
}

To bulk-merge via DataEntityDataReader:

using (var connection = new DuckDBConnection(ConnectionString))
{
    var people = GetPeople(10000);
    using (var reader = new DataEntityDataReader<Person>(people))
    {
        var mergedRows = connection.BulkMerge("Person", reader);
    }
}

Field Qualifiers

By default, the primary column is used as the qualifier. To override, pass a list of Field objects in the qualifiers argument.

using (var connection = new DuckDBConnection(ConnectionString))
{
    var people = GetPeople(10000);
    var mergedRows = connection.BulkMerge<Person>(people,
        qualifiers: e => new { e.LastName, e.DateOfBirth });
}

Use indexed columns from the target table as qualifiers to maximize performance.

Column Mappings

Add column mappings using the DuckDbBulkInsertMapItem class.

var mappings = new List<DuckDbBulkInsertMapItem>();

// Add the mappings
mappings.Add(new DuckDbBulkInsertMapItem("SourceId", "DestinationId"));
mappings.Add(new DuckDbBulkInsertMapItem("SourceName", "DestinationName"));
mappings.Add(new DuckDbBulkInsertMapItem("SourceIsActive", "DestinationIsActive"));

// Execute
using (var connection = new DuckDBConnection(ConnectionString))
{
    var people = GetPeople(10000);
    var mergedRows = connection.BulkMerge(people,
        mappings: mappings);
}

Targeting a Table

To target a specific table, pass the literal table name.

using (var connection = new DuckDBConnection(ConnectionString))
{
    var people = GetPeople(10000);
    var mergedRows = connection.BulkMerge("Person", people);
}

Async Method

An equivalent BulkMergeAsync method is also available.

using (var connection = new DuckDBConnection(ConnectionString))
{
    var mergedRows = await connection.BulkMergeAsync(people);
}