Operations · 5 min read

BulkMerge

This method merges all rows from the client application into the database in bulk — inserting new rows and updating existing ones based on the defined qualifiers. It is supported for CockroachDB.

Call Flow Diagram

The diagram below shows the flow when calling this operation.

flowchart TD
    Client["Client<br/>(RepoDB)"] -->|BulkMerge| Source["Entities /<br/>DataTable /<br/>DbDataReader"]
    Source --> Pseudo["Create Pseudo Table<br/>(Memory by default) + index<br/>on qualifier columns"]
    Pseudo --> Copy["CockroachDbBulkCopy"]
    Copy -->|Write| PseudoTable[("Pseudo Table")]
    PseudoTable --> Decision{"identityBehavior ==<br/>ReturnIdentity?"}
    Decision -->|NO| UpdateStep["UPDATE ... FROM Pseudo<br/>(matched rows)"]
    UpdateStep --> InsertStep["INSERT ... WHERE NOT EXISTS ...<br/>ORDER BY row order<br/>(unmatched rows)"]
    InsertStep --> Table[("Target Table")]
    Decision -->|YES| PreAssign["Pre-assign identities on Pseudo<br/>via nextval(...) for unmatched rows"]
    PreAssign --> Upsert["INSERT ... OVERRIDING SYSTEM VALUE<br/>ON CONFLICT DO UPDATE<br/>RETURNING identity"]
    Upsert --> Table
    Upsert -->|"assign identities<br/>back to entities"| Client
    PseudoTable -->|Drop| Cleanup(["Pseudo Table Dropped"])

Use Case

Use this method to merge rows at high speed. It leverages CockroachDbBulkCopy, a native binary COPY-based bulk writer from RepoDb.Connector.CockroachDb.

For merging 1,000 or more rows, prefer this method over MergeAll.

A pseudo (staging) table — with an index on the qualifier columns — is always created first. The library writes to it via CockroachDbBulkCopy internally, then cascades the changes to the target table — see Operations (CockroachDB) for the underlying mechanics.

Special Arguments

The qualifiers, mappings, bulkCopyTimeout, batchSize, identityBehavior and pseudoTableType arguments are available for this operation.

qualifiers defines the fields used to match existing rows, corresponding to the WHERE/ON CONFLICT clause. Defaults to the primary (or identity) column if not specified.

mappings (via CockroachDbBulkInsertMapItem) defines explicit column mappings between the source properties and the destination columns, with an optional CockroachDbType override per mapping. When omitted, columns are auto-mapped by name (case-insensitive).

bulkCopyTimeout overrides the command timeout, in seconds.

batchSize overrides the number of rows sent to the server per batch. When not set, all items are sent at once.

identityBehavior (via CockroachDbBulkImportIdentityBehavior) controls whether newly generated identity values are set back on the data entities. Disabled (KeepIdentity) by default.

pseudoTableType (via CockroachDbBulkImportPseudoTableType) controls the kind of staging table used internally.

The identity column, if any, is always left out of the INSERT column list — a brand-new row’s identity property is typically an unset default (e.g. 0), not a real value meant to be inserted as-is.

Identity Setting Alignment

When identityBehavior is KeepIdentity (the default), the upsert runs as two plain statements: an UPDATE ... FROM for the rows that already exist, followed by an INSERT ... WHERE NOT EXISTS ... for the rows that don’t.

When identityBehavior is ReturnIdentity, the identity for the rows that will be inserted is pre-assigned first: UPDATE PseudoTable SET identity = nextval(pg_get_serial_sequence(target, identity_column)) WHERE NOT EXISTS (SELECT 1 FROM target WHERE <qualifiers>). A single INSERT ... OVERRIDING SYSTEM VALUE ... ON CONFLICT (qualifiers) DO UPDATE ... RETURNING identity then performs the whole upsert and returns every row’s identity — the pre-assigned one for new rows, the existing one for updated rows — in one round trip.

This single-statement RETURNING path is more efficient than the multi-statement (snapshot, then update, then insert) approach some other providers require for the same feature.

Usability

Given a list of Person models containing both existing and new rows, the following example bulk-merges them into the Person table.

using (var connection = new CockroachDbConnection(connectionString))
{
    var mergedRows = connection.BulkMerge(people);
}

To specify a batch size:

using (var connection = new CockroachDbConnection(connectionString))
{
    var mergedRows = connection.BulkMerge(people, batchSize: 100);
}

When batchSize is not set, all rows are sent to the server in a single batch.

DataTable

using (var connection = new CockroachDbConnection(connectionString))
{
    var table = ConvertToDataTable(people);
    var mergedRows = connection.BulkMerge("Person", table);
}

Dictionary/ExpandoObject

using (var sourceConnection = new CockroachDbConnection(sourceConnectionString))
{
    var result = sourceConnection.QueryAll("Person");
    using (var destinationConnection = new CockroachDbConnection(destinationConnectionString))
    {
        var mergedRows = destinationConnection.BulkMerge("Person", result,
            qualifiers: Field.From("LastName", "DateOfBirth"));
    }
}

DataReader

using (var sourceConnection = new CockroachDbConnection(sourceConnectionString))
{
    using (var reader = sourceConnection.ExecuteReader("SELECT * FROM \"Person\" WHERE (\"IsActive\" = true);"))
    {
        using (var destinationConnection = new CockroachDbConnection(destinationConnectionString))
        {
            var rows = destinationConnection.BulkMerge("Person", reader);
        }
    }
}

To bulk-merge via DataEntityDataReader:

using (var connection = new CockroachDbConnection(connectionString))
{
    var people = GetPeople(10000);
    using (var reader = new DataEntityDataReader<Person>(people))
    {
        var mergedRows = connection.BulkMerge("Person", reader);
    }
}

Field Qualifiers

By default, the primary column is used as the qualifier. To override, pass a list of Field objects in the qualifiers argument.

using (var connection = new CockroachDbConnection(connectionString))
{
    var people = GetPeople(10000);
    var mergedRows = connection.BulkMerge<Person>(people,
        qualifiers: e => new { e.LastName, e.DateOfBirth });
}

Use indexed columns from the target table as qualifiers to maximize performance.

Column Mappings

Add column mappings using the CockroachDbBulkInsertMapItem class.

var mappings = new List<CockroachDbBulkInsertMapItem>();

// Add the mappings
mappings.Add(new CockroachDbBulkInsertMapItem("SourceId", "DestinationId"));
mappings.Add(new CockroachDbBulkInsertMapItem("SourceName", "DestinationName"));
mappings.Add(new CockroachDbBulkInsertMapItem("SourceIsActive", "DestinationIsActive"));

// Execute
using (var connection = new CockroachDbConnection(connectionString))
{
    var people = GetPeople(10000);
    var mergedRows = connection.BulkMerge(people,
        mappings: mappings);
}

Targeting a Table

To target a specific table, pass the literal table name.

using (var connection = new CockroachDbConnection(connectionString))
{
    var people = GetPeople(10000);
    var mergedRows = connection.BulkMerge("Person", people);
}

Async Method

An equivalent BulkMergeAsync method is also available.

using (var connection = new CockroachDbConnection(connectionString))
{
    var mergedRows = await connection.BulkMergeAsync(people);
}