Operations · 5 min read

BulkMerge

This method merges all rows from the client application into the database in bulk — inserting new rows and updating existing ones based on the defined qualifiers. It is supported for Aurora MySQL.

Call Flow Diagram

The diagram below shows the flow when calling this operation.

flowchart TD
    Client["Client<br/>(RepoDB)"] -->|BulkMerge| Source["Entities /<br/>DataTable /<br/>DbDataReader"]
    Source --> Pseudo["Create Pseudo Table<br/>+ index on qualifiers"]
    Pseudo --> Copy["AuroraDbBulkCopy<br/>(LOAD DATA LOCAL)"]
    Copy -->|Write| PseudoTable[("Pseudo Table")]
    PseudoTable --> Decision{"identityBehavior ==<br/>ReturnIdentity?"}
    Decision -->|NO| UpdateStep["UPDATE ... INNER JOIN Pseudo<br/>(matched rows)"]
    UpdateStep --> InsertStep["INSERT ... SELECT ...<br/>LEFT JOIN ... WHERE ... IS NULL<br/>(unmatched rows)"]
    InsertStep --> Table[("Target Table")]
    Decision -->|YES| CopyMatched["Copy existing identity onto<br/>matched pseudo rows"]
    CopyMatched --> PreAssign["Pre-assign identity via session<br/>variable for unmatched rows"]
    PreAssign --> Upsert["UPDATE ... INNER JOIN (matched)<br/>+ INSERT ... SELECT (unmatched)"]
    Upsert --> Table
    Upsert --> Report["SELECT identity ...<br/>ORDER BY row order"]
    Report -->|"assign identities<br/>back to entities"| Client
    PseudoTable -->|Drop| Cleanup(["Pseudo Table Dropped"])

Use Case

Use this method to merge rows at high speed. It leverages AuroraDbBulkCopy, a LOAD DATA LOCAL INFILE-based bulk writer from RepoDb.Connector.AuroraDb.MySqlConnector.

For merging 1,000 or more rows, prefer this method over MergeAll.

A pseudo (staging) table — with an index on the qualifier columns — is always created first. The library writes to it via AuroraDbBulkCopy internally, then cascades the changes to the target table — see Operations (AuroraDB MySqlConnector) for the underlying mechanics.

Special Arguments

The qualifiers, mappings, bulkCopyTimeout, batchSize, identityBehavior and pseudoTableType arguments are available for this operation.

qualifiers defines the fields used to match existing rows, corresponding to the JOIN ON clause. Defaults to the primary (or identity) column if not specified.

mappings (via AuroraDbBulkInsertMapItem) defines explicit column mappings between the source properties and the destination columns, with an optional AuroraDbType override per mapping. When omitted, columns are auto-mapped by name (case-insensitive).

bulkCopyTimeout overrides the command timeout, in seconds.

batchSize overrides the number of rows sent to the server per batch. When not set, all items are sent at once.

identityBehavior (via AuroraDbBulkImportIdentityBehavior) controls whether newly generated identity values are set back on the data entities. Disabled (KeepIdentity) by default.

pseudoTableType (via AuroraDbBulkImportPseudoTableType) is accepted but currently has no effect — every staged call resolves to Physical regardless of the value passed.

MySQL’s INSERT ... ON DUPLICATE KEY UPDATE only fires on an actual unique/primary key constraint, not an arbitrary qualifier list, so it cannot be used here. The upsert is instead split into an UPDATE ... INNER JOIN for matched rows and an anti-joined INSERT ... SELECT for unmatched ones.

Identity Setting Alignment

When identityBehavior is KeepIdentity (the default), the upsert runs as two plain statements: an UPDATE <real> T INNER JOIN <pseudo> S ON (qualifiers) SET ... for matched rows, followed by an INSERT ... SELECT ... LEFT JOIN <real> T ON (qualifiers) WHERE T.<qualifier> IS NULL anti-join for unmatched rows.

When identityBehavior is ReturnIdentity, this runs as five steps: (1) matched rows first copy their existing identity from the real table onto the pseudo row; (2) unmatched rows get a fresh identity pre-assigned via the same session-variable technique BulkInsert uses, seeded from MAX(identityColumn) + 1; (3) the matched-row UPDATE ... INNER JOIN runs; (4) the unmatched-row INSERT ... SELECT (anti-joined) runs, carrying the pre-assigned identities as literals; (5) a final SELECT identity ... ORDER BY __RepoDbBulkRowOrder__ reports every row’s identity — existing or newly assigned — in original bulk-load order.

Requires AllowUserVariables=True on the connection string — AuroraDbConnection defaults this to false. The anti-join assumes the first qualifier column is never legitimately null on a real, matched row (true for the typical case of qualifying on a primary/unique key).

Usability

Given a list of Person models containing both existing and new rows, the following example bulk-merges them into the Person table.

using (var connection = new AuroraDbConnection(connectionString))
{
    var mergedRows = connection.BulkMerge(people);
}

To specify a batch size:

using (var connection = new AuroraDbConnection(connectionString))
{
    var mergedRows = connection.BulkMerge(people, batchSize: 100);
}

When batchSize is not set, all rows are sent to the server in a single batch.

DataTable

using (var connection = new AuroraDbConnection(connectionString))
{
    var table = ConvertToDataTable(people);
    var mergedRows = connection.BulkMerge("Person", table);
}

Dictionary/ExpandoObject

using (var sourceConnection = new AuroraDbConnection(sourceConnectionString))
{
    var result = sourceConnection.QueryAll("Person");
    using (var destinationConnection = new AuroraDbConnection(destinationConnectionString))
    {
        var mergedRows = destinationConnection.BulkMerge("Person", result,
            qualifiers: Field.From("LastName", "DateOfBirth"));
    }
}

DataReader

using (var sourceConnection = new AuroraDbConnection(sourceConnectionString))
{
    using (var reader = sourceConnection.ExecuteReader("SELECT * FROM `Person` WHERE (`IsActive` = 1);"))
    {
        using (var destinationConnection = new AuroraDbConnection(destinationConnectionString))
        {
            var rows = destinationConnection.BulkMerge("Person", reader);
        }
    }
}

To bulk-merge via DataEntityDataReader:

using (var connection = new AuroraDbConnection(connectionString))
{
    var people = GetPeople(10000);
    using (var reader = new DataEntityDataReader<Person>(people))
    {
        var mergedRows = connection.BulkMerge("Person", reader);
    }
}

Field Qualifiers

By default, the primary column is used as the qualifier. To override, pass a list of Field objects in the qualifiers argument.

using (var connection = new AuroraDbConnection(connectionString))
{
    var people = GetPeople(10000);
    var mergedRows = connection.BulkMerge<Person>(people,
        qualifiers: e => new { e.LastName, e.DateOfBirth });
}

Use indexed columns from the target table as qualifiers to maximize performance.

Column Mappings

Add column mappings using the AuroraDbBulkInsertMapItem class.

var mappings = new List<AuroraDbBulkInsertMapItem>();

// Add the mappings
mappings.Add(new AuroraDbBulkInsertMapItem("SourceId", "DestinationId"));
mappings.Add(new AuroraDbBulkInsertMapItem("SourceName", "DestinationName"));
mappings.Add(new AuroraDbBulkInsertMapItem("SourceIsActive", "DestinationIsActive"));

// Execute
using (var connection = new AuroraDbConnection(connectionString))
{
    var people = GetPeople(10000);
    var mergedRows = connection.BulkMerge(people,
        mappings: mappings);
}

Targeting a Table

To target a specific table, pass the literal table name.

using (var connection = new AuroraDbConnection(connectionString))
{
    var people = GetPeople(10000);
    var mergedRows = connection.BulkMerge("Person", people);
}

Async Method

An equivalent BulkMergeAsync method is also available.

using (var connection = new AuroraDbConnection(connectionString))
{
    var mergedRows = await connection.BulkMergeAsync(people);
}