Operations · 4 min read

BulkInsert

This method inserts all rows from the client application into the database in bulk. It is supported for Aurora MySQL.

Call Flow Diagram

The diagram below shows the flow when calling this operation.

flowchart TD
    Client["Client<br/>(RepoDB)"] -->|BulkInsert| Source["Entities /<br/>DataTable /<br/>DbDataReader"]
    Source --> Decision{"identityBehavior ==<br/>ReturnIdentity?"}
    Decision -->|NO| Direct["AuroraDbBulkCopy<br/>(LOAD DATA LOCAL)"]
    Direct -->|Write| Table[("Target Table")]
    Decision -->|YES| Pseudo["Create Pseudo Table<br/>+ make identity column nullable"]
    Pseudo --> Staged["AuroraDbBulkCopy<br/>(LOAD DATA LOCAL)"]
    Staged -->|Write| PseudoTable[("Pseudo Table")]
    PseudoTable --> Seed["SELECT MAX(identity) + 1<br/>FROM Target Table"]
    Seed --> Assign["Session variable pre-assigns<br/>identity per pseudo row"]
    Assign --> Insert["INSERT INTO Target<br/>SELECT ... FROM Pseudo"]
    Insert --> Table
    Insert --> Report["SELECT identity ...<br/>ORDER BY row order"]
    Report -->|"assign identities<br/>back to entities"| Client
    PseudoTable -->|Drop| Cleanup(["Pseudo Table Dropped"])

Use Case

Use this method to insert rows at high speed. It leverages AuroraDbBulkCopy, a LOAD DATA LOCAL INFILE-based bulk writer from RepoDb.Connector.AuroraDb.MySqlConnector.

For inserting 1,000 or more rows, prefer this method over InsertAll.

Rows are written straight to the target table. A staging table is only used when identityBehavior is set to ReturnIdentity (see below) — see Operations (AuroraDB MySqlConnector) for the underlying mechanics.

Special Arguments

The mappings, bulkCopyTimeout, batchSize, identityBehavior and pseudoTableType arguments are available for this operation.

mappings (via AuroraDbBulkInsertMapItem) defines explicit column mappings between the source properties and the destination columns, with an optional AuroraDbType override per mapping. When omitted, columns are auto-mapped by name (case-insensitive).

bulkCopyTimeout overrides the command timeout, in seconds.

batchSize overrides the number of rows sent to the server per batch. When not set, all items are sent at once.

identityBehavior (via AuroraDbBulkImportIdentityBehavior) controls whether newly generated identity values are set back on the data entities. Disabled (KeepIdentity) by default. Enabling this (ReturnIdentity) routes the operation through a staging table, where identity values are pre-assigned via a session variable before the rows are copied into the target table.

pseudoTableType (via AuroraDbBulkImportPseudoTableType) is accepted but currently has no effect — every staged call resolves to Physical regardless of the value passed.

The DbDataReader overload has no identityBehavior argument — a forward-only, single-pass reader cannot be rewound to correlate generated identity values back onto a source row, so ReturnIdentity is not supported for that overload.

Identity Setting Alignment

When identityBehavior is ReturnIdentity, the library stages into a pseudo table (whose identity column is made nullable, since CREATE TABLE ... AS SELECT carries over the real table’s NOT NULL), then reads SELECT MAX(identityColumn) + 1 FROM <realTable> to seed the next value. A session user variable then assigns a distinct, increasing identity to every staged row (SET @repodb_seq := (seed - 1); UPDATE pseudo SET identity = (@repodb_seq := @repodb_seq + 1);), and the rows are copied into the real table via a plain INSERT ... SELECT, carrying their pre-assigned identity as literal values — MySQL always accepts an explicit value into an AUTO_INCREMENT column. A final SELECT identity ... ORDER BY __RepoDbBulkRowOrder__ reports every value back in the original bulk-load order.

Requires AllowUserVariables=True on the connection string — AuroraDbConnection defaults this to false. Reading the seed and assigning it are two separate round trips, leaving a small race window against a concurrent writer to the same table.

Usability

The following example defines a method that produces a list of Person objects, then bulk-inserts 10,000 rows into the Person table.

private IEnumerable<Person> GetPeople(int count = 1000)
{
    for (var i = 0; i < count; i++)
    {
        yield return new Person
        {
            Name = $"Person-{i}",
            IsActive = true,
            CreatedDateUtc = DateTime.UtcNow
        };
    }
}

using (var connection = new AuroraDbConnection(connectionString))
{
    var people = GetPeople(10000);
    var insertedRows = connection.BulkInsert(people);
}

To specify a batch size:

using (var connection = new AuroraDbConnection(connectionString))
{
    var people = GetPeople(10000);
    var insertedRows = connection.BulkInsert(people, batchSize: 100);
}

When batchSize is not set, all rows are sent to the server in a single batch.

To return the newly generated identity values:

using (var connection = new AuroraDbConnection(connectionString))
{
    var people = GetPeople(10000);
    var insertedRows = connection.BulkInsert(people,
        identityBehavior: AuroraDbBulkImportIdentityBehavior.ReturnIdentity);
}

DataTable

using (var connection = new AuroraDbConnection(connectionString))
{
    var people = GetPeople(10000);
    var table = ConvertToDataTable(people);
    var insertedRows = connection.BulkInsert("Person", table);
}

Dictionary/ExpandoObject

using (var sourceConnection = new AuroraDbConnection(sourceConnectionString))
{
    var result = sourceConnection.QueryAll("Person");
    using (var destinationConnection = new AuroraDbConnection(destinationConnectionString))
    {
        var insertedRows = destinationConnection.BulkInsert("Person", result);
    }
}

DataReader

using (var sourceConnection = new AuroraDbConnection(sourceConnectionString))
{
    using (var reader = sourceConnection.ExecuteReader("SELECT * FROM `Person`;"))
    {
        using (var destinationConnection = new AuroraDbConnection(destinationConnectionString))
        {
            var rows = destinationConnection.BulkInsert("Person", reader);
        }
    }
}

To bulk-insert via DataEntityDataReader:

using (var connection = new AuroraDbConnection(connectionString))
{
    var people = GetPeople(10000);
    using (var reader = new DataEntityDataReader<Person>(people))
    {
        var insertedRows = connection.BulkInsert("Person", reader);
    }
}

Column Mappings

Add column mappings using the AuroraDbBulkInsertMapItem class.

var mappings = new List<AuroraDbBulkInsertMapItem>();

// Add the mappings
mappings.Add(new AuroraDbBulkInsertMapItem("SourceId", "DestinationId"));
mappings.Add(new AuroraDbBulkInsertMapItem("SourceName", "DestinationName"));
mappings.Add(new AuroraDbBulkInsertMapItem("SourceIsActive", "DestinationIsActive"));

// Execute
using (var connection = new AuroraDbConnection(connectionString))
{
    var people = GetPeople(10000);
    var insertedRows = connection.BulkInsert(people,
        mappings: mappings);
}

Targeting a Table

To target a specific table, pass the literal table name.

using (var connection = new AuroraDbConnection(connectionString))
{
    var people = GetPeople(10000);
    var insertedRows = connection.BulkInsert("Person", people);
}

Async Method

An equivalent BulkInsertAsync method is also available.

using (var connection = new AuroraDbConnection(connectionString))
{
    var people = GetPeople(10000);
    var insertedRows = await connection.BulkInsertAsync(people);
}