Reshaping collections

Requires Pro (ReschemaCollection). See Plans and Pro features.

Reshaping rewrites every document in a collection. It is the most dangerous thing in this application, so the warnings come before the walkthrough.

#What you can lose

#The two operations

Both are on a collection's context menu.

• Rename a field — everywhere in the collection, including a nested path like profile.city.
• Extract an array — turn each element of an array field into a document in another collection, with a back-reference.

They share the same machinery, because that machinery is the risky part of both.

#Nothing runs before a dry run

Dry run walks the whole collection — not a sample — and counts:

• how many documents will change,
• how many are already in the target shape,
• how many are rejected and why,
• a few before/after pairs.

You approve numbers you have seen, not a description. The Rename button is enabled only after a successful dry run, and editing any path or collision policy throws the measurement away — otherwise it would be describing a change the button no longer performs.

The dry run and the real run are the same pass over the same documents in the same order. The only difference is whether the planned writes are sent. Two separate passes would drift, and the numbers you approved would stop describing the execution.

#How a run behaves

• A batch at a time, never the collection. Documents are read by a cursor in ascending _id order and written in batches (500 by default). Memory holds a batch, not a collection.
• The _id order is a requirement, not an optimisation. Without it, "three batches of four were written" does not identify any part of the collection. With it, the result reports the last transformed _id: everything up to that point is in the new shape, the rest is untouched, and running it again continues from there.
• A partial failure is reported precisely. Each batch is an ordered bulk write, so a failure leaves a prefix rather than a scatter, and the number of writes the server accepted is converted back into whole documents.
• Cancelling stops on a batch boundary, never inside one, and comes back with counters rather than as an error.
• Re-running is safe by design. A transformation must recognise its own output and report a document already in the target shape as unchanged. That property is what makes resuming safe.
• Refusal instead of guessing. Field names containing a dot, starting with $, empty or blank are rejected before anything is read. A document the transformation cannot express exactly is rejected individually — the run continues, and the rejections come back counted, with a reason and an example _id. Skipping quietly is not an option here.
• Field names are compared case-sensitively. A collection can hold name and Name, and folding them together would delete one.

#Renaming a field

• A path, not just a name. profile.city works; each segment has to be a valid field name, so a..b, a.$set and a trailing dot are rejected before the run.
• A field inside an array is refused, not "renamed". $rename does nothing at all to such a field, so a run reporting success over a collection it did not touch is exactly the outcome this feature must not produce. It is refused twice: the syntax (items.$[].sku, items.0.sku) before the run, and a document whose path actually passes through an array during the scan.
• Collisions default to not running. The dry run counts exactly how many documents have the source field, how many have the target, and how many have both. You then choose: do not run (the default), skip those documents, or overwrite. The collision count is taken again at run time, because the collection may have gained a colliding document in between.
• A validator naming the field refuses the run, because it would start rejecting renamed documents half way through a batch. An index on the field warns — including an index on something inside it, such as profile.city when you rename profile — and rebuilding it is left to you.

#Extracting an array

The first operation that writes in two places.

• You name everything. The back-reference field, the field that wraps scalar elements, the position field — the dialog proposes (customers → customerId) and leaves them editable, but invents nothing silently. A name nobody chose gives you a collection nobody knows how to query.
• Scalars are wrapped or refused. An element that is a document becomes a document. A string or a number needs a field name; without one the document is rejected with a count and an example, not guessed.
• A value that is not an array is rejected with its type. A single value is not a one-element array. Empty arrays and missing fields produce nothing and are counted separately — "800 documents do not have this field" is a result you need before concluding the operation failed.
• New _ids by default. An element's own identity is often unique only inside its array. Keeping the element's _id is your choice, and a duplicate then fails loudly: the ordered insert stops at the first error, so you know exactly how many went in and no element disappears quietly.
• Removing the array is off by default. Extracting while keeping the array is the reversible half, and you can check the result. Extracting and removing in one step is a migration. Only that second half has to care about a validator (refused) or indexes on the array (warned); the first changes nothing in the source.
• Writes go to the target first, then the source. A run that dies in between has copied values without removing them, and a re-run can finish that. The other order loses them. If inserting into the target fails part way, the source is updated only for documents whose elements all landed — a document is whole or absent, and that is what makes "everything up to this _id" true.
• Re-running counts the target too. With the array removed, the operation is naturally idempotent — the filter stops matching what it has already done. With the array kept, a second run would double the target, so by default it does not run: TableCore counts the documents in the target carrying a back-reference and tells you how many there are. Continue is exact rather than heuristic: inserts go in ascending source _id order, so the largest back-reference in the target names the last document the previous run reached. Continuing without a back-reference is refused — there is nothing to correlate on.
• The back-reference index is created before the documents. On an empty collection it is cheap; after filling it, it is the same thing slowly and with blocking.

#Before you start

1. Back up the database.
2. Look at the collection in the schema explorer so you know how many documents actually carry the field.
3. Dry run. Read the numbers, especially the collisions and the rejections.
4. Run it, and note the last transformed _id from the result.
5. Check the indexes and the validator afterwards.