NodeComplete Migration Source

NodeComplete is a migration source plugin that is used for generated upgrade migrations from d6 or 7.

NodeComplete vs classic migrations

With "classic" migrations, Node, NodeRevision, and NodeEntityTranslation classes were used to create 3 migrations per content type. NodeComplete combines these into a single migration.

See this change record for details: https://www.drupal.org/node/3105503. Classic migrations can be generated by configuration in settings.php: $settings['migrate_node_migrate_type_classic'] = TRUE;

NodeComplete class

See /core/modules/src/Plugin/migrate/source, d6 and d7 directories. They contain the NodeComplete class for Drupal 6 and 7, plus the classic migration classes.

NodeComplete extends NodeRevision, which extends Node, which extends FieldableEntity > DrupalSqlBase > SqlBase > SourcePluginBase.

NodeEntityTranslation extends FieldableEntitiy, used in classic migrations for a separate translation migration.

From the top:

SourcePluginBase is an abstract class in Migrate that sets up some methods for processing rows.

SqlBase is an abstract class in Migrate that has methods to query a database and process rows.

DrupalSqlBase is an abstract class in Migrate Drupal with utility functions for getting info from a Drupal database. For example, moduleExists() can check if a module was enabled in the source database.

FieldableEntity is an abstract class in Migrate Drupal for getting node and field values. It uses field config in the source database to define available fields and fetch values.

*Abstract classes cannot be used directly, and are extended to implement them. They are different than interfaces in that they provide some working methods to be used in child classes.

Node

The Node classes (d6 and 7) in the Node module have queries to fetch content from the database.

It can handle translations, but not revisions. Only the current revision is migrated.

source:
  plugin: d7_node
  node_type: page
  node_type: [page, test] # Migrate multiple types into one.

public function query() {
  // Select node in its last revision.
  $query = $this->select('node_revision', 'nr')
    ->fields('n', [
      'nid',
      'type',
      'language',
      'status',
      'created',
      'changed',
      'comment',
      'promote',
      'sticky',
      'tnid',
      'translate',
    ])
    ->fields('nr', [
      'vid',
      'title',
      'log',
      'timestamp',
    ]);
    $query->addField('n', 'uid', 'node_uid');
    $query->addField('nr', 'uid', 'revision_uid');
    $query->innerJoin('node', 'n', static::JOIN);
    ...

The static::JOIN is using a constant for the JOIN ON (const JOIN = '[n].[vid] = [nr].[vid]';)

When building this query, it checks for enabled translations on the new site and includes the language of the translated nid.

// If the content_translation module is enabled, get the source langcode
// to fill the content_translation_source field.
if ($this->moduleHandler->moduleExists('content_translation')) {
  $query->leftJoin('node', 'nt', '[n].[tnid] = [nt].[nid]');
  $query->addField('nt', 'language', 'source_langcode');
}
$this->handleTranslations($query);

Including translations in the migration is optional, use translations: false in the source plugin config.

This is how the query is altered:

protected function handleTranslations(SelectInterface $query) {
  // Check whether or not we want translations.
  if (empty($this->configuration['translations'])) {
    // No translations: Yield untranslated nodes, or default translations.
    $query->where('[n].[tnid] = 0 OR [n].[tnid] = [n].[nid]');
  }
  else {
    // Translations: Yield only non-default translations.
    $query->where('[n].[tnid] <> 0 AND [n].[tnid] <> [n].[nid]');
  }
}

The fields() method defines available migration fields:

public function fields() {
  $fields = [
    'nid' => $this->t('Node ID'),
    'type' => $this->t('Type'),
    'title' => $this->t('Title'),
    'node_uid' => $this->t('Node authored by (uid)'),
    'revision_uid' => $this->t('Revision authored by (uid)'),
    'created' => $this->t('Created timestamp'),
    'changed' => $this->t('Modified timestamp'),
    'status' => $this->t('Published'),
    'promote' => $this->t('Promoted to front page'),
    'sticky' => $this->t('Sticky at top of lists'),
    'revision' => $this->t('Create new revision'),
    'language' => $this->t('Language (fr, en, ...)'),
    'tnid' => $this->t('The translation set id for this node'),
    'timestamp' => $this->t('The timestamp the latest revision of this node was created.'),
  ];
  return $fields;
}

prepareRow() handles field values, translations and checks for an alternative title field from the title module.

tnid will equal nid if there are no translations for the node.

...

// Get Field API field values.
foreach ($this->getFields('node', $type) as $field_name => $field) {
  // Ensure we're using the right language if the entity and the field are
  // translatable.
  $field_language = $entity_translatable && $field['translatable'] ? $language : NULL;
  $row->setSourceProperty($field_name, $this->getFieldValues('node', $field_name, $nid, $vid, $field_language));
}

// Make sure we always have a translation set.
if ($row->getSourceProperty('tnid') == 0) {
  $row->setSourceProperty('tnid', $row->getSourceProperty('nid'));
}

// If the node title was replaced by a real field using the Drupal 7 Title
// module, use the field value instead of the node title.
if ($this->moduleExists('title')) {
  $title_field = $row->getSourceProperty('title_field');
  if (isset($title_field[0]['value'])) {
    $row->setSourceProperty('title', $title_field[0]['value']);
  }
}

NodeRevision

Changes the Node migration query by altering the join between node and revision tables.

const JOIN = 'n.nid = nr.nid AND n.vid <> nr.vid'; # NodeRevision

const JOIN = '[n].[vid] = [nr].[vid]'; #Node

The Node join matches the current node revision with it's record in the revision table. The NodeRevision join matches them by node id and skips the current version.

It also adds revision fields to the source.

return parent::fields() + [
  'vid' => $this->t('The primary identifier for this version.'),
  'log' => $this->t('Revision Log message'),
  'timestamp' => $this->t('Revision timestamp'),
];

NodeComplete

Modifies the query to join entity_translation_revision and include fields.

By setting nid, vid, and language as IDs, each source row can be an untranslated or default language node, a revision, or a translation. On import, nodes and revisions will be created as needed.

public function getIds() {
  return [
    'nid' => [
      'type' => 'integer',
      'alias' => 'n',
    ],
    'vid' => [
      'type' => 'integer',
      'alias' => 'nr',
    ],
    'language' => [
      'type' => 'string',
      'alias' => 'n',
    ],
  ];
}

Process plugins

These process plugins help when mixing classic and NodeComplete migrations.

NodeCompleteNodeLookup

Returns nid.

NodeCompleteNodeRevisionLookup

Returns vid.

NodeCompleteNodeTranslationLookup

Returns language code.

Migration plugins

D7NodeTranslation (and d6)

Extends default Migration plugin, but will generate followup migrations.

You will see this used in generated NodeComplete migrations.

class: Drupal\node\Plugin\migrate\D7NodeTranslation

Destination Plugins

EntityContentComplete

Extends EntityContentBase.

Most of the work is done in getEntity().

This part checks for a revision id from the source or uses the processed revision id.

$old_destination_id_values is called by import(), which passes mapped ids from a previous migration. For example when using --update.

import() is called by MigrateExecutable, which does the lookup.

$revision_id = $old_destination_id_values
  ? $old_destination_id_values[1]
  : $row->getDestinationProperty($this->getKey('revision'));

This first check determines if a revision should be created or updated. 

// If we are re-running a migration with set revision IDs and the
// destination revision ID already exists then do not create a new revision.
if (!empty($revision_id) && ($entity = $this->storage->loadRevision($revision_id))) {
  $entity->setNewRevision(FALSE);
}

If not, check if the destination ID is a valid entity, and set it to update and create a new revision.

elseif (($entity_id = $row->getDestinationProperty($this->getKey('id'))) && ($entity = $this->storage->load($entity_id))) {
  // We want to create a new entity. Set enforceIsNew() FALSE is  necessary
  // to properly save a new entity while setting the ID. Without it, the
  // system would see that the ID is already set and assume it is an update.
  $entity->enforceIsNew(FALSE);
  // Intentionally create a new revision. Setting new revision TRUE here may
  // not be necessary, it is done for clarity.
  $entity->setNewRevision(TRUE);
}

If the entity does not exist, create a new one.

else {
  // Attempt to set the bundle.
  if ($bundle = $this->getBundle($row)) {
    $row->setDestinationProperty($this->getKey('bundle'), $bundle);
  }

  // Stubs might need some required fields filled in.
  if ($row->isStub()) {
    $this->processStubRow($row);
  }
  $entity = $this->storage->create($row->getDestination());
  $entity->enforceIsNew();
}

Then update the entity. Handle translation update times, and map destination ids via updateEntity().

// We need to update the entity, so that the destination row IDs are
// correct.
$entity = $this->updateEntity($entity, $row);
$entity->isDefaultRevision(TRUE);
if ($entity instanceof EntityChangedInterface && $entity instanceof ContentEntityInterface) {
  // If we updated any untranslatable fields, update the timestamp for the
  // other translations.
  /** @var \Drupal\Core\Entity\ContentEntityInterface|\Drupal\Core\Entity\EntityChangedInterface $entity */
  foreach ($entity->getTranslationLanguages() as $langcode => $language) {
    // If we updated an untranslated field, then set the changed time for
    // for all translations to match the current row that we are saving.
    // In this context, getChangedTime() should return the value we just
    // set in the updateEntity() call above.
    if ($entity->getTranslation($langcode)->hasTranslationChanges()) {
      $entity->getTranslation($langcode)->setChangedTime($entity->getChangedTime());
    }
  }
}