Migrating XOS Models

This document describes an XOS toolchain that can be used to generate basic migrations for service models. The autogenerated migrations can later be extended with custom logic to support more complex scenarios.

NOTE: The following assumes you have downloaded the entire source code tree in ~/cord.

Installing the Toolchain

The XOS migration toolchain consists of a set of python libraries. There is a helper script to install the toolchain. Just run:

cd ~/cord/orchestration/xos
bash scripts/setup_venv.sh

Generating Migrations

Once the toolchain is available, you will be able to generate migrations for the services (or the core) using the xos-migrate command. Execute the command to see the available options:

usage: xos-migrate [-h] -s SERVICE_NAMES [-r REPO_ROOT] [-x XOS_ROOT]
                   [--services-dir SERVICES_ROOT] [--check] [-v]

XOS Migrations

optional arguments:
  -h, --help            show this help message and exit
  -r REPO_ROOT, --repo REPO_ROOT
                        Path to the CORD repo root (defaults to '../..').
                        Mutually exclusive with '--xos'.
  -x XOS_ROOT, --xos-dir XOS_ROOT
                        Path to directory of the XOS repo. Incompatible with '
                        --repo'.
  --services-dir SERVICES_ROOT
                        Path to directory of the XOS services root.
                        Incompatible with '--repo'.Note that all the services
                        repo needs to be siblings
  --check               Check if the migrations are generated for a given
                        service. Does not apply any change.
  -v, --verbose         increase log verbosity

required arguments:
  -s SERVICE_NAMES, --service SERVICE_NAMES
                        The name of the folder containing the service in
                        cord/orchestration/xos-services

For example, if the code you want to migrate is in ~/Sites and you want to generate migrations for core and fabric, then run the following command:

xos-migrate -r ~/Sites/cord -s core -s fabric

NOTE: The command above is equivalent to:

xos-migrate --xos-dir ~/Sites/cord/orchestration/xos --services-dir ~/Sites/cord/orchestration/xos-services/ -s core -s fabric

If no migrations were present for your service, you will see a new folder migrations (a sibling of models) that contains the file 0001-initial.py. If the service already had migrations you will see a new file in that folder, for example: 0002-fabricservice_fullname.py

NOTE: All the migration files need to be committed together with the code as they will be loaded into the core together with the models.

Customizing Migrations

The autogenerated migrations operate on only the database tables. They do not make any modifications to the data in those tables. You can write custom code to make such changes, as illustrated in the following example.

Assume you already have a model called FabricService that has two properties: first_name and last_name, and as part of your changes you want to add a third property called full_name, defined as first_name + last_name.

The following is the migration code that the tool will automatically generate:

class Migration(migrations.Migration):

    dependencies = [
        ('fabric', '0001_initial'),
    ]

    operations = [
        migrations.AddField(
            model_name='fabricservice',
            name='full_name',
            field=models.TextField(blank=True, null=True),
        ),
    ]

To migrate the data, you need to add a custom operation in your migration. This can be done defining a custom method forwards as:

def forwards(apps, schema_editor):
  MyModel = apps.get_model('myapp', 'MyModel')
    for row in MyModel.objects.all():
        row.full_name = "%s %s" % (row.first_name, row.last_name)
        row.save(update_fields=['full_name'])

and adding it to the operations list.

The following is a complete example of the customized migration:

def forwards(apps, schema_editor):
    MyModel = apps.get_model('myapp', 'MyModel')
    for row in MyModel.objects.all():
        row.full_name = "%s %s" % (row.first_name, row.last_name)
        row.save(update_fields=['full_name'])
        
class Migration(migrations.Migration):

    dependencies = [
        ('fabric', '0001_initial'),
    ]

    operations = [
        migrations.AddField(
            model_name='fabricservice',
            name='full_name',
            field=models.TextField(blank=True, null=True),
        ),
        migrations.RunPython(forwards),
    ]

For more information about migrations you can refer to the official Django guide.

Additional information about migration can be found in the following:

• https://realpython.com/django-migrations-a-primer/#creating-migrations
• https://realpython.com/data-migrations/

Development Workflow

It is sometimes necessary to make multiple changes to the models during development. In order to continuously upgrade the service to proceed with development we suggest you generate a new migration every time the models are changed. This is required to upgrade the service multiple times during the development loop (as the core expects new migrations).

This will probably lead to multiple migration files by the time your feature is complete, for example:

- 0003-modelA-fieldA.py
- 0004-modelA-fieldB.py
...
- 0007-modelB-fieldX.py

To maintain clarity, however, we suggest you submit a single migration as part of a patch. To do that, simply remove all the migrations you have generated as part of your development and run the xos-migrate tool again. This will generate a single migration file for all your changes.

Validating Migration Status

Once you are done with development, you should make sure that all the necessary migrations are generated and checked in. To do that, run the xos-migrate tool using the --check flag. Here is an example:

xos-migrate -s fabric --check

The XOS core can be checked in place (without the entire source tree checked out by the repo tool) with:

xos-migrate -x xos -s core --check