Generating migrations for XOS and Services

This document describes how to use the XOS toolchain to generate basic migrations for service models. The autogenerated migrations can later be extended with custom logic to support more complex scenarios.

NOTE from now on we assume you have obtained the code as described in this guide and the entire tree is now available in ~/cord

Install the XOS Toolchain

The XOS toolchain consists of a set of python libraries. There is an helper script to install the toolchain in the XOS repo, so just execute:

cd ~/cord/orchestration/xos
bash scripts/setup_venv.sh

Generate the migrations

Once the toolchain is available, you'll be able to generate migrations for the services (or the core). The xos-migrate command is now available and you can see the options by running:

usage: xos-migrate [-h] -s SERVICE_NAMES [-r REPO_ROOT] [--check] [-v]

XOS Migrations

optional arguments:
  -h, --help            show this help message and exit
  -r REPO_ROOT, --repo REPO_ROOT
                        The location of the folder containing the CORD repo
                        root (default to ~/cord)
  --check               Check if the migrations are generated for a given
                        service. Does not apply any change.
  -v, --verbose         increase log verbosity

required arguments:
  -s SERVICE_NAMES, --service SERVICE_NAMES
                        The name of the folder containing the service in
                        cord/orchestration/xos_services

Supposing that your code is living under ~/Sites/cord and you want to generate migrations for core and fabric, this is the command you'll run:

xos-migrate -r ~/Sites/cord -s core -s fabric

If no migrations were present for your service you'll see a new folder migrations that is a sibling of models containing a file 0001-initial.py. If the service already had migrations you'll see a new file in that folder, for example: 0002-fabricservice_fullname.py

NOTE that all the migration files needs to be committed together with the code as they'll be loaded into the core together with the models.

Customize the migrations

The autogenerated migrations only operates on the database altering tables, they won't make any modifications to the data. You can write custom code to do that, following this example.

We assume you already have a model called FabricService that has two properties: first_name and last_name.

As part of your changes you want to add a third property called full_name that is defined as first_name + last_name.

This is the migration code that the tool will generate:

class Migration(migrations.Migration):

    dependencies = [
        ('fabric', '0001_initial'),
    ]

    operations = [
        migrations.AddField(
            model_name='fabricservice',
            name='full_name',
            field=models.TextField(blank=True, null=True),
        ),
    ]

To migrate the data, you need to add a custom operation in your migration. This can be done defining a custom method forwards as:

def forwards(apps, schema_editor):
  MyModel = apps.get_model('myapp', 'MyModel')
    for row in MyModel.objects.all():
        row.full_name = "%s %s" % (row.first_name, row.last_name)
        row.save(update_fields=['full_name'])

and adding it to the operations list.

Here is a complete example of the customized migration:

def forwards(apps, schema_editor):
    MyModel = apps.get_model('myapp', 'MyModel')
    for row in MyModel.objects.all():
        row.full_name = "%s %s" % (row.first_name, row.last_name)
        row.save(update_fields=['full_name'])
        
class Migration(migrations.Migration):


    dependencies = [
        ('fabric', '0001_initial'),
    ]

    operations = [
        migrations.AddField(
            model_name='fabricservice',
            name='full_name',
            field=models.TextField(blank=True, null=True),
        ),
        migrations.RunPython(forwards),
    ]

For more informations about migrations you can refer to the official Django guide.

Here are some good reads about migrations:

• https://realpython.com/django-migrations-a-primer/#creating-migrations
• https://realpython.com/data-migrations/

Development workflow

During development multiple changes to the models are often necessary. In order to continuously upgrade the service to proceed with development we suggest to generate a new migration every time the models are changed. This is required to upgrade the service multiple times during the development loop (as the core expects new migrations).

This will probably lead to multiple migration files by the time your feature is complete, for example:

- 0003-modelA-fieldA.py
- 0004-modelA-fieldB.py
...
- 0007-modelB-fieldX.py

However, in order to maintain clarity, we suggest to submit a single migration as part of a patch. To do that you can simply remove all the migrations you have generated as part of your development and run the xos-migrate tool again. This will generate a single migration file for all your changes.

Validate migration status

Once you are done with development you want to make sure that all the necessary migrations are generated and checked in. To do that you can run the xos-migrate tool using the --check flag. Here is an example:

xos-migrate -s fabric --check