SEBA-738 implement SetDirty
Change-Id: Ia46096faaed7c45f72f873044f8e18b3a1a091b3
diff --git a/README.md b/README.md
index 7122cc6..0cf24c0 100644
--- a/README.md
+++ b/README.md
@@ -74,6 +74,7 @@
* `cordctl model delete <modelName> <id>` ... delete models.
* `cordctl model create <modelName> --set-json <json>` ... create a new model.
* `cordctl model sync <modelName> <id>` ... wait for a model to be synchronized.
+* `cordctl model setdirty <modelName> <id>` ... set a model dirty so it will be synchronized.
### Listing model types
@@ -95,6 +96,13 @@
Supported operators in the filters include `=`, `!=`, `>`, `<`, `>=`, `<=`.
+The core also permits models to be filtered based on state, and the `--state` argument can be used to filter based on a state. States include `all`, `dirty`, `deleted`, `dirtypol`, and `deletedpol`. `default` is a synonym for `all`. For example,
+
+```bash
+# List deleted ONOSApps
+cordctl model list ONOSApp --state deleted
+```
+
### Updating models
The `model update` command is a flexible way to update one or more models. The most basic syntax uses one or more model IDs. For example,
@@ -144,9 +152,9 @@
### Syncing Models
-All XOS operations are by nature asynchronous. When a model instance is created or updated, a synchronizer will typically run at a later time, enacting that model by applying its changes to some external component. After this is complete, the synchronizer updates the timestamps and other metadata to convey that the synchronization is complete.
+All XOS operations are by nature asynchronous. When a model instance is created or updated, a synchronizer will typically run at a later time, enacting that model by applying its changes to some external component. After this is complete, the synchronizer updates the timestamps and other metadata to convey that the synchronization is complete.
-Asynchronous operations are often inconvenient for test infrastructure, automation scripts, and even human operators. `cordctl` offers some features for synchronous behavior.
+Asynchronous operations are often inconvenient for test infrastructure, automation scripts, and even human operators. `cordctl` offers some features for synchronous behavior.
The first is the `model sync` command that can sync models based on ID or based on a filter. For example,
@@ -164,10 +172,38 @@
cordctl model create ONOSApp --sync --set-field name=myapp,app_id=org.opencord.myapp
```
+### Dirtying Models
+
+XOS determines when a model is dirty by comparing timestamps. Each model has an `updated` timestamp that indicates the last time the model was
+updated. If this timestamp is newer than the `enacted` or `policed` timestamps then sync steps or policies will be run respectively. There is
+no direct way to modify the `updated` timestamp via the API since timestamps are managed by the `XOS` core. `cordctl` provides the `setdirty` command to cause models to be dirtied without altering the other fields of the model. When `setdirty` is used, the `updated` timestamp will be set to the current time. The `setdirty` command may be used with either an ID or a filter or the `--all` flag.
+
+```bash
+# Set model dirty based on ID
+cordctl model setdirty ONOSApp 17
+
+# Set model dirty based on a field filter
+cordctl model setdirty ONOSApp --filter name=olt
+
+# Set all ONOSApp models dirty
+cordctl model setdirty ONOSApp --all
+```
+
+If you wish to query which models are dirty (for example, to verify that a previous `setdirty` worked as expected) then the `--state dirty` argument may be applied to the `model list` command. For example,
+
+```bash
+# Get a list of dirty ONOS Apps
+cordctl model list ONOSApp --state dirty
+```
+
+> Note: Not all models have syncsteps or policies implemented for them. Some models may implicitly cause related models to become
+> dirty. For example, dirtying the head of a service instance chain may cause the whole chain to be dirtied. This behavior is dependent on
+> the model. For models that do not implement syncsteps or policies and do not have the side-effect of dirtying related models, the `setdirty`
+> command has no practical value, and the models may remain in perpetual dirty state.
## Development Environment
-To run unit tests, `go-junit-report` and `gocover-obertura` tools must be installed. One way to do this is to install them with `go get`, and then ensure your `GOPATH` is part of your `PATH` (editing your `~/.profile` as necessary).
+To run unit tests, `go-junit-report` and `gocover-obertura` tools must be installed. One way to do this is to install them with `go get`, and then ensure your `GOPATH` is part of your `PATH` (editing your `~/.profile` as necessary).
```bash
go get -u github.com/jstemmer/go-junit-report
diff --git a/commands/models.go b/commands/models.go
index 81acb18..f95177e 100644
--- a/commands/models.go
+++ b/commands/models.go
@@ -19,8 +19,10 @@
import (
"context"
"fmt"
+ "github.com/fullstorydev/grpcurl"
flags "github.com/jessevdk/go-flags"
"github.com/jhump/protoreflect/dynamic"
+ "google.golang.org/grpc"
"sort"
"strings"
"time"
@@ -41,6 +43,7 @@
ShowFeedback bool `long:"showfeedback" description:"Show feedback fields in default output"`
ShowBookkeeping bool `long:"showbookkeeping" description:"Show bookkeeping fields in default output"`
Filter string `short:"f" long:"filter" description:"Comma-separated list of filters"`
+ State string `short:"s" long:"state" description:"Filter model state [DEFAULT | ALL | DIRTY | DELETED | DIRTYPOL | DELETEDPOL]"`
Args struct {
ModelName ModelNameString
} `positional-args:"yes" required:"yes"`
@@ -66,6 +69,7 @@
OutputOptions
Unbuffered bool `short:"u" long:"unbuffered" description:"Do not buffer console output and suppress default output processor"`
Filter string `short:"f" long:"filter" description:"Comma-separated list of filters"`
+ All bool `short:"a" long:"all" description:"Operate on all models"`
Args struct {
ModelName ModelNameString
} `positional-args:"yes" required:"yes"`
@@ -91,6 +95,7 @@
Unbuffered bool `short:"u" long:"unbuffered" description:"Do not buffer console output and suppress default output processor"`
Filter string `short:"f" long:"filter" description:"Comma-separated list of filters"`
SyncTimeout time.Duration `long:"synctimeout" default:"600s" description:"Timeout for synchronization"`
+ All bool `short:"a" long:"all" description:"Operate on all models"`
Args struct {
ModelName ModelNameString
} `positional-args:"yes" required:"yes"`
@@ -99,12 +104,26 @@
} `positional-args:"yes" required:"no"`
}
+type ModelSetDirty struct {
+ OutputOptions
+ Unbuffered bool `short:"u" long:"unbuffered" description:"Do not buffer console output and suppress default output processor"`
+ Filter string `short:"f" long:"filter" description:"Comma-separated list of filters"`
+ All bool `short:"a" long:"all" description:"Operate on all models"`
+ Args struct {
+ ModelName ModelNameString
+ } `positional-args:"yes" required:"yes"`
+ IDArgs struct {
+ ID []int32
+ } `positional-args:"yes" required:"no"`
+}
+
type ModelOpts struct {
- List ModelList `command:"list"`
- Update ModelUpdate `command:"update"`
- Delete ModelDelete `command:"delete"`
- Create ModelCreate `command:"create"`
- Sync ModelSync `command:"sync"`
+ List ModelList `command:"list"`
+ Update ModelUpdate `command:"update"`
+ Delete ModelDelete `command:"delete"`
+ Create ModelCreate `command:"create"`
+ Sync ModelSync `command:"sync"`
+ SetDirty ModelSetDirty `command:"setdirty"`
}
type ModelStatusOutputRow struct {
@@ -153,6 +172,81 @@
}
}
+// Convert a user-supplied state filter argument to the appropriate enum name
+func GetFilterKind(kindArg string) (string, error) {
+ kindMap := map[string]string{
+ "default": FILTER_DEFAULT,
+ "all": FILTER_ALL,
+ "dirty": FILTER_DIRTY,
+ "deleted": FILTER_DELETED,
+ "dirtypol": FILTER_DIRTYPOL,
+ "deletedpo": FILTER_DELETEDPOL,
+ }
+
+ // If no arg then use default
+ if kindArg == "" {
+ return kindMap["default"], nil
+ }
+
+ val, ok := kindMap[strings.ToLower(kindArg)]
+ if !ok {
+ return "", fmt.Errorf("Failed to understand model state %s", kindArg)
+ }
+
+ return val, nil
+}
+
+// Common processing for commands that take a modelname and a list of ids or a filter
+func GetIDList(conn *grpc.ClientConn, descriptor grpcurl.DescriptorSource, modelName string, ids []int32, filter string, all bool) ([]int32, error) {
+ err := CheckModelName(descriptor, modelName)
+ if err != nil {
+ return nil, err
+ }
+
+ // we require exactly one of ID, --filter, or --all
+ exclusiveCount := 0
+ if len(ids) > 0 {
+ exclusiveCount++
+ }
+ if filter != "" {
+ exclusiveCount++
+ }
+ if all {
+ exclusiveCount++
+ }
+
+ if (exclusiveCount == 0) || (exclusiveCount > 1) {
+ return nil, fmt.Errorf("Use either an ID, --filter, or --all to specify which models to operate on")
+ }
+
+ queries, err := CommaSeparatedQueryToMap(filter, true)
+ if err != nil {
+ return nil, err
+ }
+
+ if len(ids) > 0 {
+ // do nothing
+ } else {
+ models, err := ListOrFilterModels(context.Background(), conn, descriptor, modelName, FILTER_DEFAULT, queries)
+ if err != nil {
+ return nil, err
+ }
+ ids = make([]int32, len(models))
+ for i, model := range models {
+ ids[i] = model.GetFieldByName("id").(int32)
+ }
+ if len(ids) == 0 {
+ return nil, fmt.Errorf("Filter matches no objects")
+ } else if len(ids) > 1 {
+ if !Confirmf("Filter matches %d objects. Continue [y/n] ? ", len(models)) {
+ return nil, fmt.Errorf("Aborted by user")
+ }
+ }
+ }
+
+ return ids, nil
+}
+
func (options *ModelList) Execute(args []string) error {
conn, descriptor, err := InitReflectionClient()
if err != nil {
@@ -166,12 +260,17 @@
return err
}
+ filterKind, err := GetFilterKind(options.State)
+ if err != nil {
+ return err
+ }
+
queries, err := CommaSeparatedQueryToMap(options.Filter, true)
if err != nil {
return err
}
- models, err := ListOrFilterModels(context.Background(), conn, descriptor, string(options.Args.ModelName), queries)
+ models, err := ListOrFilterModels(context.Background(), conn, descriptor, string(options.Args.ModelName), filterKind, queries)
if err != nil {
return err
}
@@ -280,7 +379,7 @@
}
}
} else {
- models, err = ListOrFilterModels(context.Background(), conn, descriptor, modelName, queries)
+ models, err = ListOrFilterModels(context.Background(), conn, descriptor, modelName, FILTER_DEFAULT, queries)
if err != nil {
return err
}
@@ -347,43 +446,10 @@
defer conn.Close()
- err = CheckModelName(descriptor, string(options.Args.ModelName))
- if err != nil {
- return err
- }
-
- if (len(options.IDArgs.ID) == 0 && len(options.Filter) == 0) ||
- (len(options.IDArgs.ID) != 0 && len(options.Filter) != 0) {
- return fmt.Errorf("Use either an ID or a --filter to specify which models to delete")
- }
-
- queries, err := CommaSeparatedQueryToMap(options.Filter, true)
- if err != nil {
- return err
- }
-
modelName := string(options.Args.ModelName)
-
- var ids []int32
-
- if len(options.IDArgs.ID) > 0 {
- ids = options.IDArgs.ID
- } else {
- models, err := ListOrFilterModels(context.Background(), conn, descriptor, modelName, queries)
- if err != nil {
- return err
- }
- ids = make([]int32, len(models))
- for i, model := range models {
- ids[i] = model.GetFieldByName("id").(int32)
- }
- if len(ids) == 0 {
- return fmt.Errorf("Filter matches no objects")
- } else if len(ids) > 1 {
- if !Confirmf("Filter matches %d objects. Continue [y/n] ? ", len(models)) {
- return fmt.Errorf("Aborted by user")
- }
- }
+ ids, err := GetIDList(conn, descriptor, modelName, options.IDArgs.ID, options.Filter, options.All)
+ if err != nil {
+ return err
}
modelStatusOutput := InitModelStatusOutput(options.Unbuffered, len(ids))
@@ -466,43 +532,10 @@
defer conn.Close()
- err = CheckModelName(descriptor, string(options.Args.ModelName))
- if err != nil {
- return err
- }
-
- if (len(options.IDArgs.ID) == 0 && len(options.Filter) == 0) ||
- (len(options.IDArgs.ID) != 0 && len(options.Filter) != 0) {
- return fmt.Errorf("Use either an ID or a --filter to specify which models to sync")
- }
-
- queries, err := CommaSeparatedQueryToMap(options.Filter, true)
- if err != nil {
- return err
- }
-
modelName := string(options.Args.ModelName)
-
- var ids []int32
-
- if len(options.IDArgs.ID) > 0 {
- ids = options.IDArgs.ID
- } else {
- models, err := ListOrFilterModels(context.Background(), conn, descriptor, modelName, queries)
- if err != nil {
- return err
- }
- ids = make([]int32, len(models))
- for i, model := range models {
- ids[i] = model.GetFieldByName("id").(int32)
- }
- if len(ids) == 0 {
- return fmt.Errorf("Filter matches no objects")
- } else if len(ids) > 1 {
- if !Confirmf("Filter matches %d objects. Continue [y/n] ? ", len(models)) {
- return fmt.Errorf("Aborted by user")
- }
- }
+ ids, err := GetIDList(conn, descriptor, modelName, options.IDArgs.ID, options.Filter, options.All)
+ if err != nil {
+ return err
}
ctx, cancel := context.WithTimeout(context.Background(), options.SyncTimeout)
@@ -523,6 +556,34 @@
return nil
}
+func (options *ModelSetDirty) Execute(args []string) error {
+ conn, descriptor, err := InitReflectionClient()
+ if err != nil {
+ return err
+ }
+
+ defer conn.Close()
+
+ modelName := string(options.Args.ModelName)
+ ids, err := GetIDList(conn, descriptor, modelName, options.IDArgs.ID, options.Filter, options.All)
+ if err != nil {
+ return err
+ }
+
+ modelStatusOutput := InitModelStatusOutput(options.Unbuffered, len(ids))
+ for i, id := range ids {
+ updateMap := map[string]interface{}{"id": id}
+ err := UpdateModel(conn, descriptor, modelName, updateMap)
+ UpdateModelStatusOutput(&modelStatusOutput, i, id, "Dirtied", err, true)
+ }
+
+ if !options.Unbuffered {
+ FormatAndGenerateOutput(&options.OutputOptions, DEFAULT_SYNC_FORMAT, DEFAULT_SYNC_FORMAT, modelStatusOutput.Rows)
+ }
+
+ return nil
+}
+
func (modelName *ModelNameString) Complete(match string) []flags.Completion {
conn, descriptor, err := InitReflectionClient()
if err != nil {
diff --git a/commands/models_test.go b/commands/models_test.go
index 2b44f8f..c4f6c47 100644
--- a/commands/models_test.go
+++ b/commands/models_test.go
@@ -172,6 +172,49 @@
testutils.AssertJSONEqual(t, got.String(), expected)
}
+func TestModelListDirty(t *testing.T) {
+ // use `python -m json.tool` to pretty-print json
+ expected := `[
+ {
+ "controller_kind": "",
+ "controller_replica_count": 0,
+ "creator_id": 0,
+ "default_flavor_id": 0,
+ "default_image_id": 0,
+ "default_isolation": "",
+ "default_node_id": 0,
+ "description": "",
+ "enabled": false,
+ "exposed_ports": "",
+ "id": 2,
+ "max_instances": 0,
+ "mount_data_sets": "",
+ "name": "mockslice2",
+ "network": "",
+ "principal_id": 0,
+ "service_id": 0,
+ "site_id": 1,
+ "trust_domain_id": 0
+ }
+ ]`
+
+ got := new(bytes.Buffer)
+ OutputStream = got
+
+ var options ModelOpts
+ options.List.Args.ModelName = "Slice"
+ options.List.OutputAs = "json"
+ options.List.State = "dirty"
+ err := options.List.Execute([]string{})
+
+ if err != nil {
+ t.Errorf("%s: Received error %v", t.Name(), err)
+ return
+ }
+
+ testutils.AssertJSONEqual(t, got.String(), expected)
+}
+
func TestModelUpdate(t *testing.T) {
expected := `[{"id":1, "message":"Updated"}]`
@@ -374,3 +417,23 @@
testutils.AssertJSONEqual(t, got.String(), expected)
}
+
+func TestModelSetDirty(t *testing.T) {
+ expected := `[{"id":1, "message":"Dirtied"}]`
+
+ got := new(bytes.Buffer)
+ OutputStream = got
+
+ var options ModelOpts
+ options.SetDirty.Args.ModelName = "Slice"
+ options.SetDirty.OutputAs = "json"
+ options.SetDirty.IDArgs.ID = []int32{1}
+ err := options.SetDirty.Execute([]string{})
+
+ if err != nil {
+ t.Errorf("%s: Received error %v", t.Name(), err)
+ return
+ }
+
+ testutils.AssertJSONEqual(t, got.String(), expected)
+}
diff --git a/commands/orm.go b/commands/orm.go
index a817ea5..5938698 100644
--- a/commands/orm.go
+++ b/commands/orm.go
@@ -33,10 +33,22 @@
)
// Flags for calling the *WithRetry methods
-const GM_QUIET = 1
-const GM_UNTIL_FOUND = 2
-const GM_UNTIL_ENACTED = 4
-const GM_UNTIL_STATUS = 8
+const (
+ GM_QUIET = 1
+ GM_UNTIL_FOUND = 2
+ GM_UNTIL_ENACTED = 4
+ GM_UNTIL_STATUS = 8
+)
+
+// Valid choices for FilterModels `Kind` argument
+const (
+ FILTER_DEFAULT = "DEFAULT"
+ FILTER_ALL = "ALL"
+ FILTER_DIRTY = "SYNCHRONIZER_DIRTY_OBJECTS"
+ FILTER_DELETED = "SYNCHRONIZER_DELETED_OBJECTS"
+ FILTER_DIRTYPOL = "SYNCHRONIZER_DIRTY_POLICIES"
+ FILTER_DELETEDPOL = "SYNCHRONIZER_DELETED_POLICIES"
+)
type QueryEventHandler struct {
RpcEventHandler
@@ -451,7 +463,7 @@
// queries is a map of <field_name> to <operator><query>
// For example,
// map[string]string{"name": "==mysite"}
-func FilterModels(ctx context.Context, conn *grpc.ClientConn, descriptor grpcurl.DescriptorSource, modelName string, queries map[string]string) ([]*dynamic.Message, error) {
+func FilterModels(ctx context.Context, conn *grpc.ClientConn, descriptor grpcurl.DescriptorSource, modelName string, kind string, queries map[string]string) ([]*dynamic.Message, error) {
ctx, cancel := context.WithTimeout(ctx, GlobalConfig.Grpc.Timeout)
defer cancel()
@@ -472,7 +484,7 @@
},
Elements: queries,
Model: model_md,
- Kind: "DEFAULT",
+ Kind: kind,
}
err = grpcurl.InvokeRPC(ctx, descriptor, conn, "xos.xos.Filter"+modelName, headers, h, h.GetParams)
if err != nil {
@@ -497,17 +509,17 @@
}
// Call ListModels or FilterModels as appropriate
-func ListOrFilterModels(ctx context.Context, conn *grpc.ClientConn, descriptor grpcurl.DescriptorSource, modelName string, queries map[string]string) ([]*dynamic.Message, error) {
- if len(queries) == 0 {
+func ListOrFilterModels(ctx context.Context, conn *grpc.ClientConn, descriptor grpcurl.DescriptorSource, modelName string, kind string, queries map[string]string) ([]*dynamic.Message, error) {
+ if (len(queries) == 0) && (kind == FILTER_DEFAULT) {
return ListModels(ctx, conn, descriptor, modelName)
} else {
- return FilterModels(ctx, conn, descriptor, modelName, queries)
+ return FilterModels(ctx, conn, descriptor, modelName, kind, queries)
}
}
// Get a model from XOS given a fieldName/fieldValue
func FindModel(ctx context.Context, conn *grpc.ClientConn, descriptor grpcurl.DescriptorSource, modelName string, queries map[string]string) (*dynamic.Message, error) {
- models, err := FilterModels(ctx, conn, descriptor, modelName, queries)
+ models, err := FilterModels(ctx, conn, descriptor, modelName, FILTER_DEFAULT, queries)
if err != nil {
return nil, err
}
diff --git a/commands/orm_test.go b/commands/orm_test.go
index aa1b7c4..7ab30bf 100644
--- a/commands/orm_test.go
+++ b/commands/orm_test.go
@@ -161,7 +161,7 @@
qm := map[string]string{"id": "=1"}
- m, err := FilterModels(context.Background(), conn, descriptor, "Slice", qm)
+ m, err := FilterModels(context.Background(), conn, descriptor, "Slice", FILTER_DEFAULT, qm)
assert.Equal(t, err, nil)
assert.Equal(t, len(m), 1)
diff --git a/mock/data.json b/mock/data.json
index 4e1e7ee..fb168f6 100644
--- a/mock/data.json
+++ b/mock/data.json
@@ -27,6 +27,7 @@
"output": {
"id": 1,
"name": "mockslice1",
+ "description": "an up-to-date slice",
"site_id": 1,
"updated": 1234.0,
"enacted": 1234.0
@@ -38,6 +39,7 @@
"output": {
"id": 2,
"name": "mockslice2",
+ "description": "a dirty slice",
"site_id": 1,
"updated": 1234.0,
"enacted": 900.0
@@ -70,7 +72,8 @@
},
{
"method": "FilterSlice",
- "input": {"elements": [{"operator": 0, "name": "id", "iValue": 1}]},
+ "input": {"kind": 0,
+ "elements": [{"operator": 0, "name": "id", "iValue": 1}]},
"output": {
"items": [{
"id": 1,
@@ -81,7 +84,8 @@
},
{
"method": "FilterSlice",
- "input": {"elements": [{"operator": 0, "name": "name", "sValue": "mockslice2"}]},
+ "input": {"kind": 0,
+ "elements": [{"operator": 0, "name": "name", "sValue": "mockslice2"}]},
"output": {
"items": [{
"id": 2,
@@ -92,12 +96,24 @@
},
{
"method": "FilterSlice",
- "input": {"elements": [{"operator": 0, "name": "id", "iValue": 77}]},
+ "input": {"kind": 0,
+ "elements": [{"operator": 0, "name": "id", "iValue": 77}]},
"output": {
"items": []
}
},
{
+ "method": "FilterSlice",
+ "input": {"kind": 2},
+ "output": {
+ "items": [{
+ "id": 2,
+ "name": "mockslice2",
+ "site_id": 1
+ }]
+ }
+ },
+ {
"method": "UpdateSlice",
"input": {"id": 1, "name": "mockslice1_newname"},
"output": {
@@ -108,6 +124,15 @@
},
{
"method": "UpdateSlice",
+ "input": { "id": 1 },
+ "output": {
+ "id": 1,
+ "name": "mockslice1",
+ "site_id": 1
+ }
+ },
+ {
+ "method": "UpdateSlice",
"input": { "id": 77, "name": "mockslice1_newname"},
"error": { "code": 2, "message": "Slice matching query does not exist."}
},