bgpd, doc, lib, zebra: nexthop-tracking in zebra

0. Introduction

This is the design specification for next hop tracking feature in
Quagga.

1. Background

Recursive routes are of the form:

   p/m --> n
  [Ex: 1.1.0.0/16 --> 2.2.2.2]

where 'n' itself is resolved through another route as follows:

   p2/m --> h, interface
  [Ex: 2.2.2.0/24 --> 3.3.3.3, eth0]

Usually, BGP routes are recursive in nature and BGP nexthops get
resolved through an IGP route. IGP usually adds its routes pointing to
an interface (these are called non-recursive routes).

When BGP receives a recursive route from a peer, it needs to validate
the nexthop. The path is marked valid or invalid based on the
reachability status of the nexthop.  Nexthop validation is also
important for BGP decision process as the metric to reach the nexthop
is a parameter to best path selection process.

As it goes with routing, this is a dynamic process. Route to the
nexthop can change. The nexthop can become unreachable or
reachable. In the current BGP implementation, the nexthop validation
is done periodically in the scanner run. The default scanner run
interval is one minute. Every minute, the scanner task walks the
entire BGP table. It checks the validity of each nexthop with Zebra
(the routing table manager) through a request and response message
exchange between BGP and Zebra process. BGP process is blocked for
that duration. The mechanism has two major drawbacks:

(1) The scanner task runs to completion. That can potentially starve
    the other tasks for long periods of time, based on the BGP table
    size and number of nexthops.

(2) Convergence around routing changes that affect the nexthops can be
    long (around a minute with the default intervals). The interval
    can be shortened to achieve faster reaction time, but it makes the
    first problem worse, with the scanner task consuming most of the
    CPU resources.

"Next hop tracking" feature makes this process event-driven. It
eliminates periodic nexthop validation and introduces an asynchronous
communication path between BGP and Zebra for route change notifications
that can then be acted upon.

2. Goal

Stating the obvious, the main goal is to remove the two limitations we
discussed in the previous section. The goals, in a constructive tone,
are the following:

- fairness: the scanner run should not consume an unjustly high amount
  of CPU time. This should give an overall good performance and
  response time to other events (route changes, session events,
  IO/user interface).

- convergence: BGP must react to nexthop changes instantly and provide
  sub-second convergence. This may involve diverting the routes from
  one nexthop to another.

3. Overview of the changes

The changes are in both BGP and Zebra modules.  The short summary is
the following:

- Zebra implements a registration mechanism by which clients can
   register for next hop notification. Consequently, it maintains a
   separate table, per (VRF, AF) pair, of next hops and interested
   client-list per next hop.

- When the main routing table changes in Zebra, it evaluates the next
   hop table: for each next hop, it checks if the route table
   modifications have changed its state. If so, it notifies the
   interested clients.

- BGP is one such client. It registers the next hops corresponding to
   all of its received routes/paths. It also threads the paths against
   each nexthop structure.

- When BGP receives a next hop notification from Zebra, it walks the
   corresponding path list. It makes them valid or invalid depending
   on the next hop notification. It then re-computes best path for the
   corresponding destination. This may result in re-announcing those
   destinations to peers.

4. Design

4.1. Modules

The core design introduces an "nht" (next hop tracking) module in BGP
and "rnh" (recursive nexthop) module in Zebra. The "nht" module
provides the following APIs:

bgp_find_or_add_nexthop() : find or add a nexthop in BGP nexthop table
bgp_find_nexthop() : find a nexthop in BGP nexthop table
bgp_parse_nexthop_update() : parse a nexthop update message coming
                              from zebra

The "rnh" module provides the following APIs:

zebra_add_rnh() : add a recursive nexthop
zebra_delete_rnh() : delete a recursive nexthop
zebra_lookup_rnh() : lookup a recursive nexthop

zebra_add_rnh_client() : register a client for nexthop notifications
                         against a recursive nexthop

zebra_remove_rnh_client(): remove the client registration for a
                            recursive nexthop

zebra_evaluate_rnh_table(): (re)evaluate the recursive nexthop table
                            (most probably because the main routing
                            table has changed).

zebra_cleanup_rnh_client(): Cleanup a client from the "rnh" module
                            data structures (most probably because the
                            client is going away).

4.2. Control flow

The next hop registration control flow is the following:

<====      BGP Process       ====>|<====      Zebra Process      ====>
                                  |
receive module     nht module     |  zserv module        rnh module
----------------------------------------------------------------------
              |                   |                  |
bgp_update_   |                   |                  |
      main()  | bgp_find_or_add_  |                  |
              |        nexthop()  |                  |
              |                   |                  |
              |                   | zserv_nexthop_   |
              |                   |       register() |
              |                   |                  | zebra_add_rnh()
              |                   |                  |

The next hop notification control flow is the following:

<====     Zebra Process    ====>|<====      BGP Process       ====>
                                |
rib module         rnh module   |     zebra module        nht module
----------------------------------------------------------------------
              |                 |                   |
meta_queue_   |                 |                   |
    process() | zebra_evaluate_ |                   |
              |     rnh_table() |                   |
              |                 |                   |
              |                 | bgp_read_nexthop_ |
              |                 |          update() |
              |                 |                   | bgp_parse_
              |                 |                   | nexthop_update()
              |                 |                   |

4.3. zclient message format

ZEBRA_NEXTHOP_REGISTER and ZEBRA_NEXTHOP_UNREGISTER messages are
encoded in the following way:

/*
 *     0                   1                   2                   3
 *  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 * |     AF                        |  prefix len   |
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 * .      Nexthop prefix                                           .
 * .                                                               .
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 * .                                                               .
 * .                                                               .
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 * |     AF                        |  prefix len   |
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 * .      Nexthop prefix                                           .
 * .                                                               .
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 */

ZEBRA_NEXTHOP_UPDATE message is encoded as follows:

/*
 *     0                   1                   2                   3
 *  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 * |     AF                        |  prefix len   |
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 * .      Nexthop prefix getting resolved                          .
 * .                                                               .
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 * |        metric                                                 |
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 * |  #nexthops    |
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 * | nexthop type  |
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 * .      resolving Nexthop details                                .
 * .                                                               .
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 * .                                                               .
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 * | nexthop type  |
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 * .      resolving Nexthop details                                .
 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 */

4.4. BGP data structure

Legend:

/\   struct bgp_node: a BGP destination/route/prefix
\/

[ ]  struct bgp_info: a BGP path (e.g. route received from a peer)

 _
(_)  struct bgp_nexthop_cache: a BGP nexthop

   /\         NULL
   \/--+        ^
       |        :
       +--[ ]--[ ]--[ ]--> NULL
   /\           :
   \/--+        :
       |        :
       +--[ ]--[ ]--> NULL
                :
  _             :
 (_).............

4.5. Zebra data structure

rnh table:

           O
          / \
         O   O
            / \
           O   O

        struct rnh
        {
          u_char flags;
          struct rib *state;
          struct list *client_list;
          struct route_node *node;
        };

5. User interface changes

quagga# show ip nht
3.3.3.3
 resolved via kernel
 via 11.0.0.6, swp1
 Client list: bgp(fd 12)
11.0.0.10
 resolved via connected
 is directly connected, swp2
 Client list: bgp(fd 12)
11.0.0.18
 resolved via connected
 is directly connected, swp4
 Client list: bgp(fd 12)
11.11.11.11
 resolved via kernel
 via 10.0.1.2, eth0
 Client list: bgp(fd 12)

quagga# show ip bgp nexthop
Current BGP nexthop cache:
 3.3.3.3 valid [IGP metric 0], #paths 3
  Last update: Wed Oct 16 04:43:49 2013

 11.0.0.10 valid [IGP metric 1], #paths 1
  Last update: Wed Oct 16 04:43:51 2013

 11.0.0.18 valid [IGP metric 1], #paths 2
  Last update: Wed Oct 16 04:43:47 2013

 11.11.11.11 valid [IGP metric 0], #paths 1
  Last update: Wed Oct 16 04:43:47 2013

quagga# show ipv6 nht
quagga# show ip bgp nexthop detail

quagga# debug bgp nht
quagga# debug zebra nht

6. Sample test cases

     r2----r3
    /  \  /
  r1----r4

- Verify that a change in IGP cost triggers NHT
  + shutdown the r1-r4 and r2-r4 links
  + no shut the r1-r4 and r2-r4 links and wait for OSPF to come back
    up
  + We should be back to the original nexthop via r4 now
- Verify that a NH becoming unreachable triggers NHT
  + Shutdown all links to r4
- Verify that a NH becoming reachable triggers NHT
  + no shut all links to r4

7. Future work

- route-policy for next hop validation (e.g. ignore default route)
- damping for rapid next hop changes
- prioritized handling of nexthop changes ((un)reachability vs. metric
  changes)
- handling recursion loop, e.g.
   11.11.11.11/32 -> 12.12.12.12
   12.12.12.12/32 -> 11.11.11.11
   11.0.0.0/8 -> <interface>
- better statistics
Addresses upstream comments.

"show ip bgp nexthop detail" couldn't display multiple NHs due to a bug.
Fix that.

Fix reference counts for the nexthop cache entries

Signed-off-by: Pradosh Mohapatra <pmohapat@cumulusnetworks.com>
Signed-off-by: Daniel Walton <dwalton@cumulusnetworks.com>
Signed-off-by: Dinesh Dutt <ddutt@cumulusnetworks.com>
Signed-off-by: Donald Sharp <sharpd@cumulusnetworks.com>
Signed-off-by: Vivek Venkatraman <vivek@cumulusnetworks.com>

Fix reference counts for the nexthop cache entries.

Signed-off-by: Vivek Venkatraman <vivek@cumulusnetworks.com>

Edited-by: Paul Jakma <paul.jakma@hpe.com>
- Fix nexthop_ipv6_add defs in rib.h not having been modified with rib_ prefix.
- Remove rib_lookup_and_pushup, appears not to be used except for
  !HAVE_NETLINK && HAVE_STRUCT_IFALIASREQ case of ioctl.c::if_set_prefix,
  so it's not being used at all on platform with most testing of RIB.
diff --git a/bgpd/bgp_nht.c b/bgpd/bgp_nht.c
new file mode 100644
index 0000000..21e1411
--- /dev/null
+++ b/bgpd/bgp_nht.c
@@ -0,0 +1,477 @@
+/* BGP Nexthop tracking
+ * Copyright (C) 2013 Cumulus Networks, Inc.
+ *
+ * This file is part of GNU Zebra.
+ *
+ * GNU Zebra is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; either version 2, or (at your option) any
+ * later version.
+ *
+ * GNU Zebra is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with GNU Zebra; see the file COPYING.  If not, write to the Free
+ * Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA
+ * 02111-1307, USA.
+ */
+
+#include <zebra.h>
+
+#include "command.h"
+#include "thread.h"
+#include "prefix.h"
+#include "zclient.h"
+#include "stream.h"
+#include "network.h"
+#include "log.h"
+#include "memory.h"
+#include "nexthop.h"
+#include "filter.h"
+
+#include "bgpd/bgpd.h"
+#include "bgpd/bgp_table.h"
+#include "bgpd/bgp_route.h"
+#include "bgpd/bgp_attr.h"
+#include "bgpd/bgp_nexthop.h"
+#include "bgpd/bgp_debug.h"
+#include "bgpd/bgp_nht.h"
+
+extern struct zclient *zclient;
+extern struct bgp_table *bgp_nexthop_cache_table[AFI_MAX];
+
+static void register_nexthop(struct bgp_nexthop_cache *bnc);
+static void unregister_nexthop (struct bgp_nexthop_cache *bnc);
+static void evaluate_paths(struct bgp_nexthop_cache *bnc);
+static int make_prefix(int afi, struct bgp_info *ri, struct prefix *p);
+static void path_nh_map(struct bgp_info *path, struct bgp_nexthop_cache *bnc,
+			int keep);
+
+int
+bgp_find_nexthop (struct bgp_info *path, int *changed, int *metricchanged)
+{
+  struct bgp_nexthop_cache *bnc = path->nexthop;
+
+  if (!bnc)
+    return 0;
+
+  if (changed)
+    *changed = CHECK_FLAG(bnc->change_flags, BGP_NEXTHOP_CHANGED);
+
+  if (metricchanged)
+    *metricchanged = CHECK_FLAG(bnc->change_flags,
+				BGP_NEXTHOP_METRIC_CHANGED);
+
+  return (CHECK_FLAG(bnc->flags, BGP_NEXTHOP_VALID));
+}
+
+void
+bgp_unlink_nexthop (struct bgp_info *path)
+{
+  struct bgp_nexthop_cache *bnc = path->nexthop;
+
+  if (!bnc)
+    return;
+
+  path_nh_map(path, NULL, 0);
+
+  if (LIST_EMPTY(&(bnc->paths)))
+    {
+      if (BGP_DEBUG(nht, NHT))
+	{
+	  char buf[INET6_ADDRSTRLEN];
+	  zlog_debug("bgp_unlink_nexthop: freeing bnc %s",
+		     bnc_str(bnc, buf, INET6_ADDRSTRLEN));
+	}
+      unregister_nexthop(bnc);
+      bnc->node->info = NULL;
+      bgp_unlock_node(bnc->node);
+      bnc_free(bnc);
+    }
+}
+
+int
+bgp_find_or_add_nexthop (afi_t afi, struct bgp_info *ri, int *changed,
+			 int *metricchanged)
+{
+  struct bgp_node *rn;
+  struct bgp_nexthop_cache *bnc;
+  struct prefix p;
+
+  if (make_prefix(afi, ri, &p) < 0)
+    return 1;
+  rn = bgp_node_get (bgp_nexthop_cache_table[afi], &p);
+
+  if (!rn->info)
+    {
+      bnc = bnc_new();
+      rn->info = bnc;
+      bnc->node = rn;
+      bgp_lock_node(rn);
+      register_nexthop(bnc);
+    }
+  bnc = rn->info;
+  bgp_unlock_node (rn);
+  path_nh_map(ri, bnc, 1);
+
+  if (changed)
+    *changed = CHECK_FLAG(bnc->change_flags, BGP_NEXTHOP_CHANGED);
+
+  if (metricchanged)
+    *metricchanged = CHECK_FLAG(bnc->change_flags,
+				BGP_NEXTHOP_METRIC_CHANGED);
+
+  if (CHECK_FLAG(bnc->flags, BGP_NEXTHOP_VALID) && bnc->metric)
+    (bgp_info_extra_get(ri))->igpmetric = bnc->metric;
+  else if (ri->extra)
+    ri->extra->igpmetric = 0;
+
+  return (CHECK_FLAG(bnc->flags, BGP_NEXTHOP_VALID));
+}
+
+void
+bgp_parse_nexthop_update (void)
+{
+  struct stream *s;
+  struct bgp_node *rn;
+  struct bgp_nexthop_cache *bnc;
+  struct nexthop *nexthop;
+  struct nexthop *oldnh;
+  struct nexthop *nhlist_head = NULL;
+  struct nexthop *nhlist_tail = NULL;
+  uint32_t metric;
+  u_char nexthop_num;
+  struct prefix p;
+  int i;
+
+  s = zclient->ibuf;
+
+  memset(&p, 0, sizeof(struct prefix));
+  p.family = stream_getw(s);
+  p.prefixlen = stream_getc(s);
+  switch (p.family)
+    {
+    case AF_INET:
+      p.u.prefix4.s_addr = stream_get_ipv4 (s);
+      break;
+    case AF_INET6:
+      stream_get(&p.u.prefix6, s, 16);
+      break;
+    default:
+      break;
+    }
+
+  rn = bgp_node_lookup(bgp_nexthop_cache_table[family2afi(p.family)], &p);
+  if (!rn || !rn->info)
+    {
+      if (BGP_DEBUG(nht, NHT))
+	{
+	  char buf[INET6_ADDRSTRLEN];
+	  prefix2str(&p, buf, INET6_ADDRSTRLEN);
+	  zlog_debug("parse nexthop update(%s): rn not found", buf);
+	}
+      if (rn)
+        bgp_unlock_node (rn);
+      return;
+    }
+
+  bnc = rn->info;
+  bgp_unlock_node (rn);
+  bnc->last_update = bgp_clock();
+  bnc->change_flags = 0;
+  metric = stream_getl (s);
+  nexthop_num = stream_getc (s);
+
+  /* debug print the input */
+  if (BGP_DEBUG(nht, NHT))
+    {
+      char buf[INET6_ADDRSTRLEN];
+      prefix2str(&p, buf, INET6_ADDRSTRLEN);
+      zlog_debug("parse nexthop update(%s): metric=%d, #nexthop=%d", buf,
+		 metric, nexthop_num);
+    }
+
+  if (metric != bnc->metric)
+    bnc->change_flags |= BGP_NEXTHOP_METRIC_CHANGED;
+
+  if(nexthop_num != bnc->nexthop_num)
+    bnc->change_flags |= BGP_NEXTHOP_CHANGED;
+
+  if (nexthop_num)
+    {
+      bnc->flags |= BGP_NEXTHOP_VALID;
+      bnc->metric = metric;
+      bnc->nexthop_num = nexthop_num;
+
+      for (i = 0; i < nexthop_num; i++)
+	{
+	  nexthop = nexthop_new();
+	  nexthop->type = stream_getc (s);
+	  switch (nexthop->type)
+	    {
+	    case ZEBRA_NEXTHOP_IPV4:
+	      nexthop->gate.ipv4.s_addr = stream_get_ipv4 (s);
+	      break;
+	    case ZEBRA_NEXTHOP_IFINDEX:
+	    case ZEBRA_NEXTHOP_IFNAME:
+	      nexthop->ifindex = stream_getl (s);
+	      break;
+            case ZEBRA_NEXTHOP_IPV4_IFINDEX:
+	    case ZEBRA_NEXTHOP_IPV4_IFNAME:
+	      nexthop->gate.ipv4.s_addr = stream_get_ipv4 (s);
+	      nexthop->ifindex = stream_getl (s);
+	      break;
+#ifdef HAVE_IPV6
+            case ZEBRA_NEXTHOP_IPV6:
+	      stream_get (&nexthop->gate.ipv6, s, 16);
+	      break;
+            case ZEBRA_NEXTHOP_IPV6_IFINDEX:
+	    case ZEBRA_NEXTHOP_IPV6_IFNAME:
+	      stream_get (&nexthop->gate.ipv6, s, 16);
+	      nexthop->ifindex = stream_getl (s);
+	      break;
+#endif
+            default:
+              /* do nothing */
+              break;
+	    }
+
+	  if (nhlist_tail)
+	    {
+	      nhlist_tail->next = nexthop;
+	      nhlist_tail = nexthop;
+	    }
+	  else
+	    {
+	      nhlist_tail = nexthop;
+	      nhlist_head = nexthop;
+	    }
+
+	  /* No need to evaluate the nexthop if we have already determined
+	   * that there has been a change.
+	   */
+	  if (bnc->change_flags & BGP_NEXTHOP_CHANGED)
+	    continue;
+
+	  for (oldnh = bnc->nexthop; oldnh; oldnh = oldnh->next)
+	      if (nexthop_same_no_recurse(oldnh, nexthop))
+		  break;
+
+	  if (!oldnh)
+	    bnc->change_flags |= BGP_NEXTHOP_CHANGED;
+	}
+      bnc_nexthop_free(bnc);
+      bnc->nexthop = nhlist_head;
+    }
+  else
+    {
+      bnc->flags &= ~BGP_NEXTHOP_VALID;
+      bnc_nexthop_free(bnc);
+      bnc->nexthop = NULL;
+    }
+
+  evaluate_paths(bnc);
+}
+
+/**
+ * make_prefix - make a prefix structure from the path (essentially
+ * path's node.
+ */
+static int
+make_prefix (int afi, struct bgp_info *ri, struct prefix *p)
+{
+  memset (p, 0, sizeof (struct prefix));
+  switch (afi)
+    {
+    case AFI_IP:
+      p->family = AF_INET;
+      p->prefixlen = IPV4_MAX_BITLEN;
+      p->u.prefix4 = ri->attr->nexthop;
+      break;
+#ifdef HAVE_IPV6
+    case AFI_IP6:
+      if (ri->attr->extra->mp_nexthop_len != 16
+	  || IN6_IS_ADDR_LINKLOCAL (&ri->attr->extra->mp_nexthop_global))
+	return -1;
+
+      p->family = AF_INET6;
+      p->prefixlen = IPV6_MAX_BITLEN;
+      p->u.prefix6 = ri->attr->extra->mp_nexthop_global;
+      break;
+#endif
+    default:
+      break;
+    }
+  return 0;
+}
+
+/**
+ * sendmsg_nexthop -- Format and send a nexthop register/Unregister
+ *   command to Zebra.
+ * ARGUMENTS:
+ *   struct bgp_nexthop_cache *bnc -- the nexthop structure.
+ *   int command -- either ZEBRA_NEXTHOP_REGISTER or ZEBRA_NEXTHOP_UNREGISTER
+ * RETURNS:
+ *   void.
+ */
+static void
+sendmsg_nexthop (struct bgp_nexthop_cache *bnc, int command)
+{
+  struct stream *s;
+  struct prefix *p;
+  int ret;
+
+  /* Check socket. */
+  if (!zclient || zclient->sock < 0)
+    return;
+
+  p = &(bnc->node->p);
+  s = zclient->obuf;
+  stream_reset (s);
+  zclient_create_header (s, command, VRF_DEFAULT);
+  stream_putw(s, PREFIX_FAMILY(p));
+  stream_putc(s, p->prefixlen);
+  switch (PREFIX_FAMILY(p))
+    {
+    case AF_INET:
+      stream_put_in_addr (s, &p->u.prefix4);
+      break;
+#ifdef HAVE_IPV6
+    case AF_INET6:
+      stream_put(s, &(p->u.prefix6), 16);
+      break;
+#endif
+    default:
+      break;
+    }
+  stream_putw_at (s, 0, stream_get_endp (s));
+
+  ret = zclient_send_message(zclient);
+  /* TBD: handle the failure */
+  if (ret < 0)
+    zlog_warn("sendmsg_nexthop: zclient_send_message() failed");
+  return;
+}
+
+/**
+ * register_nexthop - register a nexthop with Zebra for notification
+ *    when the route to the nexthop changes.
+ * ARGUMENTS:
+ *   struct bgp_nexthop_cache *bnc -- the nexthop structure.
+ * RETURNS:
+ *   void.
+ */
+static void
+register_nexthop (struct bgp_nexthop_cache *bnc)
+{
+  /* Check if we have already registered */
+  if (bnc->flags & BGP_NEXTHOP_REGISTERED)
+    return;
+  sendmsg_nexthop(bnc, ZEBRA_NEXTHOP_REGISTER);
+  SET_FLAG(bnc->flags, BGP_NEXTHOP_REGISTERED);
+}
+
+/**
+ * unregister_nexthop -- Unregister the nexthop from Zebra.
+ * ARGUMENTS:
+ *   struct bgp_nexthop_cache *bnc -- the nexthop structure.
+ * RETURNS:
+ *   void.
+ */
+static void
+unregister_nexthop (struct bgp_nexthop_cache *bnc)
+{
+  /* Check if we have already registered */
+  if (!CHECK_FLAG(bnc->flags, BGP_NEXTHOP_REGISTERED))
+    return;
+
+  sendmsg_nexthop(bnc, ZEBRA_NEXTHOP_UNREGISTER);
+  UNSET_FLAG(bnc->flags, BGP_NEXTHOP_REGISTERED);
+}
+
+/**
+ * evaluate_paths - Evaluate the paths/nets associated with a nexthop.
+ * ARGUMENTS:
+ *   struct bgp_nexthop_cache *bnc -- the nexthop structure.
+ * RETURNS:
+ *   void.
+ */
+static void
+evaluate_paths (struct bgp_nexthop_cache *bnc)
+{
+  struct bgp_node *rn;
+  struct bgp_info *path;
+  struct bgp *bgp = bgp_get_default();
+  int afi;
+
+  LIST_FOREACH(path, &(bnc->paths), nh_thread)
+    {
+      if (!(path->type == ZEBRA_ROUTE_BGP &&
+	    path->sub_type == BGP_ROUTE_NORMAL))
+	continue;
+
+      rn = path->net;
+      afi = family2afi(rn->p.family);
+
+      /* Path becomes valid/invalid depending on whether the nexthop
+       * reachable/unreachable.
+       */
+      if ((CHECK_FLAG(path->flags, BGP_INFO_VALID) ? 1 : 0) !=
+	  (CHECK_FLAG(bnc->flags, BGP_NEXTHOP_VALID) ? 1 : 0))
+	{
+	  if (CHECK_FLAG (path->flags, BGP_INFO_VALID))
+	    {
+	      bgp_aggregate_decrement (bgp, &rn->p, path,
+				       afi, SAFI_UNICAST);
+	      bgp_info_unset_flag (rn, path, BGP_INFO_VALID);
+	    }
+	  else
+	    {
+	      bgp_info_set_flag (rn, path, BGP_INFO_VALID);
+	      bgp_aggregate_increment (bgp, &rn->p, path,
+				       afi, SAFI_UNICAST);
+	    }
+	}
+
+      /* Copy the metric to the path. Will be used for bestpath computation */
+      if (CHECK_FLAG(bnc->flags, BGP_NEXTHOP_VALID) && bnc->metric)
+	(bgp_info_extra_get(path))->igpmetric = bnc->metric;
+      else if (path->extra)
+	path->extra->igpmetric = 0;
+
+      if (CHECK_FLAG(bnc->flags, BGP_NEXTHOP_METRIC_CHANGED) ||
+	  CHECK_FLAG(bnc->flags, BGP_NEXTHOP_CHANGED))
+	SET_FLAG(path->flags, BGP_INFO_IGP_CHANGED);
+
+      bgp_process(bgp, rn, afi, SAFI_UNICAST);
+    }
+  RESET_FLAG(bnc->change_flags);
+}
+
+/**
+ * path_nh_map - make or break path-to-nexthop association.
+ * ARGUMENTS:
+ *   path - pointer to the path structure
+ *   bnc - pointer to the nexthop structure
+ *   make - if set, make the association. if unset, just break the existing
+ *          association.
+ */
+static void
+path_nh_map (struct bgp_info *path, struct bgp_nexthop_cache *bnc, int make)
+{
+  if (path->nexthop)
+    {
+      LIST_REMOVE(path, nh_thread);
+      path->nexthop->path_count--;
+      path->nexthop = NULL;
+    }
+  if (make)
+    {
+      LIST_INSERT_HEAD(&(bnc->paths), path, nh_thread);
+      path->nexthop = bnc;
+      path->nexthop->path_count++;
+    }
+}