Agenda

• HA in OpenStack's control plane
  • Brief summary
  • Questions for the Pacemaker community (you!)
• HA in OpenStack's compute plane
  • Brief summary
  • Questions for the Pacemaker community (you!)
• General questions for the Pacemaker community

L3 HA in Neutron (networking service)

L3 HA in Neutron — analysis

• neutron HA is tricky, but out of the scope of this talk
• See https://ethercalc.openstack.org/Pike-Neutron-L3-HA for analysis of various failure modes

Other caveats

• cinder-volume (block storage backend) active-active support still under development
• General phobia of Pacemaker in some parts of the OpenStack community
  • So keepalived often used instead for VIPs

HAProxy and VIPs

Red Hat OpenStack Platform control plane

Pacemaker manages:

• Galera and Redis (multi-state)
• RabbitMQ (cloned)

Active-active API services now controlled by systemd

• systemd does auto-restart on crash
• Simplifies cluster
• Prepares for containerized future

Quick poll

How's control plane HA going for you so far?

Questions relating to control plane

• Shouldn't we use an active-active HAProxy architecture?
• How should we handle stateless active-active API services?
• Should we tackle Pacemaker phobia? If so, how?
• Should we start moving resources out of core cluster onto remotes?
• Does Pacemaker need better maintenance capabilities?
• How do we handle multi-site clouds?

Fixing the HAProxy bottleneck

Standard active-passive HAProxy architecture:

• One VIP
• One HAProxy always takes all the load

Instead, why not have:

• One VIP per HAProxy node
• VIPs float as before
• DNS round-robin between them

Any downsides?

How should we handle API services?

Option 1: Use OCF RAs

https://launchpad.net/openstack-resource-agents

• Pros
  • Fairly robust functional monitoring (in theory)
• Cons
  • Duplicates a lot of logic / data already packaged by vendor
    • location of binary / pid / config file, start parameters, uid / guid, …
  • Layer boundary violation — requires knowledge of packaging
  • Monitoring logic probably duplicates probes from "real" monitoring systems (Nagios, Sensu, …)
  • Outdated assumptions (e.g. permanent DB connection)
  • Maintained by me

How should we handle API services?

Option 2: Pacemaker systemd: resources

Pacemaker auto-restarts service on crash

• Pros
  • Fairly easy to set up cluster
  • Poor man's failure notifications for free via Pacemaker UIs
• Cons
  • Does not handle malfunctioning services, only crashing ones

How should we handle API services?

Option 3: systemd only, no Pacemaker, as per Red Hat

• http://blog.clusterlabs.org/blog/2016/next-openstack-ha-arch

• systemd auto-restarts service on crash

• Pros

  • Simplifies cluster
  • Prepares for containerized future
• Cons
  • Assumes all services can robustly tolerate dependencies going down
  • Failures not surfaced via UI → requires separate monitoring / alerting component
  • Does not handle malfunctioning services, only crashing ones

How should we handle API services?

Option 4: Enhance systemd to allow health checks

• Pros
  • Eliminates all duplication
  • Cluster configuration is trivial
• Cons
  • Requires systemd development

How should we handle API services?

Option 5: OCF RAs wrapping systemctl / service(8)

• start / stop / status delegated to systemd

• monitor implemented in OCF RA

  • (or better, delegate to script reusable by other monitoring frameworks)

• Pros

  • Eliminates most duplication between RA and package
  • Clean separation of concerns (monitoring vs. control)
• Cons
  • Cluster configuration remains relatively complex
  • Have to poll; can't interact with systemd via dbus signalling
  • Would have to duplicate Pacemaker's logic for dealing with systemd quirks

How should we handle API services?

Option 6: enhance Pacemaker to support systemd: / ocf: hybrids

• start / stop / status via systemd

• monitor via OCF RA

  • (or better, delegate to script reusable by other monitoring frameworks)

• Pros

  • Eliminates most duplication between RA and package
  • Clean separation of concerns (monitoring vs. control)
• Cons
  • Cluster configuration remains relatively complex
  • More work for Ken … actually, not really!

How should we handle API services?

Option 7: use undocumented container meta-attribute

Pair systemd resource with monitor-only OCF RA

• Pros
  • Eliminates all duplication
  • No modification to Pacemaker or systemd required
• Cons
  • Cluster configuration remains relatively complex

Should we start moving resources out of core cluster onto remotes?

http://blog.clusterlabs.org/blog/2016/composable-openstack-ha

• Pros:
  • much more flexible architecture
  • avoids need for multiple clusters
  • avoids problems with cross-cluster ordering
• Cons
  • liveness check reduced to depending on single TCP connection?

Can we auto-promote remotes to core to maintain quorum?

Resource maintenance

• Sometimes need to restart services (e.g. reload config)
• Can't put a single resource on a single node in maintenance mode
• Can only put a whole node or a whole service in maintenance mode

Can we do better? Do we need to?

How do we handle multi-site clouds?

• e.g. multiple regions, shared Keystone
• Keystone reads can be distributed across sites
• Keystone writes are more difficult
  • Need to ensure they get directed to one site
  • Can do this with HAProxy
• booth is obvious candidate for deciding master site

NovaCompute / NovaEvacuate OCF agents

Pros

• Has been production-ready for a "long" time
• Commercial support available
• OCF RAs upstream in openstack-resource-agents repo (sort of)

Cons

• Known limitations (not bugs):
  • Only handles failure of compute node, not of VMs, or nova-compute
  • Some corner cases still problematic, e.g. if nova fails during recovery

What else is missing?

Design goal: Operability

Design goal: Configurability

Different cloud operators will want to support different SLAs with different workflows:

• Choose which VMs to auto-resurrect
• Reserve some hypervisor hosts for recovery
• Retry thresholds on recovery of processes and VM instances
• Configurable workflows
  • notify cloud operator
  • notify external monitoring system (Vitrage, Nagios, Sensu, …)
  • fence
  • request external service to perform some recovery (resurrect VMs)

Design goal: Intelligent, Context-aware Recovery

In some failure scenarios, VMs are still perfectly healthy but unmanageable.

Should they be automatically killed? Depends on the workload.

Design goal: Intelligent, Context-aware Recovery (2)

• Fault deduplication (via Vitrage)
• Set host to maintenance mode until recovery is complete
• Respect ephemeral storage boundaries where applicable

Design goal: Performance

• Quick response to failures
  • Push not pull — don't poll
  • Notifications on message bus
• Fault prioritization / pre-emption
  • e.g. host faults pre-empt instance faults
  • Pre-emption is visible to operators

Upgradability

Masakari to the rescue!

Modular architecture

• Key areas identified in previous Design Summits:
  • Host monitoring / recovery
  • VM monitoring / recovery
  • Process monitoring / recovery
• Agreed to split into independent components
• https://etherpad.openstack.org/p/newton-instance-ha

Specs

openstack-resource-agents-specs repository

• Host monitoring
• Host recovery
• VM monitoring
• VM recovery
• libvirtd OCF RA
  • to take action if migration-threshold reached
• NovaCompute OCF RA
  • ditto

Questions relating to compute plane

• How do we handle repeated failures of clones?
• How do we handle failures on different networks?
• How does the recovery service notify Pacemaker when recovery is complete?

How to handle repeated failure of clones

e.g. NovaCompute

• start-failure-is-fatal=false
• migration-threshold=3

After third failure, Pacemaker gives up!

We need something more proactive.

General Questions

• Can we make Pacemaker less scary?
• How can we get better testing?
  • https://review.openstack.org/#/c/443504/
• Indirect probes?

Debugging / trouble-shooting

• Logs are often incredibly hard to understand
• Less cryptic log messages would help a lot
• Currently all log messages are targeted at developers
  • but those should be the DEBUG-level ones
• messages logged at higher levels than DEBUG should be targeted at users
  i.e. don't assume in-depth knowledge of Pacemaker internals!
  • https://bugs.clusterlabs.org/show_bug.cgi?id=5294

Better documentation needed

Already some great steps in the right direction, e.g.

• http://crmsh.github.io/history-guide/
• https://hackweek.suse.com/12/projects/852

but often documentation / man pages out of date or incomplete.

• challenge: find documentation which explains crm_simulate -SL
  • https://bugs.clusterlabs.org/show_bug.cgi?id=5294
• a document explaining how the various timeout/migration parameters affect pengine is sorely needed
  • something like http://ourobengr.com/ha/#worstcase but not just for STONITH

Can we have indirect liveness probes?

Like with a gossip protocol

e.g. 5-node cluster, nodes 2/3/4 lose connection to node 5

• node 1 can still reach 5
• liveness (or even cluster events) can be gossiped to all nodes

General Disclaimer

This document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should be relied upon in making purchasing decisions. SUSE makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for SUSE products remains at the sole discretion of SUSE. Further, SUSE reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All SUSE marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.