OpenStack Nova

This is placement update 18-19. 18 was skipped because I was on holiday. With this issue I'm going to start cross-posting to my blog to increase exposure, double up the archiving, and get the content showing up on the OpenStack Planet. One upshot of ... [More] this change is that the content will now be formatted more fully as Markdown. I'll be travelling next week so there won't be one of these for weeks 20 or 21, unless someone else feels like it. Most Important We're continuing to hope that granular and nested resource providers will be fully merged by Summit (a bit more than a week from now). Not clear if this will happen as last I checked it seemed we have multiple threads of changes in progress, many of which will merge conflict with one another. But then again, I may be out of date, it's been difficult to find all those threads while trying to catch up this week. If you're going to be at summit there are (at least) two placement-related forum sessions: https://etherpad.openstack.org/p/YVR-placement-extraction https://etherpad.openstack.org/p/YVR-cinder-placement Please add to those etherpads if you have thoughts. Also a summit, Ed and Eric will be presenting Placement, Present and Future, in Nova and Beyond. What's Changed Granular requests can now be made to GET /allocation_candidates (meaning resourcesN and requiredN are now accepted). A bug with the safe_connect handler masking real problems has been fixed. The spec for Network Bandwidth Resource Provider has finally merged after a lot of thinking and discussion. The spec for Return resources of entire trees in Placement has merged. This allows the inclusion of resource providers which are not providing inventory, but are part of the current tree, in the provider summaries of a /allocation_candidates response. There are some new specs (see the end of the specs list, below) which extend required traits handling to be able to say "I need at least one of these traits". Bugs Placement related bugs not yet in progress: 16, -1 on two weeks ago In progress placement bugs 10, +2 two weeks ago Specs Total two weeks ago: 11. Now: 13 https://review.openstack.org/#/c/549067/ VMware: place instances on resource pool (using update_provider_tree) https://review.openstack.org/#/c/552924/ Proposes NUMA topology with RPs https://review.openstack.org/#/c/544683/ Account for host agg allocation ratio in placement https://review.openstack.org/#/c/552105/ Support default allocation ratios https://review.openstack.org/#/c/438640/ Spec on preemptible servers https://review.openstack.org/#/c/557065/ Proposes Multiple GPU types https://review.openstack.org/#/c/555081/ Standardize CPU resource tracking https://review.openstack.org/#/c/509042/ Propose counting quota usage from placement https://review.openstack.org/#/c/560174/ Add history behind nullable project_id and user_id https://review.openstack.org/#/c/565565/ update add-consumer-generation to focus on API https://review.openstack.org/#/c/565730/ Placement: any traits in allocation_candidate query https://review.openstack.org/#/c/565741/ Placement: support mixing required traits with any traits https://review.openstack.org/#/c/559718/ [WIP] Support Placement in Cinder Main Themes Nested providers in allocation candidates Unfortunately I'm really unclear on what the current state of this is. If someone else can give a quick overview that would be excellent. There's code in progress at both of the following topics, some of it is old and in merge conflict: https://review.openstack.org/#/q/topic:bp/nested-resource-providers-allocation-candidates https://review.openstack.org/#/q/topic:bp/nested-resource-providers Mirror nova host aggregates to placement This makes it so some kinds of aggregate filtering can be done "placement side" by mirroring nova host aggregates into placement aggregates. https://review.openstack.org/#/q/topic:bp/placement-mirror-host-aggregates This is still in progress but is still on its attention break. Consumer Generations This allows multiple agents to "safely" update allocations for a single consumer. The code is in progress: https://review.openstack.org/#/q/topic:bp/add-consumer-generation There's also a related change for ensuring consumer records. Granular Ways and means of addressing granular requests when dealing with nested resource providers. Granular in this sense is grouping resource classes and traits together in their own lumps as required. Topic is: https://review.openstack.org/#/q/topic:bp/granular-resource-requests The sole active change in that now is work in progress to get it working from the Nova side. Extraction I've created patches that adjust devstack and zuul config to use the separate placement database connection. devstack zuul db connection All of these things could merge without requiring any action by users. Instead they allow people to use different connections, but don't require it. Jay has made a first pass at an os-resource-classes which I thought was potentially more heavyweight than required, but other people should have a look too. As mentioned above there will be a forum session about extraction. In the meantime, some of the low hanging fruit on extraction is duplicating and extracting to their own files the various fixtures and base test classes that are required by both the functional and unit tests. And making them not import from the nova hierarchy. Other 17 entries two weeks ago. 19 now. Some of the older items in this list are not getting much attention. That's a shame. The list is ordered the way it is on purpose. https://review.openstack.org/#/c/546660/ Purge comp_node and res_prvdr records during deletion of cells/hosts https://review.openstack.org/#/q/topic:bp/placement-osc-plugin-rocky A huge pile of improvements to osc-placement https://review.openstack.org/#/c/524425/ General policy sample file for placement https://review.openstack.org/#/c/527791/ Get resource provider by uuid or name (osc-placement) https://review.openstack.org/#/c/477478/ placement: Make API history doc more consistent https://review.openstack.org/#/c/556669/ Handle agg generation conflict in report client https://review.openstack.org/#/c/537614/ Add unit test for non-placement resize https://review.openstack.org/#/c/493865/ cover migration cases with functional tests https://review.openstack.org/#/q/topic:bug/1732731 Bug fixes for sharing resource providers https://review.openstack.org/#/q/topic:bp/placement-return-all-resources return resoruces of entire trees in placement https://review.openstack.org/#/q/topic:libvirt-report-local-disk-only-if-no-sharing sharing disk in libvirt https://review.openstack.org/#/c/535517/ Move refresh time from report client to prov tree https://review.openstack.org/#/c/561770/ PCPU resource class https://review.openstack.org/#/c/565886/ Add nova-manage placement heal_allocations CLI https://review.openstack.org/#/c/567506/ Add random sleep between retry calls to placement https://review.openstack.org/#/c/566166/ rework how we pass candidate request information https://review.openstack.org/#/c/564876/ add root parent NULL online migration https://review.openstack.org/#/q/topic:bp/bandwidth-resource-provider add resource_requests field to RequestSpec https://review.openstack.org/#/q/topic:bp/alloc-candidates-member-of member_of related cleanups End Please followup with the many things I've missed. [Less]

In Part 2, I summarized the iterations applied to the kernel (the piece of CUDA code that executes on GPU) to remove the bottlenecks encountered during profiling, like using shared memory to avoid non-coalesced memory access. In this part, I will ... [More] talk about the final version of the kernel and using the GPU in other […] The post An experiment with Gnocchi – (the database) – Part 3 appeared first on SUSE Communities. [Less]

"Upgrading from Juno to Ocata seems like climbing Everest, when in reality it’s more like walking up stairs," says Yahoo's James Penick. The post OpenStack upgrades on a massive scale: Inside Yahoo appeared first on Superuser.

Teach a man to fish and you feed him for a lifetime.  It’s an often quoted proverb when talking about the value of personal development.  Give someone the opportunity to gain the skills and develop the talents they need and you’ll equip that person ... [More] to survive and thrive. If that person happens to work for […] The post OpenStack and the Art of Fishing appeared first on SUSE Communities. [Less]

A few weeks back in Dublin, Ireland, OpenStack engineers gathered from dozens of countries and companies to discuss the next release of OpenStack. This is always my favorite OpenStack event, because I get to do interviews with the various teams, to ... [More] talk about what they did in the just-released version (Queens, in this case) and what they have planned for the next one (Rocky). [Less]

I just published my series on Graph Databases a few hours ago, and have already had lots of hits. I guess people found the topic as interesting as I did! One reader, Jay Pipes, is one of the core developers of the Placement service I mentioned in ... [More] that series, and probably the most heavily opinionated … Continue reading "Graph Database Follow-up" [Less]

In the previous two posts in this series, I wrote about graph databases in general, and why I think that they are much more suited to modeling the data in the Placement service than relational DBs such as MySQL. In this post I’ll show a few examples of common Placement issues and how they are … Continue reading "Placement Graph Examples"

It is not my intent here to create a full tutorial on using graph databases; I just want to convey enough understanding so that you might see why they are a better fit for placement than a relational DB. The most basic thing to understand is that your whiteboard design is your database design. Here’s … Continue reading "Graph Database Basics"

The work on the OpenStack Placement service has been getting more and more complex in the last few months, as the service has matured to handle more complex resources. At first, things were pretty basic: you had a ResourceProvider, and it had ... [More] Inventory of various ResourceClasses. You told Placement how much you needed of one … Continue reading "Hammer and Nail" [Less]