Openstack RDO packstack Kilo install stuck on CENTOS 7 – MariaDB conflict

kilologoBackground: I got stuck on RDO packstack Kilo install onto CENTOS7. I got some strange log entries I didn’t understand – I post the problem and the solution here for my records and maybe to help someone else.

For my RDO packstack Kilo install setup, I followed the normal Quick Start  steps on a bare metal installed version of CENTOS 7 (CentOS-7-x86_64-LiveKDE-1503).

I ran the following step in the install guide:

# packstack --allinone

The packstack installed failed with the following error on the console:

Error: Execution of '/usr/bin/rpm -e mariadb-server-5.5.41-2.el7_0.x86_64' returned 1: error: Failed dependencies:
You will find full trace in log /var/tmp/packstack/20150704-200908-

This is an error pointing to the following log file entry:

Error: Execution of '/usr/bin/rpm -e mariadb-server-5.5.41-2.el7_0.x86_64' returned 1: error: Failed dependencies:
        mariadb-server is needed by (installed) akonadi-mysql-1.9.2-4.el7.x86_64

This is only my second install using packstack, and it is my first attempt at installing RDO packstack Kilo; so, I wasn’t familiar with the troubleshoot this kind of problem. It didn’t make sense to me that packstack would be trying to uninstall (rpm -e) MariaDB. The error indicates that the uninstall wouldn’t work because another package akonadi-mysql was using it — right? is that how I read this?  FYI:  Akonadi is a storage service that runs under KDE… I wasn’t familiar with it.

Browsing the RDO Workaround archives , I found a short article on this topic: Packstack fails when mariadb-server is installed. The body of the article shows a log file message that is different enough from mine that I didn’t find it right away.

I tried the recommended fix to uninstall MariaDB and re-run RDO packstack Kilo:

# yum remove mariadb-server
# packstack  --answer-file=packstack-answers-20150704-200908.txt

This fixed my problem!

I was able to run packstack to completion and bring up the openstack horizon dashboard. Note: I didn’t re-run packstack –allinone; I followed the packstack workflow requirement to re-run an install using the answer file previously generated.

Since the workaround was written for Juno and Fedora 20, I thought the fix was just a coincidence, but in reading the details, the original problem was triggered by installing KDE ontop of Fedora 20. Since I picked a KDE distribution of CENTOS7, maybe my problem and the original workaround write-up are related?!

Respectfully submitted,
Jack

Setting up Virtual Servers in Openstack environment

Background on Setting up Virtual Servers in Openstack environment: Once I got my Openstack environment setup, and I was able to create a couple of instances, I had to figure out the easiest way of managing IP Addresses and sub-domain names for web access to each of my instances.

I needed web access to my openstack host. I needed web access to each of my instances, which are running virtually on the same host. Further, since I am running all of this on one server in my home network, I need to somehow map all of this to one external IP address.

This is nothing too new to me. I have lots of vintages of Linux servers in my basement, and I sort of know the ropes around setting up NAT-ing, Virtual Servers, and proxies. My question was: what’s the best practice? What would be the easiest?

I couldn’t find anything directly on this (let me know if you have a reference). So here’s what I decided to do.

Enable Openstack Dashboard Network Access

By default, the Openstack Horizon Django configuration strictly controls who can get access. It’s roughly localhost only. For testing purposes, I went into the settings file and removed all restrictions:

$ cd /etc/openstack-dashboard
$ vi local_settings
...
#ALLOWED_HOSTS = ['horizon.example.com', ]
ALLOWED_HOSTS = ['*']
ZZ
$ systemctl restart httpd.service

I tried making a restrictive list, but it kept getting in my way. When done setting up, I will lock this up.

I then verified from a different PC in the same subnet that http://192.168.100.154/dashboard works.

Map Openstack Host to External IP Address

Using my home router, I configured an address mapping between port 80 and my Openstack host. Here’s the screen shot:

NetgearRouterPortForwarding012215

Port Forwarding Table from my home router

Now verify that http://my.external.IP.address/dashboard works.

Setup an Openstack Subdomain Name

One of my domain names, jackkozik.net points to my home router’s IP address. I setup an Openstack subdomain name, using my zoneedit account — I used os.jackkozik.net. Sorry no screen shot. I am (perhaps too liberally) showing my domain name, but I am reluctant to show my IP addresses. Zoneedit is pretty quick, but distributing a new subdomain address takes anywhere from 0 to 60 minutes.

I then edit Openstack’s virtual server configuration adding os.jackkozik.net as a ServerAlias, as follows:

$ cd /etc/httpd/conf.d
$ vi 15-horizon_vhost.conf
...
<VirtualHost *:80>
ServerName kozik4.lan
ServerAlias os.jackkozik.net # Add this line
...
ZZ
$ systemctl restart httpd.service

Then assuming the subdomain has had a chance to get distributed, verify http://os.jackkozik.net/dashboard.

Openstack’s install scripts automatically setup this VirtualHost.

Create Subdomains for Each of My Instances

Within Openstack, I create instances that are automatically assigned IP addresses from a pool in the range of 192.168.100.100-119. Of course these instances are accessible from my home network (eg http://192.168.100.100 displays a nice Apache default screen). But I only have one external IP address and I need a mechanism to for external web access.

Absent any better approach (I hope to find one!), I am using Apache’s ProxyPass capability. I have used this for physical servers, why not use it for virtual machines?!

For starters, I created another subdomain in zoneedit. I decided to name each external instance with a letter followed by the least significant digits from the IP address. My first subdomain is named f100: That is, it is a Fedora instance and it’s running an instance mapped to 192.168.100.100. In zoneedit, I enter the subdomain f100.jackkozik.net, and I put the same external IP address that I used for os.jackkozik.net.

Within the Apache configuration files, I created a virtual server named f100.jackkozik.net, and used ProxyPass to map it to (redirect it to?) the web server running on 192.168.100.100. See the following config file:

$ cd /etc/httpd/conf.d
$ vi openstackInstances_vhost.conf
# This file configures all the proxy modules:
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
<VirtualHost *:80>
ServerName f100.jackkozik.net
ProxyPreserveHost On
ProxyPass / http://192.168.100.100/
ProxyPassReverse / http://192.168.100.100/
</VirtualHost>
ZZ
$ systemctl restart httpd.service

I created this file and put it in the conf.d directory. It automatically gets read whenever the apache web server starts.

From here, allowing enough time for zoneedit to work, I verified that http://f100.jackkozik.net worked from both my home network and from an outside network (I sometime use my desktop PC at work test this; more commonly I use the Chrome browser on my Android phone).

I edit this file for each new instance I setup.

So each of my instances think they are sitting on the internet, but really the Openstack host Apache server and my home network router’s NAT function are fooling it.

Fix ALLOWED_HOSTS

Finally, once I got everything working, I fixed ALLOWED_HOSTS to permit any traffic from my home subnet and only allow requests from URL os.jackkozik.net from the Internet. See following:

$ cd /etc/openstack-dashboard
$ vi local_settings
...
ALLOWED_HOSTS = ['localhost', 'os.jackkozik.net', 'kozik4.lan', ]
#ALLOWED_HOSTS = ['*']
ZZ
$ systemctl restart httpd.service
Excalibur

Walking Las Vegas Strip

I was in Las Vegas for the week. My meetings started at 8am and most nights ended at 10pm, so I didn’t get alot of time to really do anything.

My last night, after 10pm, I went for a walk on the Las Vegas Strip. I exited through the taxi stand area, turned right / West and walked along Tropicana Ave.
At the light with Las Vegas Blvd, I crossed the street and turned right. From there I walked North passing Excalibur, New York-New York, Monte Carlo, Mandarin Oriental, City Center Place, Planet Hollywood, Harley Davidson Cafe, Cosmopolitan, Aria, Bellagio (saw the nice fountain show!), Paris, Caesar’s Palace, Flamingo, and Bally’s.
From Bally’s I took the Monorail ($5) back to the MGM. This was a fun way to see the city lights!

Icehouse

Openstack Icehouse on home network – part 2

In  my previous post titled OpenStack Icehouse on Fedora 20 using packstack on home PCI list the steps I followed to install Openstack Icehouse on home network.  This was mostly working from the command line of the Openstack PC.

This posting is part 2.  With the installation from part 1, I can now use the Openstack Dashboard.  It is a nice web interface that contains straight forward menus.  In the sections below, I will show screen captures from a web browser connected to the Dashboard; I will also show some command line text from a putty window.

This writeup will show steps for setting up a public subnet that links Openstack to my home network and setting up a private subnet for my guest instances to use.  Then I will show how to create an instance using the bare-bones cirros image.  And finally, I’ll install into the image repository a Fedora 20 cloud image and spin it up with a basic web server.

NetworkTopology

Since Openstack and the Openstack Dashboard are new to me,  I do all of this with the generous help of the references listed below, especially recognizing Seth Jenning’s excellent  Openstack Icehouse on Fedora 20 using RDO video.

Create a public network.

Starting with the Openstack Dashboard, logged in as admin, create a public network.  My home network is 192.168.100.0/24.  The IP address 192.168.100.163 is the address of my home router, gateway to the internet. My Openstack Icehouse host is 192.168.100.154.  And Openstack needs a subnet, referred to as public, that sits in this address range.

The naming convention of calling the network 192.168.100.0/24 public started with the packstack install scripts.  In the context of Openstack, this network is the one with a gateway to the internet and thus it is referred to as public, even though we know the internet defines 192.168.x.x IP addresses as private.

Openstack Dashboard Menu: Admin->Networks->Create Network

AdminNetworksCreateNetwork081514

Openstack Dashboard Menu: Admin->Networks->public->Create Subnet

AdminNetworksPublicCreateSubnet1

Openstack Dashboard Menu: Admin->Networks->public->Create Subnet 2

AdminNetworksPublicCreateSubnet2

Create a private subnet

Next, create  a subnet that is private to the Openstack host.  The addresses must be different from the public_subnet.  These addresses will never leave the Openstack host and it’s underlying Open vSwitch network address space.   The references use 10.0.0.0/24 as the network and 10.0.0.1 as the gateway address — and that’s what I use below.  Further, the guest instances each need to be given an IP address, and 10.0.0.2, 10.0.0.20 address are what I choose to be the range for a DHCP address pool.

Openstack Dashboard Menu: Project->Network->Network Topology->Create Network — “private”

ProjectPublicNetworktopologyCreateNetwork1

Openstack Dashboard Menu: Project->Network->Network Topology->Create Network->Subnet

ProjectNetworktopologyCreateNetwork2

Openstack Dashboard Menu: Project->Network->Network Topology->Create Network->Subnet Details

ProjectNetworktopologyCreateNetwork3

Create a router

Ok, there’s a public and a private subnet defined.  Openstack Dashboard has a really simple way to connect them together with a router function.  Create a router, define a default gateway and then add interfaces to private subnets.

Openstack Dashboard Menu: Projects->Network Topology->Create Router

ProjectNetworktopologyCreateRouter

Openstack Dashboard Menu: Project->Routers->Set Gateway

PorjectsRoutersSetGateway

Openstack Dashboard Menu: Project->Routers->router->Add Interface

ProjectRoutersRouterAddInterface

Verify Network Topology

All the steps upto this point were building a network into which virtual machines connect to the home LAN.  Run the Dashboard command below to see two subnets connect to a router.  The public subnet is on the home LAN.  The private subnet is the address space where the guests instances will connect.

Openstack Dashboard Menu: Project->Network Topology

ProjectsNetworkTopology

Setup Security Group Rules

The references suggest that for trial/learning purposes, the Security Group Rules should be wide open.  The idea is while learning the technology, the security settings can obscure basic setup issues.  In the long run, this needs to be managed more carefully.

First, remove the default rules that packstack setup, then install rules that permit incoming and outgoing TCP/UDP/ICMP — all ports.

Openstack Dashboard Menu: Project->Compute->Access and Security->default

Delete all the default rules and rebuild the rules so that the Security Group Rules table looks as follows:

ProjectComputeAccessSecurityDefaultAddRules

Setup an ssh key pair

Using the normal ssh tools, make an ssh public and private key pair.  The Openstack Dashboard lets you cut / paste your own public key into the project.  The instances that get created will have the  public key pre-installed into it.  To access instances that Openstack creates, use the private key as an option in an ssh command line.

Go to the root login of the Openstack host, at the command prompt create a key and copy the public key into the clipboard.

[root@kozik4 ~]# ssh-keygen -t rsa -f cloud.key
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in cloud.key.
Your public key has been saved in cloud.key.pub.
The key fingerprint is:
36:37:4d:d3:5b:82:45:53:48:98:24:00:8b:0b:3b:06 root@kozik4.lan
The key's randomart image is:
+--[ RSA 2048]----+
|      .......==o.|
|     . .   .o+.. |
|E . . .     + o .|
| . o .     o . + |
|  + .   S o . .  |
| . .   . o .     |
|                 |
|                 |
|                 |
+-----------------+
[root@kozik4 ~]# ls
anaconda-ks.cfg  ifconfig5.out         packstack-answers-20140803-201418.txt
cloud.key        installpackstack.log  packstack.log
cloud.key.pub    keystonerc_admin      rdorelease.log
ifconfig1.out    keystonerc_demo       runpackstack.log
ifconfig2.out    ovs1.out              yumupdate.log
ifconfig3.out    ovs2.out
ifconfig4.out    ovs3.out
[root@kozik4 ~]# cat cloud.key.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDijKn/k5ejNii3SaNugO75Njz1LQHyDDwI5blZO4+CZTRL/O/8czffrUUfK8+j3QjAx7MByNJVkj8YCGtOAYv5wCFEhzkRhqNNJlH235L++QV6ai/XPD7b0VcqhCjQTkDIfyBMp7fZO+D0BGdvTjBiQXIJdZLqZWV2j9qH8EHHS55OlOXpAAcMHvRRgWFtMdn5YSLUcq8X5HRtvfesLL7quJmNDc8/rS6mhmL/NFU56r+SJpHvr7N59U7ywNejLgFp6hfz4zZw3nWDH9y+by1zdWbNfATIO362SRue+FvuF060ss4Ciesuqw5v3tJMeyq9JM41lu8fQaIeBqoJTB43 root@kozik4.lan
[root@kozik4 ~]#

In putty screen like the above example, select the text output from the ‘cat cloud.key.pub’ command and paste into the Openstack Dashboard as follows.

Openstack Dashboard Menu: Project->Access & Security->Key Pairs->Import Key Pair

ProjectAccessSecurityImportKeyPair2

Launch Test Instance ‘cirros’ and assign floating IP

To help verify that the Openstack packstack installed correctly, spin-up the barebones cirros image.  This image is a really small linux distribution.  I’ve never heard of cirros, but I get the purpose of it.  My initial setup had troubles, and cirros helped me trouble shoot basic setup problems.  I was glad the initial install pulled it in.

The following steps startup an instance, link it to the private subnet, and map the private IP address of the instance to a floating IP address on the public subnet.  Floating IP addresses were new to me, and it wasn’t obvious how they should be used, at first.  I think of it as a generalized NATing function, that lets me hide my home network topology from the Openstack instances.

Openstack Dashboard Menu: Project->Images->Launch->Details

ProjectImagesLaunchDetails

Connect the instance to the private subnet.  Note: the web page below requires you to drag the private line and drop it into the Selected Networks cyan-colored bar.

Openstack Dashboard Menu: Project->Images->Launch->Networking

ProjectImagesLaunchNetworking

Verify the test instance is running:

Openstack Dashboard Menu: Project->Instances

ProjectInstances

Allocate a Floating IP address

Openstack Dashboard Menu: Projects->Access & Security->Floating IPs->Allocate IP to Project

ProjectAccessSecurityFloatingIPAllocate

Associate the instance’s private IP address to an IP address on the public subnet.

Openstack Dashboard Menu: Project->Access & Security->Floating IP->Associate

ProjectAccessSecurityFloatingIPManageFloatingAssociations

Verify that the instance has two IP addresses and is running ok.

Openstack Dashboard Menu: Project->Instance

ProjectInstanceswIP

Go back to the 192.168.100.154 host putty prompt (root login, home directory).  Verify that we can setup an ssh connection to the new instance. The default login id is cirros. The Instances web page above tells us to use the 192.168.100.101.

# ssh -i cloud.key cirros@192.168.100.101
The authenticity of host '192.168.100.101 (192.168.100.101)' can't be established.
RSA key fingerprint is 34:51:4c:22:c3:67:d3:47:38:83:c2:ee:55:0f:4b:e5.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.100.101' (RSA) to the list of known hosts.

$ ifconfig
eth0      Link encap:Ethernet  HWaddr FA:16:3E:62:7D:36
          inet addr:10.0.0.3  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe62:7d36/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1

$ ping yahoo.com
PING yahoo.com (98.139.183.24): 56 data bytes
64 bytes from 98.139.183.24: seq=0 ttl=47 time=49.484 ms
64 bytes from 98.139.183.24: seq=1 ttl=47 time=121.366 ms
64 bytes from 98.139.183.24: seq=2 ttl=47 time=81.164 ms

Note: the ifconfig shows that the instance only knows about the private_subnet address 10.0.0.3. Also, an important test to verify: check that the instance can access the internet, I used ping yahoo.com.

Create a Fedora 20 instance

f20-changeThe cirros instance installation steps above helped to verify that basic functionality worked.  But cirros is not a linux distribution I want to use; I want to use the latest version of Fedora.  In this section I repeat some of the steps from the previous section to get a Fedora 20 instance started.  There’s enough different here that I wanted to document it.

From the Fedora In the Cloud web page, right click the 64-bit qcow2 image and “Copy Link Address.”  The Images page has an option to import new images using a URL.

FedoraProjectCloud

Create an image from this URL:

Openstack Dashboard Menu: Project->Create Image

ProjectImageCreateImage

Following the same steps as the cirros image, launch the Fedora 20 image.

Openstack Dashboard Menu: Project->Instance->Launch Instance

ProjectInstancesLaunchInstanceFedora

Be sure to click the Networking tab and select private subnet, then click on Launch.

Once the instance is running, allocate a Floating IP.  Go to the Projects->Access & Security->Floating IP menu. First run Allocate IP to Project then Manage Floating IP Associations for the Fedora 20 instance… just like we did for the cirros instance.

The Instance Dashboard page now shows 2 instances.

Openstack Dashboard Menu: Project->Instance

ProjectInstanceswFedora

And the Network Topology page gives a nice picture of how everything is wired together.

Openstack Dashboard Menu:  Project->Network->Network Topology

ProjectNetworkNetworkTopologyFedora

So, just like with cirros, go to the Openstack host root login  prompt and ssh to the Fedora instance.  The Instance page above show 192.168.100.102 as the IP address.

# ssh -i cloud.key fedora@192.168.100.102
The authenticity of host '192.168.100.102 (192.168.100.102)' can't be established.
RSA key fingerprint is 2e:b3:7b:6b:06:43:cf:d5:95:95:49:38:5f:ab:20:39.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.100.102' (RSA) to the list of known hosts.
[fedora@fedora-20 ~]$ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.0.4  netmask 255.255.255.0  broadcast 10.0.0.255
        inet6 fe80::f816:3eff:fe9b:baf5  prefixlen 64  scopeid 0x20																					
        ether fa:16:3e:9b:ba:f5  txqueuelen 1000  (Ethernet)

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10
        loop  txqueuelen 0  (Local Loopback)

[fedora@fedora-20 ~]$ ping yahoo.com
PING yahoo.com (206.190.36.45) 56(84) bytes of data.
64 bytes from ir1.fp.vip.gq1.yahoo.com (206.190.36.45): icmp_seq=1 ttl=49 time=1         06 ms
64 bytes from ir1.fp.vip.gq1.yahoo.com (206.190.36.45): icmp_seq=2 ttl=49 time=1         56 ms
64 bytes from ir1.fp.vip.gq1.yahoo.com (206.190.36.45): icmp_seq=3 ttl=49 time=1         02 ms

Note:  the Fedora guest instance only knows the private address 10.0.0.4.  The default login id, fedora, has sudo permissions and access to the root login is done with a ‘sudo su -’ command. It is important also to verify that it can talk to the outside world, and thus verify that ping yahoo.com works.

To further test my Fedora 20 instance, I switched over to the Fedora 20 instance root login and installed apache:

[fedora@fedora-20 ~]$ sudo su -
[root@fedora-20 ~]# yum groupinstall "Web Server"
[root@fedora-20 ~]# systemctl enable httpd.service
[root@fedora-20 ~]# systemctl start  httpd.service

From here, I go to another PC in my home network and verify that the default web server works.

ApacheDefaultWebPage

References