juju juju-upgrade -m controller woes

[Daniel Bidwell]

I have a juju controller, uncloud, in a container in a vmware vm.  The
controller has a manually provisioned machine deployed happily.  I
tried to manually provision a second machine.  It is stuck in pending
with no progress after a day of waiting.  I can ssh to ***@host just
fine with the ssh keys installed.

I am provisioning from a machine with juju 2.3.2 and the controller has
version 2.2.6 on it (now unsupported).  I have attempted to do "juju
upgrade-juju -m uncloud" and it happily says that it is upgrading to
2.3.2, but I never see any progress with that either (after hours or
days).

What is the secret sauce for making these thing work?  For some reason,
whenever I have trouble with my juju environment, I have to destory it
and set up the controllers from scrap.  I guess I just don't have the
juju for juju :-(  I do like being able to develop charms for my apps
and deploy them though.

[John Meinel]

I'm a bit curious what you mean by "manually provisioned machine". Are you
saying that you used the VMWare APIs directly to launch an instance, and
then "juju bootstrap IP" to start using that machine, and then "juju
add-machine ssh:IP" to add the second machine?

You could try doing "juju upgrade-juju --debug" to see what at least the
client thinks it is trying to do. I wonder if it is a case where your VM
doesn't actually have egress access, but your client does. And so Jujud on
the VM is trying to do something like download the new agent, which your
client saw and said was available, but the controller isn't actually able
to download.

John
=:->

I have a juju controller, uncloud, in a container in a vmware vm. The
controller has a manually provisioned machine deployed happily. I
tried to manually provision a second machine. It is stuck in pending
fine with the ssh keys installed.
I am provisioning from a machine with juju 2.3.2 and the controller has
version 2.2.6 on it (now unsupported). I have attempted to do "juju
upgrade-juju -m uncloud" and it happily says that it is upgrading to
2.3.2, but I never see any progress with that either (after hours or
days).
What is the secret sauce for making these thing work? For some reason,
whenever I have trouble with my juju environment, I have to destory it
and set up the controllers from scrap. I guess I just don't have the
juju for juju :-( I do like being able to develop charms for my apps
and deploy them though.
--
--
Juju-dev mailing list
Modify settings or unsubscribe at: https://lists.ubuntu.com/
mailman/listinfo/juju-dev

[Daniel Bidwell]

I have a vmware vm on my public network that is my juju host.  In it I
have an lxc container that is also bridged to my public network.  I
bootstrapped it with:

juju bootstrap manual/ipaddress modelname

I also created vms with public IP addresses that I setup an ubuntu
account and added as:

juju add-machine ssh:***@ipaddress

all machines have public ip address and full routing.

The controller container was running juju version 2.2.6 (unsupported)
while the main machine had been upgraded to 2.3.2.

juju status produces:

Model    Controller  Cloud/Region  Version  SLA
default  uncloud     manual        2.2.6    unsupported

App                   Version  Status  Scale  Charm                 Store  Rev  OS      Notes
au-webcluster-member           active      1  au-webcluster-member  local    0  ubuntu  
aubase1                        active      1  aubase1               local    0  ubuntu  

Unit                     Workload  Agent  Machine  Public address  Ports  Message
au-webcluster-member/0*  active    idle   1        webc4                  Web cluster member for instance test completed
aubase1/0*               active    idle   1        webc4                  

Machine  State    DNS      Inst id         Series  AZ  Message
0        down     amzsend  manual:amzsend  xenial      Manually provisioned machine
1        started  webc4    manual:webc4    xenial      Manually provisioned machine
2        pending  webc5    manual:webc5    xenial      Manually provisioned machine

machine 2 failed to "add".  The upgrade-juju command now says:

ERROR some agents have not upgraded to the current model version 2.2.6:
machine-2

but juju doesn't let me do anything with machine-2 until it is finished
with it.

I suspect that my only option is to build a new container/controller
and redeploy my machines on it before removing the old
container/controller.

Post by John Meinel
I'm a bit curious what you mean by "manually provisioned machine".
Are you saying that you used the VMWare APIs directly to launch an
instance, and then "juju bootstrap IP" to start using that machine,
and then "juju add-machine ssh:IP" to add the second machine?
You could try doing "juju upgrade-juju --debug" to see what at least
the client thinks it is trying to do. I wonder if it is a case where
your VM doesn't actually have egress access, but your client does.
And so Jujud on the VM is trying to do something like download the
new agent, which your client saw and said was available, but the
controller isn't actually able to download.
John
=:->

I have a juju controller, uncloud, in a container in a vmware vm. 
The
controller has a manually provisioned machine deployed happily.  I
tried to manually provision a second machine.  It is stuck in
pending
just
fine with the ssh keys installed.
I am provisioning from a machine with juju 2.3.2 and the controller has
version 2.2.6 on it (now unsupported).  I have attempted to do
"juju
upgrade-juju -m uncloud" and it happily says that it is upgrading to
2.3.2, but I never see any progress with that either (after hours or
days).
What is the secret sauce for making these thing work?  For some
reason,
whenever I have trouble with my juju environment, I have to destory it
and set up the controllers from scrap.  I guess I just don't have
the
juju for juju :-(  I do like being able to develop charms for my apps
and deploy them though.
--
--
Juju-dev mailing list
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman
/listinfo/juju-dev

[John Meinel]

In 2.3 there is "juju upgrade-juju --ignore-agent-versions" but I think
that is only respected if the target controller already is 2.3 (eg, this
would help if you were trying to go from 2.3 to 2.4, but won't help going
2.2 to 2.3).

There is a separate command line utility that you can build in the juju
source tree called "juju-force-upgrade" which can work to bypass the agent
version check.

It might also be possible to "juju remove-machine --force 2" which would
let you remove that machine. However, I'm not 100% sure that you can remove
a machine that is currently pending provisioning.

Its possible that "juju debug-log --replay [-m controller]" would be
enlightening as to why machine 2 is failing to be provisioned (or looking
at machine-0.log on the controller machine).

Is there a reason you are manually provisioning instances and then
registering them rather than using the cloud as a VMWare cloud? (juju
add-cloud vmware).

John
=:->

I have a vmware vm on my public network that is my juju host. In it I
have an lxc container that is also bridged to my public network. I
juju bootstrap manual/ipaddress modelname
I also created vms with public IP addresses that I setup an ubuntu
all machines have public ip address and full routing.
The controller container was running juju version 2.2.6 (unsupported)
while the main machine had been upgraded to 2.3.2.
Model Controller Cloud/Region Version SLA
default uncloud manual 2.2.6 unsupported
App Version Status Scale Charm
Store Rev OS Notes
au-webcluster-member active 1 au-webcluster-
member local 0 ubuntu
aubase1 active 1 aubase1
local 0 ubuntu
Unit Workload Agent Machine Public
address Ports Message
au-webcluster-member/0* active idle 1 webc4 Web
cluster member for instance test completed
aubase1/0* active idle 1 webc4
Machine State DNS Inst id Series AZ Message
0 down amzsend manual:amzsend xenial Manually
provisioned machine
1 started webc4 manual:webc4 xenial Manually
provisioned machine
2 pending webc5 manual:webc5 xenial Manually
provisioned machine
machine-2
but juju doesn't let me do anything with machine-2 until it is finished
with it.
I suspect that my only option is to build a new container/controller
and redeploy my machines on it before removing the old
container/controller.

Post by John Meinel
I'm a bit curious what you mean by "manually provisioned machine".
Are you saying that you used the VMWare APIs directly to launch an
instance, and then "juju bootstrap IP" to start using that machine,
and then "juju add-machine ssh:IP" to add the second machine?
You could try doing "juju upgrade-juju --debug" to see what at least
the client thinks it is trying to do. I wonder if it is a case where
your VM doesn't actually have egress access, but your client does.
And so Jujud on the VM is trying to do something like download the
new agent, which your client saw and said was available, but the
controller isn't actually able to download.
John
=:->

Post by Daniel Bidwell
I have a juju controller, uncloud, in a container in a vmware vm.
The
controller has a manually provisioned machine deployed happily. I
tried to manually provision a second machine. It is stuck in pending
fine with the ssh keys installed.
I am provisioning from a machine with juju 2.3.2 and the controller has
version 2.2.6 on it (now unsupported). I have attempted to do "juju
upgrade-juju -m uncloud" and it happily says that it is upgrading to
2.3.2, but I never see any progress with that either (after hours or
days).
What is the secret sauce for making these thing work? For some reason,
whenever I have trouble with my juju environment, I have to destory it
and set up the controllers from scrap. I guess I just don't have the
juju for juju :-( I do like being able to develop charms for my apps
and deploy them though.
--
--
Juju-dev mailing list
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman
/listinfo/juju-dev

--