Since I ran into another wall with trying to use Ansible Vault under Bash on Ubuntu on Windows10 (this time, chmod wouldn’t change the permissions on the .vault_pass.txt file from 755 to 600 – or any other permissions set for that matter), I went back to my Linux-based setup to try out the Ansible Vault solution I’d devised.
Here, I ended up unable to communicate with the VM using Ansible because SSH from Ubuntu to the Debian8 box had an incompatibility – to wit, when I ran ssh email@example.com -p 2222, the command eventually timed out with the error “ssh_exchange_identification: read: Connection reset by peer”.
This is yet another piece of evidence that someone very recently (I believe between the 8.5.2 and the 8.6.0 versions of the box on Atlas) made breaking changes to the OpenSSH and/or OpenSSL configuration of the box. One change I’ve figured out is they disabled PasswordAuthentication in the /etc/ssh/sshd_config file.
This problem? Looks like (based on my read of articles like this one) the ssh client and server can’t agree on some cryptographic parameter. Fun. Cause there’s only about a million combinations of these parameters to play with.
[I also pursued ideas like the solution to this report, but currently the Debian8 box’s /etc/hosts.deny is still empty of uncommented entries. Or the “is sshd running” idea from this report, but /var/log/auth.log definitely includes “[date] jessie sshd: Server listening on 0.0.0.0 port 22”.]
OK, so what’s the fastest way to isolate the set of parameters that are being offered and demanded between the client and server?
Running the ssh client with -vvv parameter doesn’t help much – it enumerates the “key_load_public” attempts (rsa, rsa-cert, dsa, dsa-cert, ecdsa, ecdsa-cert, ed25519, ed25519-cert), then “Enabling compatibility mode for protocol 2.0” and the SSH version “Local version string SSH-2.0-OpenSSH_7.2ps Ubuntu-4ubuntu2.1”, then fires off the “connection reset by peer” error again. Dpkg -l reports that openssh-client is “1:7.2ps-4ubuntu2.1”.
What’s the server’s version of OpenSSH? According to dpkg -l, it’s “1;6.7p1-5+deb8u3. Is that right – 1.6.7? And if so, how do I find out if there’s a cryptographic configuration incompatibility between 1.7.2 and 1.6.7? [Certainly I can see that we have no such “connection reset by peer” issue between my Win10 Bash on Ubuntu shell, running 1.6.6p1 of openssh-client and the Debian8 box’s 1.6.7p1, so cryptographic compatibility between 1.6.6 and 1.6.7 is a reasonable assumption.] Or better, is it possible to upgrade the Debian8 box’s openssh-server to something later than 1.6.7 – preferably (but not exclusively) 1.7.2?
On the server, I can crawl through the /etc/ssh/sshd_config” file to look for configured parameters (RSAAuthentication yes for example), but that doesn’t tell me what the OpenSSH defaults are, and doesn’t tell me what’s necessarily being asked of OpenSSL either (which might be swallowing the actual error).
I started to pursue the idea of upgrading OpenSSH, so I ran sudo apt-get update to prepare for updating everything in the VM. That’s when I noticed I wasn’t getting any network connectivity, as it spat back “Could not resolve ‘security.debian.org'” and “Could not resolve ‘httpredir.debian.org'”.
Vbox Mgr indicates I’m using NAT networking (the default), which has worked for me in the past – and works fine for the same Vagrant box running on my Win10 VirtualBox/Vagrant instance (sudo apt-get update “Fetched 529 kB in 3s (142kB/s)”). Further, the Ubuntu host for this VM has no problem reaching the network.
So I tried changing to Bridged Adapter in Vbox Manager. Nope, no difference. Why does the same Vagrant box work fine under Windows but not under Ubuntu? Am I cursed?
Back to the root problem
Let me review: I’m having a problem getting Ansible to communicate with the VM over SSH. So let’s get creative:
- Can Ansible be coerced into talking to the target without SSH?
- Can Ansible use password authentication instead of public key authentication for SSH?
- Can the Ubuntu client be downgraded from 1.7.2 to 1.6.7 openssh-client?
Of course! The “connection reset by peer” issue isn’t a matter of deep crypto at all – unless I’m misreading this, the fact that the Ubuntu SSH client takes nearly a minute to return the “connection reset” error and the fact that the Debian VM doesn’t seem to have any IP networking ability off the host…adds up to SSH client not even connecting to the VM’s sshd?
Boy do I feel dumb. This has nothing to do with crypto – it’s simple layer 3 issues.
Reminds me of a lesson I learned 20 years ago, and seem to re-learn every year or three: “When you hear hooves, think horses not zebras.”
Then how do we establish where the problem is – Virtualbox, Ubuntu, Debian or something else?
- If it’s a problem in the Debian VM, then download a different Vagrant box
- If it’s a problem in the Virtualbox setting, keep trying different network settings until one breaks through
- If it’s a problem in the Ubuntu host, look for reasons why there’d be a block between 127.0.0.1 (host to VM or vice-versa)
What other evidence do we have? Well, when I run vagrant up from the Ubuntu host, it gets to “default: SSH auth method: private key” then eventually reports “Timed out while waiting for the machine to boot. This means that Vagrant was unable to communicate with the guest machine within the configured (“config.vm.boot_timeout” value) time period.” Makes me more suspicious of the VM.
Searching the Vagrant boxes registry, mosaicpro/html looks like it’s desktop (not locked-down server) oriented, so I tried that one. Watched it boot, then report “default: Warning: Remote connection disconnect. Retrying…” over and over for a few minutes. The console via Vbox Mgr looked like the Ubuntu VM was trying to configure networking (even though DHCP had offered it an address of 10.2.0.15 – which must’ve been the NAT adapter, since my home network runs on 192.168.1/24). But oddly, networking from within the client was working fine after that – ping out to my home router (192.168.1.1) returned fine. OK, then I’m *definitely* suspecting that Debian/jessie64 (8.6.0) box.
Vagrant/Debian downgrade anyone?
So, after all this, can I download a previous version of the Debian/jessie64 box (e.g. 8.5.2, not this troublesome 8.6.0)? Let’s try it, using this article as basis.
(I went one step further and ran the initial command as vagrant add debian-8.5.2 https://atlas.hashicorp.com/debian/boxes/jessie64/versions/8.5.2/providers.virtualbox.box – and amazingly, this variation seemed to work!)
And here’s some promising results:
- lsb_release -a reports the 8.5.2 box as “Debian GNU/Linux 8.5 (jessie)”, vs the 8.6.0 box as “Debian GNU/Linux 8.6 (jessie)”
- A quick look at the /etc/ssh/sshd_config from the 8.5.2 box shows there is *no* insertion of the PasswordAuthentication configuration parameter (let alone setting it to “no” like in the 8.6.0 box)
- Network connectivity from the 8.5.2 box to my home router is awesome (vs the 8.6.0 box that can’t seem to ping out of a wet paper bag)
If you’re a Vagrant + Virtualbox user, stay FAR away from the 8.6.0 version of the debian/jessie64 box (unless you’re prepared to fight with these same issues I have, and probably other ‘security lockdown’ ideas that I haven’t even uncovered yet, but are almost surely there).