Troubleshooting another SSH blocker (networking?) in debian/jessie64

Since I ran into another wall with trying to use Ansible Vault under Bash on Ubuntu on Windows10 (this time, chmod wouldn’t change the permissions on the .vault_pass.txt file from 755 to 600 – or any other permissions set for that matter), I went back to my Linux-based setup to try out the Ansible Vault solution I’d devised.

Here, I ended up unable to communicate with the VM using Ansible because SSH from Ubuntu to the Debian8 box had an incompatibility – to wit, when I ran ssh vagrant@ -p 2222, the command eventually timed out with the error “ssh_exchange_identification: read: Connection reset by peer”.

This is yet another piece of evidence that someone very recently (I believe between the 8.5.2 and the 8.6.0 versions of the box on Atlas) made breaking changes to the OpenSSH and/or OpenSSL configuration of the box.  One change I’ve figured out is they disabled PasswordAuthentication in the /etc/ssh/sshd_config file.

This problem?  Looks like (based on my read of articles like this one) the ssh client and server can’t agree on some cryptographic parameter.  Fun.  Cause there’s only about a million combinations of these parameters to play with.

[I also pursued ideas like the solution to this report, but currently the Debian8 box’s /etc/hosts.deny is still empty of uncommented entries.  Or the “is sshd running” idea from this report, but /var/log/auth.log definitely includes “[date] jessie sshd[366]: Server listening on port 22”.]

OK, so what’s the fastest way to isolate the set of parameters  that are being offered and demanded between the client and server?

Running the ssh client with -vvv parameter doesn’t help much – it enumerates the “key_load_public” attempts (rsa, rsa-cert, dsa, dsa-cert, ecdsa, ecdsa-cert, ed25519, ed25519-cert), then “Enabling compatibility mode for protocol 2.0” and the SSH version “Local version string SSH-2.0-OpenSSH_7.2ps Ubuntu-4ubuntu2.1”, then fires off the “connection reset by peer” error again.  Dpkg -l reports that openssh-client is “1:7.2ps-4ubuntu2.1”.

What’s the server’s version of OpenSSH?  According to dpkg -l, it’s “1;6.7p1-5+deb8u3.  Is that right – 1.6.7?  And if so, how do I find out if there’s a cryptographic configuration incompatibility between 1.7.2 and 1.6.7?  [Certainly I can see that we have no such “connection reset by peer” issue between my Win10 Bash on Ubuntu shell, running 1.6.6p1 of openssh-client and the Debian8 box’s 1.6.7p1, so cryptographic compatibility between 1.6.6 and 1.6.7 is a reasonable assumption.]  Or better, is it possible to upgrade the Debian8 box’s openssh-server to something later than 1.6.7 – preferably (but not exclusively) 1.7.2?

On the server, I can crawl through the /etc/ssh/sshd_config” file to look for configured parameters (RSAAuthentication yes for example), but that doesn’t tell me what the OpenSSH defaults are, and doesn’t tell me what’s necessarily being asked of OpenSSL either (which might be swallowing the actual error).

Aside/Weirdness: networking

I started to pursue the idea of upgrading OpenSSH, so I ran sudo apt-get update to prepare for updating everything in the VM.  That’s when I noticed I wasn’t getting any network connectivity, as it spat back “Could not resolve ‘'” and “Could not resolve ‘'”.

Vbox Mgr indicates I’m using NAT networking (the default), which has worked for me in the past – and works fine for the same Vagrant box running on my Win10 VirtualBox/Vagrant instance (sudo apt-get update “Fetched 529 kB in 3s (142kB/s)”).  Further, the Ubuntu host for this VM has no problem reaching the network.

So I tried changing to Bridged Adapter in Vbox Manager.  Nope, no difference.  Why does the same Vagrant box work fine under Windows but not under Ubuntu?  Am I cursed?

Back to the root problem

Let me review: I’m having a problem getting Ansible to communicate with the VM over SSH.  So let’s get creative:

  • Can Ansible be coerced into talking to the target without SSH?
  • Can Ansible use password authentication instead of public key authentication for SSH?
  • Can the Ubuntu client be downgraded from 1.7.2 to 1.6.7 openssh-client?

Lightbulb moment

Of course!  The “connection reset by peer” issue isn’t a matter of deep crypto at all – unless I’m misreading this, the fact that the Ubuntu SSH client takes nearly a minute to return the “connection reset” error and the fact that the Debian VM doesn’t seem to have any IP networking ability off the host…adds up to SSH client not even connecting to the VM’s sshd?

Boy do I feel dumb.  This has nothing to do with crypto – it’s simple layer 3 issues.

Reminds me of a lesson I learned 20 years ago, and seem to re-learn every year or three: “When you hear hooves, think horses not zebras.”

Then how do we establish where the problem is – Virtualbox, Ubuntu, Debian or something else?

  • If it’s a problem in the Debian VM, then download a different Vagrant box
  • If it’s a problem in the Virtualbox setting, keep trying different network settings until one breaks through
  • If it’s a problem in the Ubuntu host, look for reasons why there’d be a block between (host to VM or vice-versa)

What other evidence do we have?  Well, when I run vagrant up from the Ubuntu host, it gets to “default: SSH auth method: private key” then eventually reports “Timed out while waiting for the machine to boot.  This means that Vagrant was unable to communicate with the guest machine within the configured (“config.vm.boot_timeout” value) time period.”  Makes me more suspicious of the VM.

Searching the Vagrant boxes registry, mosaicpro/html looks like it’s desktop (not locked-down server) oriented, so I tried that one.  Watched it boot, then report “default: Warning: Remote connection disconnect. Retrying…” over and over for a few minutes.  The console via Vbox Mgr looked like the Ubuntu VM was trying to configure networking (even though DHCP had offered it an address of – which must’ve been the NAT adapter, since my home network runs on 192.168.1/24).  But oddly, networking from within the client was working fine after that – ping out to my home router ( returned fine.  OK, then I’m *definitely* suspecting that Debian/jessie64 (8.6.0) box.

Vagrant/Debian downgrade anyone?

So, after all this, can I download a previous version of the Debian/jessie64 box (e.g. 8.5.2, not this troublesome 8.6.0)?  Let’s try it, using this article as basis.

(I went one step further and ran the initial command as vagrant add debian-8.5.2 – and amazingly, this variation seemed to work!)

And here’s some promising results:

  • lsb_release -a reports the 8.5.2 box as “Debian GNU/Linux 8.5 (jessie)”, vs the 8.6.0 box as “Debian GNU/Linux 8.6 (jessie)”
  • A quick look at the /etc/ssh/sshd_config from the 8.5.2 box shows there is *no* insertion of the PasswordAuthentication configuration parameter (let alone setting it to “no” like in the 8.6.0 box)
  • Network connectivity from the 8.5.2 box to my home router is awesome (vs the 8.6.0 box that can’t seem to ping out of a wet paper bag)

Final Lesson

If you’re a Vagrant + Virtualbox user, stay FAR away from the 8.6.0 version of the debian/jessie64 box (unless you’re prepared to fight with these same issues I have, and probably other ‘security lockdown’ ideas that I haven’t even uncovered yet, but are almost surely there).

What’s the (remaining) value of cryptographic hashing of small values these days?

A friend forwarded this to me today – one more in a long narrative of the incredibly reduced value of hashing
to make it hard for anyone seeing the hash to determine the original data being
hashed (for small-sized inputs).  Hashing
a password/passphrase, hashing a “unique identifier” – these
approaches to obscuring (for lack of a better word) the password/unique ID seem
effectively moot to me.  I honestly don’t
know that there’s any real value in performing the hash and then storing or
exchanging it – frankly, the difference in your risk between
“sending/storing the password” and “sending/storing the hash of
the password” seems pretty small.
Even small inputs + small salts seem doomed, given these
massive advanced in FP calculation arrays. 
Further, it seems like “stored, single-value salts” are just
as pointless, given the amount of research that attackers generally put into
discovering these stored/fixed salt values – so storing a hugely long salt
value just feels wrong to me for many threat scenarios.
Do any of the SHA-3 algorithms take massively more time
on these hash-calculating clusters?  If
not, what other options can our products use for protecting small values like
passwords and unique identifiers?
I get that there are still some attack vectors against
which a hashed password is a useful mitigation vs. the raw password
itself.  It just seems like we’re getting
further and further away from the knee-jerk “hash it and you’re much better
off” that I was taught at the feet of my cryptographic elders.
Note: I believe there’s still value in using
modern/advanced hash functions to predict the integrity of a known piece of
information (digital signatures, message authentication):
  • e.g. hash a large document and sign it to later assert
    with some degree of confidence that the document hasn’t been tampered with
  • e.g. compare the previously-stored password hash to
    determine if the supplicant has possession of that password

“A presentation at the Passwords^12 Conference in
Oslo, Norway, has moved the goalposts on password cracking yet again.
Speaking on Monday, researcher Jeremi Gosney (a.k.a epixoip) demonstrated a rig
that leveraged the Open Computing Language (OpenCL) framework and a technology
known as Virtual Open Cluster (VCL) to run the HashCat password cracking
program across a cluster of five, 4U servers equipped with 25 AMD Radeon GPUs
communicating at 10 Gbps and 20 Gbps over Infiniband switched fabric. Gosney’s
system elevates password cracking to the next level, and effectively renders
even the strongest passwords protected with weaker encryption algorithms, like
Microsoft’s LM and NTLM, obsolete. In a test, the researcher’s system was able
to generate 348 billion NTLM password hash checks per second. That renders even
the most secure password vulnerable to compute-intensive brute force and
wordlist (or dictionary) attacks. A 14 character Windows XP password hashed
using LM for example, would fall in just six minutes, said Per Thorsheim,
organizer of the Passwords^12 Conference. For some context: In June,
Poul-Henning Kamp, creator of the md5crypt() function used by FreeBSD and
other, Linux-based operating systems, was forced to acknowledge that the
hashing function is no longer suitable for production use — a victim of GPU-powered
systems that could perform ‘close to 1 million checks per second on COTS
(commercial off the shelf) GPU hardware,’ he wrote. Gosney’s cluster cranks out
more than 77 million brute force attempts per second against MD5crypt.”
 URL –