So SSH is awesome, Passwordless SSH using Key-based Authentication is even more awesome...but what if you have dozens and dozens of servers, and what if multiple users need to access said server? Do you really want to waste time copying SSH keys out to EVERY server you have in your environment?? Sure, you COULD use configuration management systems like Puppet or bcfg2 to handle this for you, but those are a real pain to set up.

Now, most shops I've worked in, have had LDAP servers in place for authentication between servers...my current workplace is no different. So that got me thinking, What if I could store SSH Keys within an LDAP directory? Then wherever I go on the network, my public key goes with me. Cool concept right?

Well, turns out it's been done before, but much of the documentation is spotty, lost to the ever-whirling eddies of the interwebz, or it just plain sucks and is a chore to read. So, here's me throwing my hat into the ring to show you all how to set this up.

For the purposes of this tutorial, we'll be assuming you're running a CentOS5/RHEL5-based environment, because that's what I'm running here at the office.

So first you'll need a few things: The sources for OpenSSH, the patch that hooks openldap-devel into OpenSSH at compile-time, the actual LDAP schema for OpenLDAP (there's actually one for Sun LDAP as well, if you're running Solaris) that sets up the LdapPublicKey ObjectType and sshPublicKey attribute, and you'll also need to install the openldap-devel and pam-devel packages.

So to start off, let's install those dependencies:

$ yum -y install gcc openldap-devel pam-devel zlib-devel openssl-devel

Once those are installed, go grab the OpenSSH sources:

$ wget -qO - http://ftp.jaist.ac.jp/pub/OpenBSD/OpenSSH/portable/openssh-5.4p1.tar.gz | tar zxf -

See what I did there? Favourite little bash trick of mine...download and unpack a tarball all in one shot. ;)

So now dive into the directory, download the patch and apply it:

$ cd openssh-5.4p1
$ wget -q http://openssh-lpk.googlecode.com/svn/trunk/patch/contrib/contrib-openssh-lpk-5.4p1-0.3.13.patch 
$ patch < contrib-openssh-lpk-5.4p1-0.3.13.patch

With the patch applied, it's time to do the ol' configure/make shuffle (Make sure you use these EXACT configure flags!):

$ ./configure --with-ldap --sysconfdir=/etc/ssh --prefix=/usr --with-pam
$ make

Now, we're going to stop here a second...normally you'd issue a make install here, but not yet...chances are OpenSSH is ALREADY installed, and as such, if you try to run a make install, it will fail miserably and leave the old (non-LPK'd) binaries behind. So for this next part, make sure you have some kind of local or out-of-band access to the machine you're working on, because at this point, you have to wipe ssh clean from the system.

BUT WAIT!!! DON'T DO ANYTHING JUST YET!!

 There's one more thing you should do here, and that's back up your current sshd configs as well as your sshd init script, because make certainly isn't going to do that for you (at least not in the case of the init script).

$ cp /etc/ssh /some/dir
$ cp /etc/init.d/sshd /some/dir

Now that you've got all that backed up, wipe out the existing OpenSSH install from a local console:

$ yum -y erase openssh-server

Once that's done, you can finally install the new version:

$ make install

Once the new version is installed, you're 3/4 of the way done on the OpenSSH side. Next up is to move the sshd init script back:

$ mv /dir/where/you/put/it/sshd /etc/init.d

Next up is the sshd_config file that was set up for you by make. Unless you have other custom configurations specific to your environment, you can leave the config alone save for the section regarding LPK, which should read as follows:

Of course I'm assuming you know at least SOMETHING about LDAP, so you'll know what else to change in there.

At this point, effectively you're done and OpenSSH will start without issue. It will however complain at you because there's a directory missing that it uses for privilege separation:

$ mkdir -p /var/empty/sshd/etc

And that's the SSH side taken care of...now for the LDAP server.

First, download the schema to your /etc/openldap/schema directory:

$ wget -q http://openssh-lpk.googlecode.com/files/openssh-lpk_openldap.schema -O /etc/openldap/schema/openssh-lpk_openldap.schema

Now just add an include line in your /etc/openldap/slapd.conf for the new schema:

include /etc/openldap/schema/core.schema
include /etc/openldap/schema/cosine.schema
include /etc/openldap/schema/inetorgperson.schema
include /etc/openldap/schema/nis.schema
include /etc/openldap/schema/openssh-lpk_openldap.schema

Once that's done, you're 90% of the way there; just restart openldap and then functionally, you're done. Though I'm assuming you have some kind of management interface for LDAP. (In our environment at the office, we use phpLDAPadmin.

So log into phpLDAPadmin and browse to the user you want to attach an SSH key to:

Now, under the user's "objectClass" entry, click "Add Value":

You should now see the following screen:

Select "ldapPublicKey" from the list and then hit "Add new ObjectClass".

Now that the ldapPublicKey ObjectClass is available to this user, you can add the sshPublicKey attribute to the user and paste their key in. Just hit "Add new attribute" as shown below, and you will get a dropdown menu.

In the dropdown menu, select "sshPublicKey" and you will see the dropdown change into the following:

Paste the user's key in here, then hit "Update Object", and that's it! You're done...the user now has their Public SSH key stored in your LDAP directory, and as long as their companion private key is available on the machine they're connecting from, they will never need to use a password to log into any machine that's pointed at your LDAP server.

So ESXi is wonderful and all, everyone loves it (except XenServer fanbois like me). But whenever you have to do something low-level with it. You get cock-blocked by millions of people saying "you can't do that". I can't do what?! It's Linux! I can do whatever I damn well please!!!

And that's just it...ESXi is nothing more than a Linux install that has been severely tweaked and screwed with. BUT IT'S STILL LINUX; thus it follows all the same rules.

Now if you've been around the block before with Linux, you've probably seen about half a dozen ways at least to reset a forgotten root password. Well, those evil bastards at VMWare have gone on some serious console ninja shit to block your keyboard-fu. So none of the techniques you're used to will work...except for one.

Basically what you're going to do, is go through a bit of a convoluted process to wipe the password field out of /etc/shadow on the ESXi host's boot volume. So fire up your favourite linux live cd (Personally I like Trinity Rescue Kit) and let's get started...

Step 1: Load RAID drivers

Now YMMV on this one, but this was definitely a step I had to go through, as all the servers at the office are Dell PowerEdge and use Dell's PERC RAID controller cards. So if you're in a similar situation, you'll need to execute the following:

modprobe megaraid_sas

That'll fire up the drivers for your PERC card...now as I said...YMMV. Google your RAID controller to find the right driver for it if you need to.

Step 2: Mount ESXi boot volume

Now here's where we REALLY get into things...and here's where you're probably gonna have a serious WTF moment. I'm not going to hold your hand when it comes to mounting stuff in Linux...you're a big-boy console jockey now...if you can't do this you should really just reinstall Windows now or buy a Mac and ignore everything I'm saying.

At any rate, to speed up the process, /dev/sdX5 is likely what you're going end up mounting...unless as before, you're working with a RAID, in which case you will need to mount every single drive in that RAID array manually.

Step 3: WTF?! Tarballs...Tarballs everywhere!!

Um...yeah...sooo...you were probably expecting a normal filesystem when you mounted those partitions, weren't you? Well...instead you got tarballs. Lots of them. Can you guess what's in 'em? If you said "filesystem", you're right. If you said "candy" however...you're horribly horribly wrong, hopelessly stupid and should really go back to playing FarmVille now.

TL;DR: VMWare like to screw with people who actually KNOW THEIR SHIT. And one way they screw with you is by sticking the entire ESXi filesystem in a bunch of tarballs. So what now? Well...

Step 4: Unpack root filesystem and modify /etc/shadow

Yup, you ACTUALLY have to unpack the / filesystem from a tarball-within-a-tarball (Yo Dawg?!), I recommend doing this within /tmp. Anyways, you're looking for a tarball called state.tgz. Untar that and you'll find local.tgz (again I say, "Yo Dawg?!"). In here is the entirety of what would normally be /etc. So dive into the etc folder, and edit the shadow file...at this point, the rest should be self-evident...but I'll keep going for the sake of all the little n00blets out there. 

Now the /etc/shadow file SHOULD look like this:

What we want to edit of course, is the root line, which you just strip the encrypted password data out of, then save the file, and THEN...

Step 5: Repack /etc and copy back to boot volume:

Do I need to explain more? *sigh* fine. While you're still in the /tmp/ directory, do this:

tar czf local.tgz etc; tar czf state.tgz local.tgz; cp -f state.tgz /path/to/mounted_esxi_boot_volume

Now again: If you're working with a RAIDed setup, YOU MUST COPY state.tgz to EVERY physical volume the RAID array is using, otherwise when the machine is restarted, the RAID controller will wipe out all your hard work.

And that's it...just reboot, and your ESXi host will have a blank root password!

Now as an added tip, if you're smart, you'll keep that modified state.tgz handy on a USB stick or network drive or something in case your genius brain has an old-people moment again and forgets the root password...AGAIN.

So, this goes out to all my fellow IT nerds: I'm looking at setting up a colo here in the Toronto area. The purpose? A little sandbox environment for me to tinker in.

There exists one problem however: I alone can't afford the monthly costs of running said colo, nor can I afford the upfront hardware costs. 

So consider this blog post a formal proposition. Any of you out there who are interested and willing to chip in some money, I will gladly give you full root access to any physical machine I install in this proposed lab setup. Any hardware you want to add to the rack, I will gladly do so as well, so long as you are the one paying for it. I will impose on you no restrictions regarding what you can or cannot run in the lab, with the single restriction being that I refuse to play host to any racist/hate-related or (of course) child exploitation content of any kind. If you're a creep, GTFO and delete yourself from my friends lists immediately. Otherwise, I don't care what you run so long as you help pay for things.

If this sounds like something you'd be interested in, please let me know.

I love open source. I really do. Just ask anyone. And part of what I love about Open Source is the community. I always know that somewhere out there, someone else is having the same problem I am. Or if you're lucky, you'll even bump into the people who wrote the software you're working with.

Having said that, a big part of the FOSS community is the Freenode IRC Network; and they really are a godsend. Innumerable channels devoted to thousands of FOSS projects with over 70,000 users logging on every day with countless questions that need to be answered. Freenode is so massive that it is now owned by Ubuntu benefactor and FOSS Think Tank Extraordinare, Canonical.

Yes, Freenode is awesome, I am truly grateful for it, and its buzzing throngs of FOSS geeks like me are the first thing I turn to when I get stumped. Yet there is a small problem, something that irks me whenever I show up in any given chat room on Freenode. I suppose given that this is all free, I shouldn't complain too much...but I just have to get this off my chest.

There are just too many stupid answers given on Freenode, and the stupidest of all are what I have termed "baby-and-bathwater" solutions; because like the aforementioned "throw the baby out with the bath water" cliche, these head-scratching non-answers do just the same thing. Rather than an elegant solution, you are fluffed off and told, more or less, that the only answer is to throw the baby out with the bath water.

Let me give you all a couple of examples, which I only just heard in these past couple weeks.

Situation 1: Though I just got a brand new LG Optimus 3D smartphone, just prior to this I had an old HTC Dream, and I was looking to, after much thought, hand-wringing and trepidation, root the phone and flash it with CyanogenMod 6 (as CM7 apparently didn't support the Dream).

<TuxOtaku> Hey I'm looking to root my HTC Dream and install CM6 on it...but I have a couple of questions. 1. How likely is it that I'll completely brick this phone and render it totally unusable? 2. Given the 911 update that Rogers pushed through, does that affect the rooting process at all.

<dumbassuser1> get a new phone, yours sucks.

See what I mean? Here's another situation...

Situation 2: In this instance, I was working on deploying puppet at work, and part of what puppet was going to do was a bit of post-install server hardening. So I only needed the hardening manifest to run once and then never again.

<TuxOtaku> Hey I'm trying to set up a manifest in puppet, but I only want it to run once on any given node. The purpose for this manifest is to do some simple server hardening and install a couple of packages. Once it's done I don't want puppet trying to execute anything from this manifest ever again. Near as I can figure, this involves me creating a dummy file with puppet and then using some manner of conditional statement to tell puppet to check for the existance of this file, and then saying that if the file does exist, don't run anything else in the manifest.

<dumbassuser1> Don't use puppet.

Again with the useless non-answers. And these aren't the only times I've ever seen this. This is a problem that seems to be endemic to the immediate nature of IRC help channels. 

I think the overall point I'm trying to make here is that this behaviour is a huge faux-pas. If you can't help in earnest, don't irritate newcomers with non-answers that do no one any end of good.

</rant>

So as I've mentioned, I am a HUGE Minecraft fan...and recently I purchased the Alpha version and played around with the server software a little. Problems with CPU usage and network latency got me thinking though...would it be possible to create a redundant network of linked minecraft alpha servers?

This idea came also from the notion that IRC servers can do exactly this, and since the underlying concepts behind how IRC servers do it is rather simple, I believe it could be done...and I will outline now just how I think it should all work out, but first let's tackle the major problems in building such a network.

Problems:

1. Level/World Data: From what I have seen, the biggest problem faced in constructing this network is that minecraft stores level/world data within .dat files. This data would need to exist or be accessible across each server in the network in order for this entire concept to be in any way feasible.

2. Uptime monitoring: Some manner of service will be needed to monitor the uptime and health of each server in the network, and an accompanying service will be needed to talk to each server on the network in the event that one or more servers become unreachable.

3. "Talk between servers": Another big problem is that, while yes, world data could potentially be the same between servers, that doesn't mean the entire network will have the same users on every server. This could potentially be catastrophic to level data if the problem is not addressed properly. The reason IRC networks can do what they can is because users connecting through one server can talk to users connecting through ANY OTHER server on the network. A similar method MUST be employed here in order for this entire concept to work at all.

Solutions:

TBA. I'm still working out how to tackle the aforementioned problems, but if you have any ideas, please feel free to comment on this post.