Active Directory Authentication and RHEL, pt1

Ok, so it, again, has been a couple of months since my last technical post. The whole RHEL cluster thing has been tossed to the side for now so there was nothing to report.

In searching for the command to extend the Satellite Server's database, I came across my own blog entry for just that. I realized that I've neglected my own blog so I figured I should add something.

One project I've been working on over the past month or so has to do with overcoming the expected abysmal connectivity from Red China during the 2008 Olympics. The application folks were turned on to Aspera from our colleagues at Fox News. This software is, essentially, a fork of OpenSSH. It is designed to send file transfers through a UDP based data channel (they call it "fasp"). This has the effect of beating TCP based transfers over high-latency connections. Basically, Aspera transfers beats FTP hands down. It beats it so well, transfers can move at wire speed and overwhelm all other traffic. Fortunately, Aspera has built in throttling mechanisms into fasp that help guard against killing networks.

So, what does this have to do with Active Directory authentication on RHEL? Plenty for us. Since we'd be placing this thing on the bad-bad internet expecting reporters from Beijing to upload hours worth of Star Spangled Banner from any ol' internet cafe, authentication is a concern. The speed of this project forced us to put it in place before getting any SA or Help Desk resources on board with support. I don't like the idea of an SA or engineer getting called at 3am because some luddite in Beijing can't figure out their password. Linking in logins to the Aspera server to AD was a critical component.

Since Aspera runs on Solaris, RHEL and Windows, we expected to stick it on Windows and call it a day. Alas, there was one problem: lockout DoS attacks. Windows natively doesn't protect against this. RHEL does with pam_tally. RHEL also can handle LDAP authentication with an AD server. In trying to stick with a non-complicated roll-out in order to make ongoing SA support easy, RHEL became the OS of choice here.

So, there we were with the following solution: Aspera and RHEL with pam_tally and Active Directory authentication. Unfortunately, a lot became trial and error since there seems to be a dearth of real world examples on how to get RHEL to talk to an AD server. It turns out to be very straightforward and simple.

The weird thing about Aspera's software, however, is that they spent a lot of effort making FASP and then slapped on a generic web interface. They expect you to hire consolutants to make the Web interface professional looking. We tried in vain to get Apache to authenticate via LDAP without opening ourselves up to the lockout DoS. We even tried to compile in "mod_auth_pam" to take advantage of pam_tally, but it seems to refuse to use LDAP over SSL through pam, even though LDAP in the clear worked fine. I believe it has something to do with getting non-privileged UID access to the OpenSSL libraries or maybe the originating LDAPS port.

In the intrest of getting up and running in time for capturing team USA kick ass on video, we gave up trying to figure out the issue and decided to stick with a regular "htpasswd" authentication scheme. In the end, I left the mod_auth_pam in just to use pam_tally.


What follows is the fruit of our labors in this endeavor.



Install RHEL 5

Install Apache

Download the Aspera Enterprise Server for RHEL from the Aspera web site.

http://www.asperasoft.com/downloads/index.html

First, install the RPM as you would any other RPM. The bulk of the software installs in /opt, though binaries drop into the usual places under /usr. Next, configure the license with the provided license key by placing the keys into “/opt/aspera/etc/fasplicense.txt”.

Configure the Aspera Enterprise Server Software

Make a unix account with the same name (say, “ASPERA”). It is ok to lock the account from logins and even create a shell of “/bin/false”. Set the home directory to be the directory where all file transfers will work out of (“/home/ASPERA” is sufficient). We will use a UID of 7000 in our setup.

Now, add user ASPERA's UID and home directory to “/opt/aspera/etc/docroot” on one line, separated by a “:”.

# cat > docroot
7000:/home/ASPERA ^D
# cat docroot
7000:/home/ASPERA
#

Set the world writable and executable permissions for user videotransfer’s home directory so that everyone can access its contents.

# chmod 777 /home/ASPERA

Finally, you will need to hack one of Aspera’s perl scripts to allow everyone to share the same directory for file transfers. Near line 826 of “/opt/aspera/var/webtools/scripts/aspera-dirlist.pl”, change the command

$sought = $id.':'; To read $sought = '7000:';

Whereas “7000” is the UID of the unix account “ASPERA”.

Edit “/opt/aspera/etc/aspera.conf” to reflect these settings for bandwidth caps and target rate flows.

Under "Central":

TargetRateCapPerFlow="YOURLICENSEDCAP"

Under "Web":
BandwidthCap="YOURLICENSEDCAP"
TargetRate="YOURLICENSEDCAP"

Be sure to restart the aspera service.

# /usr/sbin/service asperacentral restart

Configure Apache

Within “/opt/aspera/etc”, create group login and password in the” htpasswd” format within a file called “webpasswd”. We choose to use “ASPERA” as the username.

# htpasswd -b webpasswd ASPERA SOMEPASSWORD

Add thes “ASPERA” stanza to the bottom of “/etc/httpd/conf/httpd.conf”. We want to be able to have everyone reach the Aspera directory without using "http://asperaserver.corp.com/aspera/user" all the time. Keeping root pointed at "aspera/user" is a good idea for this.

(httpd.conf entry to follow on another post)

Set DocumentRoot to be the Aspera directory.

DocumentRoot "/opt/aspera/var/webtools"

Copy “.htaccess” from “/opt/aspera/var/webtools/user” to “/opt/aspera/var/webtools”.


Configure LDAP

Create a new “/etc/ldap.conf” with the following entries. LDAP will be used to talk to the Active Directory servers, over SSL, to provide password authentication. You need to have a read only AD account created for the initial binding. This user account is used to lookup accounts to provide the password hash.

host WINDOWSDC1.domain.corp.com WINDOWSDC2.domain.corp.com

base dc=domain,dc=corp,dc=com binddn CN=READONLYUSER,OU=Users,OU=USA,DC=domain,DC=corpm,DC=com

bindpw SOMEPASSWORD

timelimit 120

bind_timelimit 120

idle_timelimit 3600

pam_login_attribute samaccountname

ssl on

tls_cacertdir /etc/pki/tls/certs

debug 5

Download your Certificate Authority certificates. Be sure that they are in PEM format. Ours come as PC friendly pkcs7.

# openssl pkcs7 -inform DER -in CA.p7b -print_certs -text -out CA.pem

Our CA certs have mulitple entries. The certificates from our pem file had to be separated out into different files. Fortunately someone addressed this issue. This PERL script can do the work for you.

#!/usr/bin/perl
#
# Splits a certficate file with multiple entries up into
# one certificate perl file
#
# Artistic License
#
# v0.0.1 Nick Burch
#
my $filename = shift;
unless($filename) die("Usage:\n cert-split.pl <certificate-file>\n");
open INP, "<$filename" or die("Unable to load \"$filename\"\n"); my $ifile = ""; my $thisfile = ""; while(<inp>) {
$ifile .= $_;
$thisfile .= $_;
if($_ =~ /^\-+END(\s\w+)?\sCERTIFICATE\-+$/) {
print "Found a complete certificate:\n";
print `echo "$thisfile" | openssl x509 -noout -issuer -subject`;
print "\n";
print "What file should this be saved to?\n";
my $fname = <>;
open CERT, ">$fname";
print CERT $thisfile;
close CERT;
$thisfile = "";
print "Certificate saved\n\n";
}
}
close INP;
print "Completed\n";


Next, move these new PEM files to “/etc/pki/tls/certs” and softlink their x509 hash values to their names. Hashes are calculated with “/usr/bin/openssl x509 –noout –hash –in FILENAME”. OpenSSL uses the x509 hash of a signature to match against a filename of the same name in order to know which certificate to use to verify authenticity. Observe the soft linking example below.

# cp *.pem /etc/pki/tls/certs; cd /etc/pki/tls/certs
# ls -alF
total 376
drwxr-xr-x 2 root root 4096 Jul 2 11:59 ./
drwxr-xr-x 7 root root 4096 Jul 2 10:22 ../
-rw-r--r-- 1 root root 249373 Jan 12 2007 ca-bundle.crt
-rw-r--r-- 1 root root 5054 Jul 2 10:31 ca1.pem
-rw-r--r-- 1 root root 4066 Jul 2 10:31 ca2.pem
-rw-r--r-- 1 root root 5060 Jul 2 10:31 ca3.pem
-rw-r--r-- 1 root root 5055 Jul 2 10:31 ca4.pem
-rw-r--r-- 1 root root 5054 Jul 2 10:31 ca5.pem
-rw-r--r-- 1 root root 610 Jan 12 2007 make-dummy-cert
-rw-r--r-- 1 root root 1832 Jan 12 2007 Makefile
# for i in `ls *pem`; do ln -s $i `openssl x509 -noout -hash -in $i`.0; done
# ls -alF
total 396
drwxr-xr-x 2 root root 4096 Jul 2 12:06 ./
drwxr-xr-x 7 root root 4096 Jul 2 10:22 ../
lrwxrwxrwx 1 root root 9 Jul 2 12:06 184d21f0.0 -> ca1.pem lrwxrwxrwx 1 root root 9 Jul 2 12:06 3cea7904.0 -> ca4.pem lrwxrwxrwx 1 root root 9 Jul 2 12:06 64cc7d0a.0 -> ca2.pem lrwxrwxrwx 1 root root 9 Jul 2 12:06 a4a62ce2.0 -> ca5.pem -rw-r--r-- 1 root root 249373 Jan 12 2007 ca-bundle.crt
lrwxrwxrwx 1 root root 9 Jul 2 12:06 d32b38ca.0 -> ca3.pem -rw-r--r-- 1 root root 5054 Jul 2 10:31 ca1.pem
-rw-r--r-- 1 root root 4066 Jul 2 10:31 ca2.pem
-rw-r--r-- 1 root root 5060 Jul 2 10:31 ca3.pem
-rw-r--r-- 1 root root 5055 Jul 2 10:31 ca4.pem
-rw-r--r-- 1 root root 5054 Jul 2 10:31 ca5.pem
-rw-r--r-- 1 root root 610 Jan 12 2007 make-dummy-cert
-rw-r--r-- 1 root root 1832 Jan 12 2007 Makefile
#

Configure PAM

In “/etc/pam.d/sshd”, configure the pam stack to include pam_tally and pam_ldap. The pam_tally module will prevent failed password attempts on this server from triggering an account lockout from the Active Directory. If a user enters in a wrong password five times (“deny=5” in the option list), they will be locked out from this server only for 61 minutes (one minute longer than Active Directory’s automatic reset). After 61 minutes, pam_tally will automatically reset and allow them to try again. The pam_ldap module will allow users to authenticate their passwords from Active Directory rather than from “/etc/shadow”. Password management for those accounts should then come from the Dow Jones Help Desk, as with all other Active Directory passwords. The order in which these modules are referenced within “/etc/pam.d/sshd” is important. The example below allows a local user account to authenticate against “/etc/shadow” after trying against the Active Directory server. This allows for normal Unix logins.

# cat /etc/pam.d/sshd
#%PAM-1.0
auth requisite pam_tally.so onerr=fail deny=5\ unlock_time=3660
auth sufficient pam_ldap.so
auth include system-auth
account required pam_nologin.so
account requisite pam_tally.so
account sufficient pam_ldap.so
account include system-auth
password include system-auth
session optional pam_keyinit.so force revoke
session include system-auth
session required pam_loginuid.so #

Create User Accounts
Create user accounts for every person expected to use this software. The user account names must match their Windows Acitve Directory name and their shell must be “/bin/aspshell”. Their unix passwords in “/etc/shadow” can remain locked for added security.

Thoughts on the current financial crisis

Here is the content of an email I sent out to some friends explaining the current financial crisis in the US:

"Bailout" is certainly the wrong name for this.

Picture this: you, me, and Frank are playing 5 card stud, no peek, jacks or a better split the pot. Anyone who rolls a 2 of clubs immediately has to fold and loses their ante and bets. The ante is high, say, the title to your car. Are you going to play? A rational person probably would not, even though you have a poor chance of being dealt a 2 of clubs.

That's the game we're playing. No one knows who has a rotten hand, and that is keeping everyone from playing.

So, with that game in mind, pricture this: you're the CFO of a small bank and you are sitting on $1 million dollars. You can't keep that in your safe, so you have to invest it somewhere. You shop around and find an investment that is likely to yield 1% or two better than a Treasury bond and is given a high rating by S&P and Moody's. It is backed, in part or in whole, by mortgages. You buy it, not just because it seems safe, but you were encouraged by the fact everyone was buying them. You did your job: maximizing contemporaneous shareholder value (biggest share price now).

Meanwhile, some circus monkeys in an investment firm somewhere weren't entirely truthful in creating an investment vehicle that was based on a pool of mortgages. When working with the rating agencies to gauge the overall risk, they kinda hid the truth behind the mortgages they were bundling. The guys working the ratings agencies are folks who didn't do well enough in business school to go to work for an investment bank, so they ended up working for essentially (apologies to my co-workers) newspapers (S&P and Moody's) and are not the sharpest pencils in the box. They even allowed themselves to be "wined and dined" by the firms while working on the ratings, because at $50,000 a year, a stay in a nice hotel with a lobster dinner is a nice change of scenery. Between mouthfuls of lobster, they gave their thumbs up.

These investment vehicles were then divided up and resold in packages with other investments by some other unwitting but enterprising young business school guy working at a different investment firm, which were in turn chopped up again by some other guy and resold with other entirely different assets. So now there is this investment vehicle that might be backed in part by someone who will not make his mortgage, but for the most part, the investment is probably backed by entirely other things that have nothing to do with home mortgages.

You bought this investment from the $1 million dollars you had before. Now, there's no way for you to know just what portion of that $1 million dollars you will see evaporate because someone can't pay their mortgage. It could be, literally, $1, or it could be the whole $1 million. You just don't know and you can't know as it would probably take $1 million dollars to figure it out. If you and everyone else who bought these things pooled together, you all would lose maybe 2% (I'm making this number up, but it is much less than 15%) of your overall investment due to deliquent mortgages. It's really nothing over all, but no one knows for sure how much of that 2% they own themselves.

No one is going to buy it from you because no one can put a finger on its worth. Worse still, you can't use it as collateral for a loan from a bigger bank because no one will take it. In January you could use it as the basis for $10 million in loans to small business. Thanks to the recent "mark to market" account rule, you now can't use that investment vehicle anymore and you've essentially had to remove $10 million dollars from the economy. That $10 million dollars is no longer available to the corner cabinet shop to borrow to make this month's payroll while they're waiting on their customers to pay for the cabinets they've ordered.

Now, the government is coming along saying they're willing to buy this investment from you based upon an auction. It'll probably mean you will have to sell your $1 million investment for something like $500,000. $500,000 is better than $0, and it allows you to get back in the business of using at least $500,000 for collateral or loaning out $5 million dollars to that cabinet shop to make payroll.

Should you lose your promised compensation package because you unloaded this to the government? That's unfair, in my opinion. In fact, it creates what is called an agency problem. If I'm that CFO, I might think twice about participating, and I might tell my boss that we're better off waiting for this whole thing to blow over as we'll likely to get more than $500,000 for it. Meanwhile, that cabinet maker has to close down because he can't make payroll.

In my opinion, yes, the guys that made the crappy bundles in the first place and the rating agencies bear most of the blame. The thing about the ratings agencies is that they are free and independent and there are only two of them. The whole world relies on them so you can't beat them up just yet and you can't do it so publicly. God, the horror that would entail if they had to stop doing ratings suddenly!

As far as nailing the guys who did the bundling, that market is so complicated and intertwined, the guilty will all be dead before we figure out who did what. I remember a year or two ago specialists had to be brought in to meet with the Fed to explain how these bundles worked. It went even above THEIR heads!

Yes, this is directly related to Fannie and Freddie [Mac]. They are the largest holders of these investments. The more "illiquid" these papers became, the more pressure they had to come up with other assets for collateral, until they had no other assets available to use.

The systemic risk is based on the fact no one knows who has what exposure so no one is willing to part with their cash. Investment banks who trade with each other thousands of times a day, using blind faith, are eyeing each other suspiciously. Banks are told they need to come up with more collateral than they ever thought they would need. In the end, it is the small business guy who can't get a loan to operate. Our economy is based on the short-term use of cash. I borrow today, loan tomorrow, and we all get to do things like make payroll, buy office equipment, build inventory, etc. off of this practice. That's the lifeblood of business and it is quickly drying up.