Bizzaro Issue of the Day

The toughest job on earth is to be the MIS Administrator at a technology consulting company. Can you imagine having to put up with all the monster ego’s and know-it-all’s that inhabit a relatively successful consultancy?! Just think about all the people telling you how to do your job and bitching that they should have administrator access to the environment because “they do this for clients”. This is the life of my friend and network admin Charlie (his last name has been omitted to protect the innocent).

And with all that said, it is he who ran into a strange issue in our internal AD and Exchange environment that neither I, nor a very experienced friend (whose name will be full omitted because I doubt he wants to be associated with this blog at Microsoft Consulting Services, have ever seen before.

Specifically he deleted a Contact a few weeks ago from the local Domain NC (Naming Context, or directory partition for those who care). Yet this contact object persisted in the Exchange GAL. Now, the GAL in Exchange 200x is, in fact, the AD Global Catalog and both Outlook 2000+ and the Exchange 200x DS proxy (which is used by legacy clients such as Outlook 98) query the GC to build and return GAL requests and entries. Even online (meaning non OAB) queries to the GAL were returning this contact, so it appeared this object persisted in the GC despite being deleted from the DomainNC in which it was homed. This is very, very strange, to say the least.

Now here is where it gets really weird. The GC partial replica set for the DomainNC and the actual DomainNC the GC is a member of are one and the same, so addition and deletion of that contact object should have happened immediately. However since it didn’t, Charlie went and (in an attempt to clean up the GAL after two week of this object lingering) used the Exchange System Manager to delete the contact by expanding the All Global Address Lists node and selected the properties of the Default Global Address List and from the General Tab of the properties page he clicked the Preview button to list the entire GAL. Now, thanks to uber-smart-person Mark Russinovich and his tool Process Explorer, I was able in realtime to view the netstat activity of the System Manager MMC snapin and validate that, in fact, the GAL preview is sourced from the Global Catalog. HOWEVER, any action such as viewing the properties of the object causes the MMC process to open a direct STANDARD ldap connection to a DC in the Domain the object is homed in. Once the objects are listed in the preview window, the GC connection is closed.

What this means is that when Charlie deleted the object, he in fact deleted it from the DomainNC for a second time! In fact, if you try to delete and object directly from the GC (using LDP or ADSI edit) you will be refused the action by the server. So the big question is what was up with that contact and why was it hidden from the standard AD Users and Computers yet was still visible from the GC AND was clearly still and instantiated AD object (since the delete option would have failed if the object wasn’t in the DomainNC). Also, why didn’t it delete the way it was supposed to?

Well, I have a couple of theories, but at this point that’s all they are. The first is that the ADC replicated the object back from Exchange 5.5, but that makes no sense since the object should have been deleted on the 5.5 side as soon as the ADC CA picked up the deletion from the AD and it doesn’t explain why the object was not visible from ADUC.

The second theory is that the object was deleted (meaning that it was in the deleted objects container in the domain NC), but that the change to a deleted object was somehow not represented in the GC instance of the object. Since the object was tombstoned in the DomainNC and hadn’t yet been garbage collected, it would technically still exist. Perhaps the second delete action caused the proper deletion of the GC instance of the object to occur. If this was the case, then that would point to a major inconsistency in the AD and one hell of a bug. I find this explanation far less likely than the first, though neither one really makes any sense at all.

Had the time existed for me to properly diagnosis the issue, I would have used LDP with the “Return Deleted Objects” object control turned on to see if there was (a.) a second instance of the contact and (b.) if there was a deleted instance of the contact. This brings me to my first tip of the day:

Need to view tombstoned objects in the AD? Use LDP with the “Return Deleted Objects” control enabled by selecting the Options menu and Controls. In the load predefined drop down list, select Return Deleted Objects. You can now browse you tombstoned objects to your hearts delight.

My second tip of the day is if you need to set certain AD objects so that they are always created with a certain set of entries in the objects ACL, you can do this by opening the AD Schema MMC snap-in, select the properties of the class object and edit the ACE’s on the Default Permission tab. What you are doing is changing the values of the defaultSecurityDescriptor attribute on the class object. Now, any instance of that object that is created will have the security descriptor you just specified (and if you uncheck inheritance, the object will get that descriptor regardless of container ACL’ing or inheritance). Remember to reload the schema to get your changes to take effect and, as always, be extremely careful when modifying the AD schema.

Well that’s enough geeking for one day.

A quick RANT!

Okay, folks, listen up, it’s RANT TIME (not that anyone reads this blog):

STOP RUNNING YOUR BOX AS an ADMIN. In short, don’t use an account that has been granted local administrator privileges on your Windows NT 2K, XP or 2k3 system. And you should NEVER, Ever, EVER run as a Domain Admin as your primary interactive login (I see this bad idea enacted FAR to often).

Now, I can hear the bitching and moaning about installing software and whatnot. Guess what, you can use the runas command either via the command prompt (runas /user:username applicationname) or by shift-right clicking the icon or executable you need to run. Just about ANY application can be installed via this method and, if you need to perform administrative tasks, using the runas command is THE way to do it.

Let me give you a little example of why running as a local admin is so dangerous (if you don’t already believe me). Download ftp://blog.ntpower.com/bsod.zip. Now login as a regular non-privileged account and run it. Nothing happens. Now login as an admin (make sure you save anything important) and run it. Wow, wasn’t that cool, that little program just drop-kicked your machine and now you are looking at a nice shade of blue.

So what just happened? Well, you just ran a small program written by Mr. Windows Internals himself, Mark Russinovich that demand loads a device driver that immediately calls KeBugCheckEx, which in turn tells the system to crash. Now the point here isn’t that this utility just blue screened your machine, it’s that it loaded a device driver that can do VERY bad things to and with your computer. This driver could just have easily instantiated raw sockets, inspect network traffic, disable other drivers and services, key log your system and just about ANYTHING else you can thing of. More importantly, that application needn’t stop with just loading a device driver; it could inject code into other processes, launch processes in the system context, compromise any service account credentials stored for a given service and attack the local SAM, along with many other wonderfully awful things. That folks, is why it’s called an ADMINISTRATOR account.

As you can imagine, running as a Domain Admin (or any significantly privileged AD account) interactively makes the situation an order of magnitude worse. Take everything I mentioned above, and multiply it by the number of computers in your organization.

So stop running as Admin. It simply isn’t necessary, and it is REALLY dangerous. In fact, the vast majority of worms and other malware would be stopped in their tracks if people would simply use a standard user account. So run Windows Update, keep your AV up to date and get your (and if your an Admin, your users) account(s) out of the Administrators group.

Before I retire, a couple of quick tricks:

1. If you need to edit your network settings, you will notice that there is no runas option if you right click on any of the network connection objects (this is a major oversight by Microsoft if you ask me). You can do a runas at the command prompt on netsh, but netsh at times is like using a sledgehammer to crack an egg. A quick trick is simply to create a standard shortcut to Internet Explorer iexplore.exe (not the desktop icon) and do a shift+rightclick runas on that. Use that IE instance running in the privileged account to browse to your network connections (remember, IE and explorer.exe are essentially the same thing). Now you can edit your network connections as an Admin with out logging out. Remember, close that IE session after you are done using it as you don’t accidentally want to browse the web with it.

2. There are some crappy applications out there (usually home grown, aka amateur hour) that simply don’t run properly without Admin privileges. These turds can be tricked in most cases with symbolic links from HKLM to HKCU or specific loosening of keys or directories. I am not going to go into how to do this except to say that Microsoft has a great application compatibility toolkit to help users and IT pro’s get craptacular applications running with LPN (least privileges necessary… a new acronym to add to ones lexicon). You can get the latest version of the application compatibility toolkit from:

http://www.microsoft.com/downloads/details.aspx?FamilyID=7fc46855-b8a4-46cd-a236-3159970fde94&DisplayLang=en

End Rant

GreyMatter, how to install on IIS6 for those who care

So I went through all the trouble of setting up GreyMatter (the free blogging software from the GreyMatter project at http://www.greymatterforums.com/) only to decide that blogger.com was less trouble than http redirection (to get past my ISP’s blocking port 80). HTTP redirection, as we all know, sucks.

But since I went through all the trouble, I might as well help someone else who goes down this path. Now, being a minion of the Evil Empire, I was not satisfied with simply setting up a ‘nix box running Apache. No I had to go and try to run GM on IIS 6.0 (which the forum moderators at greymatterforums.com will tirelessly tell you is not supported). If you are trying to set it up on ‘nix, then just follow the direction that come with the software. If on the other hand you insist on being an imperial shill like me, read on.

In truth, it wasn’t all that difficult, but it certainly isn’t for the unacquainted especially if you are trying to do it right. What do I mean by right, you ask?

Well first of all, right would be NOT running it in the default site or the default application pool. Right would also be making sure that you ACL the directories and files correctly. There is also a little bit of editing of the default CGI scripts that come with GreyMatter.

So here it goes, how to install GreyMatter 1.3 on IIS 6.0:

1. First off, create a new directory off the root or in inetpub (we will call it GM from here on out).

2. Create a new site, set that site to use the new directory (GM) you have created as its home directory.

3. Create a new, standard user account and place it in the local machine \IIS_WPG group. Use the Local Security MMC to grant “Adjust memory quotas for a process” and “Replace a process level token” rights to the account you just created (if you are really fly you can accomplish this through an AD GPO, which might make sense if you are configuring multiple IIS servers to run CGI based apps as this is a common configuration requirement for running perl/cgi apps on IIS 6).

4. Create a new application pool (call it something obvious) and configure it to use the account you created for its credentials.

5. ACL the GM Directory to grant the same account you created for the application pool modify permissions to the directory. Make sure local machine \Users has Read&Execute permissions.

6. Create a cgi-bin subdirectory underneath GM.

7. Create an emoticons and archives subdirectory under GM also.

8. Copy the GreyMatter CGI scripts to the cgi-bin subdirectory.

9. ACL the following files to grant local machine \Users Modify permissions (roughly equivalent to chmod’ing to 666): gm-authors.cgi, gm-banlist.cgi, gm-config.cgi, gm-counter.cgi, gm-cplog.cgi, gm-entrylist.cgi, gm-templates (frankly, I don’t think this is safe from a code perspective – using script files to hold data is asking for trouble, but it’s how it’s coded and free is free).

10. Copy the other GM files to their appropriate places, namely the emoticons to the so named directory and the index.html and gm-icon.gif to the GM directory.

11. ACL the index.html file in the GM directory to give local machine \Users Modify permissions and ACL the Archives directory to grant local machine /Users Write privileges.

12. Now, go download the latest rev of ActivePerl (http://www.activestate.com/Products/Download/Download.plex?id=ActivePerl)

13. Install Active Perl.

14. Through IIS manager, modify the site root Home Directory setting to remove the default application and set the execute permissions to none. Choose all the nodes and apply this setting.

15. Now select properties the cgi-bin directory through IIS Manager; on the Directory tab create a new application, with its starting point in cgi-bin and make sure you specify the application pool you created earlier.

16. One the cgi-bin directory, be sure to set the execute option to Scripts only (remember we disabled it for all nodes in step 14.

17. Click “Configuration” and add a new script mapping for “.cgi”; the executable path should be ‘C:\Perl\bin\perl.exe “%s” %s ‘ (substitute your install path as needed) and the verbs should be limited to “GET,HEAD,POST”. DO NOT allow all verbs. Having done this step you should actually see the little folder icon for cgi-bin turn into a little cog to note that IIS now recognizes the cgi-bin folder for the site as the “application”.

18. Last but not least, edit the gm.cgi, gm-comments.cgi, gm-karma.cgi and gm-upload.cgi files replacing the first line (#!/../perl) with ‘ chdir “C:\\blogroot\\html\\cgi-bin” ‘. Replacing the perl path is okay as IIS already knows the path to the perl executable based on the script mapping (smart) but instead needs the local working path for cgi-bin set (not so smart).

Having completed these steps, you should now have a working blog. Try it out by access the /cgi-bin/gm.cgi and configuring you GreyMatter install.

Now, why have I gone through all the extra effort of creating a separate app pool, and doing all the ACL’ing and what-not? Because it’s a hell of a lot safer. Worst case scenario, my blog gets hacked, not the whole machine and all the sites on it. Compromising one app pool and the paths it has access to (via the credentials configured) is not a total loss.

Now do keep in mind, in order for CGI to run, a fairly dangerous privilege is granted to the app pool’s account, namely CreateProcessAsUser(); aka “Replace a process level token”. However, in this case, the new process will inherit the environment of the calling process which is not terribly privileged and in the case of how the app pool is using it, its simply calling the IUSR_ account. If something were to happen to grant malicious code access to the app pool in the context of the account it runs in, that code would have to pass credentials for some really cool account when calling CreateProcessAsUser(). All in all, app pools and the ability to sand-box applications relatively effectively in IIS 6 illustrates why it is about 1 BILLION times better than the utterly crap IIS 4.x/5.x (and why you’d be a foolio to stick with IIS 5.x).

Also note that if you happen to have googled onto this blog looking for help with CGI apps on IIS 6, take out the GM stuff above and you have a pretty good set of instructions on how to configure common CGI apps to run under IIS 6.

The funny thing is I am not a web developer or admin; when you work with Exchange 200x day in and day out, you are forced to learn a LOT about IIS. I will have to remind myself to post my notes on using the GUI metabase editor that comes with IIS 6 resource kit to set custom metabase ACL’ing for delegating permissions to the IIS Manager. I am sure there are people out there that want that info.

Well, that’s it for tonight.

Finally… A Blog

Yes, I have finally given in and joined the ranks of bloggers. So just what will my blog be about? Absolutely nothing all that interesting unless you are a dweeb like me.

Anyhow, I primarily intend to use this to post useful technical tidbits relating to my job (so that I can remember just what I am, and have been, doing as of late). I might also go on the occasional rant.

Of course, I might also just drop this whole thing if I think its a waste of time.

Who knows…

-Ned