I happen to find out yesterday that none of my outgoing mails were leaving the Exchange outbound message queues.  After taking a quick peek at the event log I saw a long list of red messages – Event ID 5774 Net Logon DNS failed access errors.  Somehow, my DNS forward lookup zones were messed up, and Active Directory was in a tizzy over it.  This caused Exchange to sulk and thus no mail was leaving the server. Naturally, I checked Google and found an article about reinstalling DNS zones in Active Directory (link). 

I do not confess to being an AD nor an Exchange 2000 guru, so I followed the instructions as best as I could, and eventually fixed my problem (well it at least appears that way for now).  Below is a synopsis of the steps I followed.  Be sure that you know what you’re doing before following these instructions, they worked for me, but not every situation is the same and there is a possibility that you might toast your server.  If possible backup the server first. Also, I only tried these steps on a Windows 2000 server domain controller with Exchange 2000 on the same box – there is no saying how these steps will behave in a different configuration.

Haven’t scared you off? Good, here we go…

1. Open up Administrative Tools in the Control panel (If I’ve lost you already then I would suggest calling in an expert).
2. Double click the DNS applet to view the currently installed forward lookup zones and reverse look up zones.
3. Right click each zone, click properties and change active directory zones types to “Standard Primary”.
    (Before making and changes, make a note of the zone settings for later).
4. Double click the services applet, find the DNS Server service and stop it.
5. Stop the Net Logon service too.
6. In %SystemRoot%System32DNS remove the “.dns” files corresponding to the zones you just changed in step #3.
7. Double click the Active Directory Users and Computers applet.
8. Click the view menu and make sure that Advanced Features is checked.
9. In the treeview on the left navigate to SystemMicrosoftDNS
10. Remove the zones (note: ignore the scary dialog about removing entries from Exchange if you have Exchange installed, it didn’t seem to break anything).
11. Remove the zones from the DNS server applet.
12. From a command prompt execute ipconfig /flushdns.

The article, I read, mentioned removing and reinstalling the DNS server service at one point, which cannot hurt (unless you have custom configured your DNS entries).  I’ll not document these steps because if you’ve got this far then you should know how to do this already.  The DNS service can be uninstalled from “Add Remove Windows Components” in the control panel.

12. With a fresh install of the DNS server continue with the following steps.
13. Make sure the DHCP client is running.
14. Remove all DNS IP addresses from the active network TCP/IP settings, and set the main DNS server address to the IP address of this domain controller.

NOTE : The Dynamic Host Configuration Protocol (DHCP) client service needs to be running on each of these computers to register the records in Dynamic DNS. It is not relevant if the computer is a DHCP client or not. You must have this service set to “start” and the “Start up” type set to “automatic.” The DHCP client service is what registers records in Dynamic DNS. (Refer to the description in the Computer Management snap-in.) – Yes this one perplexed me at first, but it seems to make sense.

15. If not already started, start the DNS Server service and Net Logon service.
16. Open the DNS server applet again and add the forward zones and reverse zones back from step #3, their type should be AD integrated.
17. From the command line execute ipconfig /flushdns followed by ipconfig /registerdns.
18. Make sure that you can still access the Active Directory Users and Computers applet.
19. Reboot the server for kicks, and all may be good again.

After I had completed the above I no longer received Net Logon errors, and pending mail in the Exchange SMTP queues started to leave the server.  I’ll check back in a few hours to see if I see anymore problems listed in the event log, but for now the outlook appears sunny.

My laptop was behaving strangely this evening – applications
were spontaneously hanging, system was freezing when logging out, etc.  A
closer look at my Windows XP event log told me that STISVC.EXE (still
imaging service) was failing to start, and a whole load of errors were
reported about the COM+ subsystem. Uh-oh!

So, I reinstalled COM+ (as per my instructions),
which fixed the COM+ errors, but did nothing for the random system
hangs.  So, after backing up the user profiles on my machine, I
pulled up Ghost, found my last good image from June and restored my
machine. A quick break of 15 minutes, while Ghost does it’s magic,
after which my machine was stable again.  All I needed to do then
was run windows update, update my anti virus application, and restore
the user profiles, presto – a working machine.

The moral of this story boys and girls – always have a recent backup
of your data files, and and a working operating system image stashed
away somewhere.

The problem has to be hardware related because a reinstall of a
previous working Ghost image was experiencing the same “hanging”
symptoms as before, but yet my server worked great in safe mode (no
surprise since most of the services are not running in safe
mode).  So, here is what I did to resolve my problem:

While in SM, I made a note of all the services with start up type set
to “Automatic”, and then disabled them all.  I noticed that most
of the services, not running, were using svchost.exe
-  a service host for non-service applications. During earlier
testing I had seen several instances of this executable running in the
task manager, and killing a few of the instances would sometimes
resolve the hanging issue (most of the time the system would become
unstable).  I wondered if svchost.exe was hung up trying to access a broken piece of hardware, and all other requests using the same executable were then deadlocked.

After rebooting into normal mode, I was able to install Windows 2003
SP1 and remaining updates from windows update (this would at least replace svchost.exe, if it happened to be corrupted). Finally, I enabled the
services that I had disabled earlier, leaving out those services that I
did not require (such as the sound system and print spooler
services).  I was going to enable all the services to see which
one caused the problem, and possibly pinpoint a hardware component
failure, but my server was behaving, and running faster, so I decided
not to toy with it.

I figured I should start backing up some of the more recent changes to my data in case my server failed catastrophically (I
usually keep a good back up, but it was out of date), so I started dumping files onto
DVD.  Midway through burning files my E drive failed. Fortunately
I was able to get the drive back with a cold reboot, and there was very little
data on it, so I was able to copy over everything to my D drive, and
continue burning everything to DVD.

Presently, everything seems to be working.  Windows update works
fine, so does RDP, and my E drive has not failed since I cleaned it out
(but I’m not using it).  I am tempted to experiment with svchost.exe
and the services list some more, to see which service causes the system
to hang, but I would like to think that the problem was related to the
secondary disk failure, and having look no further.  I am in need
of some more disk space on my server, so I plan to replace this disk
anyway. 

Hopefully I’ve seen the end of my problem for now, and can spend
tonight enjoying a glass of wine in front of the TV – not in the server
room.

I ran into problems with one of my Windows 2003 servers
yesterday.  It all started when I had connected remotely (via RDP)
from the office and installed the latest Microsoft updates.  The
upgrade went smoothly, but after a reboot I was no longer able to
connect to my server remotely.  Later, when I got home I went down to my server
room to investigate the problem.  I was half expecting to see a BSoD, but I
was able to log onto the console and everything looked peachy, except for RDP not working.

The next step was to check out the network settings, perhaps my card
had gone phut, or maybe the upgrade had changed the network
settings.  However, I found out that I was not able to pull
up the network settings dialog, without mmc.exe
hanging.  Rebooting into safe mode enabled me to check the
network, and all the settings were as they should be, but once I rebooted
into normal mode, I could not get at the network properties once again.

At this point I decided to cut a big corner and restore the system from
a ghost image that I had made back in January.  My machine has
been working fine since January, until yesterday, so I know the image
would work.  I waited a brief pause while Norton Ghost did it’s thing,
and then I was able to boot back into my system.  The network
hanging issue had gone away – hurrah! All I needed to do was run
windows update again, and run an upgrade on my anti virus software -
sounds easy enough – oh no!  Both windows update and the anti virus
application were suffering from the same “hanging” problem that I had
been experiencing with the network settings.

Since I could count on the Ghost image being correct, I figured that
the problem must be hardware related – time to start disabling hardware
components.  I cracked open the case, added a new network card,
disabled the on board LAN, disabled the on board modem, and disabled
the
on board sound card.   After another reboot I was able to run
the anti virus update successfully, windows update was still not
cooperating.

I wondered if bad disk sectors or faulty memory could be the
problem.  So I spent the next couple of hours running check disk
(checking for bad sectors).  On another machine I attempted to
download memory checker software, but I didn’t find anything that worked.
I had used memtest86 in the past, but for some reason this application
did not even boot on my server, never mind check the memory.

At 2am this morning I decided that I should shelve this problem for
another day (after some sleep I may have some fresh ideas).  RDP
seemed to be working again, as did my anti virus software (I ran a full
scan), but windows update is still broken – I don’t even get to see
the welcome screen, IE just goes into a tailspin when I visit the site, which makes me think that the problem is IE
related. However, I can browse other web sites.

So, question time…..

1. Does anyone know of any good hardware diagnostic software, which
does not cost an arm and a leg (free would be desirable), will not
destroy the data on my drives, and doesn’t require me to pull apart my
machine?

2. Has anyone out there experienced similar problems with windows update?

3. Can you share any thoughts on why random applications and system
configuration dialogs (they appear to be network related) may hang for no
reason, but work fine in safe mode.

4. Do any log files exist, which would give me an indication as to the
problem?  I
checked the windows update log, but it doesn’t appear to indicate an error. 
Also, the event log doesn’t show up any errors or warnings.

5. Please, no wise ass cracks about replacing Windows with
Linux, this post is not an excuse to start another OS debate. I really
want some insight into fixing my server.

http://www.microsoft.com/windowsserversystem/virtualserver/evaluation/vsmt.mspx

Note, this software is in beta.