Jay's Journal - Bug in Adobe Reader/Acrobat 9.0 causes crash when the user's Application Data is on a network share.

Warning for any system administrators thinking of deploying the newly released Adobe Reader 9.0, or it's big brother Adobe Acrobat 9.0: if your users have Group Policy applied that uses Group Policy Folder Redirection to move their Application Data directory to a network share (with or without Roaming Profiles), it will cause Adobe Reader/Acrobat to crash almost immediately after launching. I have had this confirmed by Adobe UK Support as a known issue, and there is currently no ETA for a fix.

The issue has been reported by a handful of users on the Adobe Forums and some of my users ran into it after I began a test roll-out of Acrobat on our site yesterday.

These are the exact error messages we're seeing:

Microsoft Visual C++ Debug Library 

Runtime Error! 

Program: C:\Program Files\Adobe\Acrobat 9.0\Acrobat\Acrobat.exe 

The application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

and when attempting to view PDF files in Internet Explorer:

Microsoft Visual C++ Debug Library 

Debug Error! 

Program: C:\Program Files\Internet Explorer\iexplore.exe 

abnormal program termination 

(Press Retry to debug the application)

We often see odd problems with software running on our network due to the various configuration options we use that set our workstations apart from the average home computer (and a lot of computers in other schools, for that matter), and often trip up programs that haven't been tested on domain workstations. I ran a trace using ProcMon to see if I could work out what was going on:

98959 11:29:15.6155113 Acrobat.exe 972 CreateFile \Device\Mup\ NAME INVALID Desired Access: Read Data/List Directory, Synchronize, Disposition: Open, Options: Directory, Synchronous IO Non-Alert, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a 
99021 11:29:15.6181294 Acrobat.exe 972 CreateFile \\mydomain\users\ SUCCESS Desired Access: Read Data/List Directory, Synchronize, Disposition: Open, Options: Directory, Synchronous IO Non-Alert, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened
99100 11:29:15.6219431 Acrobat.exe 972 QueryStandardInformationFile \\mydomain\users\ SUCCESS AllocationSize: 0, EndOfFile: 0, NumberOfLinks: 1, DeletePending: False, Directory: True
99111 11:29:15.6222464 Acrobat.exe 972 CloseFile \\mydomain\users\ SUCCESS
99182 11:29:15.6254897 Acrobat.exe 972 CreateFile \\mydomain\users\ NAME COLLISION Desired Access: Read Data/List Directory, Synchronize, Disposition: Create, Options: Directory, Synchronous IO Non-Alert, Open Reparse Point, Attributes: N, ShareMode: Read, Write, AllocationSize: 0
99187 11:29:15.6257927 Acrobat.exe 972 RegOpenKey HKU\S-1-5-21-1708537768-2052111302-682003330-4640\Software\Adobe\Adobe Acrobat\9.0 SUCCESS Desired Access: Read
99189 11:29:15.6258324 Acrobat.exe 972 RegQueryValue HKU\S-1-5-21-1708537768-2052111302-682003330-4640\Software\Adobe\Adobe Acrobat\9.0\Crashlog NAME NOT FOUND Length: 144
99191 11:29:15.6258536 Acrobat.exe 972 RegCloseKey HKU\S-1-5-21-1708537768-2052111302-682003330-4640\Software\Adobe\Adobe Acrobat\9.0 SUCCESS
99192 11:29:15.6259012 Acrobat.exe 972 QueryNameInformationFile C:\Program Files (x86)\Adobe\Acrobat 9.0\Acrobat\Acrobat.exe SUCCESS Name: \Program Files (x86)\Adobe\Acrobat 9.0\Acrobat\Acrobat.exe
99193 11:29:15.6260191 Acrobat.exe 972 RegOpenKey HKLM\Software\Wow6432Node\Microsoft\Windows\Windows Error Reporting\Escalation NAME NOT FOUND Desired Access: Read

The suspicious line here is in red: Acrobat issued a CreateFile on the root of the DFS share hosting the redirected AppData, with CreationDisposition set to CREATE_ALWAYS; in other words, trying to overwrite the share... unsurprisingly this resulted in a NAME COLLISION since it already exists, and a call to Windows Error Reporting immediately followed.

As I had tested the software on my own laptop I first assumed that perhaps only non-Administrators were affected, but quickly put paid to that theory after checking in with a non-admin who was also testing the software who only had the problem on certain machines. After checking in with further users, I found that only our desktop workstations were affected, not laptops, regardless of which user logged in. Since the same user was affected on a desktop but not a laptop, I surmised the difference was likely to do with something specific the fact that our desktop computers are configured to redirect most users' Application Data directory to a network share, in order to keep consistent program settings regardless of where they log on. The laptops are not configured in this way, so that users can more easily access their accounts while off-site and not connected to the network.

Taking a page from Mark Russinovich's book (or blog, to be more precise), I setup one of my Windows Vista testbench workstations to generate a crash dump for Acrobat in order to analyse, and from reading through the stack trace it appeared the problem stems from the PDFLTerm() function in AdobePDFL.dll (remember this is a stack, so the cause of the issue is usually below all the exceptions):

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForMultipleObjects+0xc
kernel32!WaitForMultipleObjectsEx+0x11d
kernel32!WaitForMultipleObjects+0x18
kernel32!WerpReportFaultInternal+0x16d
kernel32!WerpReportFault+0x70
kernel32!UnhandledExceptionFilter+0x1b5
msvcr80!abort+0xeb msvcr80!terminate+0x4d
Acrobat+0x422e
ntdll!__RtlUserThreadStart+0x6f
ntdll!_EH4_CallFilterFunc+0x12
ntdll!_except_handler4+0x8e
ntdll!ExecuteHandler2+0x26
ntdll!ExecuteHandler+0x24
ntdll!KiUserExceptionDispatcher+0xf
kernel32!RaiseException+0x58
msvcr80!_CxxThrowException+0x46
Acrobat_69630000!DllCanUnloadNow+0x68e27
Acrobat_69630000!PDFLTerm+0x645ce
Acrobat_69630000!DllCanUnloadNow+0x663df msvcr80!_NLG_Return

Armed with all of this, I quickly located the forum post I mentioned earlier where other users had found the same issue with Adobe Reader 9.0 under the same circumstances. As users of the free download of Reader, they were out on their own as Adobe have no obligation to give them one-to-one support. However, as someone who had just spent several thousand pounds buying Adobe software, including Acrobat, they would have to talk to me, so I called their UK support line.

As soon as I read out the error message to the agent I talked to (the very helpful Andrew Mitchell, for reference), he told me that he'd heard the exact same problem yesterday and that Adobe were aware of a problem producing this error. I mentioned my diagnostics and told him I suspected the AdobePDFL.dll, which he confirmed was the case. He put me on hold for a couple of minutes to talk to a senior technician, and when he returned, confirmed more of what I'd already found out.

Unfortunately, Adobe were only made aware of the issue very recently, and there is currently no ETA for a fix. I later got a follow-up email from Andrew which stated the following:

"Unfortunately because of the new-ness of the software we are unable to provide you with a direct solution to this issue as all solutions are experimental at this time."

Translation: we're still testing the patch for this. I was told I will be contacted again when a fix is available, but for the meantime, I would strongly advise you do not deploy Adobe Reader 9.0 or Adobe Acrobat 9.0 if your users' have their Application Data directory stored on a network share.