Fighting the MSVCRT.DLL hell

If you are C/C++ developer that needs to create applications using contemporary Microsoft Visual C++ compilers that are linked to the MSVCRT.DLL only, then you should read this article. However if you are lucky and you are creating self contained applications which you totally control and which do not allow third party plug-ins then this article is probably of no interest to you.

I was intrigued by the fact that majority of Microsoft products are linked only to MSVCRT.DLL and that those CRT’s are not the same as one used or produced by Visual Studio 6.0 compiler and CRT sources. There are few articles you can find on the Internet about the topic, but I didn’t found any of them actually helpful, so I decided to dig in and create something usable.

With Visual Studio .NET back in 2002 Microsoft introduced a new CRT DLL and all hell broke loose. Since then each new version of Visual Studio ships with a new CRT DLL making the applications or libraries bound to that particular DLL. Here is the table that shows those version

• MSVCRT70.DLL Visual Studio .NET
• MSVCRT71.DLL Visual Studio 2003
• MSVCRT80.DLL Visual Studio 2005
• MSVCRT90.DLL Visual Studio 2008
• MSVCRT100.DLL Visual Studio 2010

The major problem that arises when using multiple CRT versions is explained in this and this MSDN article.

As you can see, one thing that Microsoft tries to convince you is that MSVCRT.DLL is of no interest to you and that you should use the one that comes with Visual Studio, and even more that you should make sure that each and every component is build using exactly the same version of C compiler. Nice in theory but if it happens that you need to embed already produced binaries compiled by someone else then this is not an option. So if MSVCRT.DLL is of no interest to you how come that lots of Microsoft programs are bound to that DLL instead eating their own dog food? I suppose there is a bit truth in that old latin say “Quod licet Iovi, non licet bovi”

Few things that MSDN article doesn’t mention are inherent cross bounday objects like stdio objects. If you redirect your stdout results can be surprising.

DLL

#include
#include

__declspec(dllexport) void __stdcall dllhi(void)
{
    printf("Hi from DLL\n");
}

Test program

#include
#include

__declspec(dllimport) void __stdcall dllhi(void);

void redirect(const char *fname)
{
    /* Reassign "stdout" to fname */
    freopen(fname, "w", stdout);
}

void sayhi(void)
{
    printf("This will go to the test.out\n");
}

int main(int argc, char *argv[])
{
    redirect("test.out");
    sayhi();
    dllhi();

    return 0;
}

Compile DLL

    cl /MD /DLL dll.c

Compile Test program

    cl /MD tst.c dll.lib

If you compile both DLL and Test  program with the same compiler everything works as expected. However if you compile tst.c with VS2003 and dll.c with VS2008 the application won’t even initialize complaining about Runtime initialization. Other way around works if you compile tst.c with VS2008 and dll.c with VS2003 but the message from DLL won’t be present in the output file. The reason is because each MSVCRTnn.DLL maintains its own copy of those system objects like stdio and stderr streams.

First problem is more easily handled although the error message box doesn’t say much about the reason of failure, but the second is more subtle. Everything seems to be working but you are loosing the data from the DLL that would go to redirected file. This means that each and every CRT stdio function is unsafe when crossing DLL boundaries even if you don’t pass them directly. Also almost all functions that deal with memory allocation are unsafe as well. This means that you cannot free memory that was allocated in a DLL. You cannot use Posix file descriptors as well. For example using an descriptor obtained by calling open() in DLL would result in access violation for any operation on that descriptor performed from the object file linked with different CRT. Just like for stdio and stderr each CRT maintains it’s own list of pseudo file handles. In worse case your data can end up in unexpected location if it happens that you call open() from different DLL’s.

So what happens if you link with system MSVCRT.DLL. Well it doesn’t solve the issues, but it won’t crash your application neither will your DLL (if that’s what you are building) won’t crash the host.

To fight the information loss across different CRT DLL boundaries one solution is to update the data in all CRT DLL’s currently loaded in process. This can be very complex as shown on PostgreSQL example. As you can see they try to load each and every possible MSVCRTnn.DLL and obtain a pointer to putenv() function and calling that function. However this still won’t work for a simple printf() with redirected stdio stream. Acutally I’m not even sure there is a solution for that, beside using the same CRT of course.

The only safe solution is to use the same CRT for your application and your application’s modules. This has some drawbacks of course, and the major one is that you simply must ensure that all dependencies are using the same compiler generation and thus linking to the same CRT DLL. Within Apache Httpd we were using good old Visual Studio 6.0 for producing official binaries, but recently there were some discussion to switch to Visual Studio 2010. One of the major reasons for that is because you just cannot download Visual Studio 6.0 any more, even if you’ve purchased MSDN subscription. It’s not maintained any more as well, and it cannot create 64 bit binaries so it’s logical to move forward to something more advanced.

There is however a solution to use the system MSVCRT.DLL without the need to ship the CRT redistributables which mandate using some sort of installer and Microsoft insist that this should be .msi. Solution is to use the Windows Driver Kit which is freely downloadable from MSDN and can be used for producing a binaries linked with MSVCRT.DLL. Driver kit is meant to be used for producing OS drivers and since this is very sensitive part of your system it must be very stable and error prone. You might ask yourself like I did: OK if drivers which first premise is stability must be compiled to MSVCRT.DLL then why everything else must be compiled with MSVCRTnn.DLL. Weird statement and I really have no clue why Microsoft decided to go that way. At least I didn’t find any technological reason for such a move. Well, I understand that MSVCRT.DLL in Windows XP is different the in Windows 7, but that could at least make sure that higher MSVCRTnn.DLL use data structures from MSVCRT.DLL each of them maintaining its own copy or parallel implementation.

Anyhow, I decided to go with the Windows DDK path. However this created a whole bunch of problems. One of the the first one I faced was that msvcrt.lib from DDK doesn’t export few symbols for functions or data that is in the MSVCRT.DLL like getpid(), _environ and _wenviron. I would like to think that this is just a bug rather then intentional. This can be solved pretty easy by creating a library which will export those symbols, but the drawback is that you must link to that library. At the end this means you must modify you make files to include that thunk library. Create a file named msvcrt_compat.def with the following content:

EXPORTS
    _getpid=msvcrt._getpid
    _environ=msvcrt._environ
    _wenviron=msvcrt._wenviron

Then create a thunk library with those missing symbols using

    lib /NOLOGO /NODEFAULTLIB /DEF:msvcrt_compat.def /MACHINE:X86 /NAME:msvcrt.dll /OUT:msvcrt_compat.lib

All you have to do now is to link your existing application with msvcrt_compat..lib and msvcrt_win2003.obj or msvcrt_winxp.obj from DDK. I have made a toolkit as part of my next pet project that does that automatically for you. It creates a custom Visual Studio compilation using Windows Driver Kit and Windows Software Development Kit. It’s currently hosted on sourceforge and it contains some additional stuff which you are probably not interested in, like optionally installing Cygwin and Perl. Anyhow for those that just need a compiler they will need the following.

• Windows Server 2003 R2 Platform SDK
  Make complete install inside default directory
  C:\Program Files\Microsoft Platform SDK for Windows Server 2003 R2
• Windows Software Development Kit for Windows 7
  Make complete install inside default directory.
  C:\Program Files\Microsoft SDKs\Windows\v7.0\
  It will also install a subset of Visual Studio 2008 (9.0) at
  C:\Program Files\Microsoft Visual Studio 9.0
• Windows Driver Kit version 7.1.0
  This is a DDK version for Windows 7 and Windows Server 2008r2 There are two versions of those at MSDN, so make sure to download version 7.1.0 or later. It installs in: c:\WinDDK\7600.16385.1
• Toolkit for making a custom MSVC distribution
  svn co https://myomake.svn.sourceforge.net/svnroot/myomake/trunk/misc/tools/windows/cmsc
  Make sure you read the README.txt files

If you install all the mentioned packages and checkout the toolkit from MyoMake sourceforge project, open a command prompt and cd to tools directory. Execute cmsc_compile.bat script and it will create a workable compiler distribution for you. Note that you are probably not allowed to redistribute that to someone else according to the Microsoft license terms (don’t say you haven’t be warned). You can call cmsc_makedist.bat which will create a .zip file which you can use for backup purposes or for your dev box. I have created a setup script for command line compilation (setenv.bat) which  when called set’s up the correct compiler in a similar way SetEnv.cmd does with Platform SDK. It also sets EXTRA_LIBS environment variable which you can use to add required thunk at link time as well as BUILD_CPU with selected architecture (i386, amd64 or ia64). After calling setenv.bat /x86 the upper examples can be compiled

    cl /MD /DLL dll.c %EXTRA_LIBS%
    cl /MD tst.c dll.lib %EXTRA_LIBS%

Note that this is not a users manual for that custom compilation script. I’m not even sure it will survive the next DDK version, but who knows. Microsoft might surprise us. Anyhow I’d like to hear your experiences with the subject. When making a comment make sure you read the rules.