Get in touch
2543934363
gsatactical@gmail.com

GLOBAL SECURITY AGENCY (GSA)

How-to: Reversing and debugging ISAPI modules

ron • June 27, 2023

Recently, I had the privilege to write a detailed analysis of CVE-2023-34362 , which is series of several vulnerabilities in the MOVEit file transfer application that lead to remote code execution. One of the several vulnerabilities involved an ISAPI module - specifically, the MoveITISAPI.dll ISAPI extension. One of the many vulnerabilities that comprised the MOVEit RCE was a header-injection issue, where the ISAPI application parsed headers differently than the .net application. This point is going to dig into how to analyze and reverse engineer an ISAPI-based service!

This wasn’t the first time in the recent past I’d had to work on something written as an ISAPI module, and each time I feel like I have to start over and remember how it’s supposed to work. This time, I thought I’d combine my hastily-scrawled notes with some Googling, and try to write something that I (and others) can use in the future. As such, this will be a quick intro to ISAPI applications from the angle that matters to me - how to reverse engineer and debug them!

I want to preface this with: I’m not a Windows developer, and I’ve never run an IIS server on purpose. That means that I am approaching this with brute-force ignorance! I don’t have a lot of background context nor do I know the correct terminology for a lot of this stuff. Instead, I’m going to treat these are typical DLLs from typical applications, and approach them as such.

What is ISAPI?

You can think of ISAPI as IIS’s equivalent to Apache or Nginx modules - that is, they are binaries written in a low-level language such as C, C++, or Delphi (no really, the Wikipedia page lists Delphi !) that are loaded into the IIS memory space as shared libraries. Since they’re low-level code, they can suffer from issues commonly found in low-level code, such as memory corruption. You’ve probably used Microsoft-supplied ISAPI modules without realizing it - they are used behind the scenes for .aspx applications, for example!

I found this helpful overview of ISAPI , which links to the other pages I mention below. It has a deprecation warning, but AFAICT no replacement page, so I can say it existed in June/2023 in case you need to use the Internet Archive to fetch it.

Depending on the application, ISAPI modules can either handle incoming requests by themselves (“ ISAPI extensions ”) or modify requests en route to their final handler (“ ISAPI filters ”). They’re both implemented as .dll files, but you can distinguish one from the other by looking at the list of exported functions (in a .dll file, an “exported function” is a function that can be called by the service that loads the .dll file). You can view exports in IDA Pro or Ghidra or other tools, but for these examples I found a simple CLI tool written in Ruby tool called pedump , which you can install via the Rubygems command gem install pedump .

Here are the exported functions in MOVEitISAPI.dll , which is the ISAPI module included with the MOVEit file transfer application:

 $ pedump -E ./MOVEitISAPI.dll

=== EXPORTS ===

# module "MOVEitISAPI.dll"
# flags=0x0  ts="2106-02-07 06:28:15"  version=0.0  ord_base=1
# nFuncs=3  nNames=3

  ORD ENTRY_VA  NAME
    1    9deb0  GetExtensionVersion
    2    9dfe0  HttpExtensionProc
    3    9dff0  TerminateExtension 

The two that we’re interested in are GetExtensionVersion and HttpExtensionProc , which we’ll dig into later.

Let’s contrast the ISAPI module with an ISAPI filter - MOVEitFilt.dll :

 $ pedump -E ./MOVEitFilt.dll 

=== EXPORTS ===

# module "MOVEitFilt.dll"
# flags=0x0  ts="2106-02-07 06:28:15"  version=0.0  ord_base=1
# nFuncs=2  nNames=2

  ORD ENTRY_VA  NAME
    1     1500  GetFilterVersion
    2     1540  HttpFilterProc 

The filter has two other functions - GetFilterVersion and HttpFilterProc .

So basically, you can figure out what type of ISAPI module you’re looking at by looking at the functions exported. In theory, there’s no reason why an ISAPI module can’t be both, but I’m not sure if anybody does that! Maybe for a CTF challenge I’ll try to develop an ISAPI Filter Extension that modifies its own requests :)

Finding ISAPI modules

Let’s say you’re working on an application that runs on IIS, and you want to identify the attack surface - ie, find ISAPI modules. The probably-correct way to answer this is to open up the IIS manager ( InetMgr.exe ) and look at the configuration. But that’s boring!

Recalling my original promise, to use “brute-force ignorance”, let’s use some commandline tools to figure out what’s going on!

The IIS process is called w3wp.exe , so let’s use wmic to find all instances of w3wp.exe running (note that I’m using MOVEit as an example, but this will work on other applications as well):

 C:\Users\Administrator>wmic process where "ExecutablePath like '%\\w3wp.exe'" get CommandLine 

Which outputs:

 c:\windows\system32\inetsrv\w3wp.exe -ap "moveitdmz Pool" -v "v4.0" -l "webengine4.dll" -a \\.\pipe\iisipm78b02bd1-595b-4442-9df0-a04b9d58775b -h "C:\inetpub\temp\apppools\moveitdmz Pool\moveitdmz Pool.config" -w "" -m 0 -t 20 -ta 0
c:\windows\system32\inetsrv\w3wp.exe -ap "moveitdmz ISAPI Pool" -v "v4.0" -l "webengine4.dll" -a \\.\pipe\iisipm2c508efc-6670-4302-a0b7-1ffb65ac196d -h "C:\inetpub\temp\apppools\moveitdmz ISAPI Pool\moveitdmz ISAPI Pool.config" -w "" -m 0 -t 20 -ta 0 

In this case, there are two IIS services running, with two different configurations: one is called the moveitdmz Pool and the other is called the moveitdmz ISAPI Pool . We can probably guess that the latter is what we want, but it turns out that the configurations are identical. I’m sure there’s some meaningful difference, but this is precisely where my knowledge of IIS ends so let’s just move on. :)

If we pick one of those configuration files and search it for “isapi”, we’ll find a ton of matches, because stuff like ASP are implemented as ISAPI modules. But if we look and search hard enough, we’ll find our modules:

 [...] <isapiFilters> 
 <clear 
 /> 
 <filter 
 name= 
 "MOVEit Filter" 
 path= 
 "C:\MOVEitTransfer\MOVEitISAPI\MOVEitFilt.dll" 
 enabled= 
 "true" 
 /> 
 </isapiFilters> 
[...] <handlers 
 accessPolicy= 
 "Execute" 
 > 
 <clear 
 /> 
 <add 
 name= 
 "MOVEitISAPIExtension" 
 path= 
 "MOVEitISAPI.dll" 
 verb= 
 "GET,POST" 
 modules= 
 "IsapiModule" 
 scriptProcessor= 
 "C:\MOVEitTransfer\MOVEitISAPI\MOVEitISAPI.dll" 
 requireAccess= 
 "Execute" 
 /> 
 </handlers> 
[...] 

This is well and good, but reading configuration files is hard and prone to missing stuff.

My recommendation, and personal approach, is to just copy the entire application to a Linux system, then use grep :

 $ grep -Er 'Get(Extension|Filter)Version'
grep: MOVEitISAPI/MOVEitISAPI.dll: binary file matches
grep: MOVEitISAPI/MOVEitFilt.dll: binary file matches 

Then work your way backwards to IIS to see how they’re served. Easy! :)

Reverse engineering ISAPI extensions

Let’s switch from generically talking about ISAPI modules to talking specifically about ISAPI Extensions - the type of module that serves a page directly, as opposed to the modules that filter requests. Filters will be similar, just different function names.

Microsoft provides an overview of ISAPI extensions here , which I’m going to use a bit.

Loading the .dll

ISAPI modules are shared libraries (ie, .dll files) that are loaded into the address space of IIS. When any .dll file is loaded (ISAPI or otherwise), the first function that’s called is always DllMain :

  BOOL 
 WINAPI 
 DllMain 
 ( 
 HINSTANCE 
 hinstDLL 
 , 
 // handle to DLL module 
 DWORD 
 fdwReason 
 , 
 // reason for calling function 
 LPVOID 
 lpvReserved 
 ) 
 // reserved 
 

It’s the .dll equivalent to main() in a typical C application, except that it’s called multiple times in the .dll’s lifecycle. The fdwReason argument specifies why it’s being called:

  • DLL_PROCESS_DETACH (0) - The .dll is being unloaded
  • DLL_PROCESS_ATTACH (1) - The .dll is being loaded
  • DLL_THREAD_ATTACH (2) - The process the .dll is loaded into is creating a new thread (all .dll files are alerted when this happens)
  • DLL_THREAD_DETACH (3) - A thread in the process is exiting cleanly

This isn’t anything special with ISAPI modules, and there’s a good chance that the DllMain function isn’t used at all - it simply has to return true . If it IS used, you’ll most likely see the DLL_PROCESS_ATTACH and DLL_PROCESS_DETACH reasons being used to initialize and clean up.

GetExtensionVersion()

After the ISAPI module is loaded, IIS will call the GetExtensionVersion() exported function. You can read about the function in Microsoft’s documentation , but the important part is the definition:

  BOOL 
 WINAPI 
 GetExtensionVersion 
 ( 
 HSE_VERSION_INFO 
 * 
 pVer 
 ); 
 

HSE_VERSION_INFO is a fairly simple structure with just two fields:

  typedef 
 struct 
 _HSE_VERSION_INFO 
 { 
 DWORD 
 dwExtensionVersion 
 ; 
 CHAR 
 lpszExtensionDesc 
 [ 
 HSE_MAX_EXT_DLL_NAME_LEN 
 ]; 
 // 256 
 } 
 HSE_VERSION_INFO 
 , 
 * 
 LPHSE_VERSION_INFO 
 ; 
 

As far as I know, these are free-form values. I added a struct called HSE_VERSION_INFO into IDA Pro, and it already knew the structure:

 00000000 HSE_VERSION_INFO struc ; (sizeof=0x104, align=0x4, copyof_483)
00000000 dwExtensionVersion dd ?
00000004 lpszExtensionDesc db 256 dup(?)
00000104 HSE_VERSION_INFO ends 

And it let me decorate rcx (the pVer argument on a 64-bit machine) with the proper field names (still using MOVEitISAPI.dll as an example):

 .text:000000018009DEB0 ; BOOL __stdcall GetExtensionVersion(HSE_VERSION_INFO *pVer)
.text:000000018009DEB0                 public GetExtensionVersion
.text:000000018009DEB0 GetExtensionVersion proc near           ; DATA XREF: .rdata:off_180AD21C8↓o
.text:000000018009DEB0                                         ; .pdata:0000000180BDE048↓o
.text:000000018009DEB0
.text:000000018009DEB0 var_18          = dword ptr -18h
.text:000000018009DEB0
.text:000000018009DEB0                 sub     rsp, 38h
.text:000000018009DEB4                 mov     [rcx+HSE_VERSION_INFO.dwExtensionVersion], 80000h
.text:000000018009DEBA                 movups  xmm0, xmmword ptr cs:aMoveitisapiExt ; "MOVEitISAPI Extension"
.text:000000018009DEC1                 movups  xmmword ptr [rcx+HSE_VERSION_INFO.lpszExtensionDesc], xmm0
.text:000000018009DEC5                 mov     eax, dword ptr cs:aMoveitisapiExt+10h ; "nsion"
.text:000000018009DECB                 mov     dword ptr [rcx+(HSE_VERSION_INFO.lpszExtensionDesc+10h)], eax
.text:000000018009DECE                 movzx   eax, word ptr cs:aMoveitisapiExt+14h ; "n"
.text:000000018009DED5                 mov     word ptr [rcx+(HSE_VERSION_INFO.lpszExtensionDesc+14h)], ax
[...] 

This function is also called exactly once in the ISAPI module lifecycle, which means it can (and often IS) used to initialize variables. Keep an eye out in both DllMain and GetExtensionVersion for initialized variables!

HttpExtensionProc()

The real meat of an ISAPI extension is HttpExtensionProc() , which is executed each time somebody accesses the extension. It’s where all the interesting stuff is going to happen.

The definition of the function is:

  DWORD 
 WINAPI 
 HttpExtensionProc 
 ( 
 LPEXTENSION_CONTROL_BLOCK 
 lpECB 
 ); 
 

Once again, it takes exactly one argument, which is stored in rcx (on a 64-bit host), or on top of the stack (on 32-bit). We’ll stick to 64-bit.

The argument is a pointer to a EXTENSION_CONTROL_BLOCK structure, which has the following definition:

  typedef 
 struct 
 _EXTENSION_CONTROL_BLOCK 
 EXTENSION_CONTROL_BLOCK 
 { 
 DWORD 
 cbSize 
 ; 
 DWORD 
 dwVersion 
 ; 
 HCONN 
 connID 
 ; 
 DWORD 
 dwHttpStatusCode 
 ; 
 CHAR 
 lpszLogData 
 [ 
 HSE_LOG_BUFFER_LEN 
 ]; 
 LPSTR 
 lpszMethod 
 ; 
 LPSTR 
 lpszQueryString 
 ; 
 LPSTR 
 lpszPathInfo 
 ; 
 LPSTR 
 lpszPathTranslated 
 ; 
 DWORD 
 cbTotalBytes 
 ; 
 DWORD 
 cbAvailable 
 ; 
 LPBYTE 
 lpbData 
 ; 
 LPSTR 
 lpszContentType 
 ; 
 BOOL 
 ( 
 WINAPI 
 * 
 GetServerVariable 
 ) 
 (); 
 BOOL 
 ( 
 WINAPI 
 * 
 WriteClient 
 ) 
 (); 
 BOOL 
 ( 
 WINAPI 
 * 
 ReadClient 
 ) 
 (); 
 BOOL 
 ( 
 WINAPI 
 * 
 ServerSupportFunction 
 ) 
 (); 
 } 
 EXTENSION_CONTROL_BLOCK 
 ; 
 

Once again, if you add a struct called EXTENSION_CONTROL_BLOCK to IDA Pro, it’s aware of the structure and size of all the fields:

 00000000 EXTENSION_CONTROL_BLOCK struc ; (sizeof=0xC0, align=0x8, copyof_485)
00000000 cbSize          dd ?
00000004 dwVersion       dd ?
00000008 ConnID          dq ?                    ; offset
00000010 dwHttpStatusCode dd ?
00000014 lpszLogData     db 80 dup(?)
00000064                 db ? ; undefined
00000065                 db ? ; undefined
00000066                 db ? ; undefined
00000067                 db ? ; undefined
00000068 lpszMethod      dq ?                    ; offset
00000070 lpszQueryString dq ?                    ; offset
00000078 lpszPathInfo    dq ?                    ; offset
00000080 lpszPathTranslated dq ?                 ; offset
00000088 cbTotalBytes    dd ?
0000008C cbAvailable     dd ?
00000090 lpbData         dq ?                    ; offset
00000098 lpszContentType dq ?                    ; offset
000000A0 GetServerVariable dq ?                  ; offset
000000A8 WriteClient     dq ?                    ; offset
000000B0 ReadClient      dq ?                    ; offset
000000B8 ServerSupportFunction dq ?              ; offset
000000C0 EXTENSION_CONTROL_BLOCK ends 

The most interesting fields are:

  • dwHttpStatusCode - The HTTP status code that’ll be returned (you’ll see 0xc8 a lot, which is HTTP/200)
  • lpszMethod - Will be GET or POST (or other methods), some modules will distinguish and others won’t
  • lpszQueryString - The HTTP query string (ie, what comes after the ? in the URL)
  • lpszPathInfo - What comes after the ISAPI module in the path (ie, https://example.org/isapimodule.dll/pathgoeshere )
  • cbTotalBytes - The size of the HTTP body, if any (typically used in a POST)
  • cbAvailable - The number of bytes that have already been received
  • lpbData - A buffer of data that has already been received (more data might be queued up, if it’s longer)
  • lpszContentType - The request’s content-type

Additionally, four function pointers are passed in that structure:

  • GetServerVariable - Used to retrieve information about the connection or server
  • WriteClient - Send data to the client
  • ReadClient - Receive data from the client
  • ServerSupportFunction - Other stuff the the previous callbacks don’t do

Once you know the structure of the incoming data, you can identify a lot of what’s going on in the module; for example, this code:

 .text:000000018009B98D                 mov     r8d, 1000h
.text:000000018009B993                 lea     rdx, [rsp+15F38h+var_5838]
.text:000000018009B99B                 mov     rcx, [r14+EXTENSION_CONTROL_BLOCK.lpszQueryString]
.text:000000018009B99F                 call    sub_18006FF00 

Appears to be copying the query string. We can all but confirm that in the next line, which uses var_5838 :

 .text:000000018009B9A4                 mov     r9d, 400h
.text:000000018009B9AA                 lea     r8, [rsp+15F38h+ep_buffer] ; buffer
.text:000000018009B9B2                 lea     rdx, aEp        ; "ep"
.text:000000018009B9B9                 lea     rcx, [rsp+15F38h+var_5838] ; querystring
.text:000000018009B9C1                 call    get_field_from_querystring_maybe 

It passes what looks like the query string, and the literal string “ep”, to another function. Without ever looking at that function, I named it get_field_from_querystring_maybe . That’s later confirmed with:

 .text:000000018009BA7B                 lea     r8, [rsp+15F38h+var_5838]
.text:000000018009BA83                 lea     rdx, aQueryStringS ; "Query string: %s"
.text:000000018009BA8A                 mov     ecx, 3Ch
.text:000000018009BA8F                 call    log_function_maybe

.text:000000018009BA94                 mov     r9d, 40h ; '@'
.text:000000018009BA9A                 lea     r8, [rsp+15F38h+action_buffer]
.text:000000018009BAA2                 lea     rdx, parameter_name ; "action"
.text:000000018009BAA9                 lea     rcx, [rsp+15F38h+var_5838] ; <-- Query string
.text:000000018009BAB1                 call    get_field_from_querystring_maybe 

We can also find the callback functions, like GetServerVariable , being used to read values from the environment:

 .text:000000018009BABC                 mov     [rsp+15F38h+length], 20h ; ' '
.text:000000018009BAC4                 lea     r9, [rsp+15F38h+length] ; lpdwSize
.text:000000018009BAC9                 lea     r8, [rsp+15F38h+remote_addr_buffer]
.text:000000018009BAD1                 lea     rdx, szVariableName ; "REMOTE_ADDR"
.text:000000018009BAD8                 mov     rcx, [r14+EXTENSION_CONTROL_BLOCK.ConnID] ; hConn
.text:000000018009BADC                 call    [r14+EXTENSION_CONTROL_BLOCK.GetServerVariable] 

From here, it’s a pretty typical Windows application, and can be reversed as such. That could be a good or bad thing, depending on your comfort level.. but one thing we CAN do is attach a debugger. Let’s see how!

Debugging

Thanks to the magic of “this being a normal .dll file”, we can debug this just like any program with a .dll file.

First, we need to figure out which process is actually serving that .dll. You could look at configs and stuff, but that’s boring. You can bruteforce and debug every w3wp.exe process, and that’s what I normally do, but I actually found a better way while writing this blog.. you can use tasklist /m <DLL> to check which processes have a specific .dll loaded:

 C:\Users\Administrator>tasklist /m MOVEitISAPI.dll

Image Name                     PID Modules
========================= ======== ============================================
w3wp.exe                      5248 MOVEitISAPI.dll 

That’s kinda magic, and could have saved me SO much trouble in the past!

Anyways, once you know the PID (in this case, 5248), you can attach a debugger such as windbg . When you attach, you should see the ISAPI modules loaded into memory as if they’re standard .dll files (because they are):

 [...]
ModLoad: 00007ff8`8a240000 00007ff8`8a2bb000   C:\MOVEitTransfer\MOVEitISAPI\MOVEitFilt.dll
ModLoad: 00007ff8`69860000 00007ff8`6a4c7000   \\?\C:\MOVEitTransfer\MOVEitISAPI\MOVEitISAPI.dll
[...] 

Due to ASLR, the addresses probably won’t match the addresses you see in other tools, but that’s a starting point!

You can use the x command to get a list of the exported addresses:

 0:012> x MOVEitISAPI!*
00007ff8`698fde80 MOVEitISAPI!GetExtensionVersion (<no parameter info>)
00007ff8`698fdfb0 MOVEitISAPI!HttpExtensionProc (<no parameter info>)
00007ff8`698fdfc0 MOVEitISAPI!TerminateExtension (<no parameter info>) 

You can also put a breakpoint on the HttpExtensionProc function (be sure to pass in the module name, since there will be multiple ISAPI modules with the same names):

 0:012> bp MOVEitISAPI!HttpExtensionProc
0:012> bl
     0 e Disable Clear  00007ff8`698fdfb0     0001 (0001)  0:**** MOVEitISAPI!HttpExtensionProc 

Then send some request:

 $ curl -ik 'https://10.0.0.193/moveitisapi/moveitisapi.dll' --data 'This is my postdata' 

And observe in the debugger, using the field offsets we saw earlier (I imagine you can use dx to dump the whole object if you load the definition into windbg, which I don’t know how to do):

 Breakpoint 0 hit
MOVEitISAPI!HttpExtensionProc:
00007ff8`698fdfb0 e96bd8ffff      jmp     MOVEitISAPI+0x9b820 (00007ff8`698fb820)

0:005> ds rcx+0x70
000001c8`64b69f78  "/moveitisapi/moveitisapi.dll"

0:005> ds rcx+0x78
000001c8`64b69f98  "C:\MOVEitTransfer\MOVEitISAPI\moveitisapi.dll"

0:005> ds rcx+0x90
000001c8`64b68346  "application/x-www-form-urlencoded"

0:005> ds rcx+0x88
000001c8`64b69f60  "This is my postdata" 

From there, you can use break-on-access ( ba ) and other stuff to track the data, if desired! Whatever you want to do!

By ZenBusiness Admin September 19, 2023
The new season is a great reason to make and keep resolutions. Whether it’s eating right or cleaning out the garage, here are some tips for making and keeping resolutions.
By ZenBusiness Admin September 19, 2023
There are so many good reasons to communicate with site visitors. Tell them about sales and new products or update them with tips and information.
By ZenBusiness Admin September 19, 2023
Write about something you know. If you don’t know much about a specific topic that will interest your readers, invite an expert to write about it.
By Richard Bejtlich June 25, 2023
Cybersecurity is a social and policy problem, not a scientific or technical problem. Cybersecurity is also a wicked problem. In a landmark 1973 article, Dilemmas in a General Theory of Planning , urban planners Horst W. J. Rittel and Melvin M. Webber described wicked problems in these terms:
By Richard Bejtlich October 31, 2020
#securityonepercent
By Richard Bejtlich October 23, 2020
MITRE ATT&CK
By Richard Bejtlich April 7, 2020
disturbing story today
By Richard Bejtlich March 27, 2020
Google Books
More Posts
Share by: