[pull] master from kevoreilly:master (#44)

* Update callback.py

* Update callback.py

* Package summary/description in UI (kevoreilly#2313)

- Following on from kevoreilly#2220 now we display the package summary and description in the UI
- In views.py:
  - Added code to parse the python analysis package modules, pulling out summary and description
  - Updated `get_form_data()` to return list of dicts, not list of strings
  - When sorting the package names, no longer be case sensitive
- Updated the `submission/index.html` template to expect a dict per package
- Added some tests
- Tweak to `web/settings.py` so it can find the templates, even during test execution

* Update cape2.sh

* Add SslKeyLogFile aux module and PcapNg processing module (kevoreilly#2312)

* Add SslKeyLogFile aux module and PcapNg processing module
SslKeyLogFile sets the SSLKEYLOGFILE environment var on the guest and collects the resulting log file. This is especially useful when detonating inside of browsers.

PcapNg takes SSL/TLS keys from tlsdump.log and SslKeyLogFile and injects them into the detonation PCAP using the `editcap` binary. The file is made available to download via the UI.

* Update views.py

* style: Automatic code formatting

* Monitor update: Add capability to dynamically unhook previously hooked functions (unhook-apis option takes colon-separated list e.g. unhook-apis=NtSetInformationThread:NtDelayExecution)

* Themida detonation shim

* Update index.html

* Update submit.rst

* Only process PCAPs with httpreplay when tlsdump.log exists (kevoreilly#2315)

These changes prevent PCAPs from being processed by httpreplay (Pcap2 processing module) when there are not TLS keys available. This is because httpreplay processing is very time intensive due to its pure-python implementation.

Httpreplay's core use is to decrypt TLS traffic so it can be processed by Suricata. If there are no TLS keys available, there is no requirement to use it for processing.

For context, when CAPE attempted to process a ~250MB PCAP with httpreplay, it took ~960 seconds. Without httpreplay, it took ~16 seconds.

* style: Automatic code formatting

* Stealc detection: loosen yara pattern slightly

---------

Co-authored-by: doomedraven <[email protected]>
Co-authored-by: Robin Koumis (SecureWorks) <[email protected]>
Co-authored-by: Josh Feather <[email protected]>
Co-authored-by: GitHub Actions <[email protected]>
Co-authored-by: Kevin O'Reilly <[email protected]>

analyzer/windows/data/yara/Themida.yar

@@ -0,0 +1,11 @@
+rule Themida
+{
+    meta:
+        author = "kevoreilly"
+        description = "Themida detonation shim"
+        cape_options = "unhook-apis=NtSetInformationThread,force-sleepskip=0"
+    strings:
+        $code = {FC 31 C9 49 89 CA 31 C0 31 DB AC 30 C8 88 E9 88 D5 88 F2 B6 08 66 D1 EB 66 D1 D8 73 09}
+    condition:
+        uint16(0) == 0x5A4D and all of them
+}

changelog.md

@@ -1,3 +1,7 @@
+### [10.09.2024]
+* Monitor update: Add capability to dynamically unhook previously hooked functions (unhook-apis option takes colon-separated list e.g. unhook-apis=NtSetInformationThread:NtDelayExecution)
+* Themida detonation shim
+
 ### [03.09.2024]
 * Monitor updates:
     * Add HtaFile to CLSIDS handled in WMI/DCOM (e.g. FormBook) - thanks Will Metcalf

data/yara/CAPE/Stealc.yar

@@ -6,7 +6,7 @@ rule Stealc
         cape_type = "Stealc Payload"
         hash = "77d6f1914af6caf909fa2a246fcec05f500f79dd56e5d0d466d55924695c702d"
     strings:
-        $nugget1 = {68 04 01 00 00 6A 00 FF 15 [4] 50 FF 15 [4] 8B F?}
+        $nugget1 = {68 04 01 00 00 6A 00 FF 15 [4] 50 FF 15}
         $nugget2 = {64 A1 30 00 00 00 8B 40 0C 8B 40 0C 8B 00 8B 00 8B 40 18 89 45 FC}
     condition:
         uint16(0) == 0x5A4D and all of them

docs/book/src/usage/submit.rst

@@ -166,6 +166,7 @@ some options (in this case a command line argument for the malware)::
 - ``during_script_args``: Command line arguments for during_script. Example: during_script_args=file1 file2 file3
 - ``pwsh``: - for ps1 package: prefer PowerShell Core, if available in the vm
 - ``check_shellcode``: - Setting check_shellcode=0 will disable checking for shellcode during package identification and extracting from archive
+- ``unhook-apis``: - capability to dynamically unhook previously hooked functions (unhook-apis option takes colon-separated list e.g. unhook-apis=NtSetInformationThread:NtDelayExecution)
 
 .. _webpy:
 

modules/processing/network.py

@@ -28,6 +28,7 @@
 import dns.resolver
 from dns.reversename import from_address
 
+import utils.profiling as profiling
 from data.safelist.domains import domain_passlist_re
 from lib.cuckoo.common.abstracts import Processing
 from lib.cuckoo.common.config import Config
@@ -78,6 +79,7 @@
 sys.path.append(CUCKOO_ROOT)
 
 TLS_HANDSHAKE = 22
+PCAP_BYTES_HTTPREPLAY_WARN_LIMIT = 30 * 1024 * 1024
 
 Keyed = namedtuple("Keyed", ["key", "obj"])
 Packet = namedtuple("Packet", ["raw", "ts"])
@@ -922,6 +924,11 @@ def run(self):
             log.debug('The PCAP file does not exist at path "%s"', self.pcap_path)
             return {}
 
+        httpreplay_start = profiling.Counter()
+        log.info("starting processing pcap with httpreplay")
+        if os.path.getsize(self.pcap_path) > PCAP_BYTES_HTTPREPLAY_WARN_LIMIT:
+            log.warning("httpreplay processing may timeout due to pcap size")
+
         r = httpreplay.reader.PcapReader(open(self.pcap_path, "rb"))
         r.tcp = httpreplay.smegma.TCPPacketStreamer(r, self.handlers)
 
@@ -1060,6 +1067,8 @@ def run(self):
 
                 results[f"{protocol}_ex"].append(tmp_dict)
 
+        log.info("finished processing pcap with httpreplay")
+        log.debug("httpreplay processing time: %s", (profiling.Counter() - httpreplay_start))
         return results
 
 
@@ -1117,7 +1126,10 @@ def run(self):
 
         if HAVE_HTTPREPLAY:
             try:
-                p2 = Pcap2(self.pcap_path, self.get_tlsmaster(), self.network_path).run()
+                p2 = {}
+                tls_master = self.get_tlsmaster()
+                if tls_master:
+                    p2 = Pcap2(self.pcap_path, tls_master, self.network_path).run()
                 if p2:
                     results.update(p2)
             except Exception:

utils/profiling.py

@@ -0,0 +1,30 @@
+import time
+from dataclasses import dataclass, field
+
+
+@dataclass
+class Counter:
+    """Profiler that counts real and CPU time."""
+
+    real: float = field(default_factory=time.perf_counter)
+    cpu: float = field(default_factory=time.process_time)
+
+    def __sub__(self, other):
+        real = self.real - other.real
+        cpu = self.cpu - other.cpu
+        return Counter(real, cpu)
+
+    def __add__(self, other):
+        real = self.real + other.real
+        cpu = self.cpu + other.cpu
+        return Counter(real, cpu)
+
+    def __str__(self) -> str:
+        return f"{self.real:.2f}s (cpu {self.cpu:.2f}s)".format(self.real, self.cpu)
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc_value, exc_tb):
+        elapsed = Counter() - self
+        self.__dict__.update(**elapsed.__dict__)

web/templates/submission/index.html

@@ -538,6 +538,11 @@
                                                     <td><code>check_shellcode</code></td>
                                                     <td>Setting check_shellcode=0 will disable checking for shellcode during package identification and extracting from archive</td>
                                                 </tr>
+                                                <tr>
+                                                    <td><code>unhook-apis</code></td>
+                                                    <td>Capability to dynamically unhook previously hooked functions (unhook-apis option takes colon-separated list e.g. unhook-apis=NtSetInformationThread:NtDelayExecution)</td>
+                                                </tr>
+
                                             </tbody>
                                         </table>
                                     </div>