diff --git a/LICENSE b/LICENSE new file mode 100644 index 00000000000..cbe5ad16704 --- /dev/null +++ b/LICENSE @@ -0,0 +1,437 @@ +Attribution-NonCommercial-ShareAlike 4.0 International + +======================================================================= + +Creative Commons Corporation ("Creative Commons") is not a law firm and +does not provide legal services or legal advice. Distribution of +Creative Commons public licenses does not create a lawyer-client or +other relationship. Creative Commons makes its licenses and related +information available on an "as-is" basis. Creative Commons gives no +warranties regarding its licenses, any material licensed under their +terms and conditions, or any related information. Creative Commons +disclaims all liability for damages resulting from their use to the +fullest extent possible. + +Using Creative Commons Public Licenses + +Creative Commons public licenses provide a standard set of terms and +conditions that creators and other rights holders may use to share +original works of authorship and other material subject to copyright +and certain other rights specified in the public license below. The +following considerations are for informational purposes only, are not +exhaustive, and do not form part of our licenses. + + Considerations for licensors: Our public licenses are + intended for use by those authorized to give the public + permission to use material in ways otherwise restricted by + copyright and certain other rights. Our licenses are + irrevocable. Licensors should read and understand the terms + and conditions of the license they choose before applying it. + Licensors should also secure all rights necessary before + applying our licenses so that the public can reuse the + material as expected. Licensors should clearly mark any + material not subject to the license. This includes other CC- + licensed material, or material used under an exception or + limitation to copyright. More considerations for licensors: + wiki.creativecommons.org/Considerations_for_licensors + + Considerations for the public: By using one of our public + licenses, a licensor grants the public permission to use the + licensed material under specified terms and conditions. If + the licensor's permission is not necessary for any reason--for + example, because of any applicable exception or limitation to + copyright--then that use is not regulated by the license. Our + licenses grant only permissions under copyright and certain + other rights that a licensor has authority to grant. Use of + the licensed material may still be restricted for other + reasons, including because others have copyright or other + rights in the material. A licensor may make special requests, + such as asking that all changes be marked or described. + Although not required by our licenses, you are encouraged to + respect those requests where reasonable. More considerations + for the public: + wiki.creativecommons.org/Considerations_for_licensees + +======================================================================= + +Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International +Public License + +By exercising the Licensed Rights (defined below), You accept and agree +to be bound by the terms and conditions of this Creative Commons +Attribution-NonCommercial-ShareAlike 4.0 International Public License +("Public License"). To the extent this Public License may be +interpreted as a contract, You are granted the Licensed Rights in +consideration of Your acceptance of these terms and conditions, and the +Licensor grants You such rights in consideration of benefits the +Licensor receives from making the Licensed Material available under +these terms and conditions. + + +Section 1 -- Definitions. + + a. Adapted Material means material subject to Copyright and Similar + Rights that is derived from or based upon the Licensed Material + and in which the Licensed Material is translated, altered, + arranged, transformed, or otherwise modified in a manner requiring + permission under the Copyright and Similar Rights held by the + Licensor. For purposes of this Public License, where the Licensed + Material is a musical work, performance, or sound recording, + Adapted Material is always produced where the Licensed Material is + synched in timed relation with a moving image. + + b. Adapter's License means the license You apply to Your Copyright + and Similar Rights in Your contributions to Adapted Material in + accordance with the terms and conditions of this Public License. + + c. BY-NC-SA Compatible License means a license listed at + creativecommons.org/compatiblelicenses, approved by Creative + Commons as essentially the equivalent of this Public License. + + d. Copyright and Similar Rights means copyright and/or similar rights + closely related to copyright including, without limitation, + performance, broadcast, sound recording, and Sui Generis Database + Rights, without regard to how the rights are labeled or + categorized. For purposes of this Public License, the rights + specified in Section 2(b)(1)-(2) are not Copyright and Similar + Rights. + + e. Effective Technological Measures means those measures that, in the + absence of proper authority, may not be circumvented under laws + fulfilling obligations under Article 11 of the WIPO Copyright + Treaty adopted on December 20, 1996, and/or similar international + agreements. + + f. Exceptions and Limitations means fair use, fair dealing, and/or + any other exception or limitation to Copyright and Similar Rights + that applies to Your use of the Licensed Material. + + g. License Elements means the license attributes listed in the name + of a Creative Commons Public License. The License Elements of this + Public License are Attribution, NonCommercial, and ShareAlike. + + h. Licensed Material means the artistic or literary work, database, + or other material to which the Licensor applied this Public + License. + + i. Licensed Rights means the rights granted to You subject to the + terms and conditions of this Public License, which are limited to + all Copyright and Similar Rights that apply to Your use of the + Licensed Material and that the Licensor has authority to license. + + j. Licensor means the individual(s) or entity(ies) granting rights + under this Public License. + + k. NonCommercial means not primarily intended for or directed towards + commercial advantage or monetary compensation. For purposes of + this Public License, the exchange of the Licensed Material for + other material subject to Copyright and Similar Rights by digital + file-sharing or similar means is NonCommercial provided there is + no payment of monetary compensation in connection with the + exchange. + + l. Share means to provide material to the public by any means or + process that requires permission under the Licensed Rights, such + as reproduction, public display, public performance, distribution, + dissemination, communication, or importation, and to make material + available to the public including in ways that members of the + public may access the material from a place and at a time + individually chosen by them. + + m. Sui Generis Database Rights means rights other than copyright + resulting from Directive 96/9/EC of the European Parliament and of + the Council of 11 March 1996 on the legal protection of databases, + as amended and/or succeeded, as well as other essentially + equivalent rights anywhere in the world. + + n. You means the individual or entity exercising the Licensed Rights + under this Public License. Your has a corresponding meaning. + + +Section 2 -- Scope. + + a. License grant. + + 1. Subject to the terms and conditions of this Public License, + the Licensor hereby grants You a worldwide, royalty-free, + non-sublicensable, non-exclusive, irrevocable license to + exercise the Licensed Rights in the Licensed Material to: + + a. reproduce and Share the Licensed Material, in whole or + in part, for NonCommercial purposes only; and + + b. produce, reproduce, and Share Adapted Material for + NonCommercial purposes only. + + 2. Exceptions and Limitations. For the avoidance of doubt, where + Exceptions and Limitations apply to Your use, this Public + License does not apply, and You do not need to comply with + its terms and conditions. + + 3. Term. The term of this Public License is specified in Section + 6(a). + + 4. Media and formats; technical modifications allowed. The + Licensor authorizes You to exercise the Licensed Rights in + all media and formats whether now known or hereafter created, + and to make technical modifications necessary to do so. The + Licensor waives and/or agrees not to assert any right or + authority to forbid You from making technical modifications + necessary to exercise the Licensed Rights, including + technical modifications necessary to circumvent Effective + Technological Measures. For purposes of this Public License, + simply making modifications authorized by this Section 2(a) + (4) never produces Adapted Material. + + 5. Downstream recipients. + + a. Offer from the Licensor -- Licensed Material. Every + recipient of the Licensed Material automatically + receives an offer from the Licensor to exercise the + Licensed Rights under the terms and conditions of this + Public License. + + b. Additional offer from the Licensor -- Adapted Material. + Every recipient of Adapted Material from You + automatically receives an offer from the Licensor to + exercise the Licensed Rights in the Adapted Material + under the conditions of the Adapter's License You apply. + + c. No downstream restrictions. You may not offer or impose + any additional or different terms or conditions on, or + apply any Effective Technological Measures to, the + Licensed Material if doing so restricts exercise of the + Licensed Rights by any recipient of the Licensed + Material. + + 6. No endorsement. Nothing in this Public License constitutes or + may be construed as permission to assert or imply that You + are, or that Your use of the Licensed Material is, connected + with, or sponsored, endorsed, or granted official status by, + the Licensor or others designated to receive attribution as + provided in Section 3(a)(1)(A)(i). + + b. Other rights. + + 1. Moral rights, such as the right of integrity, are not + licensed under this Public License, nor are publicity, + privacy, and/or other similar personality rights; however, to + the extent possible, the Licensor waives and/or agrees not to + assert any such rights held by the Licensor to the limited + extent necessary to allow You to exercise the Licensed + Rights, but not otherwise. + + 2. Patent and trademark rights are not licensed under this + Public License. + + 3. To the extent possible, the Licensor waives any right to + collect royalties from You for the exercise of the Licensed + Rights, whether directly or through a collecting society + under any voluntary or waivable statutory or compulsory + licensing scheme. In all other cases the Licensor expressly + reserves any right to collect such royalties, including when + the Licensed Material is used other than for NonCommercial + purposes. + + +Section 3 -- License Conditions. + +Your exercise of the Licensed Rights is expressly made subject to the +following conditions. + + a. Attribution. + + 1. If You Share the Licensed Material (including in modified + form), You must: + + a. retain the following if it is supplied by the Licensor + with the Licensed Material: + + i. identification of the creator(s) of the Licensed + Material and any others designated to receive + attribution, in any reasonable manner requested by + the Licensor (including by pseudonym if + designated); + + ii. a copyright notice; + + iii. a notice that refers to this Public License; + + iv. a notice that refers to the disclaimer of + warranties; + + v. a URI or hyperlink to the Licensed Material to the + extent reasonably practicable; + + b. indicate if You modified the Licensed Material and + retain an indication of any previous modifications; and + + c. indicate the Licensed Material is licensed under this + Public License, and include the text of, or the URI or + hyperlink to, this Public License. + + 2. You may satisfy the conditions in Section 3(a)(1) in any + reasonable manner based on the medium, means, and context in + which You Share the Licensed Material. For example, it may be + reasonable to satisfy the conditions by providing a URI or + hyperlink to a resource that includes the required + information. + 3. If requested by the Licensor, You must remove any of the + information required by Section 3(a)(1)(A) to the extent + reasonably practicable. + + b. ShareAlike. + + In addition to the conditions in Section 3(a), if You Share + Adapted Material You produce, the following conditions also apply. + + 1. The Adapter's License You apply must be a Creative Commons + license with the same License Elements, this version or + later, or a BY-NC-SA Compatible License. + + 2. You must include the text of, or the URI or hyperlink to, the + Adapter's License You apply. You may satisfy this condition + in any reasonable manner based on the medium, means, and + context in which You Share Adapted Material. + + 3. You may not offer or impose any additional or different terms + or conditions on, or apply any Effective Technological + Measures to, Adapted Material that restrict exercise of the + rights granted under the Adapter's License You apply. + + +Section 4 -- Sui Generis Database Rights. + +Where the Licensed Rights include Sui Generis Database Rights that +apply to Your use of the Licensed Material: + + a. for the avoidance of doubt, Section 2(a)(1) grants You the right + to extract, reuse, reproduce, and Share all or a substantial + portion of the contents of the database for NonCommercial purposes + only; + + b. if You include all or a substantial portion of the database + contents in a database in which You have Sui Generis Database + Rights, then the database in which You have Sui Generis Database + Rights (but not its individual contents) is Adapted Material, + including for purposes of Section 3(b); and + + c. You must comply with the conditions in Section 3(a) if You Share + all or a substantial portion of the contents of the database. + +For the avoidance of doubt, this Section 4 supplements and does not +replace Your obligations under this Public License where the Licensed +Rights include other Copyright and Similar Rights. + + +Section 5 -- Disclaimer of Warranties and Limitation of Liability. + + a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE + EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS + AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF + ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS, + IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION, + WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR + PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS, + ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT + KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT + ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU. + + b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE + TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION, + NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT, + INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES, + COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR + USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN + ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR + DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR + IN PART, THIS LIMITATION MAY NOT APPLY TO YOU. + + c. The disclaimer of warranties and limitation of liability provided + above shall be interpreted in a manner that, to the extent + possible, most closely approximates an absolute disclaimer and + waiver of all liability. + + +Section 6 -- Term and Termination. + + a. This Public License applies for the term of the Copyright and + Similar Rights licensed here. However, if You fail to comply with + this Public License, then Your rights under this Public License + terminate automatically. + + b. Where Your right to use the Licensed Material has terminated under + Section 6(a), it reinstates: + + 1. automatically as of the date the violation is cured, provided + it is cured within 30 days of Your discovery of the + violation; or + + 2. upon express reinstatement by the Licensor. + + For the avoidance of doubt, this Section 6(b) does not affect any + right the Licensor may have to seek remedies for Your violations + of this Public License. + + c. For the avoidance of doubt, the Licensor may also offer the + Licensed Material under separate terms or conditions or stop + distributing the Licensed Material at any time; however, doing so + will not terminate this Public License. + + d. Sections 1, 5, 6, 7, and 8 survive termination of this Public + License. + + +Section 7 -- Other Terms and Conditions. + + a. The Licensor shall not be bound by any additional or different + terms or conditions communicated by You unless expressly agreed. + + b. Any arrangements, understandings, or agreements regarding the + Licensed Material not stated herein are separate from and + independent of the terms and conditions of this Public License. + + +Section 8 -- Interpretation. + + a. For the avoidance of doubt, this Public License does not, and + shall not be interpreted to, reduce, limit, restrict, or impose + conditions on any use of the Licensed Material that could lawfully + be made without permission under this Public License. + + b. To the extent possible, if any provision of this Public License is + deemed unenforceable, it shall be automatically reformed to the + minimum extent necessary to make it enforceable. If the provision + cannot be reformed, it shall be severed from this Public License + without affecting the enforceability of the remaining terms and + conditions. + + c. No term or condition of this Public License will be waived and no + failure to comply consented to unless expressly agreed to by the + Licensor. + + d. Nothing in this Public License constitutes or may be interpreted + as a limitation upon, or waiver of, any privileges and immunities + that apply to the Licensor or You, including from the legal + processes of any jurisdiction or authority. + +======================================================================= + +Creative Commons is not a party to its public +licenses. Notwithstanding, Creative Commons may elect to apply one of +its public licenses to material it publishes and in those instances +will be considered the “Licensor.” The text of the Creative Commons +public licenses is dedicated to the public domain under the CC0 Public +Domain Dedication. Except for the limited purpose of indicating that +material is shared under a Creative Commons public license or as +otherwise permitted by the Creative Commons policies published at +creativecommons.org/policies, Creative Commons does not authorize the +use of the trademark "Creative Commons" or any other trademark or logo +of Creative Commons without its prior written consent including, +without limitation, in connection with any unauthorized modifications +to any of its public licenses or any other arrangements, +understandings, or agreements concerning use of licensed material. For +the avoidance of doubt, this paragraph does not form part of the +public licenses. + +Creative Commons may be contacted at creativecommons.org. diff --git a/README.md b/README.md new file mode 100644 index 00000000000..c134cb2ba5e --- /dev/null +++ b/README.md @@ -0,0 +1,26 @@ +# Documentation site for Velociraptor + +This is the documentation site for Velociraptor - digging deeper! + +## Building this site + +The site uses hugo. To develop on the site, simply clone this +repository, and run at the top level: + +``` +hugo serve +``` + +This will bring up a local web server where you can see changes. + + +Shield: [![CC BY-NC-SA 4.0][cc-by-nc-sa-shield]][cc-by-nc-sa] + +This work is licensed under a +[Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License][cc-by-nc-sa]. + +[![CC BY-NC-SA 4.0][cc-by-nc-sa-image]][cc-by-nc-sa] + +[cc-by-nc-sa]: http://creativecommons.org/licenses/by-nc-sa/4.0/ +[cc-by-nc-sa-image]: https://licensebuttons.net/l/by-nc-sa/4.0/88x31.png +[cc-by-nc-sa-shield]: https://img.shields.io/badge/License-CC%20BY--NC--SA%204.0-lightgrey.svg diff --git a/content/docs/api/_index.md b/content/docs/api/_index.md new file mode 100644 index 00000000000..7b5cf191d1c --- /dev/null +++ b/content/docs/api/_index.md @@ -0,0 +1,6 @@ +--- +title: "The API" +date: 2021-06-27T04:29:34Z +draft: false +weight: 70 +--- diff --git a/content/docs/extending_vql/_index.md b/content/docs/extending_vql/_index.md new file mode 100644 index 00000000000..aa939060b1c --- /dev/null +++ b/content/docs/extending_vql/_index.md @@ -0,0 +1,6 @@ +--- +title: "Extending VQL" +date: 2021-06-27T04:29:26Z +draft: false +weight: 50 +--- diff --git a/content/docs/forensic/_index.md b/content/docs/forensic/_index.md index b21fb7920b0..9ce038264b3 100644 --- a/content/docs/forensic/_index.md +++ b/content/docs/forensic/_index.md @@ -11,303 +11,54 @@ possible. Velociraptor's strength lies in the wide array of VQL plugins and functions that are geared towards making DFIR investigations and detections effective. +{{% children "description"=true "style"="h3" %}} -Digging deeper in Windows -2 - -Module overview -Velociraptor implements many forensic capabilities in VQL -This module will focus on typical forensic analysis and deep inspection capabilities. We will learn how to put the capabilities together to produce effective artifacts and when to use those. -This module will not use Velociraptor’s GUI or even the client/server mode since we are focused on the techniques themselves. Later we can leverage the same VQL across the network at scale, and effectively hunt for artifacts across our infrastructure - keep this in mind through this module. - -NTFS Analysis -42 - -NTFS overview -NTFS is the standard Windows filesystem. -All files are represented in a Master File Table -Files can contain multiple attributes: -Filename (Long name/Short name) -Data attribute – contains file data -I30 attribute (contains directory listing) -Data attributes may be compressed or sparse -Filename attributes contain their own timestamps - - -43 - -The Master File Table -The NTFS file system contains a file called the master file table, or MFT. There is at least one entry in the MFT for every file on an NTFS file system volume, including the MFT itself. All information about a file, including its size, time and date stamps, permissions, and data content, is stored either in MFT entries, or in space outside the MFT that is described by MFT entries. - -https://docs.microsoft.com/en-us/windows/win32/fileio/master-file-table - -44 - -NTFS Concepts -https://www.fireeye.com/blog/threat-research/2012/10/incident-response-ntfs-indx-buffers-part-4-br-internal.html -45 - -Velociraptor’s NTFS support -Velociraptor has 2 accessors providing access to NTFS -ntfs - Supports Alternate Data Streams in directory listings. -lazy_ntfs - much faster but does not detect ADS. - -Due to these accessors it is possible to operate on files in the NTFS volume using all the usual plugins. -46 - -47 - -The NTFS accessor makes NTFS specific information available in the Data field. For regular files it includes the inode string. -The NTFS accessor considers all paths to begin with a device name. For convenience the accessor also accepts a drive letter. - -48 -Volume Shadow Copies -NTFS allows for a special copy on write snapshot feature called “Volume Shadow Copy”. - -Create a VSS copy on your own machine using WMI: - -On Windows server OS you can use: -vssadmin create shadow - -Checking for VSC -Ensure your system contains a volume shadow copy -49 - -NTFS accessor and VSS -When a VSS copy is created, it is accessible via a special device. Velociraptor allows the VSS copies to be enumerated by listing them at the top level of the filesystem. -At the top level, the accessor provides metadata about each device in the “Data” column, including its creation time. This is essentially the same output as vssadmin list shadows -50 +Velociraptor +Digging Deeper! +1 -51 +Forensic Analysis with VQL Pt2 -52 -Find all VSS copies of the event logs - -53 -We can glob the VSS just as if they were a directory - -Makes it easy to fetch every version of a certain file (e.g. a log file). - -54 -Operating on VSS -Simply use the VSS device name as a prefix to all paths and the ntfs accessor will parse it instead. - -You can use it to analyze older versions of the drive! - -55 - -Parsing the MFT -You can download the entire $MFT file from the endpoint using the ntfs accessor, then process it offline. +Digging even deeper in Windows +2 -You can also parse the $MFT on the endpoint using Velociraptor. +Module overview +Velociraptor implements many forensic capabilities in VQL +This module will focus on typical forensic analysis and deep inspection capabilities. We will learn how to put the capabilities together to produce effective artifacts. +We will use the Mitre Att&ck framework for guidance. -This is most useful when you need to pass over all the files in the disk - it is more efficient than a recursive glob and might recover deleted files. -56 -57 +3 -58 -Exercise: Find all .exe on the drive -Efficiently find all .exe on disk that were created after Jan 20, 2020 +Evidence of execution +4 -MFT Entries -An MFT Entry can have multiple attributes and streams -The previous plugin just shows high level information about each MFT entry - we can dig deeper with the parse_ntfs() plugin which accepts an mft ID. -Choose a file of interest in the previous output and inspect it deeper. +29 59 -60 -An inode is a triple of mft id, type id and id - -e.g. 974-16-0 - -representing a stream of data - -61 -NTFS timestamps -An MFT entry can have up to 16 timestamps! -Timestamps are critical to forensic investigations -Determine when files were copied -When files were modified -And sometimes we can determine when a file was accessed -In NTFS there are timestamps -In $STANDARD_INFORMATION stream (usually only 1) -In the $FILENAME stream (sometimes 2 or 3) -In the $I30 stream of the parent directory (see later) - -Timestomping -62 -Attackers sometimes change the timestamps of files to make them less obvious. E.g make malware look like it was installed many years ago. - -For the next exercise we will stomp over some times. Use the provided powershell to stomp over Velociraptor.exe’s timestamps. - - - -Timestomp a file -63 -$file = 'C:\Program Files\Velociraptor\Velociraptor.exe' -$stomp = Get-Date 2007-07-07 -$(Get-Item $file).creationtime = $stomp -$(Get-Item $file).lastaccesstime = $stomp -$(Get-Item $file).lastwritetime = $stomp -Get-ChildItem $file | Select *, Fullname, *Time* - -powershell -executionpolicy bypass "& .\stomp.ps1" - -64 -Before -After - - -65 -Timestomping uses the API to change the times of a file but this only changed the $STANDARD_INFORMATION stream. The real times are still present on the $FILENAME attributes. - -66 - -Exercise: Detect timestomping -Write an artifact that detects when a file has had its time stomped. - -Note: This is not necessarily a smoking gun - many installers will update a file’s timestamps during installation. - -http://www.forensickb.com/2009/02/detecting-timestamp-changing-utlities.html -67 - -68 -Many binaries are timestomped naturally because they come from CAB or MSI files. -To eliminate noise you can narrow the created time from the $FILE_NAME attribute. - -Created0x30 is the real time the file was created. - -69 -Timeline analysis -We can get a timeline by sorting the table on the modified or birth timestamps. - -It is more efficient to narrow the time of interest first. - -When post processing large tables it is better to work in stages. - -Exercise: Build a timeline -Collect Windows.NTFS.MFT from your system -Post process by building a timeline -What happened on the machine in today's session? -What files were modified? -Prefetch -Link files -Logfiles - - -70 - -71 -Many binaries are timestomped naturally because they come from CAB or MSI files. -To eliminate noise you can narrow the created time from the $FILE_NAME attribute - -The $I30 INDX stream -In NTFS a directory is simply an MFT entry with $I30 streams. The streams contains a B+ tree of the MFT entries in the directory. - -Since INDX streams are a B+ tree when a record is deleted, the tree will be reordered. Sometimes this leaves old entries in the slack space. -72 - -73 -INDX stream is allocated in 4096 byte blocks. Contains information about the directory contents. - -74 -INDX stream is allocated in 4096 bytes. Contains information about the directory contents. - -Carving INDX headers -https://www.fireeye.com/blog/threat-research/2012/10/incident-response-ntfs-indx-buffers-part-4-br-internal.html -75 - -Exercise: Experiment with $I30 carving -Add and remove files from a directory and observe which files can be carved from the $I30 stream. -See previous slide to verify the process. -76 - -Exercise: Write an artifact -Sometimes we need to prove that a file used to exist in a directory - just the presence of the name and timestamps is significant! - -Example: -FIN8 deletes prefetch files https://attack.mitre.org/techniques/T1107/ - -Write an artifact that recovers the filenames of deleted files in directories. -77 - -Exercise: Write an artifact - - - -SELECT * FROM foreach( - row={ - SELECT FullPath, Data.mft AS MFT - FROM glob(globs=DirectoryGlobs, accessor="ntfs") - WHERE IsDir - }, - query={ - SELECT FullPath, Name, NameType, Size, AllocatedSize, - IsSlack, SlackOffset, Mtime, Atime, Ctime, Btime, MFTId - FROM parse_ntfs_i30(device=FullPath, inode=MFT) -}) -78 - -The USN journal -Update Sequence Number Journal or Change journal is maintained by NTFS to record filesystem changes. -Records metadata about filesystem changes. -Resides in the path $Extend\$UsnJrnl:$J - -79 - -USN Journal -Records are appended to the file at the end -The file is sparse - periodically NTFS will remove the range at the start of the file to make it sparse -Therefore the file will report a huge size but will actually only take about 30-40mb on disk. -When collecting the journal file, Velociraptor will collect the sparse file. -80 - -81 - -82 -Velociraptor uploads only ranges with data. An index file contains the ranges offsets. -Downloading the file from the "Uploaded Files" tab will pad the sparse regions. - -OR - -Exporting the data in a zip file will include both the sparse file and the idx file. - +Conclusions +In this module we learned about more ways we can recover information from a Windows system +The SRUM database contains system telemetry about program execution. This can establish that binaries ran on the system. +Other methods include prefetch files, amcache, BAM etc. 83 -Parsing USN journal -Velociraptor can parse each entry in the journal -Remember the beginning of the file is sparse, we start parsing from the first valid range. -The USN value is the offset in the file. -The journal records many interactions with each file. -The USN journal can go back a week or two -You can find evidence of files long removed! - +Conclusions +Windows event logs are critical sources of information +We have looked at the internals of Windows Event Logs and discovered that event messages are not stored in the log files. +Velociraptor can enrich event logs automatically by parsing messages out of system Dlls +Velociraptor can also watch the event logs in and event query and respond automatically to certain events. 84 -You can collect the USN journal using the Windows.Forensics.Usn artifact! - - +Conclusions +Windows Machine Instrumentation (WMI) is a powerful OS level capability for exposing system state information. +Velociraptor provides a WMI bridge allowing VQL artifacts to directly query the WMI system. +This can be used to enrich results with file data and metadata +WMI eventing is also exposed providing a way to write event driven VQL queries that respond to WMI exposed events. 85 -The USN journal can be used to gather evidence of historical file modifications! -86 -Exercise: Post process USN -Collect the USN journal from the endpoint -Establish: -Which files were downloaded to the Downloads folder? -Program execution through prefetch? -Which files were opened through link file analysis? -Conclusions -Velociraptor implements state of the art forensic analysis capabilities in the client agent -These capabilities are exposed via VQL plugins/function -Putting together these capabilities in arbitrary combinations is the real strength: -Velociraptor can enrich forensic analysis results with extra endpoint state -Artifacts can be adapted on the fly to respond to new threats -87 Conclusions In this module we learned about: diff --git a/content/docs/forensic/event_logs/_index.md b/content/docs/forensic/event_logs/_index.md new file mode 100644 index 00000000000..31834f6153e --- /dev/null +++ b/content/docs/forensic/event_logs/_index.md @@ -0,0 +1,175 @@ +--- +title: "Event Logs" +date: 2021-06-27T04:34:03Z +draft: false +weight: 80 +--- + +Windows Event Logs + +Windows event logs +Stored in files with extension of *.evtx typically in C:\Windows\System32\WinEVT\Logs\*.evtx + +File format features: +Rollover - File is divided into chunks and new chunks can overwrite older chunks +Binary XML format provides compression +Structured records with strong types +30 + +Parsing EVTX +31 +The event message is actually written in XML but Velociraptor convert it into a JSON object to make it easier to filter specific fields. + + +Event significant fields +Provider, Channel, Computer - this represents the source of the message +Event ID - An index into the message table identifying the type of this event +EventRecordID - The ID of this message within the evtx file. +UserData - An application specific blob of structured data +32 + +Event Messages +Windows Event Logs architecture does NOT store the event message in the evtx file! +This allows for event message internationalization +Saves some small amount of space in the evtx files themselves +But mostly makes it difficult to analyze offline +Grabbing all the EVTX files off the system may result in loss of event messages! +33 + +34 + +35 +The event description message contains vital context about what the event actually means. +Without the message we would need to search for the event id. + + +36 +Event message search +If you copied the event log files off the system and do not have access to the messages, you will need to figure out what does the event id mean. + +Some common event ids are documented publicly. + +Deriving event messages +Using the provider, channel and computer name lookup the registry key +HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\EventLog\\ +Read the value EventMessageFile. +This will point at a DLL path, open the resource section of this dll for a Message Table resource. This will produce a formatted string. Interpolate the UserData section into the string. +37 + +Deriving event messages +Open the DLL, and locate the resource section in the PE file of this dll, searching for a Message Table resource. + +A MESSAGE_TABLE resource is a list of strings - the Event ID is an index into this table. + +This will produce a string with expansion directives like %1, %2 etc. Interpolate the UserData section into the string. +38 + +39 + +40 +Resolving Messages +Velociraptor can automatically follow this process when parsing event logs using the parse_evtx() plugin. +Notice the UserData is expanded into the messages. + +What could go wrong? +If you just collect the EVTX files from one system to another you will lose access to message tables, because the messages are in DLL files scattered across the entire system. + +If an application is uninstalled, its message DLLs will be removed and earlier events are not able to be displayed any more. +41 + +Event Message databases +The https://github.com/Velocidex/evtx-data repository contains sqlite databases of many known message tables collected from different systems. + +The dumpevtx tool can resolve messages from these databases and the sqlite databases. +42 + +Event logs DFIR +43 + +Exercise - resolve messages +Clone the evtx-data repository to your Linux machine. Download the dumpevtx tool from the releases page. + +View the event log samples on your Linux machine using the dumpevtx tool. Note that no messages are present. + +Download the relevant sqlite message databases and resolve messages from your evtx files. +44 + +45 +The dumpevtx tool emits JSON data. You can use the jq tool to reformat the JSON data to remove un-needed fields. + + +46 +SELECT timestamp(epoch=System.TimeCreated.SystemTime) AS Timestamp, + Message, + get(field='EventData') AS EventData +FROM parse_evtx(filename=EVTX, messagedb=MSG) + +References +https://www.appliedincidentresponse.com/windows-event-log-analyst-reference/ + +https://docs.microsoft.com/en-us/windows/security/threat-protection/auditing/audit-logon + + +47 + +Disabling event logs +Event logs can be easily disabled. + + +48 + +What is the setting? +49 + +Exercise: Detect disabled logs +Write an artifact that reports the state of each log channel (enabled/disabled) + +Use the Microsoft-Windows-Bits-Client/Operational channel as an example +50 + +Convert to an artifact +51 + +Event Tracing for Windows +52 + +What is ETW +ETW is the underlying system by which event logs are generated and collected. +https://docs.microsoft.com/en-us/windows-hardware/test/weg/instrumenting-your-code-with-etw + + +53 + +ETW Providers +Show all registered ETW providers + + + +Show details about each provider + + +54 +logman query providers +logman query providers Microsoft-Windows-DNS-Client + +ETW for event driven logs +ETW and event logs are just two sides of the same coin +Log providers are just ETW providers + +In VQL watch_etw() can be used +instead of watch_evtx() + +See Windows.Sysinternals.SysmonLogForward +for an example +55 + +Exercise - Monitor DNS queries +Use ETW to monitor all clients' DNS queries. + +Stream queries to server +56 + +Exercise - Monitor DNS queries +57 + +58 diff --git a/content/docs/forensic/evidence_of_execution/_index.md b/content/docs/forensic/evidence_of_execution/_index.md new file mode 100644 index 00000000000..b61ec53015e --- /dev/null +++ b/content/docs/forensic/evidence_of_execution/_index.md @@ -0,0 +1,161 @@ +--- +title: "Evidence Of Execution" +date: 2021-06-27T04:12:21Z +draft: false +weight: 60 +--- + +Sometimes we need to find out when (or if) a particular binary was run +on the endpoint. This question can come up in a number of contexts, +such as running malware by a user, lateral movement from a threat +actor etc. + +Windows has a rich set of forensic artifacts that we can use to infer +program execution. This page covers some of the more common evidence +of execution artifacts. + +## Prefetch files + +Prefetch files are used to [keep track of executions](http://web.archive.org/web/20130315214654/http://windows.microsoft.com:80/en-US/windows7/What-is-the-prefetch-folder) + +> What is the prefetch folder? +> +> Each time you turn on your computer, +> Windows keeps track of the way your computer starts and which +> programs you commonly open. Windows saves this information as a +> number of small files in the prefetch folder. The next time you turn +> on your computer, Windows refers to these files to help speed the +> start process. + +You can see those prefetch files in the `C:\Windows\prefetch` directory + +![Prefetch files](image4.png) + +Prefetch files’ name consist of the original binary and the [hash of the application path](https://www.symantec.com/connect/blogs/prefetch-analysis-live-response). Velociraptor has a built in Prefetch file parser, that allows extracting more information from the files themselves. + +Prefetch files contain the following data (In recent Windows 10) +* The last 8 times the binary was run +* The number of times the binary was run +* The binary name +* The file size + +## Prefetch tips + +You can try to establish the original path of the executable by [brute forcing the hash](https://hiddenillusion.github.io/2016/05/10/go-prefetch-yourself/). Typically the full path of the binary is also encoded as one of the linked PE files. + +Look for particularly suspicious binaries, eg sc.exe, xcopy.exe, +psexec.exe, bitsadmin.exe and particularly random looking binary +names. Typically lower execution counts are more interesting + +Even though the prefetch file itself only records 8 times of +execution, each time a binary is executed, the system will update the +prefetch file. It may be that other artifacts record this +interaction. In particular, the USN journal might record an +interaction with the prefetch file which is not recorded in the actual +prefetch file itself (because the binary was run more than 8 times or +the prefetch file was removed as an anti-forensic method). + +Note too that the prefetch file creation time will record the time +when the program was **first** run, giving an additional timestamp to +consider. + +## Prefetch timeline + +The `Windows.Forensics.Prefetch` artifact shows all the execution +times for each file as an array. This is less useful as we normally +want to filter it by time of interest. The +`Windows.Timeline.Prefetch` artifact is more useful for that as it +breaks records into distinct rows that can be easily filtered by +timestamps. + +![Prefetch files](image10.png) + +## Background Activity Moderator + +BAM is a Windows service that Controls activity of background +applications. This service exists in Windows 10 only after Fall +Creators update – [version 1709](https://www.andreafortuna.org/dfir/forensic-artifacts-evidences-of-program-execution-on-windows-systems/). + +The service maintains binary data in the registry which keeps track on +the execution of different programs by the user. Velociraptor can +parse these timestamps using the `Windows.Forensics.Bam` artifact. + +![BAM](image30.png) + +{{% notice note %}} + +The BAM artifact is stored in the registry under a key unique to the +user SID on the system, therefore it provides valuable attribution as +to who ran the binary (which Prefetch does not provide). + +{{% /notice %}} + +## Shim cache + +Windows maintains a backward compatible set of tweaks to binaries +called “Shims”. As part of this mechanism, there is a application +compatibility database stored in the registry key + +`HKLM\SYSTEM\CurrentControlSet\Control\SessionManager\AppCompatibility\AppCompatCache` + +You can read more about the Shim cache [here](https://www.fireeye.com/content/dam/fireeye-www/services/freeware/shimcache-whitepaper.pdf) or [here](http://www.alex-ionescu.com/?p=39) or [here](https://www.andreafortuna.org/2017/10/16/amcache-and-shimcache-in-forensic-analysis/). + +The Shim cache database tracks the executables’ file name, file size and last modified time of the binary. + +Velociraptor can parse the shim cache using the `Windows.Registry.AppCompatCache` artifact. + +{{% notice note %}} +Note this is the modification time of the binary from the NTFS $STANDARD_INFORMATION stream, which might be replicated by the installer - so it might even be before the system was installed. +{{% /notice %}} + +## Amcache + +The Windows Application Experience Service tracks process creation +data in a registry file located in +`C:\Windows\AppCompat\Programs\Amcache.hve` + +This tracks the first execution of a program on the system, including +programs executed from an external storage. You can investigate the +Amcache hive using the `Windows.System.Amcache` artifact. + +Unlike the other registry based artifacts above, this registry hive is +not mounted and accessible via the Windows APIs - the service simply +uses the registry file format to store the information. We therefore +need to parse the raw registry hive file using the raw registry +accessor. + +{{% notice note %}} + +Note the key location is a URL - Velociraptor uses URL notation to +access raw registry hives as described [here]({{< ref "/docs/forensic/filesystem#raw-registry-parsing" >}}). This one uses +the ntfs file accessor to access the raw hive data since it is usually +locked at runtime. + +{{% /notice %}} + +## System Resource Usage Monitor (SRUM) + +Windows keeps a running count of application metrics using SRUM in +order to power the "App history" tab in the task manager. + +![SRUM](image12.png) + +Metrics are stored in an ESE database at the location `%windir%\System32\sru\SRUDB.dat`. You can read more about the SRUM [here](https://www.velocidex.com/blog/medium/2019-12-31_digging-into-the-system-resource-usage-monitor-srum-afbadb1a375/). + +You can examine the ESE database manually using Nirsoft [ESEDatabaseViewer](https://www.nirsoft.net/utils/ese_database_view.html). + +![SRUM](image16.png) + +The database contains multiple tables named after the GUID of the SRUM +extension that is recording data. While not all tables are fully +understood or documented it is sometimes possible to work out what +information is recorded by simple inspection of the database tables. + +Velociraptor already knows how to interpret some of the providers: + +* `{D10CA2FE-6FCF-4F6D-848E-B2E99266FA89}` is for application resource usage. +* `{DD6636C4-8929-4683-974E-22C046A43763}` is for network connection stats + +You can collect the SRUM database using the `Windows.Forensics.SRUM` +artifact. The artifact contains several sources, each attempting to +interpret a different provider table. diff --git a/content/docs/forensic/evidence_of_execution/image10.png b/content/docs/forensic/evidence_of_execution/image10.png new file mode 100644 index 00000000000..ce7d58f592e Binary files /dev/null and b/content/docs/forensic/evidence_of_execution/image10.png differ diff --git a/content/docs/forensic/evidence_of_execution/image12.png b/content/docs/forensic/evidence_of_execution/image12.png new file mode 100644 index 00000000000..2bea5c6a4be Binary files /dev/null and b/content/docs/forensic/evidence_of_execution/image12.png differ diff --git a/content/docs/forensic/evidence_of_execution/image16.png b/content/docs/forensic/evidence_of_execution/image16.png new file mode 100644 index 00000000000..e326b918a71 Binary files /dev/null and b/content/docs/forensic/evidence_of_execution/image16.png differ diff --git a/content/docs/forensic/evidence_of_execution/image30.png b/content/docs/forensic/evidence_of_execution/image30.png new file mode 100644 index 00000000000..cf5db31a411 Binary files /dev/null and b/content/docs/forensic/evidence_of_execution/image30.png differ diff --git a/content/docs/forensic/evidence_of_execution/image4.png b/content/docs/forensic/evidence_of_execution/image4.png new file mode 100644 index 00000000000..d5c5dbf56df Binary files /dev/null and b/content/docs/forensic/evidence_of_execution/image4.png differ diff --git a/content/docs/forensic/filesystem/_index.md b/content/docs/forensic/filesystem/_index.md index 5deac75314c..288aacbf617 100644 --- a/content/docs/forensic/filesystem/_index.md +++ b/content/docs/forensic/filesystem/_index.md @@ -1,5 +1,9 @@ --- title: "Searching Filenames" +description: | + One of the most common operations in DFIR is searching for files + based on their file names. + date: 2021-06-12T23:25:17Z draft: false weight: 20 diff --git a/content/docs/forensic/ntfs/_index.md b/content/docs/forensic/ntfs/_index.md new file mode 100644 index 00000000000..76df9873255 --- /dev/null +++ b/content/docs/forensic/ntfs/_index.md @@ -0,0 +1,374 @@ +--- +title: "NTFS Analysis" +date: 2021-06-26T20:10:17Z +description: | + NTFS is the standard Windows filesystem. Velociraptor contains powerful NTFS analysis capabilities. + +draft: false +weight: 50 +--- + +NTFS is the standard Windows filesystem. Velociraptor contains +powerful NTFS analysis capabilities. This section describes +Velociraptor's NTFS capabilities and does not aim to be a complete +description of NTFS itself. We will only introduce the basic and most +relevant concepts of NTFS and examine how these can be used in a +number of DFIR contexts. + +## The Master File Table + +In NTFS, all files are represented in a [Master File Table (MFT)](https://docs.microsoft.com/en-us/windows/win32/fileio/master-file-table). The +MFT is also a file within the filesystem with the special filename of +`$MFT`. While this special file is normally hidden by the API, +Velociraptor's NTFS parser makes it available to view, read or upload. + +The `$MFT` file contains a sequence of `MFT Entries`, each of a fixed +size (usually 512 bytes). These entries contain metadata about files, +called `File Attributes`. The different attributes contain different +kinds of information about each file: + +* Filename (Long name/Short name) +* Data attribute – contains file data runs. +* I30 attribute (contains directory listing) +* Security attributes such as ACLs + +![MFT](image22.png) + +Data attributes may be compressed or sparse and contain a list of `runs` that comprize the content of the file. The data content is stored elsewhere on the disk, but the location is stored within the MFT entry. + +In NTFS Each file may contain two different filenames, a long and a +short filename. Filename attributes contain their own timestamps. + +{{% notice warning %}} + +Although NTFS long and short filenames are usually closely related +(e.g. the short filename is the first part of the long filename with a +suffix such as `%1`), this is not a requirement. + +It is very easy to create a file with a completely different short +filename to its long filename. This can be problematic if you are +looking for references to the long filename from e.g. registry keys. + +In the below example, I set the shortname of the `velociraptor.exe` +binary to `runme.exe`. I can then create a service that launches +`runme.exe` instead. Tools that only show the long filename of the +directory will fail to show the file and analysis may conclude that +the service target is missing from the filesystem. + +```shell +C:\Users\test>fsutil file setshortname velociraptor.exe runme.exe +C:\Users\test>dir /x *.exe + Volume in drive C has no label. + Volume Serial Number is 9459-F443 + + Directory of C:\Users\test + +08/19/2018 11:37 PM 12,521,472 RUNME.EXE velociraptor.exe + 2 File(s) 16,140,732 bytes + 0 Dir(s) 11,783,704,576 bytes free +C:\Users\test>runme.exe -h +usage: velociraptor [] [ ...] +``` + +{{% /notice %}} + +## The `ntfs` accessor + +Velociraptor has a complete NTFS parser able to access files and +directories by parsing the raw NTFS filesystem from the raw device. To +make it easy to utilize this parser with VQL, Velociraptor implements +the `ntfs` accessor (For a description of accessors, see [here]({{< ref "/docs/forensic/filesystem/#filesystem-accessors" >}}) ). + +The `ntfs` accessor makes it possible to see and access the normally +hidden NTFS files such as `$MFT`. It also makes it possible to see +Alternate Data Streams (ADS), which are additional data streams +attached to the same MFT entry. + +![NTFS accessor](image24.png) + +The NTFS accessor makes NTFS specific information available in the +Data field. For regular files it includes the inode string, as well as +the short filename. + +When providing a path to the `ntfs` accessor, the first part of the +path is interpreted as the `drive letter` or the `device part`. + +For example providing a path starting with `C:` or `D:`, will be +converted internally to Windows device notation, for example `\\.\C:` +or `\\.\D:`. The `ntfs` accessor then uses this to open the raw +logical device so it can be parsed. + +This means that all paths returned from the `ntfs` accessor start with +the device name, e.g. `\\.\C:`. + +{{% notice tip %}} + +Since Velociraptor operated on the logical device it if not affected +by full disk encryption such as Bitlocker. Velociraptor will be able +to parse the raw NTFS filesystem regardless of the disk encryption +status. + +{{% /notice %}} + +## Volume Shadow Copies + +NTFS allows for a special copy on write snapshot feature called +`Volume Shadow Copy` or `VSS`. You can think of a VSS as a light +weight snapshot of the current filesystem without needing to copy any +data (future writes will simply be diverted to the current active +snapshot). + +On server class Windows systems, you can create a VSS copy on your own +machine using `vssadmin create shadow`, but on other Windows versions +you will need to do this via WMI: + +![Creating shadow copy](image33.png) + +When a VSS copy is created, it is accessible via a special +device. Velociraptor allows the VSS copies to be enumerated by listing +them at the top level of the filesystem. At the top level, the +accessor provides metadata about each device in the `Data` column, +including its creation time. This is essentially the same output as +`vssadmin list shadows`. In the below screenshot we can see the `Data` +column of the fixed `C:` drive and the VSS device. + +![VSS info](image28.png) + +## Operating on VSS + +Because the `ntfs` accessor treats all devices at the first top level +directory, it is possible to see the same file in all VSS copies at +the same time. For example, the following finds all VSS copies of the +event logs: + +![VSS globbing](image31.png) + +Simply use the VSS device name as a prefix to all paths and the ntfs +accessor will parse it instead. + +You can use it to analyze older versions of the drive! + +## Parsing the MFT + +Since the `ntfs` accessor allows accessing the `$MFT` file as a +regular file, you can download the entire $MFT file from the endpoint +using the ntfs accessor, then process it offline. For example using +the `Windows.Search.FileFinder` artifact with the `ntfs` accessor - or +simply using the VQL: + +```sql +SELECT upload(path="C:/$MFT", accessor="ntfs") +FROM scope() +``` + +However, in practice this is inefficient and does not scale. Typically +we want to parse the MFT in order to answer some questions about the +system, such as which files were modified within a timerange. + +Velociraptor provides access to the $MFT parser using the +`parse_mft()` plugin, so the MFT can be parsed directly on the +endpoint using Velociraptor. The plugin emits a high level summary of +each MFT entry, including its timestamps (for the +$STANDARD_INFORMATION and $FILENAME streams) and MFT ID. + +This plugin is most useful when you need to pass over all the files in +the disk - it is more efficient than a recursive glob and might +recover deleted files. For example to recover all the files with a +.exe extension from the drive: + +```sql +SELECT * FROM parse_mft(filename="C:/$MFT", accessor="ntfs") +WHERE FileName =~ ".exe$" +``` + +## MFT Entries + +An MFT Entry can have multiple attributes and streams. While +`parse_mft()` plugin emits a high level summary for each entry, +sometimes we need more information on each MFT entry. This information +is provided by the `parse_ntfs()` VQL function which accepts and MFT +ID: + +![Parse ntfs](image39.png) + +The MFT ID can be take from the output of `glob()` or `parse_mft()`. + +{{% notice tip %}} + +In the above you will sometimes see the term `inode` referred to. This +term traditionally comes from the Sleuthkit and is a string consisting +of a triple of mft id, type id and stream id, e.g. `974-16-0` +representing a stream of data + +{{% /notice %}} + +## NTFS timestamps + +A single MFT entry can have up to 16 timestamps, based on different +attributes: + +* The $STANDARD_INFORMATION attribute contains 4 timestamps (Modified, Accessed, indoe Changed, Born) +* There are often 2 $FILENAME attributes for a short name and a long name, each will have 4 further timestamps. +* The $I30 stream of the parent directory also contains 4 timestamps for the file. + +Timestamps are critical to forensic investigations as they help to +establish a timeline of activity on the system. + +### Timestomping + +Attackers sometimes change the timestamps of files to make them less +obvious. E.g make malware look like it was installed many years +ago. This makes timelines more difficult to establish and might cause +you to miss important filesystem events. + +For the next exercise we will stomp over some times. Use the following +powershell to stomp over Velociraptor.exe’s timestamps. + +```powershell +$file = 'C:\Program Files\Velociraptor\Velociraptor.exe' +$stomp = Get-Date 2007-07-07 +$(Get-Item $file).creationtime = $stomp +$(Get-Item $file).lastaccesstime = $stomp +$(Get-Item $file).lastwritetime = $stomp +Get-ChildItem $file | Select *, Fullname, *Time* +``` + +![Before](image44.png) +![After](image38.png) + + +The above script uses the API to change the times of a file but this +only changes the $STANDARD_INFORMATION stream. The real times are +still present on the $FILENAME attributes. A common detection to this +is to find files which have $STANDARD_INFORMATION times earlier then +the $FILENAME times. When the file is created, $FILENAME times are set +to the real times, then if the API is used to send the timestamps +backwards the $STANDARD_INFORMATION timestamps will appear earlier +than the $FILENAME times. + +![Timestomp detection](image42.png) + + +{{% notice warning %}} + +Although it might appear to be a solid detection to timestomping, +generally timestomping detections are not very reliable in +practice. It turns out that a lot of programs set file timestamps +after creating them into the past by design - mostly archiving +utilities like 7zip or cab will reset the file time to the times +stored in the archive. + +Conversely it might appear that the $FILENAME times are the most +reliable and should be mostly relied upon in an investigation since +they are not directly modifiable by the Win32 APIs. + +Unfortunately this is not the case - the $FILENAME attributes can be +easily modified by simply renaming the file (after timestomping) and +rename it back. Windows will copy the timestamps from the +$STANDARD_INFORMATION attribute to the $FILENAME when renaming the +file. + +{{% /notice %}} + +## Timeline analysis + +Timelines in forensic analysis are very important as they place events +in chronological order and may reveal causal relationships. We can +get a timeline by sorting the table on the modified or birth +timestamps. + +```sql +SELECT * FROM parse_mft(filename="C:/$MFT", accessor="ntfs") +WHERE Created0x30 > "2020-01-02" +ORDER BY Created0x30 +``` + +It is more efficient to narrow the time of interest first. + +## The $I30 INDX stream + +In NTFS a directory is simply an MFT entry with $I30 streams. The +streams contains a B+ tree of the MFT entries in the directory. + +Since INDX streams are a B+ tree when a record is deleted, the tree +will be reordered. Sometimes this leaves old entries in the slack +space. INDX stream is allocated in 4096 byte blocks which leaves a lot +of slack space to potentially hold residual data. + +![I30 slack](image54.png) + +Velociraptor can report on the $I30 streams and carve out headers from slack using the parse_ntfs_i30() function as duscussed in [this article](https://www.fireeye.com/blog/threat-research/2012/10/incident-response-ntfs-indx-buffers-part-4-br-internal.html). An example query: + +```sql +SELECT * FROM foreach( + row={ + SELECT FullPath, Data.mft AS MFT + FROM glob(globs=DirectoryGlobs, accessor="ntfs") + WHERE IsDir + }, + query={ + SELECT FullPath, Name, NameType, Size, AllocatedSize, + IsSlack, SlackOffset, Mtime, Atime, Ctime, Btime, MFTId + FROM parse_ntfs_i30(device=FullPath, inode=MFT) +}) +``` + +## The USN journal + +Update Sequence Number Journal or Change journal is maintained by NTFS +to record filesystem changes. Prmiarily designed to support backup +programs, the USN journal records metadata about filesystem changes. + +The journal resides in the path `$Extend\$UsnJrnl:$J` and is normally +a hidden NTFS internal file (so it can only be accessed via the `ntfs` +accessor). + +Windows appends USN records to the end of the file. However, the file +is sparse - periodically NTFS will remove the range at the start of +the file to make it sparse and preserve disk space. + +Typically the file will report a huge size but will actually only take +about 30-40mb on disk since the first part of the file is sparse. + +![The USN journal](image43.png) + +When collecting the journal file, Velociraptor will collect the sparse +file only (Velociraptor is aware of sparse files and preserves their +sparse ranges by adding an additional `.idx` file to the collection +with the ranges containing real data. You can see this in the +`Uploaded Files` tab of the collection - the file size is reported to +be very large, however only about 30mb was actually collected. + +![The USN journal collected](image47.png) + + +{{% notice tip %}} + +Downloading the file from the `Uploaded Files` tab will pad the sparse regions and produce a large file with ranges of 0 in it. On the other hand, exporting the zip file from the `Overview` tab will store the collected file and the idx range file into the zip file so will only store about 30mb. + +{{% /notice %}} + +### Parsing USN journal + +Velociraptor can parse each entry in the USN journal directly on the +endpoint. This allows for queries to target specific files or times of +interest on the endpoint. + +Since the beginning of the file is sparse, we start parsing from the +first valid range. + +The USN journal may record interactions with files that have been +removed. Many files represent evidence of system interaction (such as +lnk files or prefetch files) and the USN journal can therefore uncover +the "smoking gun" when the system was initially compromised. + +![The USN journal](image60.png) + +You can collect the USN journal using the `Windows.Forensics.Usn` +artifact. + +{{% notice tip %}} + +The USN journal contains so much valuable evidence that it might be worth carving for USN records from the raw disk. Although this is a slow process it can yield very good results if your are lucky - see [this blog post]({{< ref "/blog/2021/2021-06-16-carving-usn-journal-entries-72d5c66971da/" >}}) for more information. + +{{% /notice %}} diff --git a/content/docs/forensic/ntfs/image22.png b/content/docs/forensic/ntfs/image22.png new file mode 100644 index 00000000000..d6a168338c5 Binary files /dev/null and b/content/docs/forensic/ntfs/image22.png differ diff --git a/content/docs/forensic/ntfs/image24.png b/content/docs/forensic/ntfs/image24.png new file mode 100644 index 00000000000..00cf9952f29 Binary files /dev/null and b/content/docs/forensic/ntfs/image24.png differ diff --git a/content/docs/forensic/ntfs/image28.png b/content/docs/forensic/ntfs/image28.png new file mode 100644 index 00000000000..fdbebd42c43 Binary files /dev/null and b/content/docs/forensic/ntfs/image28.png differ diff --git a/content/docs/forensic/ntfs/image31.png b/content/docs/forensic/ntfs/image31.png new file mode 100644 index 00000000000..79beef71a0c Binary files /dev/null and b/content/docs/forensic/ntfs/image31.png differ diff --git a/content/docs/forensic/ntfs/image33.png b/content/docs/forensic/ntfs/image33.png new file mode 100644 index 00000000000..b02a56f24c9 Binary files /dev/null and b/content/docs/forensic/ntfs/image33.png differ diff --git a/content/docs/forensic/ntfs/image38.png b/content/docs/forensic/ntfs/image38.png new file mode 100644 index 00000000000..3c262f17c50 Binary files /dev/null and b/content/docs/forensic/ntfs/image38.png differ diff --git a/content/docs/forensic/ntfs/image39.png b/content/docs/forensic/ntfs/image39.png new file mode 100644 index 00000000000..4036b6ebf67 Binary files /dev/null and b/content/docs/forensic/ntfs/image39.png differ diff --git a/content/docs/forensic/ntfs/image42.png b/content/docs/forensic/ntfs/image42.png new file mode 100644 index 00000000000..65b50390a27 Binary files /dev/null and b/content/docs/forensic/ntfs/image42.png differ diff --git a/content/docs/forensic/ntfs/image43.png b/content/docs/forensic/ntfs/image43.png new file mode 100644 index 00000000000..09c13627aef Binary files /dev/null and b/content/docs/forensic/ntfs/image43.png differ diff --git a/content/docs/forensic/ntfs/image44.png b/content/docs/forensic/ntfs/image44.png new file mode 100644 index 00000000000..c36f9d07ab7 Binary files /dev/null and b/content/docs/forensic/ntfs/image44.png differ diff --git a/content/docs/forensic/ntfs/image47.png b/content/docs/forensic/ntfs/image47.png new file mode 100644 index 00000000000..5587e14e6e2 Binary files /dev/null and b/content/docs/forensic/ntfs/image47.png differ diff --git a/content/docs/forensic/ntfs/image54.png b/content/docs/forensic/ntfs/image54.png new file mode 100644 index 00000000000..11e1438221c Binary files /dev/null and b/content/docs/forensic/ntfs/image54.png differ diff --git a/content/docs/forensic/ntfs/image60.png b/content/docs/forensic/ntfs/image60.png new file mode 100644 index 00000000000..e6cdb09b829 Binary files /dev/null and b/content/docs/forensic/ntfs/image60.png differ diff --git a/content/docs/forensic/searching/_index.md b/content/docs/forensic/searching/_index.md index 56f41242b75..f900ed6c732 100644 --- a/content/docs/forensic/searching/_index.md +++ b/content/docs/forensic/searching/_index.md @@ -1,5 +1,10 @@ --- title: "Searching Content" +description: | + A powerful DFIR technique is searching bulk data for patterns. YARA is a + powerful keyword scanner that allows to search unstructured binary data + based on user provided rules. + date: 2021-06-17T02:30:41Z draft: false weight: 30 @@ -17,15 +22,17 @@ Bulk searching helps to identify evidence without needing to parse file formats ## YARA - The swiss army knife -YARA is a powerful keyword scanner -Uses rules designed to identify binary patterns in bulk data -YARA is optimized to scan for many rules simultaneously. -Velociraptor supports YARA scanning of bulk data (via accessors) and memory. +YARA is a powerful keyword scanner that allows to search unstructured +binary data based on user provided rules. YARA is optimized to scan +for many rules simultaneously, making is an excellent choice for +detecting suspicious binaries using common patterns. + +Velociraptor supports YARA scanning of bulk data (via accessors) and +memory using the `yara()` and `proc_yara()` plugins. -yara() and proc_yara() -28 +An example of a YARA rule is shown below. -YARA rules +```yara rule X { strings: $a = “hello” nocase @@ -35,64 +42,111 @@ rule X { condition: $a and ($b or $c) } -29 - -Exercise: drive by download -You suspect a user was compromised by a drive by download (i.e. they clicked and downloaded malware delivered by mail, ads etc). -You think the user used the Edge browser but you have no idea of the internal structure of the browser cache/history etc. -Write an artifact to extract potential URLs from the Edge browser directory (also where is it?) -30 - -Step 1: Figure out where to look -31 - -32 -Looks like somewhere in C:\Users\\AppData\Local\Microsoft\Edge\** - -Step 2: Recover URLs -We don't exactly understand how Edge stores data but we know roughly what a URL is supposed to look like! -Yara is our sledgehammer ! - -rule URL { - strings: $a = /https?:\\/\\/[a-z0-9\\/+&#:\\?.-]+/i - condition: any of them -} -33 - -Step 3: Let’s do this! -34 +``` -35 +The rule consists of a `strings` section and a `condition` +section. Strings represent a set of keywords which might include ASCII +or UTF16 encoded strings, as well as regular expressions. You can refer to the [Yara rules reference page](https://yara.readthedocs.io/en/stable/) to learn about how to construct rules. -36 +{{% notice tip %}} -37 -YARA best practice -You can get yara rules from many sources (threat intel, blog posts etc) -YARA is really a first level triage tool: -Depending on signature many false positives expected -Some signatures are extremely specific so make a great signal -Try to collect additional context around the hits to eliminate false positives. -Yara scanning is relatively expensive! consider more targeted glob expressions and client side throttling since usually YARA scanning is not time critical. +The `yara()` VQL plugin can accept an optional `accessor` +parameter. If the accessor is specified, the plugin will read chunks +of data from the accessor and apply the YARA rules on the string in +memory. This allows you to apply YARA rules on any data that is +available via an accessor including raw strings (using the `data` +accessor), registry values (using the `registry` accessor) or NTFS +parsed data (using the `ntfs` accessor) for example. +While this is convenient, it means that rules that example the entire +file will not work as expected. For example, the YARA `pe` module +looks at the PE header, but when the file is read in chunks, only the +first chunk contains the PE header. Similarly YARA rules that contain +an expression checking a file offset will not work because the rules +are applied to buffers in memory. -Uploading files -38 +When an accessor is not specified, the `yara()` plugin assumes the +filename refers to a filesystem path, and simply allows the YARA +library to scan the file as is. The YARA library uses `mmap()` to map +the entire file into memory and can therefore optimize the scan across +the entire file. -Collecting files -Velociraptor can collect file data. -Over the network -Locally to a collection zip file. -Driven by VQL +It is therefore much faster to not specify an accessor to the `yara()` +plugin if you just need to scan files on disk. -The upload() VQL function copies a file using an accessor to the relevant container -39 +{{% /notice %}} -Exercise -Collect all executables in users’ home directory +### Example: drive by download +You suspect a user was compromised by a drive by download (i.e. they +clicked and downloaded malware delivered by mail, ads etc). -Write your own VQL by combining glob() and upload() -40 +You think the user used the Edge browser but for this example, assume +you have no idea of the internal structure of the browser +cache/history etc. Write an artifact to extract potential URLs from +the Edge browser directory. -41 +```sql +LET YaraRule = ''' +rule URL { + strings: $a = /https?:\\/\\/[a-z0-9\\/+&#:\\?.-]+/i + condition: any of them +} +''' + +SELECT * FROM foreach( +row={ + SELECT FullPath FROM glob(globs='''C:\Users\*\AppData\Local\Microsoft\Edge\**''') +}, query={ + SELECT str(str=Strings.Data) AS Hit, + String.Offset AS Offset, + FileName + FROM yara(files=FullPath, rules=YaraRule) +}) +``` + +![URL scanning](image18.png) + +## YARA best practice + +You can get yara rules from many sources (threat intel, blog posts +etc) or you can write your own. Rules may be very specific, in which +case a hit may represent a valuable signal. If the YARA rule is too +loose, the likelihood of a false positive increases, and further +postprocessing will be required to verify the hits. + +Try to collect additional context around the hits to eliminate false +positives. You can use other plugins to help verify other aspects of +each hit before reporting it, thereby eliminating false positives. + +Yara scanning is relatively expensive since we need to read data from +disk! consider more targeted glob expressions to limit the number of +disk reads Velociraptor will need to do to evaluate the query. If you +find you do need to scan a lot of data, consider specifying client +side throttling when launching the collection or hunt (using the +Ops/Sec mechanism) - usually YARA scanning is not time critical. + + +## Uploading files + +One of the unique capabilities of Velociraptor is uploading file +content from the endpoint. While the actual mechanism of uploading the +file to the server is abstracted away, triggering a file upload from +VQL is a simple matter of calling the `upload()` function. This makes +it trivial to upload files based on any criteria of the query. + +The `upload()` function simply requires an accessor and a filename to +read the file out, and the file is uploaded to the server +automatically. Optionally the function may also take a `name` +parameter which renames the file as sent to the server. + +### Example: Collect all executables in users’ home directory + +This is a common use of compbining a `glob()` plugin with an +`upload()` function: + +```sql +SELECT upload(path=FullPath) AS Upload +FROM glob(globs='''C:\Users\*\Downloads\*''') +WHERE NOT IsDir +``` diff --git a/content/docs/forensic/searching/image18.png b/content/docs/forensic/searching/image18.png new file mode 100644 index 00000000000..e90cb534a8f Binary files /dev/null and b/content/docs/forensic/searching/image18.png differ diff --git a/content/docs/forensic/volatile/_index.md b/content/docs/forensic/volatile/_index.md new file mode 100644 index 00000000000..1309a50c623 --- /dev/null +++ b/content/docs/forensic/volatile/_index.md @@ -0,0 +1,130 @@ +--- +title: "Volatile State" +date: 2021-06-27T04:35:23Z +draft: false +weight: 90 +--- + +Volatile machine state + +Volatile state +So far we looked at disk based artifacts. +Often evidence is ephemeral and will vanish quickly. The next slides focus on evidence that only exists temporarily and may disappear quickly. +Velociraptor's unique strength is being able to quickly and efficiently capture this volatile state using automated artifacts +60 + +61 +Windows Management Instrumentation + +WMI +A framework to export internal windows state information using a query language (WQL) +Consists of classes (providers) and objects +Lots of hooks into many internal system features +Being able to inspect system state using a consistent interface allows a tool to query a wide range of services. +62 + +63 + +64 + +65 + +66 +Mutants + +Malware persistence +Malware needs to ensure there is only a single copy of it running. +A common method is to use a Mutant (Or named mutex) +Create a mutant with a constant name: + If the named mutant already exists, then exit + +Ensures only a single copy is run. +67 + +Exercise - Mutants +$createdNew = $False +$mutex = New-Object -TypeName System.Threading.Mutex( + $true, "Global\MyBadMutex", [ref]$createdNew) +if ($createdNew) { + echo "Acquired Mutex" + sleep(100) +} else { + echo "Someone else has the mutex" +} + +68 + +Enumerate the mutants +69 + +70 +Process analysis + +Windows Processes +A process is a user space task with a specific virtual memory layout + +A process has a Process ID (Pid), an initial binary on disk, an ACL Token, environment variables etc. + +Each of these properties can be inspected by Velociraptor +71 + +Process Information +Simple pslist() can reveal basic information about the process + +Who launched the binary? +Transfer metrics (network/disk activity) +Is it elevated? +Process Creation time +72 + +73 + +74 +Process Call chain + +75 +Process traversing can be done in pure VQL by recursively calling a locally defined function. + +76 +Exercise - Find elevated command shell +Write an artifact to find all currently running elevated command shells + +Report how long they have run for + +Mapped Memory +When a binary runs it links many DLLs into it + +A linked DLL is a copy on write memory mapping of a file on disk into the process memory space. + +DLLs can be linked when the program starts or dynamically +77 + +The VAD plugin +This plugin shows all the process memory regions and if the memory is mapped to file, the filename it is mapped from. + +DLLs and .NET assemblies are mapped into the process - so we can use this to get an idea of what the program is doing. +78 + +79 + +80 +Exercise - look into powershell +Without enabling powershell block logging, we can get an idea of what the script is doing by looking at its dependencies. +Write VQL to list all the DLL modules that powershell is running. + +Run our previous mutex script. +Add the following command (this is typical of C&C) +Invoke-WebRequest -Uri "https://www.google.com" -UseBasicParsing + + + +Dump mapped objects +Dump the powershell process's mapped DLLs. +The DLL winhttp.dll is responsible for making outbound http connections. + +If the http request is enabled, the process will link the winhttp.dll at runtime. +This technique works on many other programs that may be subverted for example Cobalt Strike reflective DLL injection. +81 + +Dump mapped objects +82 diff --git a/content/docs/offline_triage/_index.md b/content/docs/offline_triage/_index.md new file mode 100644 index 00000000000..370a14f19a5 --- /dev/null +++ b/content/docs/offline_triage/_index.md @@ -0,0 +1,6 @@ +--- +title: "Offline Triage" +date: 2021-06-27T04:31:24Z +draft: false +weight: 45 +--- diff --git a/content/vql_reference/_index.md b/content/vql_reference/_index.md index 9af2385e9d8..da4944a4b49 100644 --- a/content/vql_reference/_index.md +++ b/content/vql_reference/_index.md @@ -1,13 +1,16 @@ --- -title: "Search VQL Reference" +title: "VQL Reference" menutitle: VQL Reference date: 2021-06-12T05:12:26Z draft: false weight: 60 no_edit: true disableToc: true +chapter: false pre: head:
--- +Search for documentation on VQL plugins or functions. + {{% reference %}} diff --git a/layouts/partials/menu-footer.html b/layouts/partials/menu-footer.html index 1568cd6ac4e..fbd20cab4b0 100644 --- a/layouts/partials/menu-footer.html +++ b/layouts/partials/menu-footer.html @@ -1,3 +1,4 @@
Brough to you by -
2021
+
2021 + diff --git a/layouts/shortcodes/exchange.html b/layouts/shortcodes/exchange.html index 528d4eda33f..87e34c6b1ec 100644 --- a/layouts/shortcodes/exchange.html +++ b/layouts/shortcodes/exchange.html @@ -121,7 +121,7 @@

Share an artifact

for (let j=0; j ` + tag + ``); + template.find(".idea-tag").append(`` + tag + ` `); } let new_item = $(".search_results").append(template); } diff --git a/layouts/shortcodes/reference.html b/layouts/shortcodes/reference.html index 96c7e0002c9..96b92fb9559 100644 --- a/layouts/shortcodes/reference.html +++ b/layouts/shortcodes/reference.html @@ -63,10 +63,14 @@ `); - template.find(".title").append(item.name); - template.find(".description").append(item.description); + template.find(".title").append(item.name); + template.find(".description").append(item.description); template.find(".link").attr("href", getLink(item)); template.find(".type").append(item.type) + template.find(".idea-tag").append( + `` + + (item.category || "misc") + ``); + if (item.args.length > 0) { let args_template = $(`