Update VQL references (Velocidex#535)

.pyspelling.yml

@@ -20,9 +20,6 @@ matrix:
           # ```
           - open: '(?s)^(?P<open> *`{3,})'
             close: '^(?P=open)$'
-            # Ignore text between inline back ticks
-          - open: '(?P<open>`+)'
-            close: '(?P=open)'
             # Insides of URL links
           - open: '\]\('
             close: '\)'
@@ -32,8 +29,12 @@ matrix:
             # Inside HTML tags
           - open: '(?s)(?P<open>[<])'
             close: '[>]'
+            # Ignore text between inline back ticks
+          - open: '(?P<open>`+)'
+            close: '(?P=open)'
+
             # Inside yaml headers
-          - open: '(?s)^(?P<open> *-{3,})'
+          - open: '(?s)^(?P<open>---)$'
             close: '^(?P=open)$'
 
     - pyspelling.filters.markdown:

.wordlist.txt

@@ -1292,3 +1292,21 @@ CVEs
 TLDR
 musl
 vfs
+
+deduped
+LazyAny
+
+VelociraptorDarwin
+VelociraptorLinux
+VelociraptorWindows
+os
+IOPs
+ProcessEntry
+ProcessTrackerUpdate
+enrichments
+msec
+
+namespece
+cwd
+PIDs
+EBS

content/docs/server_automation/server_monitoring/_index.md

@@ -30,8 +30,7 @@ case:
 
 The full description of how to set up slack to receive messages from
 Velociraptor can be found in our blog post [Slack and
-Velociraptor]({{< ref
-"/blog/2020/2020-12-26-slack-and-velociraptor-b63803ba4b16/" >}}), but
+Velociraptor]({{< ref "/blog/2020/2020-12-26-slack-and-velociraptor-b63803ba4b16/" >}}), but
 here we cover the high level details.
 
 ### Step 1: Checking a group of clients for online status

content/vql_reference/basic/_index.md

@@ -50,7 +50,7 @@ or in condition clauses (i.e. after the `WHERE` keyword).
 |[join](join)|<span class='vql_type'>Function</span>|Join all the args on a separator|
 |[killkillkill](killkillkill)|<span class='vql_type'>Function</span>|Kills the client and forces a restart - this is very aggressive!|
 |[len](len)|<span class='vql_type'>Function</span>|Returns the length of an object|
-|[log](log)|<span class='vql_type'>Function</span>|Log the message|
+|[log](log)|<span class='vql_type'>Function</span>|Log the message and return TRUE|
 |[lowcase](lowcase)|<span class='vql_type'>Function</span>||
 |[max](max)|<span class='vql_type'>Function</span>|Finds the largest item in the aggregate|
 |[memoize](memoize)|<span class='vql_type'>Function</span>|Memoize a query into memory|

content/vql_reference/basic/len/_index.md

@@ -19,7 +19,7 @@ no_edit: true
 
 Arg | Description | Type
 ----|-------------|-----
-list|A list of items too filter|Any (required)
+list|A list of items to filter|Any (required)
 
 ### Description
 

content/vql_reference/basic/log/_index.md

@@ -26,5 +26,44 @@ level|Level to log at (DEFAULT, WARN, ERROR, INFO).|string
 
 ### Description
 
-Log the message.
+Log the message and return TRUE.
+
+The message will be logged into the query log stream (Viewable in
+the Logs tab). The `message` parameter represents a format string
+that will be expanded using the `args` parameter list if needed.
+
+Since `log()` always returns TRUE it is easy to use in a WHERE
+clause as a form of debugging (It is basically equivalent to the
+print statement of other languages).
+
+```sql
+SELECT * FROM glob(...)
+WHERE log(message="Value of OSPath is %v", args=OSPath)
+```
+
+## Deduplication
+
+Log messages will be deduped according to the `dedup`
+parameter - each distinct format string will not be emitted more
+frequently than the `dedup` parameter (by default 60 seconds).
+
+This makes it safe to use `log()` frequently without overflowing
+the logs stream.
+
+```vql
+SELECT * FROM range(end=_value)
+WHERE log(message="Value is %v", args=_value)
+```
+
+Will only emit a single message due to the format string being
+deduped.
+
+This property makes it useful to add progress logging to long
+running artifacts. The logs will be emitted every minute.
+
+```vql
+SELECT * FROM glob(...)
+WHERE log(message="Processing file %v", args=OSPath)
+```
+
 

content/vql_reference/misc/_index.md

@@ -19,10 +19,14 @@ Miscellaneous plugins not yet categorized.
 |[delete_flow](delete_flow)|<span class='vql_type'>Plugin</span>|Delete all the files that make up a flow|
 |[entropy](entropy)|<span class='vql_type'>Function</span>|Calculates shannon scale entropy of a string|
 |[flow_logs](flow_logs)|<span class='vql_type'>Plugin</span>|Retrieve the query logs of a flow|
+|[get_flow](get_flow)|<span class='vql_type'>Function</span>|Gets flow details|
 |[gunzip](gunzip)|<span class='vql_type'>Function</span>|Apply Gunzip to the data|
 |[hunt_delete](hunt_delete)|<span class='vql_type'>Plugin</span>|Delete a hunt|
 |[lru](lru)|<span class='vql_type'>Function</span>|Creates an LRU object|
 |[lzxpress_decompress](lzxpress_decompress)|<span class='vql_type'>Function</span>|Decompress an lzxpress blob|
+|[mail](mail)|<span class='vql_type'>Function</span>|Send Email to a remote server|
+|[mock_clear](mock_clear)|<span class='vql_type'>Function</span>|Resets all mocks|
+|[mock_replay](mock_replay)|<span class='vql_type'>Function</span>|Replay recorded calls on a mock|
 |[org](org)|<span class='vql_type'>Function</span>|Return the details of the current org|
 |[org_create](org_create)|<span class='vql_type'>Function</span>|Creates a new organization|
 |[org_delete](org_delete)|<span class='vql_type'>Function</span>|Deletes an Org from the server|
@@ -41,8 +45,10 @@ Miscellaneous plugins not yet categorized.
 |[query](query)|<span class='vql_type'>Plugin</span>|Evaluate a VQL query|
 |[rekey](rekey)|<span class='vql_type'>Function</span>|Causes the client to rekey and regenerate a new client ID|
 |[remap](remap)|<span class='vql_type'>Function</span>|Apply a remapping configuration to the root scope|
+|[repack](repack)|<span class='vql_type'>Function</span>|Repack and upload a repacked binary or MSI to the server|
 |[server_frontend_cert](server_frontend_cert)|<span class='vql_type'>Function</span>|Get Server Frontend Certificate|
 |[tlsh_hash](tlsh_hash)|<span class='vql_type'>Function</span>|Calculate the tlsh hash of a file|
+|[trace](trace)|<span class='vql_type'>Function</span>|Upload a trace file|
 |[user](user)|<span class='vql_type'>Function</span>|Retrieves information about the Velociraptor user|
 |[user_grant](user_grant)|<span class='vql_type'>Function</span>|Grants the user the specified roles|
 |[vfs_ls](vfs_ls)|<span class='vql_type'>Plugin</span>|List directory and build a VFS object|

content/vql_reference/misc/all/_index.md

@@ -19,7 +19,7 @@ no_edit: true
 
 Arg | Description | Type
 ----|-------------|-----
-items|The items to consider. Can be an array, subquery or stored query. Will only be lazily evaluted!|Any (required)
+items|The items to consider. Can be an array, subquery or stored query. Will only be lazily evaluated!|Any (required)
 filter|A callback to consider each item|Lambda
 regex|Optionally one or more regex can be provided for convenience|list of string
 

content/vql_reference/misc/any/_index.md

@@ -19,7 +19,7 @@ no_edit: true
 
 Arg | Description | Type
 ----|-------------|-----
-items|The items to consider. Can be an array, subquery or stored query. Will only be lazily evaluted!|Any (required)
+items|The items to consider. Can be an array, subquery or stored query. Will only be lazily evaluated!|Any (required)
 filter|A callback to consider each item|Lambda
 regex|Optionally one or more regex can be provided for convenience|list of string
 

content/vql_reference/misc/get_flow/_index.md

@@ -0,0 +1,28 @@
+---
+title: get_flow
+index: true
+noTitle: true
+no_edit: true
+---
+
+
+
+<div class="vql_item"></div>
+
+
+## get_flow
+<span class='vql_type pull-right page-header'>Function</span>
+
+
+
+<div class="vqlargs"></div>
+
+Arg | Description | Type
+----|-------------|-----
+client_id||string (required)
+flow_id||string
+
+### Description
+
+Gets flow details.
+

content/vql_reference/misc/mail/_index.md

@@ -0,0 +1,38 @@
+---
+title: mail
+index: true
+noTitle: true
+no_edit: true
+---
+
+
+
+<div class="vql_item"></div>
+
+
+## mail
+<span class='vql_type pull-right page-header'>Function</span>
+
+
+
+<div class="vqlargs"></div>
+
+Arg | Description | Type
+----|-------------|-----
+to|Recipient of the mail|list of string (required)
+from|The from email address.|string
+cc|A cc for the mail|list of string
+subject|The subject.|string
+body|The body of the mail.|string (required)
+period|How long to wait before sending the next mail - help to throttle mails.|int64
+server_port|The SMTP server port to use (default 587).|uint64
+server|The SMTP server to use (if not specified we try the config file).|string
+auth_username|The SMTP username we authenticate to the server.|string
+auth_password|The SMTP username password we use to authenticate to the server.|string
+skip_verify|Skip SSL verification(default: False).|bool
+root_ca|As a better alternative to disable_ssl_security, allows root ca certs to be added here.|string
+
+### Description
+
+Send Email to a remote server.
+

content/vql_reference/misc/mock_clear/_index.md

@@ -0,0 +1,20 @@
+---
+title: mock_clear
+index: true
+noTitle: true
+no_edit: true
+---
+
+
+
+<div class="vql_item"></div>
+
+
+## mock_clear
+<span class='vql_type pull-right page-header'>Function</span>
+
+
+### Description
+
+Resets all mocks.
+

content/vql_reference/misc/mock_replay/_index.md

@@ -0,0 +1,30 @@
+---
+title: mock_replay
+index: true
+noTitle: true
+no_edit: true
+---
+
+
+
+<div class="vql_item"></div>
+
+
+## mock_replay
+<span class='vql_type pull-right page-header'>Function</span>
+
+
+
+<div class="vqlargs"></div>
+
+Arg | Description | Type
+----|-------------|-----
+plugin|The plugin to mock|string
+function|The function to mock|string
+expected_calls|How many times plugin should be called|int
+clear|This call will clear previous mocks for this plugin|bool
+
+### Description
+
+Replay recorded calls on a mock.
+

content/vql_reference/misc/repack/_index.md

@@ -0,0 +1,32 @@
+---
+title: repack
+index: true
+noTitle: true
+no_edit: true
+---
+
+
+
+<div class="vql_item"></div>
+
+
+## repack
+<span class='vql_type pull-right page-header'>Function</span>
+
+
+
+<div class="vqlargs"></div>
+
+Arg | Description | Type
+----|-------------|-----
+target|The name of the target OS to repack (VelociraptorWindows, VelociraptorLinux, VelociraptorDarwin)|string
+exe|Alternative a path to the executable to repack|OSPath
+accessor|The accessor to use to read the file.|string
+binaries|List of tool names that will be repacked into the target|list of string
+config|The config to be repacked in the form of a json or yaml string|string (required)
+upload_name|The name of the upload to create|string (required)
+
+### Description
+
+Repack and upload a repacked binary or MSI to the server.
+

content/vql_reference/misc/trace/_index.md

@@ -0,0 +1,20 @@
+---
+title: trace
+index: true
+noTitle: true
+no_edit: true
+---
+
+
+
+<div class="vql_item"></div>
+
+
+## trace
+<span class='vql_type pull-right page-header'>Function</span>
+
+
+### Description
+
+Upload a trace file.
+

content/vql_reference/misc/write_jsonl/_index.md

@@ -19,8 +19,8 @@ no_edit: true
 
 Arg | Description | Type
 ----|-------------|-----
-filename|Output JSON files to open|string (required)
-accessor|The accessor to use. Currently only supports "auto", "file" or "".|string
+filename|CSV files to open|string (required)
+accessor|The accessor to use|string
 query|query to write into the file.|StoredQuery (required)
 
 ### Description

content/vql_reference/parsers/parse_mft/_index.md

@@ -22,6 +22,7 @@ Arg | Description | Type
 filename|The MFT file.|OSPath (required)
 accessor|The accessor to use.|string
 prefix|If specified we prefix all paths with this path.|OSPath
+start|The first entry to scan.|int64
 
 ### Description
 

content/vql_reference/parsers/split_records/_index.md

@@ -21,10 +21,12 @@ Arg | Description | Type
 ----|-------------|-----
 filenames|Files to parse.|list of OSPath (required)
 accessor|The accessor to use|string
-regex|The split regular expression (e.g. a comma)|string (required)
+regex|The split regular expression (e.g. a comma, default whitespace)|string
 columns|If the first row is not the headers, this arg must provide a list of column names for each value.|list of string
 first_row_is_headers|A bool indicating if we should get column names from the first row.|bool
 count|Only split into this many columns if possible.|int
+record_regex|A regex to split data into records (default |string
+buffer_size|Maximum size of line buffer (default 64kb).|int
 
 ### Description
 

content/vql_reference/plugin/pathspec/_index.md

@@ -24,6 +24,7 @@ DelegatePath|A delegate to pass to the accessor.|string
 Path|A path to open.|Any
 parse|Alternatively parse the pathspec from this string.|string
 path_type|Type of path this is (windows,linux,registry,ntfs).|string
+accessor|The accessor to use to parse the path with|string
 
 ### Description
 

content/vql_reference/plugin/yara/_index.md

@@ -88,7 +88,7 @@ yara DSL and the following combinations are supported `wide`,
 `wide ascii`, `wide nocase`, `wide nocase ascii`.
 
 This shorthand notation is less useful because recent Velociraptor
-versions offer a context sensivite Yara rule editor in the GUI
+versions offer a context sensitive Yara rule editor in the GUI
 (simply press ? to bring up a rule template).
 
 {{% notice note %}}

content/vql_reference/server/create_flow_download/_index.md

@@ -22,8 +22,8 @@ Arg | Description | Type
 client_id|Client ID to export.|string (required)
 flow_id|The flow id to export.|string (required)
 wait|If set we wait for the download to complete before returning.|bool
-type|Type of download to create (deperated Ignored).|string
-template|Report template to use (deperated Ignored).|string
+type|Type of download to create (deprecated Ignored).|string
+template|Report template to use (deprecated Ignored).|string
 password|An optional password to encrypt the collection zip.|string
 format|Format to export (csv,json,csv_only) defaults to both.|string
 expand_sparse|If set we expand sparse files in the archive.|bool