-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
9 changed files
with
286 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,126 @@ | ||
+++ | ||
title = "SU-CS361 MAY072024" | ||
author = ["Houjun Liu"] | ||
draft = false | ||
+++ | ||
|
||
## Generalization Error {#generalization-error} | ||
|
||
\begin{equation} | ||
\epsilon\_{gen} = \mathbb{E}\_{x \sim \mathcal{X}} \qty[\qty(f(x) - \hat{f}(x))^{2}] | ||
\end{equation} | ||
|
||
we usually instead of compute it by averaging specific points we measured. | ||
|
||
|
||
## Probabilistic Surrogate Models {#probabilistic-surrogate-models} | ||
|
||
|
||
### Gaussian Process {#gaussian-process} | ||
|
||
A [Gaussian Process](#gaussian-process) is a [Gaussian distribution]({{< relref "KBhgaussian_distribution.md" >}}) over [function]({{< relref "KBhfunction.md" >}})s! | ||
|
||
Consider a mean function \\(m(x)\\), and a covariance ([kernel]({{< relref "KBhnull_space.md" >}})) function \\(k(x, x')\\). And, for a set of objective values \\(y\_{j} \in \mathbb{R}\\), which we are trying to infer using \\(m\\) and \\(k\\). | ||
|
||
\begin{equation} | ||
\mqty[y\_1 \\\ \dots \\\ y\_{m}] \sim \mathcal{N} \qty(\mqty[m(x\_1) \\\ \dots \\\ m(x\_{m})], \mqty[k(x\_1, x\_1) & \dots & k(x\_1, x\_{m}) \\\&\dots&\\\ k(x\_{m}, x\_{1}) &\dots &k(x\_{m}, x\_{m})]) | ||
\end{equation} | ||
|
||
The choice of [kernel]({{< relref "KBhnull_space.md" >}}) makes or breaks the your ability to model your system. Its the way by which your input values are "smoothed" together to create a probabilistic estimate. | ||
|
||
|
||
#### Choice of Kernels {#choice-of-kernels} | ||
|
||
<!--list-separator--> | ||
|
||
- squared exponential kernel | ||
|
||
\begin{equation} | ||
k(x,x') = \exp \qty( \frac{-(x-x')^{2}}{2 l^{2}}) | ||
\end{equation} | ||
|
||
where, \\(l\\) is the parameter controlling the "length scale" (i.e. distance required for the function to change significantly). As \\(l\\) gets larger, there's more smoothing. | ||
|
||
<!--list-separator--> | ||
|
||
- Matérn Kernel | ||
|
||
This is a very common kernel. Look it up. | ||
|
||
|
||
#### Prediction {#prediction} | ||
|
||
Given known means and variances of the sampled points from the original system, we can compute: | ||
|
||
\begin{equation} | ||
P(Y^{\*}|Y) | ||
\end{equation} | ||
|
||
through using [conditioning Gaussian distributions]({{< relref "KBhgaussian_distribution.md#conditioning-gaussian-distributions" >}}). | ||
|
||
Specifically: | ||
|
||
\begin{equation} | ||
\mqty[\hat{y} \\\ y] \sim \mathcal{N} \qty(\mqty[m(X^{\*}) \\\ m(X)], \mqty[K(X^{\*}, X^{\*}) & K(X^{\*},X) \\\ K(X, X^{\*}) & K(X, X)]) | ||
\end{equation} | ||
|
||
|
||
#### Noisy Measurements {#noisy-measurements} | ||
|
||
We can account for zero-mean noise by adding some noise to your covariance: | ||
|
||
\begin{equation} | ||
\mqty[\hat{y} \\\ y] \sim \mathcal{N} \qty(\mqty[m(X^{\*}) \\\ m(X)], \mqty[K(X^{\*}, X^{\*}) & K(X^{\*},X) \\\ K(X, X^{\*}) & K(X, X) + v I]) | ||
\end{equation} | ||
|
||
|
||
## Surrogate Optimization {#surrogate-optimization} | ||
|
||
|
||
### Prediction Based Exploration {#prediction-based-exploration} | ||
|
||
Given your existing points \\(D\\), evaluate \\(\mu\_{x|D}\\), and optimize for the next design point that has the smallest \\(\mu\_{x|D}\\). | ||
|
||
This is **all exploitation, no exploration**. | ||
|
||
|
||
### Error Based Exploration {#error-based-exploration} | ||
|
||
Use the 95% [confidence interval]({{< relref "KBhconfidence_interval.md" >}}) from the [Gaussian Process](#gaussian-process), find the areas with with the biggest gap and then lower those. | ||
|
||
This is \*all exploration, no exploitation. | ||
|
||
|
||
### Lower Confidence Bound Exploration {#lower-confidence-bound-exploration} | ||
|
||
tradeoff between exploration and exploitation. Try to minimize: | ||
|
||
\begin{equation} | ||
LB(x) = \hat{\mu}(x) - \alpha \hat{\Sigma}(x) | ||
\end{equation} | ||
|
||
try to minimize both the LOWER BOUND as well as the optimum. This is a probabilistic generalization of the [Shubert-Piyavskill Method]({{< relref "KBhsu_cs361_apr092024.md#shubert-piyavskill-method" >}})---and no [Lipschitz Constant]({{< relref "KBhuniqueness_and_existance.md#lipschitz-condition" >}}) needed! | ||
|
||
Reminder, though, these are **probabilistic bounds**---unlike [Shubert-Piyavskill Method]({{< relref "KBhsu_cs361_apr092024.md#shubert-piyavskill-method" >}}). | ||
|
||
|
||
### Probability of Improvement Exploration {#probability-of-improvement-exploration} | ||
|
||
We define "improvement" as: | ||
|
||
\begin{equation} | ||
I(y) = \begin{cases} | ||
y\_{\min} - y, \text{if}\ y < y\_{\min} \\\\ | ||
0, \text{otherwise} | ||
\end{cases} | ||
\end{equation} | ||
|
||
And then, you choose what to look at by: | ||
|
||
\begin{equation} | ||
P(y < y\_{\min}) = \int\_{\infty}^{y\_{\min}} \mathcal{N}(y | \hat{\mu}, \hat{\Sigma}) \dd{y} | ||
\end{equation} | ||
|
||
(i.e. we want to find points that are very possible to improve). | ||
|
||
You can also do this by the expected value of improvement |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,76 @@ | ||
+++ | ||
title = "SU-ENGR76 MAY072024" | ||
author = ["Houjun Liu"] | ||
draft = false | ||
+++ | ||
|
||
Welcome to Unit 2. | ||
|
||
|
||
## Fundamental Problem of Communication {#fundamental-problem-of-communication} | ||
|
||
"The fundamental task of communication is that of reproducing at one point either exactly or approximately a message selected at another point" | ||
|
||
Now: all communication signals are subject to some noise, so we need to design systems to responds to them. | ||
|
||
Most designs center around changing transmitter and receiver, sometimes you can change the channel but often not. | ||
|
||
|
||
## communication {#communication} | ||
|
||
|
||
### analog communication {#analog-communication} | ||
|
||
1. convert sound waves into continuous-time electrical signal \\(s(t)\\) | ||
2. apply \\(s(t)\\) directly to the voltage of my channel | ||
3. on the other end, decode \\(s'(t)\\), a noisy version of the original signal | ||
4. speak \\(s'(t)\\) | ||
|
||
|
||
### digital communication {#digital-communication} | ||
|
||
1. convert sound waves into continuous-time electrical signal \\(s(t)\\) | ||
2. sample \\(s(t)\\) at a known sampling rate | ||
3. quantize the results to a fixed number of levels, turning them into discrete symbols | ||
4. use [Huffman Coding]({{< relref "KBhhuffman_coding.md" >}}) to turn them into bits | ||
5. generate a continuous-time signal of voltage using the bits | ||
6. communicate this resulting signal over the cable | ||
7. get the noisy result signal and recovering from bits from it | ||
8. decode that by using interpolation + codebook | ||
9. speak data | ||
|
||
this format allows us to generalize all communication as taking on type (Bits => Bits), i.e. we only have to design Tx and Rx. This could be much more flexible than [Analog Communication]({{< relref "KBhanalog_vs_digital_signal.md#analog-communication" >}}). | ||
|
||
communicating bits, too, can allow for control of distortion because it bounds the codomain of the output signal into bits which one cloud filter out, unlike [Analog Communication]({{< relref "KBhanalog_vs_digital_signal.md#analog-communication" >}}). | ||
|
||
Tx and Rx maps **boolean [signal]({{< relref "KBhsu_engr76_apr162024.md#signal" >}})** and map it into something that can be transmitted across a channel, meaning it usually converts binary symbols into continuous time signal to be played back. | ||
|
||
|
||
## digital encoding {#digital-encoding} | ||
|
||
"how do we map a sequence of bits 0100100.... and map it to a continuous time signal \\(X(t)\\)?" | ||
|
||
|
||
### simplest digital encoding approcah {#simplest-digital-encoding-approcah} | ||
|
||
choose some voltage \\(V\\). Assign 1-bit voltage \\(V\\), assign 0-bit voltage \\(0\\), and simply play a voltage for a set amount of time \\(t\\) and move on to the next symbol for encoding. | ||
|
||
To decode, we sample midpoints, shifted forward by \\(\frac{T}{2}\\). | ||
|
||
Two parameters: | ||
|
||
- \\(V\\): the voltage for the active signal | ||
- \\(T\\): the time to communicate each symbol | ||
|
||
Each symbol \\(T\\), if held for a smaller amount of time, communicating would take a smaller amount of time. | ||
|
||
However, if too small, the bandwidth of the communication will increase (it will spike faster)---this is bad because... | ||
|
||
1. channels can be frequency-selective (i.e. too high frequencies may not have the same propagation schemes, such as a speaker/microphones) | ||
2. channels may have frequency restriction (FCC may give you a certain band, that is, you may only be licensed to communicate a band) | ||
3. you may not be able to sample fast enough | ||
|
||
|
||
#### High-Passing {#high-passing} | ||
|
||
because you are almost never allowed to just give [Baseband Signal]({{< relref "KBhsu_engr76_may022024.md#passband-signal" >}}), we also need to be able to communicate these bits with a [Bandwidth]({{< relref "KBhsu_engr76_apr252024.md#bandwidth" >}}) limit both below and above. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
+++ | ||
title = "SU-ENGR76 Unit 2 Index" | ||
author = ["Houjun Liu"] | ||
draft = false | ||
+++ | ||
|
||
Communication System Design! | ||
|
||
[Fundamental Problem of Communication]({{< relref "KBhsu_engr76_may072024.md#fundamental-problem-of-communication" >}}) | ||
|
||
- [communication]({{< relref "KBhsu_engr76_may072024.md#communication" >}}) | ||
- [Analog Communication]({{< relref "KBhanalog_vs_digital_signal.md#analog-communication" >}}) | ||
- [Digital Communication]({{< relref "KBhanalog_vs_digital_signal.md#digital-communication" >}}) | ||
- [digital encoding]({{< relref "KBhsu_engr76_may072024.md#digital-encoding" >}}) | ||
- [simplest digital encoding approcah]({{< relref "KBhsu_engr76_may072024.md#simplest-digital-encoding-approcah" >}}) |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.