Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add guidance on using profilers from within Jupyter notebooks. #24

Merged
merged 5 commits into from
Mar 22, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"id": "e80c86cb-d60a-4e45-a34c-191aab11acf0",
"metadata": {},
"outputs": [],
"source": [
"!pip install line_profiler\n",
"%load_ext line_profiler"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "de5e278a-6826-4f42-b1af-496f5aabf996",
"metadata": {},
"outputs": [],
"source": [
"def fizzbuzz(n):\n",
" for i in range(1, n + 1):\n",
" if i % 3 == 0 and i % 5 == 0:\n",
" print(\"FizzBuzz\")\n",
" elif i % 3 == 0:\n",
" print(\"Fizz\")\n",
" elif i % 5 == 0:\n",
" print(\"Buzz\")\n",
" else:\n",
" print(i)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c54f2642-617e-4a51-b3e8-1ff1ba0e8426",
"metadata": {},
"outputs": [],
"source": [
"%lprun -f fizzbuzz fizzbuzz(100)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"id": "83b45371-9cd9-4a8f-b0ff-ceb1bfc2ecc2",
"metadata": {},
"outputs": [],
"source": [
"import time\n",
"\n",
"def a_1():\n",
" for i in range(3):\n",
" b_1()\n",
" time.sleep(1)\n",
" b_2()\n",
" \n",
"def b_1():\n",
" c_1()\n",
" c_2()\n",
"\n",
"def b_2():\n",
" time.sleep(1)\n",
" \n",
"def c_1():\n",
" time.sleep(0.5)\n",
"\n",
"def c_2():\n",
" time.sleep(0.3)\n",
" d_1()\n",
"\n",
"def d_1():\n",
" time.sleep(0.1)\n",
" "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0fc38195-3dc4-4a7c-82f6-1167f54d3182",
"metadata": {},
"outputs": [],
"source": [
"!pip install snakeviz"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c47affa0-23cc-43f8-a4f8-e73ab2f44afd",
"metadata": {},
"outputs": [],
"source": [
"%load_ext snakeviz"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "58040218-8578-47fc-9a18-476f4b27d25a",
"metadata": {},
"outputs": [],
"source": [
"%snakeviz a_1()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
43 changes: 43 additions & 0 deletions episodes/profiling-functions.md
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,42 @@ By clicking a box within the diagram, it will "zoom" making the selected box the

As you hover each box, information to the left of the diagram updates specifying the location of the method and for how long it ran.

::::::::::::::::::::::::::::::::::::: callout

## snakeviz Inside Notebooks

If you're more familiar with writing Python inside Jupyter notebooks, you might not feel comfortable using the command line for profiling. Instead, `snakeviz` can be called directly from inside notebooks, and it will automatically call `cProfile` for you.
Robadob marked this conversation as resolved.
Show resolved Hide resolved

First `snakeviz` must be installed and it's extension loaded.

```py
!pip install snakeviz
%load_ext snakeviz
```

Following this, you can either call `%snakeviz` to profile a function defined earlier in the notebook.

```py
%snakeviz my_function()
```

Or, you can create a `%%snakeviz` cell, to profile the python executed within it.

```py
%%snakeviz

def my_function():
print("Hello World!")

my_function()
```

In both cases, the full `snakeviz` profile visualisation will appear as an output within the notebook!

*You may wish to right click the top of the output, and select "Disable Scrolling for Outputs" to expand it's box if it begins too small.*

:::::::::::::::::::::::::::::::::::::::::::::

## Worked Example

:::::::::::::::::::::::::::::::::: instructor
Expand Down Expand Up @@ -310,6 +346,13 @@ The sunburst visualisation displays less text on the boxes, so it can be harder

:::::::::::::::::::::::::::::::::::::::::::::

If you followed along inside a notebook it might look like this:
Robadob marked this conversation as resolved.
Show resolved Hide resolved

![The worked example inside a notebook.](episodes/fig/snakeviz-worked-example-notebook.png){alt="A Jupyter notebook showing the worked example profiled with snakeviz." width=80%}

Because notebooks operate by creating temporary Python files, the filename (shown `1378276351.py` above) and line numbers are not too useful, but function names should still be helpful.
Robadob marked this conversation as resolved.
Show resolved Hide resolved


## Exercises

The following exercises allow you to review your understanding of what has been covered in this episode.
Expand Down
37 changes: 36 additions & 1 deletion episodes/profiling-lines.md
Original file line number Diff line number Diff line change
Expand Up @@ -238,7 +238,42 @@ Therefore it can be seen in this example, how the time spent executing each line

The `-r` argument passed to `kernprof` (or `line_profiler`) enables rich output, if you run the profile locally it should look similar to this. *This requires the optional package `rich`, it will have been installed if `[all]` was specified when installing `line_profiler` with `pip`.*

![Rich (highlighted) console output provided by `line_profiler` for the above FizzBuzz profile code.](episodes/fig/line_profiler-worked-example.png){alt="A screenshot of the `line_profiler` output from the previous code block, where the code within the line contents column has basic highlighting."}
![Rich (highlighted) console output provided by `line_profiler` for the above FizzBuzz profile code.](episodes/fig/line_profiler-worked-example-rich.png){alt="A screenshot of the `line_profiler` output from the previous code block, where the code within the line contents column has basic highlighting."}

:::::::::::::::::::::::::::::::::::::::::::::

::::::::::::::::::::::::::::::::::::: callout

## line_profiler Inside Notebooks

If you're more familiar with writing Python inside Jupyter notebooks, you might not feel comfortable using the command line for profiling. Instead, `line_profiler` can be called directly from inside notebooks. It is still necessary for the code to be profiled to be placed within a function.
Robadob marked this conversation as resolved.
Show resolved Hide resolved

First `line_profiler` must be installed and it's extension loaded.

```py
!pip install line_profiler
%load_ext line_profiler
```

Following this, you call `line_profiler` with `%lprun`.

```py
%lprun -f profiled_function_name entry_function_call()
```

The functions to be line profiled are specified with `-f <function name>`, this is repeated for each individual function that you would otherwise apply the `@profile` decorator to.

This is followed by calling the function which runs the full code to be profiled.

For the above fizzbuzz example it would be:

```py
%lprun -f fizzbuzz fizzbuzz(100)
```

This will then create an output cell with any output from the profiled code, followed by the standard output from `line_profiler`. *It is not currently possible to get the rich/coloured output from `line_profiler` within notebooks.*

![Output provided by `line_profiler` inside a Juypter notebook for the above FizzBuzz profile code.](episodes/fig/line_profiler-worked-example-notebook.png){alt="A screenshot of the line_profiler output from the previous code block inside a Jupyter notebook."}

:::::::::::::::::::::::::::::::::::::::::::::

Expand Down
Loading