Jupyter Notebooks Advanced Features¶
PYNQ notebook front end allows interactive coding, output visualizations and documentation using text, equations, images, video and other rich media.
Code, analysis, debug, documentation and demos are all alive, editable and connected in the Notebooks.
## Contents
- Live, Interactive Cell for Python Coding
- Rich Output Media
- Interactive Plots and Visualization
- Notebooks are not just for Python
Live, Interactive Python Coding¶
Guess that number game¶
Run the cell to play
Cell can be run by selecting the cell and pressing Shift+Enter
[1]:
import random
the_number = random.randint(0, 10)
guess = -1
name = input('Player what is your name? ')
while guess != the_number:
guess_text = input('Guess a number between 0 and 10: ')
guess = int(guess_text)
if guess < the_number:
print(f'Sorry {name}, your guess of {guess} was too LOW.\n')
elif guess > the_number:
print(f'Sorry {name}, your guess of {guess} was too HIGH.\n')
else:
print(f'Excellent work {name}, you won, it was {guess}!\n')
print('Done')
Player what is your name? User
Guess a number between 0 and 10: 5
Sorry User, your guess of 5 was too LOW.
Guess a number between 0 and 10: 8
Sorry User, your guess of 8 was too LOW.
Guess a number between 0 and 10: 9
Sorry User, your guess of 9 was too LOW.
Guess a number between 0 and 10: 10
Excellent work User, you won, it was 10!
Done
Generate Fibonacci numbers¶
[2]:
def generate_fibonacci_list(limit, output=False):
nums = []
current, ne_xt = 0, 1
while current < limit:
current, ne_xt = ne_xt, ne_xt + current
nums.append(current)
if output == True:
print(f'{len(nums[:-1])} Fibonacci numbers below the number '
f'{limit} are:\n{nums[:-1]}')
return nums[:-1]
limit = 1000
fib = generate_fibonacci_list(limit, True)
16 Fibonacci numbers below the number 1000 are:
[1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987]
Plotting Fibonacci numbers¶
Plotting is done using the matplotlib library
[3]:
%matplotlib inline
import matplotlib.pyplot as plt
from ipywidgets import *
limit = 1000000
fib = generate_fibonacci_list(limit)
plt.plot(fib)
plt.plot(range(len(fib)), fib, 'ro')
plt.show()
Interactive input and output analysis¶
Input and output interaction can be achieved using Ipython widgets
[ ]:
%matplotlib inline
import matplotlib.pyplot as plt
from ipywidgets import *
def update(limit, print_output):
i = generate_fibonacci_list(limit, print_output)
plt.plot(range(len(i)), i)
plt.plot(range(len(i)), i, 'ro')
plt.show()
limit=widgets.IntSlider(min=10,max=1000000,step=1,value=10)
interact(update, limit=limit, print_output=False);
Interactive debug¶
Uses ``set_trace`` from the Ipython debugger library
Type ‘h’ in debug prompt for the debug commands list and ‘q’ to exit
[ ]:
from IPython.core.debugger import set_trace
def debug_fibonacci_list(limit):
nums = []
current, ne_xt = 0, 1
while current < limit:
if current > 1000:
set_trace()
current, ne_xt = ne_xt, ne_xt + current
nums.append(current)
print(f'The fibonacci numbers below the number {limit} are:\n{nums[:-1]}')
debug_fibonacci_list(10000)
Rich Output Media¶
Display images¶
Images can be displayed using combination of HTML, Markdown, PNG, JPG, etc.
Image below is displayed in a markdown cell which is rendered at startup.
Render SVG images¶
SVG image is rendered in a code cell using Ipython display library.
[4]:
from IPython.display import SVG
SVG(filename='images/python.svg')
[4]:
Audio Playback¶
IPython.display.Audio lets you play audio directly in the notebook
[5]:
import numpy as np
from IPython.display import Audio
framerate = 44100
t = np.linspace(0,5,framerate*5)
data = np.sin(2*np.pi*220*t**2)
Audio(data,rate=framerate)
[5]:
Add Video¶
IPython.display.YouTubeVideo lets you play Youtube video directly in the notebook. Library support is available to play Vimeo and local videos as well
[6]:
from IPython.display import YouTubeVideo
YouTubeVideo('ooOLl4_H-IE')
[6]:
Video Link with image display
Add webpages as Interactive Frames¶
Embed an entire page from another site in an iframe; for example this is the PYNQ documentation page on readthedocs
[7]:
from IPython.display import IFrame
IFrame('https://pynq.readthedocs.io/en/latest/getting_started.html',
width='100%', height=500)
[7]:
Render Latex¶
Display of mathematical expressions typeset in LaTeX for documentation.
[8]:
%%latex
\begin{align} P(Y=i|x, W,b) = softmax_i(W x + b)= \frac {e^{W_i x + b_i}}
{\sum_j e^{W_j x + b_j}}\end{align}
\begin{align} P(Y=i|x, W,b) = softmax_i(W x + b)= \frac {e^{W_i x + b_i}}
{\sum_j e^{W_j x + b_j}}\end{align}
Interactive Plots and Visualization¶
Plotting and Visualization can be achieved using various available python libraries such as Matplotlib, Bokeh, Seaborn, etc.
Below is shown a Iframe of the Matplotlib website. Navigate to ‘gallery’ and choose a plot to run in the notebook
[9]:
from IPython.display import IFrame
IFrame('https://matplotlib.org/gallery/index.html', width='100%', height=500)
[9]:
Matplotlib¶
Below we run the code available under examples –> Matplotlib API –> Radar_chart in the above webpage
[10]:
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.path import Path
from matplotlib.spines import Spine
from matplotlib.projections.polar import PolarAxes
from matplotlib.projections import register_projection
def radar_factory(num_vars, frame='circle'):
"""Create a radar chart with `num_vars` axes.
This function creates a RadarAxes projection and registers it.
Parameters
----------
num_vars : int
Number of variables for radar chart.
frame : {'circle' | 'polygon'}
Shape of frame surrounding axes.
"""
# calculate evenly-spaced axis angles
theta = np.linspace(0, 2*np.pi, num_vars, endpoint=False)
def draw_poly_patch(self):
# rotate theta such that the first axis is at the top
verts = unit_poly_verts(theta + np.pi / 2)
return plt.Polygon(verts, closed=True, edgecolor='k')
def draw_circle_patch(self):
# unit circle centered on (0.5, 0.5)
return plt.Circle((0.5, 0.5), 0.5)
patch_dict = {'polygon': draw_poly_patch, 'circle': draw_circle_patch}
if frame not in patch_dict:
raise ValueError('unknown value for `frame`: %s' % frame)
class RadarAxes(PolarAxes):
name = 'radar'
# use 1 line segment to connect specified points
RESOLUTION = 1
# define draw_frame method
draw_patch = patch_dict[frame]
def __init__(self, *args, **kwargs):
super(RadarAxes, self).__init__(*args, **kwargs)
# rotate plot such that the first axis is at the top
self.set_theta_zero_location('N')
def fill(self, *args, **kwargs):
"""Override fill so that line is closed by default"""
closed = kwargs.pop('closed', True)
return super(RadarAxes, self).fill(closed=closed, *args, **kwargs)
def plot(self, *args, **kwargs):
"""Override plot so that line is closed by default"""
lines = super(RadarAxes, self).plot(*args, **kwargs)
for line in lines:
self._close_line(line)
def _close_line(self, line):
x, y = line.get_data()
# FIXME: markers at x[0], y[0] get doubled-up
if x[0] != x[-1]:
x = np.concatenate((x, [x[0]]))
y = np.concatenate((y, [y[0]]))
line.set_data(x, y)
def set_varlabels(self, labels):
self.set_thetagrids(np.degrees(theta), labels)
def _gen_axes_patch(self):
return self.draw_patch()
def _gen_axes_spines(self):
if frame == 'circle':
return PolarAxes._gen_axes_spines(self)
# The following is a hack to get the spines (i.e. the axes frame)
# to draw correctly for a polygon frame.
# spine_type must be 'left', 'right', 'top', 'bottom', or `circle`.
spine_type = 'circle'
verts = unit_poly_verts(theta + np.pi / 2)
# close off polygon by repeating first vertex
verts.append(verts[0])
path = Path(verts)
spine = Spine(self, spine_type, path)
spine.set_transform(self.transAxes)
return {'polar': spine}
register_projection(RadarAxes)
return theta
def unit_poly_verts(theta):
"""Return vertices of polygon for subplot axes.
This polygon is circumscribed by a unit circle centered at (0.5, 0.5)
"""
x0, y0, r = [0.5] * 3
verts = [(r*np.cos(t) + x0, r*np.sin(t) + y0) for t in theta]
return verts
def example_data():
# The following data is from the Denver Aerosol Sources and Health study.
# See doi:10.1016/j.atmosenv.2008.12.017
#
# The data are pollution source profile estimates for five modeled
# pollution sources (e.g., cars, wood-burning, etc) that emit 7-9 chemical
# species. The radar charts are experimented with here to see if we can
# nicely visualize how the modeled source profiles change across four
# scenarios:
# 1) No gas-phase species present, just seven particulate counts on
# Sulfate
# Nitrate
# Elemental Carbon (EC)
# Organic Carbon fraction 1 (OC)
# Organic Carbon fraction 2 (OC2)
# Organic Carbon fraction 3 (OC3)
# Pyrolized Organic Carbon (OP)
# 2)Inclusion of gas-phase specie carbon monoxide (CO)
# 3)Inclusion of gas-phase specie ozone (O3).
# 4)Inclusion of both gas-phase species is present...
data = [
['Sulfate', 'Nitrate', 'EC', 'OC1', 'OC2', 'OC3', 'OP', 'CO', 'O3'],
('Basecase', [
[0.88, 0.01, 0.03, 0.03, 0.00, 0.06, 0.01, 0.00, 0.00],
[0.07, 0.95, 0.04, 0.05, 0.00, 0.02, 0.01, 0.00, 0.00],
[0.01, 0.02, 0.85, 0.19, 0.05, 0.10, 0.00, 0.00, 0.00],
[0.02, 0.01, 0.07, 0.01, 0.21, 0.12, 0.98, 0.00, 0.00],
[0.01, 0.01, 0.02, 0.71, 0.74, 0.70, 0.00, 0.00, 0.00]]),
('With CO', [
[0.88, 0.02, 0.02, 0.02, 0.00, 0.05, 0.00, 0.05, 0.00],
[0.08, 0.94, 0.04, 0.02, 0.00, 0.01, 0.12, 0.04, 0.00],
[0.01, 0.01, 0.79, 0.10, 0.00, 0.05, 0.00, 0.31, 0.00],
[0.00, 0.02, 0.03, 0.38, 0.31, 0.31, 0.00, 0.59, 0.00],
[0.02, 0.02, 0.11, 0.47, 0.69, 0.58, 0.88, 0.00, 0.00]]),
('With O3', [
[0.89, 0.01, 0.07, 0.00, 0.00, 0.05, 0.00, 0.00, 0.03],
[0.07, 0.95, 0.05, 0.04, 0.00, 0.02, 0.12, 0.00, 0.00],
[0.01, 0.02, 0.86, 0.27, 0.16, 0.19, 0.00, 0.00, 0.00],
[0.01, 0.03, 0.00, 0.32, 0.29, 0.27, 0.00, 0.00, 0.95],
[0.02, 0.00, 0.03, 0.37, 0.56, 0.47, 0.87, 0.00, 0.00]]),
('CO & O3', [
[0.87, 0.01, 0.08, 0.00, 0.00, 0.04, 0.00, 0.00, 0.01],
[0.09, 0.95, 0.02, 0.03, 0.00, 0.01, 0.13, 0.06, 0.00],
[0.01, 0.02, 0.71, 0.24, 0.13, 0.16, 0.00, 0.50, 0.00],
[0.01, 0.03, 0.00, 0.28, 0.24, 0.23, 0.00, 0.44, 0.88],
[0.02, 0.00, 0.18, 0.45, 0.64, 0.55, 0.86, 0.00, 0.16]])
]
return data
if __name__ == '__main__':
N = 9
theta = radar_factory(N, frame='polygon')
data = example_data()
spoke_labels = data.pop(0)
fig, axes = plt.subplots(figsize=(9, 9), nrows=2, ncols=2,
subplot_kw=dict(projection='radar'))
fig.subplots_adjust(wspace=0.25, hspace=0.20, top=0.85, bottom=0.05)
colors = ['b', 'r', 'g', 'm', 'y']
# Plot the four cases from the example data on separate axes
for ax, (title, case_data) in zip(axes.flatten(), data):
ax.set_rgrids([0.2, 0.4, 0.6, 0.8])
ax.set_title(title, weight='bold', size='medium', position=(0.5, 1.1),
horizontalalignment='center', verticalalignment='center')
for d, color in zip(case_data, colors):
ax.plot(theta, d, color=color)
ax.fill(theta, d, facecolor=color, alpha=0.25)
ax.set_varlabels(spoke_labels)
# add legend relative to top-left plot
ax = axes[0, 0]
labels = ('Factor 1', 'Factor 2', 'Factor 3', 'Factor 4', 'Factor 5')
legend = ax.legend(labels, loc=(0.9, .95),
labelspacing=0.1, fontsize='small')
fig.text(0.5, 0.965, '5-Factor Solution Profiles Across Four Scenarios',
horizontalalignment='center', color='black', weight='bold',
size='large')
plt.show()
Notebooks are not just for Python¶
Access to linux shell commands¶
Starting a code cell with a bang character, e.g.
!, instructs jupyter to treat the code on that line as an OS shell commandSystem Information¶
[11]:
!cat /proc/cpuinfo
processor : 0
model name : ARMv7 Processor rev 0 (v7l)
BogoMIPS : 650.00
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpd32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x3
CPU part : 0xc09
CPU revision : 0
processor : 1
model name : ARMv7 Processor rev 0 (v7l)
BogoMIPS : 650.00
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpd32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x3
CPU part : 0xc09
CPU revision : 0
Hardware : Xilinx Zynq Platform
Revision : 0003
Serial : 0000000000000000
Verify Linux Version¶
[12]:
!cat /etc/os-release | grep VERSION
VERSION="16.04 LTS (Xenial Xerus)"
VERSION_ID="16.04"
CPU speed calculation made by the Linux kernel¶
[13]:
!head -5 /proc/cpuinfo | grep "BogoMIPS"
BogoMIPS : 650.00
Available DRAM¶
[14]:
!cat /proc/meminfo | grep 'Mem*'
MemTotal: 507892 kB
MemFree: 101684 kB
MemAvailable: 357020 kB
Network connection¶
[15]:
!ifconfig
eth0 Link encap:Ethernet HWaddr 00:18:3e:02:6d:cb
inet addr:172.19.73.161 Bcast:172.19.75.255 Mask:255.255.252.0
inet6 addr: fe80::218:3eff:fe02:6dcb/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:22262 errors:0 dropped:0 overruns:0 frame:0
TX packets:9274 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3972408 (3.9 MB) TX bytes:11258468 (11.2 MB)
Interrupt:27 Base address:0xb000
eth0:1 Link encap:Ethernet HWaddr 00:18:3e:02:6d:cb
inet addr:192.168.2.99 Bcast:192.168.2.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:27 Base address:0xb000
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:4558 errors:0 dropped:0 overruns:0 frame:0
TX packets:4558 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:3876932 (3.8 MB) TX bytes:3876932 (3.8 MB)
Directory Information¶
[16]:
!pwd
!echo --------------------------------------------
!ls -C --color
/home/xilinx/jupyter_notebooks/getting_started
--------------------------------------------
1_jupyter_notebooks.ipynb 4_base_overlay_iop.ipynb images
2_python_environment.ipynb 5_base_overlay_video.ipynb
3_jupyter_notebooks_advanced_features.ipynb 6_base_overlay_audio.ipynb
Shell commands in python code¶
[17]:
files = !ls | head -3
print(files)
['1_jupyter_notebooks.ipynb', '2_python_environment.ipynb', '3_jupyter_notebooks_advanced_features.ipynb']
Python variables in shell commands¶
By enclosing a Python expression within {}, i.e. curly braces, we can substitute it into shell commands
[18]:
shell_nbs = '*.ipynb | grep "ipynb"'
!ls {shell_nbs}
1_jupyter_notebooks.ipynb
2_python_environment.ipynb
3_jupyter_notebooks_advanced_features.ipynb
4_base_overlay_iop.ipynb
5_base_overlay_video.ipynb
6_base_overlay_audio.ipynb
Magics¶
IPython has a set of predefined ‘magic functions’ that you can call with a command line style syntax. There are two kinds of magics, line-oriented and cell-oriented. Line magics are prefixed with the % character and work much like OS command-line calls: they get as an argument the rest of the line, where arguments are passed without parentheses or quotes. Cell magics are prefixed with a double %%, and they are functions that get as an argument not only the rest of the line, but also the lines below it in a separate argument.
To learn more about the IPython magics, simple type %magic in a separate cell
Below is a list of available magics
[19]:
%lsmagic
[19]:
Available line magics:
%alias %alias_magic %autocall %automagic %autosave %bookmark %cat %cd %clear %colors %config %connect_info %cp %debug %dhist %dirs %doctest_mode %ed %edit %env %gui %hist %history %killbgscripts %ldir %less %lf %lk %ll %load %load_ext %loadpy %logoff %logon %logstart %logstate %logstop %ls %lsmagic %lx %macro %magic %man %matplotlib %mkdir %more %mv %notebook %page %pastebin %pdb %pdef %pdoc %pfile %pinfo %pinfo2 %popd %pprint %precision %profile %prun %psearch %psource %pushd %pwd %pycat %pylab %qtconsole %quickref %recall %rehashx %reload_ext %rep %rerun %reset %reset_selective %rm %rmdir %run %save %sc %set_env %store %sx %system %tb %time %timeit %unalias %unload_ext %who %who_ls %whos %xdel %xmode
Available cell magics:
%%! %%HTML %%SVG %%bash %%capture %%debug %%file %%html %%javascript %%js %%latex %%markdown %%perl %%prun %%pypy %%python %%python2 %%python3 %%ruby %%script %%sh %%svg %%sx %%system %%time %%timeit %%writefile
Automagic is ON, % prefix IS NOT needed for line magics.
Timing code using magics¶
The following examples show how to call the built-in
%time magic%time times the execution of a single statementReference: The next two code cells are excerpted from the Python Data Science Handbook by Jake VanderPlas
Time the sorting on an unsorted list¶
A list of 100000 random numbers is sorted and stored in a variable ‘L’
[20]:
import random
L = [random.random() for _ in range(100000)]
%time L.sort()
CPU times: user 550 ms, sys: 0 ns, total: 550 ms
Wall time: 550 ms
Coding other languages¶
If you want to, you can combine code from multiple kernels into one notebook.
Just use IPython Magics with the name of your kernel at the start of each cell that you want to use that Kernel for:
%%bash
%%HTML
%%python2
%%python3
%%ruby
%%perl
[22]:
%%bash
factorial()
{
if [ "$1" -gt "1" ]
then
i=`expr $1 - 1`
j=`factorial $i`
k=`expr $1 \* $j`
echo $k
else
echo 1
fi
}
input=5
val=$(factorial $input)
echo "Factorial of $input is : "$val
Factorial of 5 is : 120