ML with Python (numpy, pandas), my tools + code snippets (Part 1)

Contents:

1. Jupyter notebooks

2. Path to packages
3. PYZO editor for .py
4. Basics of numpy arrays
5. Numpy.random
6. Array operations
7. Boolean indexing
8. Fancy indexing
9. Transposition

1. Jupyter notebooks

Jupyter is part of Anaconda pack of packages, but I prefer simple commands:

$ python3 -m pip install jupyter

Jupyter notebooks run a web server that can be queried either directly from web console, or via terminal. To find what localhost:port is used by a running server:

$ jupyter notebook list

To stop a server:

$ jupyter notebook stop 8888 

To open a notebook file (with .ipynb extension), in terminal point to a notebook directory and run:

$ jupyter notebook

This starts a server, then opens a notebook's home directory in a browser. Click a notebook to run it. Keep terminal shell open and running a server as an attached process. To stop a server use Ctrl-C, or just close a terminal.

Shift-Return to run selected cell and advance to the next.

2. Paths to Python components and packages

>>> import sys  

>>> print('\n'.join(sys.path))

3. PYZO editor for .py

PYZO works for me. It is lightweight, has debugging options, and is configurable.
Supports cells, blocks of code that can be executed with a simple shortcut (Cmd-Return). Define cells by delimiting with #$$ optional commentary lines.

Some shortcuts (original and my own custom):

---------- EDITOR: 

Opt-Tab    - select previous file 

F1         - focus to shell panel 

F2         - focus to file editor 

Cmd-/      - comment selection  

Cmd-Opt-/  - uncomment  

---------- RUN:  

Cmd-R      - run file as a script (in console restarts interpreter)  

Cmd-Return - run cell with cursor (cells are delimited by #%%)   

Opt-Return - run selection  

---------- DEBUGGER:  

Cmd-B      - toggle breakpoint  

F6         - step over  

F7         - step in  

F8         - step out  

Ctrl-Cmd-Y - continue  

Ctrl-Cmd-. - stop debugging 

4. Basics of numpy arrays

Many ways to init arrays. There are two main options - shape and dtype.

>>> x = np.ndarray(shape=(2, 2), dtype=np.int8, order='C') 

>>> print(x) 

[[1 0] [1 1]]

Internal buffer is linear. Shape can be changed easily without reallocation if total element count remains the same:

>>> x.shape = (1,4) 

>>> print(x) 

[[1 0 1 1]]

Other array constructors:

>>> print(np.array((1, 2, 3))) 

[1 2 3] 

>>> print(np.zeros((2, 3))) 

[[0. 0. 0.] 

 [0. 0. 0.]] 

>>> print(np.empty((2,))) 

[7.74860419e-304 7.74860419e-304]

Array construction using list comprehensions. Note, unspecified dimension size of -1 infers the required element count in that dimension:

>>> x = np.array([(x, y) for x in [1,2,3] for y in [3,1,4] if x != y]) 

>>> print(x) 

[[1 3] 

 [1 4] 

 [2 3] 

 [2 1] 

 [2 4] 

 [3 1] 

 [3 4]] 

>>> x.shape = (2, -1) 

>>> print(x) 

[[1 3 1 4 2 3 2] 

 [1 2 4 3 1 3 4]] 

>>> x.shape = (-1) 

>>> print(x) 

[1 3 1 4 2 3 2 1 2 4 3 1 3 4]

Numpy arrays in list comprehension expressions:

>>> y = np.array([e for e in x if not e % 2]) 

>>> print(y) 

[4 2 2 2 4 4]

Range into 2D array:

>>> x = np.arange(15).reshape(5, -1).T 

>>> print(x) 

[[ 0 3 6 9 12] 

 [ 1 4 7 10 13] 

 [ 2 5 8 11 14]]

Numpy array operations perform like native C memory access ops (or parallelized vectors in GPU). Therefore much faster than Python's list comprehensions:

#%% Timing init 

import time 

x1 = np.arange(1000000) 

x2 = np.arange(1000000) 

#%% Timing native 

t0 = time.time() 

for _ in range(10): 

    x1 **= 2 

t1 = time.time() 

print("x1 time = ", t1 - t0) 

#%% Timing list comprehension 

t0 = time.time() 

for _ in range(10): 

    x2 = np.array([x ** 2 for x in x1]) 

t1 = time.time() 

print("x2 time = ", t1 - t0) 

#%%

---------------------------------------- 

x1 time = 0.015141725540161133 

x2 time = 4.744795083999634

5. Numpy.random

To generate standard numpy arrays filled with random numbers:

np.random.rand(d0,..dn)     - with each value uniformly in range [0, 1). 

np.random.randn(d0,..dn)    - with each value in Gaussian distribution, where mean = 0, variance = 1 (sigma squared). 

sigma * np.random.randn(d0,..dn) + mean    - a full normal distribution.

There are lots of other useful functions, i.e. np.random.choice(...).

6. Array operations

Slice of an array returns a data structure that defines subrange and points at the original array. Note that the second index (i.e. in [3:10]) points beyond the last element:

x1 = np.arange(12) 

print(x1) 

x1_slice = x1[3:10] 

print('x1_slice      = ', x1_slice) 

print('x1_slice[0:3] = ', x1_slice[0:3]) 

x1_slice[0:3] = 100 

print(x1) 

---------------------------------------- 

[ 0 1 2 3 4 5 6 7 8 9 10 11] 

x1_slice      = [3 4 5 6 7 8 9] 

x1_slice[0:3] = [3 4 5] 

[ 0 1 2 100 100 100 6 7 8 9 10 11]

Multi-dim array slices:

x1 = np.arange(12).reshape(4,-1) 

print(x1) 

x1_slice = x1[2:4] 

print(x1_slice) 

print(x1_slice[0][1:3]) 

---------------------------------------- 

[[  0  1  2] 

 [  3  4  5] 

 [  6  7  8] 

 [  9 10 11]]

[[ 6 7 8] 

 [ 9 10 11]] 

[7 8]

Slicing across multiple dimensions:

print(x1[:2, 1:3]) 

x1[:, 1:2] = 100 

print(x1) 

---------------------------------------- 

[[1 2] 

 [4 5]] 

[[  0 100  2]  

 [  3 100  5] 

 [  6 100  8] 

 [  9 100 11]]

7. Boolean indexing

Taken and modified from the pydata-book.

A boolean operation with array will return an array of boolean element-wise results. Boolean array when used as index will pick array elements where boolean component is True.

Boolean array size must be equal to data array size at the index:

names = np.array(['Bob', 'Joe', 'Will', 'Bob', 'Joe']) 

data = np.random.randint(1, 10, (5, 4)) 

print(names) 

print(data) 

mask = (names == 'Bob') | (names == 'Will') 

print(mask) 

print(data[mask]) 

data[mask, 1:3] = 100. 

print(data) 

data[data > 5] = 0 

print(data) 

----------------------------------------  

['Bob' 'Joe' 'Will' 'Bob' 'Joe'] 

[[1 4 3 6] 

 [6 1 3 3] 

 [7 1 6 1] 

 [3 3 9 4] 

 [4 7 8 9]] 

[ True False True True False] 

[[1 4 3 6] 

 [7 1 6 1] 

 [3 3 9 4]] 

[[  1 100 100   6] 

 [  6   1   3   3] 

 [  7 100 100   1] 

 [  3 100 100   4] 

 [  4   7   8   9]] 

[[1 0 0 0] 

 [0 1 3 3] 

 [0 0 0 1] 

 [3 0 0 4] 

 [4 0 0 0]]

8. Fancy indexing

Passing list of numbers into an indexer will treat it as an index picker. Allows assembling a new array from a combination of elements of another array:

x1 = np.arange(20).reshape((5, 4)) 

print(x1) 

x2 = x1[[1, 4, 2, 2], [0, 3, 1, 2]] 

print(x2) 

x1[[1, 4, 2, 2], [0, 3, 1, 2]] = 100 

print(x1) 

----------------------------------------   

[[ 0  1  2  3] 

 [ 4  5  6  7] 

 [ 8  9 10 11] 

 [12 13 14 15] 

 [16 17 18 19]] 

[ 4 19 9 10] 

[[  0   1   2   3] 

 [100   5   6   7] 

 [  8 100 100  11] 

 [ 12  13  14  15] 

 [ 16  17  18 100]]

9. Transposition

Transposed matrix is pointing at the same data (no copying takes place):

x1 = np.arange(15).reshape((3, 5)) 

print(x1) 

x2 = x1.T 

print(x2) 

x2[2] = 100 

print(x1) 

----------------------------------------    

[[ 0  1  2  3  4] 

 [ 5  6  7  8  9] 

 [10 11 12 13 14]] 

[[ 0 5 10] 

 [ 1 6 11] 

 [ 2 7 12] 

 [ 3 8 13] 

 [ 4 9 14]] 

[[  0  1 100  3  4] 

 [  5  6 100  8  9] 

 [ 10 11 100 13 14]]

References:

1. https://github.com/wesm/pydata-book
2. https://leemendelowitz.github.io/blog/how-does-python-find-packages.html

3. https://docs.python.org/3/installing/index.html

4. http://nbviewer.jupyter.org/github/pydata/pydata-book/blob/2nd-edition/ch02.ipynb#
5. https://docs.python.org/3/tutorial/datastructures.html#tuples-and-sequences

Microsoft's Imagine Cup Challenge - Team Heuristic Clinic

A month ago I prepared a quick team and a project for consideration at Microsoft's Imagine Cup. It was an intriguing experience to document details, to present in a video, and to polish a previously developed prototype - a wireless motion/gesture capturing app and a set of sensors.
I understood there was a slim chance to actually win the Cup, so I instead focused on setting up this project for possible future applications elsewhere.

The team is called Heuristic Clinic, the project is Applied Neural Diagnosis. Brief description can be found here: Heuristic Clinic, and the resulting certificate can be viewed here.
They were kind to offer $500 worth of Azure cloud, but I had to decline because the biggest expense is funding client-side development. Appreciated.

2018 Big Idea Challenge

Top 50 - 2018 Big Idea Challenge

This Certificate is Presented To:

alexey l

Team Heuristic Clinic

Congratulations and thank you for your hard work and dedication as a competitor in Imagine Cup 2018. You are now part of an elite international community of students who have shown remarkable creativity and innovation to push technology forward.

Pablo Veramendi
Imagine Cup Competition Manager
Microsoft Corporation

Martian sandstorm

Now a fully interactive volumetric sand storm, probably somewhere on Mars.
This marks an intermediate milestone following months of research into volume rendering. It uses finite differences solver for a fluid dynamics equation, some additional research into graphics pipeline on a Mac and iOS.
Note the shadows, they depend on volume density, and also note the unbounded nature of volumes. Other methods would produce visible banding artifacts.

Unexpected volumetric clouds

I was playing with code that I am testing to produce fluid dynamic effects, and I came across a weird cloud generating setup.
The advection part is failing at some point here and causes gas density to freeze in some regions of the map. It remains static and appears as a cloud, and I think this can be further improved to produce realistic looking real cloud. What's missing is a better phase function, and shading based on anisotropic scattering of different wavelenths.

Fun with self-shadowing volumetrics

As a preparation to a full-fledged interactive volume rendering, here is the first functional volumetric composition. It uses a few techniques to achieve a self-shadowing appearance, as you would expect from a real-life volume of transparent matter.

Volumetric hints.

A few volume rendering hints.

1. Cannot discard writing into render-targets selectively in an MRT frame-buffer. This arises, for example, when two targets are used to store two custom denormalized depth buffers. A frame-buffer has a depth attachment as well, but it is used in a regular sense of a depth buffer.

2. Volume rendering can use analytic volume buffer or a set of depth textures. To limit volume to a custom 3D shape requires preliminary step to render that shape into two depth attachments, one with inverted normals and inverted depth test:

// Enable depth test, inverted, front to back.
glDepthMask(GL_TRUE);
glEnable(GL_DEPTH_TEST);
glDepthFunc(GL_GREATER);

// Render only back faces.
glEnable(GL_CULL_FACE);
glCullFace(GL_FRONT);

// Clear the buffers.
// Note that the most distant shape will overwrite all others.
glClearDepth(0.0);
glClearColor(0.0f, 0.0f, 0.0f, 0.0f);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

//... Render the shape of the volume, i.e. a cloud.

// Enable depth test, back to front.
glDepthMask(GL_TRUE);
glEnable(GL_DEPTH_TEST);
glDepthFunc(GL_LESS);

// Render only front faces.
glEnable(GL_CULL_FACE);
glCullFace(GL_BACK);

// Clear the buffers.
glClearDepth(1.0);
glClearColor(0.0f, 0.0f, 0.0f, 0.0f);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

//... Render the same shape of the volume.

3. In a fragment shader calculate the distance to a rasterized point. Do it in fragment instead of in a vertex, hoping for speedup due to hardware interpolation. If two vertices spread out evenly in a view space, their depths will be equal and much bigger than the distance to a front clipping plane. This results in artifacts when moving towards a volume and crossing it.

4. A depth buffer attachment can now be shared between with other frame-buffers. It works for simple convex shapes as volumes that can be rendered from inside and outside. The real work is to be done elsewhere to modulate this shape with other detail, i.e. noise.

Volumetric rendering with custom volume shapes

It took me a week to figure out a problem with a vertex shader not able to interpolate a single float output.
I started with a multi-pass stage that produced a volume from a shape, then fed that volume to a volumetric pass and observed unexplainable artefacts. When flying through a scene and entering a volume the buffer that was clipped by a view frustum's near plane would mangle a default buffer value (set by glClear). Attempts to fix that by playing with glEnable(GL_DEPTH_CLAMP), glDepthRange(-1.0, 1.0) etc did not help.
What did work was switching from a float output to a whole vec3, and calculating depth in a fragment shader. Inefficient, and annoying.

Real-time atmospheric effects

This demonstrates a few atmospheric effects that I've added to a custom engine. It is real-time, using stages in shader pipeline, and very customizable (fog, lighting, composition etc).
In addition, the demo is showing Fourier transform to simulate ocean waves.
More to follow.

On UI composition, embedded view controllers, weak vars, multi-cast delegates.

Interface Builder is great for composing plain interfaces, where each view controller is represented by a single instance of view in a view hierarchy.

Embedded view controllers are different.

Even though you can create multiple Embed segues connecting different places in a storyboard to the same embedded view controller - during app initialization the framework will create as many instances of that controller as there are segues.

This is counter-intuitive at first. A collection of views is an ordered tree, where each node has a reference to its single parent. But in a storyboard the view controllers can be connected by Embed segues arbitrarily and can form cycles.

It appears Apple gets away with this by forcing each Embed segue to create a new instance of a destination view controller.

For example, the following storyboard with embedded view controllers (part of a project in a coursera.org course) has many parent-child Embed relations:




Note the three parent Container views, and two of them share a view controller.

A parent view controller will have outlets to instances of child view controllers. Those outlets are declared weak and fished out of the storyboard in an override of prepareForSegue() method. If there are multiple instances of the same embedded view controller (as in the screenshot) - you’d want them stored in an array of weak references. But the challenge with Swift 2 is that it does not support collections of weak references.

In a simple case - having separate weak refs is ok. The prepareForSegue() implementation might look like this:

    // MARK: - Navigation

    override func prepareForSegue(segue: UIStoryboardSegue, sender: AnyObject?) {

        // !!! Embed segues are processed before viewDidLoad().
        switch segue.destinationViewController {
        case let vc as MyHobbiesCollectionViewController:
            // Note that this embedded VC has multiple instances,
            // need additional id check (i.e. by segue.identifier).
            switch segue.identifier {
                case .Some("My1stHobbiesEmbedSegue"):
                    my1stHobbiesCVC = vc;
            default:
                my2ndHobbiesCVC = vc;
            }
           
            // Also prepare non-embed segues.
        case let vc as EditHobbiesViewController:
            break;
        default: break;
        }

    }

In other more advanced cases, a recommended solution is to declare a weak ref in a structure and then have an array of those structures. Hopefully Apple is working to fix this minor annoyance.

A few words about how a hierarchy of view controllers should communicate with the rest of the tree.

KVO and NSNotificationCenter are ok in general case, but they lack type-safety of messages, and require not very nice symmetry of subscrbe/unsubscribe calls. Instead, I prefer declaring protocols for each child view controller, then a parent would implement those protocols, and assign itself to weak ref delegates of its children.

Events of the child controllers will first travel up a hierarchy, where a parent will route them to peers.

This messaging system could make use of a collection of weak ref delegates, which would be referred to as multi-cast delegates. But this will have to wait till a version of Swift that implements weak ref collections natively.

Quick Auto Layout rig for a responsive tab bar control

Let’s say I need to create a tab bar with 3 tabs, each having a button and an indicator. It needs to be responsive (autoresizeable) and each tab equally sized with the others.





Using Auto Layout here is how I do it.

1. Select a view that renders a tab (here called “TabView”). Copy-paste as many as needed. Customize each copy.





2. Add missing constraints manually. Select each tab view, uncheck “Constrain to margins”, and add space constraints. The first tab needs 4 constraints, the rest - only 3.







3. Add size constraints (width or height) by selecting all tab views, then Equal Widths.







4. Finally, update all frames in a parent view.





Embed in a parent controller, declare protocols to handle taps, etc.