2015-05-05 Reproducing Olshausen's classical SparseNet (part 1)

  • This notebook tries to reproduce the learning strategy specified in the framework of the SparseNet algorithm from Bruno Olshausen. It allows to efficiently code natural image patches by constraining the code to be sparse.

  • the underlying machinery uses the dictionary learning used in the image denoising example from sklearn and our aim here is to show that a novel ingredient is necessary to reproduce Olshausen's results.

  • All these code bits are regrouped in the SHL scripts repository (where you will also find some older matlab code). You may install it using

    pip install git+https://github.com/bicv/SHL_scripts
In [1]:
import matplotlib.pyplot as plt
%matplotlib inline
%config InlineBackend.figure_format='svg'
import numpy as np
np.set_printoptions(precision=2, suppress=True)
import pandas as pd
import seaborn as sns
%load_ext autoreload
%autoreload 2
/usr/local/lib/python3.5/site-packages/matplotlib/__init__.py:872: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
  warnings.warn(self.msg_depr % (key, alt_key))
/usr/local/lib/python3.5/site-packages/matplotlib/__init__.py:872: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
  warnings.warn(self.msg_depr % (key, alt_key))
In [2]:
from shl_scripts import SHL
database = '/Users/lolo/pool/science/BICV/SHL_scripts/database/'
DEBUG_DOWNSCALE, verbose = 100, 0
DEBUG_DOWNSCALE, verbose = 1, 0
shl = SHL(database=database, DEBUG_DOWNSCALE=DEBUG_DOWNSCALE, verbose=verbose, eta_homeo=0.)
fig, ax = shl.show_dico(shl.learn_dico(), title='no homeo')
fig.show()
/Users/lolo/pool/libs/numbers/scikit-learn-sparsenet/sklearn/decomposition/dict_learning.py:152: RuntimeWarning:  Orthogonal matching pursuit ended prematurely due to linear
dependence in the dictionary. The requested precision might not have been met.

  copy_Xy=copy_cov).T

testing sklearn's dictionary learning

Basically, we will here replicate the same image denoising code from sklearn (see also below), but with:

  • no distorsion of image denoising -- we will assess the results by the coding efficiency,
  • whitening of the images (as in Olshausen) befor taking patches,
  • bigger patches,
  • a bigger set of natural images,
  • a longer learning
  • testing different transform_algorithms as learning and coding algorithms,
  • uses online learning from the third example

What differs from the original algorithm is mainly

  • the learning method is modified,
  • the way that the norm of the filters is controlled. Here, sklearn simply assumes that $ || V_k ||_2 = 1$, $\forall k$ (with $0 <= k < n_{components}$). We will see that this may be a problem.

Let's start with the learning:

In [3]:
dicos = {}
learning_algorithms = [
    ('Orthogonal Matching Pursuit 10 atom', 'OMP1_N10',
     {'learning_algorithm':'omp', 'transform_n_nonzero_coefs': 10}),
    ('Orthogonal Matching Pursuit alpha 0.9', 'OMP_tol',
     {'learning_algorithm':'omp', 'alpha': .9}),
    ('Least-angle regression 5 atoms', 'LARS',
     {'learning_algorithm':'lars', 'transform_n_nonzero_coefs': 5}),
    ('Least-angle regression 5 atoms', 'LARS',
     {'learning_algorithm':'lasso_lars', 'transform_n_nonzero_coefs': 5}),
    ('Least-angle regression 5 atoms', 'LARS',
     {'learning_algorithm':'lasso_cd', 'transform_n_nonzero_coefs': 5})]
        
for learning_title, learning_label, learning_kwargs in learning_algorithms:
    print('Dictionary learned from image patches using ' + learning_title)
    shl = SHL(database=database, DEBUG_DOWNSCALE=DEBUG_DOWNSCALE, verbose=verbose, eta_homeo=0., **learning_kwargs)
    dicos[learning_label] = shl.learn_dico()
    fig, ax = shl.show_dico(dicos[learning_label], title=learning_title)
    fig.show()
Dictionary learned from image patches using Orthogonal Matching Pursuit 10 atom
Dictionary learned from image patches using Orthogonal Matching Pursuit alpha 0.9
Dictionary learned from image patches using Least-angle regression 5 atoms
Dictionary learned from image patches using Least-angle regression 5 atoms
Dictionary learned from image patches using Least-angle regression 5 atoms
/Users/lolo/pool/libs/numbers/scikit-learn-sparsenet/sklearn/decomposition/dict_learning.py:152: RuntimeWarning:  Orthogonal matching pursuit ended prematurely due to linear
dependence in the dictionary. The requested precision might not have been met.

  copy_Xy=copy_cov).T