2015-12-11 Reproducing Olshausen's classical SparseNet (part 3)

In this notebook, we test the convergence of SparseNet as a function of different learning parameters. This shows the relative robusteness of this method according to the coding parameters, but also the importance of homeostasis to obtain an efficient set of filters:

  • first, whatever the learning rate, the convergence is not complete without homeostasis,
  • second, we achieve better convergence for similar learning rates and on a certain range of learning rates for the homeostasis
  • third, the smoothing parameter alpha_homeo has to be properly set to achieve a good convergence.
  • last, this homeostatic rule works with the diferent variants of sparse coding.

See also :

In [1]:
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
np.set_printoptions(precision=2, suppress=True)
/usr/local/lib/python3.5/site-packages/matplotlib/__init__.py:872: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
  warnings.warn(self.msg_depr % (key, alt_key))
In [2]:
from shl_scripts import SHL
DEBUG_DOWNSCALE, verbose = 10, 100
DEBUG_DOWNSCALE, verbose = 10, 0
DEBUG_DOWNSCALE, verbose = 1, 0
N_scan = 7
database = '/Users/lolo/pool/science/BICV/SHL_scripts/database/'

1. With diferent learning rates but without homeostasis

Here,we only ensure the norm ofthe filters is constant.

In [3]:
for eta in np.logspace(-3, -1.5, N_scan, base=10):
    shl = SHL(DEBUG_DOWNSCALE=DEBUG_DOWNSCALE, eta_homeo=0, database=database,
              learning_algorithm='omp', eta=eta, verbose=verbose)
    dico = shl.learn_dico()
    _ = shl.show_dico(dico, title='no homeo - eta={}'.format(eta))
/Users/lolo/pool/libs/numbers/scikit-learn-sparsenet/sklearn/decomposition/dict_learning.py:152: RuntimeWarning:  Orthogonal matching pursuit ended prematurely due to linear
dependence in the dictionary. The requested precision might not have been met.

  copy_Xy=copy_cov).T

2. Homeostasis à-la-SparseNet

In [4]:
for eta in np.logspace(-3, -1.5, N_scan, base=10):
    shl = SHL(DEBUG_DOWNSCALE=DEBUG_DOWNSCALE, database=database,
              learning_algorithm='omp', eta=eta, verbose=verbose)
    dico = shl.learn_dico()
    _ = shl.show_dico(dico, title='homeo - eta={}'.format(eta))
/Users/lolo/pool/libs/numbers/scikit-learn-sparsenet/sklearn/decomposition/dict_learning.py:152: RuntimeWarning:  Orthogonal matching pursuit ended prematurely due to linear
dependence in the dictionary. The requested precision might not have been met.

  copy_Xy=copy_cov).T