将数据拟合到指数分布_Python数据分析实战_百度阅读
[印]伊凡·伊德里斯
3834人在读
3.2　将数据拟合到指数分布指数分布（exponential distribution）是伽马分布（gamma distribution）的特殊情况，我们将会在本章遇到。指数分布可用于分析降雨的极值。它还可以用于对服务客户所花费的时间进行建模。对于零和负值，指数分布的概率分布函数（Probability Distribution Function，PDF）为零。对于正值，PDF按指数衰减：我们将使用降雨数据作为示例，这对于指数分布拟合非常适合。显然，雨量不可能是负数，我们知道总体来看瓢泼大雨比不下雨的可能性小。事实上，一整天都不下雨是很有可能的。1.操作步骤如下的步骤将降雨数据拟合到指数分布：（1）导入部分如下：（2）我创建了一个包装类，它会调用scipy.stats.expon方法。首先，调用fit（）方法：（3）下面的代码在拟合残差上调用scipy.stats.expon.pdf（）方法和scipy.stats.describe（）函数：（4）为了评估拟合，可以使用度量指标。使用如下代码段计算拟合度量指标：（5）绘制拟合图并显示分析结果，如下所示：请参见以下截图了解最终结果（代码在本书配套软件包的fitting_expon.ipynb文件中）：......
扫描二维码，继续阅读用Python为直方图绘制拟合曲线的两种方法 - 简书
用Python为直方图绘制拟合曲线的两种方法
直方图是用于展示数据的分组分布状态的一种图形，用矩形的宽度和高度表示频数分布，通过直方图，用户可以很直观的看出数据分布的形状、中心位置以及数据的离散程度等。
在python中一般采用matplotlib库的hist来绘制直方图，至于如何给直方图添加拟合曲线（密度函数曲线），一般来说有以下两种方法。
方法一：采用matplotlib中的mlab模块
mlab模块是Python中强大的3D作图工具，立体感效果极佳。在这里使用mlab可以跳出直方图二维平面图形的限制，在此基础上再添加一条曲线。在这里，我们以鸢尾花iris中的数据为例，来举例说明。
import numpy as np
import matplotlib.mlab as mlab
import matplotlib.pyplot as plt
import pandas
# Load dataset
"https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data"
names = ['sepal-length', 'sepal-width','petal-length', 'petal-width', 'class']
dataset = pandas.read_csv(url, names=names)
print(dataset.head(10))
# descriptions
print(dataset.describe())
x = dataset.iloc[:,0] #提取第一列的sepal-length变量
mu =np.mean(x) #计算均值
sigma =np.std(x)
以上为通过python导入鸢尾花iris数据，然后提取第一列的sepal-length变量为研究对象，计算出其均值、标准差，接下来就绘制带拟合曲线的直方图。
num_bins = 30 #直方图柱子的数量
n, bins, patches = plt.hist(x, num_bins,normed=1, facecolor='blue', alpha=0.5)
#直方图函数，x为x轴的值，normed=1表示为概率密度，即和为一，绿色方块，色深参数0.5.返回n个概率，直方块左边线的x值，及各个方块对象
y = mlab.normpdf(bins, mu, sigma)#拟合一条最佳正态分布曲线y
plt.plot(bins, y, 'r--') #绘制y的曲线
plt.xlabel('sepal-length') #绘制x轴
plt.ylabel('Probability') #绘制y轴
plt.title(r'Histogram : $\mu=5.8433$,$\sigma=0.8253$')#中文标题 u'xxx'
plt.subplots_adjust(left=0.15)#左边距
plt.show()
以上命令主要采用mlab.normpdf基于直方图的柱子数量、均值、方差来拟合曲线，然后再用plot画出来，这种方法的一个缺点就是画出的正态分布拟合曲线（红色虚线）并不一定能很好反映数据的分布情况，如上图所示。
方法二：采用seaborn库中的distplot绘制
Seaborn其实是在matplotlib的基础上进行了更高级的API封装，从而使得作图更加容易，在大多数情况下使用seaborn就能做出很具有吸引力的图，而使用matplotlib就能制作具有更多特色的图。应该把Seaborn视为matplotlib的补充，而不是替代物。
import seaborn as sns
sns.set_palette("hls") #设置所有图的颜色，使用hls色彩空间
sns.distplot(x,color="r",bins=30,kde=True)
plt.show()
在这里主要使用sns.distplot（增强版dist），柱子数量bins也设置为30，kde=True表示是否显示拟合曲线，如果为False则只出现直方图。
在这里注意一下它与前边mlab.normpdf方法不同的是，拟合曲线不是正态的，而是更好地拟合了数据的分布情况，如上图，因此比mlab.normpdf更为准确。
进一步设置sns.distplot，可以采用kde_kws（拟合曲线的设置）、hist_kws（直方柱子的设置），可以得到：
import seaborn as sns
import matplotlib as mpl
sns.set_palette("hls")
mpl.rc("figure", figsize=(6,4))
sns.distplot(x,bins=30,kde_kws={"color":"seagreen", "lw":3 }, hist_kws={ "color": "b" })
plt.show()
其中，lw为曲线粗细程度。
写作不易，特别是技术类的写作，请大家多多支持，关注、点赞、转发等等
统计类专业，喜欢数据分析、可视化、数据挖掘、大数据，历史、文学等
//我所经历的大数据平台发展史（三）：互联网时代 o 上篇http://www.infoq.com/cn/articles/the-development-history-of-big-data-platform-paet02 编者按：本文是松子（李博源）的大数据平台发展史...
第1章 准备工作第2章 Python语法基础，IPython和Jupyter第3章 Python的数据结构、函数和文件第4章 NumPy基础：数组和矢量计算第5章 pandas入门第6章 数据加载、存储与文件格式第7章 数据清洗和准备第8章 数据规整：聚合、合并和重塑第9章...
探索数据分析 作者：Blink
爱好：喜欢数据分析、可视化和机器学习，目前研究深度学习中。
可以结团Kaggle或者比赛喔！ 什么叫探索性数据分析？ 探索性数据分析（Exploratory Data Analysi...
算法技术解构 1、Python基础知识 （1）IPythonIPython的开发者吸收了标准解释器的基本概念，在此基础上进行了大量的改进，创造出一个令人惊奇的工具。在它的主页上是这么说的：“这是一个增强的交互式Python shell。”具有tab补全，对象自省，强大的历史...
无尽的炼狱中，生存者一群以人类灵魂喂食的魔鬼。
炙热热的岩浆，熊熊的火焰在这片大地上肆虐着。灰暗的天空中充满着死气，随处可见的骸骨点缀着大地。
“在一处辉煌璀璨的大殿中，一个女孩静静的站在中间，她的对面一位煞气萦绕的中年男子坐在白金王座上。”
这次你犯...
五一期间我去参加了一个耳穴培训班！ 本意是为公司新品&微砭耳针&去寻求学术合作，去实地了解下培训机构的专业能力，类似经常有客户跟我们合作之前会来工厂一探虚实。 对本次授课的刘老完全没有概念，也没报太多期望。一打听本次的学员，我开始有点激动了，给我做耳穴疗法启蒙的董老师赫然在...
李碧华的文字向来是凉薄的，读着读着你仿佛能见到影影绰绰的张爱玲的剪影。 已经忘了最初是因为电影才看了她的文字抑或因着文字才去看的电影，总觉得心头留着点凉，像是逼仄弄堂里潮湿的青苔，幽幽的。因西湖边那抹妩媚的精魂，涂着诱惑的红信子。 遇有情人作快乐事，一段未问是缘是劫的余烬，...
白鹿原上老一辈里又一位老人泰恒老汉死了，他死于因气愤和激动突发的心脑血管崩裂，死于白孝武婚礼的酒席，死于原上共产党人领导的土地革命正如火如荼开展起来的时候，死于自己的孙子批斗父亲正“热闹”的时期……
不知道白孝文跑回来告诉白嘉轩，“知道不知道鹿子霖这些年贪了...
昨天下午一回到家，头重脚轻，估计今晚要早睡咯！赶紧冲凉，再回到房间。 久违的空间啊…… 老妈说她收拾了一个下午，从一点到四点，把书桌，书桌上的东西和地上以及地上堆积的书都搬出去了。我在心里嘴里都向妈妈表达了最真挚的谢意！ “看到你那一堆的东西，我要晕过去了。” 这是老妈挂在...没有更多推荐了，
加入CSDN，享受更精准的内容推荐，与500万程序员共同成长！后使用快捷导航没有帐号？
查看: 1103|回复: 0
Python机器学习从原理到实践(2)：数据拟合与广义线性回归
金牌会员, 积分 1162, 距离下一级还需 1838 积分
论坛徽章:37
中的预测问题通常分为2类：回归与分类。简单的说回归就是预测数值，而分类是给数据打上标签归类。本文讲述如何用Python进行基本的数据拟合，以及如何对拟合结果的误差进行分析。本例中使用一个2次函数加上随机的扰动来生成500个点，然后尝试用1、2、100次方的多项式对该数据进行拟合。
　　拟合的目的是使得根据训练数据能够拟合出一个多项式函数，这个函数能够很好的拟合现有数据，并且能对未知的数据进行预测。
代码如下：[python] view plaincopyimport matplotlib.pyplot as pltimport numpy as npimport scipy as spfrom scipy.stats import normfrom sklearn.pipeline import Pipelinefrom sklearn.linear_model import LinearRegressionfrom sklearn.preprocessing import PolynomialFeaturesfrom sklearn import linear_model''''' 数据生成 '''x = np.arange(0, 1, 0.002)y = norm.rvs(0, size=500, scale=0.1)y = y + x**2''''' 均方误差根 '''def rmse(y_test, y):return sp.sqrt(sp.mean((y_test - y) ** 2))''''' 与均值相比的优秀程度，介于[0~1]。0表示不如均值。1表示完美预测.这个版本的实现是参考scikit-learn官网文档 '''def R2(y_test, y_true):return 1 - ((y_test - y_true)**2).sum() / ((y_true - y_true.mean())**2).sum()''''' 这是Conway&White《机器学习使用案例解析》里的版本 '''def R22(y_test, y_true):y_mean = np.array(y_true)y_mean[:] = y_mean.mean()return 1 - rmse(y_test, y_true) / rmse(y_mean, y_true)plt.scatter(x, y, s=5)degree = [1,2,100]y_test =y_test = np.array(y_test)for d in degree:clf = Pipeline([('poly', PolynomialFeatures(degree=d)),('linear', LinearRegression(fit_intercept=False))])clf.fit(x[:, np.newaxis], y)y_test = clf.predict(x[:, np.newaxis])print(clf.named_steps['linear'].coef_)print('rmse=%.2f, R2=%.2f, R22=%.2f, clf.score=%.2f' %(rmse(y_test, y),R2(y_test, y),R22(y_test, y),clf.score(x[:, np.newaxis], y)))plt.plot(x, y_test, linewidth=2)plt.grid()plt.legend(['1','2','100'], loc='upper left')plt.show()该程序运行的显示结果如下：[-0..]
rmse=0.13, R2=0.82, R22=0.58, clf.score=0.82[ 0...]
rmse=0.11, R2=0.88, R22=0.66, clf.score=0.88
[ 6. -1. 6. -1.
......rmse=0.10, R2=0.89, R22=0.67, clf.score=0.89显示出的coef_就是多项式参数。如1次拟合的结果为y = 0.x -0.
这里我们要注意这几点：1、误差分析。做回归分析，常用的误差主要有均方误差根（RMSE）和R-平方（R2）。RMSE是预测值与真实值的误差平方根的均值。这种度量方法很流行（Netflix机器学习比赛的评价方法），是一种定量的权衡方法。R2方法是将预测值跟只使用均值的情况下相比，看能好多少。其区间通常在（0,1）之间。0表示还不如什么都不预测，直接取均值的情况，而1表示所有预测跟真实结果完美匹配的情况。R2的计算方法，不同的文献稍微有不同。如本文中函数R2是依据scikit-learn官网文档实现的，跟clf.score函数结果一致。而R22函数的实现来自Conway的著作《机器学习使用案例解析》，不同在于他用的是2个RMSE的比值来计算R2。我们看到多项式次数为1的时候，虽然拟合的不太好，R2也能达到0.82。2次多项式提高到了0.88。而次数提高到100次，R2也只提高到了0.89。2、过拟合。使用100次方多项式做拟合，效果确实是高了一些，然而该模型的据测能力却极其差劲。而且注意看多项式系数，出现了大量的大数值，甚至达到10的12次方。这里我们修改代码，将500个样本中的最后2个从训练集中移除。然而在测试中却仍然测试所有500个样本。clf.fit(x[:498, np.newaxis], y[:498])这样修改后的多项式拟合结果如下：[-0..0052037 ]
rmse=0.12, R2=0.85, R22=0.61, clf.score=0.85
rmse=0.10, R2=0.90, R22=0.69, clf.score=0.90...rmse=0.21, R2=0.57, R22=0.34, clf.score=0.57仅仅只是缺少了最后2个训练样本，红线（100次方多项式拟合结果）的预测发生了剧烈的偏差，R2也急剧下降到0.57。而反观1，2次多项式的拟合结果，R2反而略微上升了。这说明高次多项式过度拟合了训练数据，包括其中大量的噪音，导致其完全丧失了对数据趋势的预测能力。前面也看到，100次多项式拟合出的系数数值无比巨大。人们自然想到通过在拟合过程中限制这些系数数值的大小来避免生成这种畸形的拟合函数。其基本原理是将拟合多项式的所有系数值之和（L1正则化）或者平方和（L2正则化）加入到惩罚模型中，并指定一个惩罚力度因子w，来避免产生这种畸形系数。这样的思想应用在了岭（Ridge）回归（使用L2正则化）、Lasso法（使用L1正则化）、弹性网（Elastic net，使用L1+L2正则化）等方法中，都能有效避免过拟合。更多原理可以参考相关资料。下面以岭回归为例看看100次多项式的拟合是否有效。将代码修改如下:clf = Pipeline([('poly', PolynomialFeatures(degree=d)),
('linear', linear_model.Ridge )])
clf.fit(x[:400, np.newaxis], y[:400])结果如下：[ 0. 0.]
rmse=0.15, R2=0.78, R22=0.53, clf.score=0.78
rmse=0.11, R2=0.87, R22=0.64, clf.score=0.87
[ 0. 2. 3. 2.
1. 1. 7. 4.
3. 2. 2. 1.
rmse=0.10, R2=0.90, R22=0.68, clf.score=0.90可以看到，100次多项式的系数参数变得很小。大部分都接近于0.另外值得注意的是，使用岭回归之类的惩罚模型后，1次和2次多项式回归的R2值可能会稍微低于基本线性回归。然而这样的模型，即使使用100次多项式，在训练400个样本，预测500个样本的情况下不仅有更小的R2误差，而且还具备优秀的预测能力。
dataguru.cn All Right Reserved.
扫一扫加入本版微信群scikit-learn v0.19.1
Please if you use the software.
1.1. Generalized Linear Models
The following are a set of methods intended for regression in which
the target value is expected to be a linear combination of the input
variables. In mathematical notion, if
is the predicted
Across the module, we designate the vector
as coef_ and
as intercept_.
To perform classification with generalized linear models, see
1.1.1. Ordinary Least Squares
fits a linear model with coefficients
to minimize the residual sum
of squares between the observed responses in the dataset, and the
responses predicted by the linear approximation. Mathematically it
solves a problem of the form:
will take in its fit method arrays X, y
and will store the coefficients
of the linear model in its
coef_ member:
&&& from sklearn import linear_model
&&& reg = linear_model.LinearRegression()
&&& reg.fit ([[0, 0], [1, 1], [2, 2]], [0, 1, 2])
LinearRegression(copy_X=True, fit_intercept=True, n_jobs=1, normalize=False)
&&& reg.coef_
array([ 0.5,
However, coefficient estimates for Ordinary Least Squares rely on the
independence of the model terms. When terms are correlated and the
columns of the design matrix
have an approximate linear
dependence, the design matrix becomes close to singular
and as a result, the least-squares estimate becomes highly sensitive
to random errors in the observed response, producing a large
variance. This situation of multicollinearity can arise, for
example, when data are collected without an experimental design.
1.1.1.1. Ordinary Least Squares Complexity
This method computes the least squares solution using a singular value
decomposition of X. If X is a matrix of size (n, p) this method has a
cost of , assuming that .
1.1.2. Ridge Regression
regression addresses some of the problems of
by imposing a penalty on the size of
coefficients. The ridge coefficients minimize a penalized residual sum
of squares,
is a complexity parameter that controls the amount
of shrinkage: the larger the value of , the greater the amount
of shrinkage and thus the coefficients become more robust to collinearity.
As with other linear models,
will take in its fit method
arrays X, y and will store the coefficients
of the linear model in
its coef_ member:
&&& from sklearn import linear_model
&&& reg = linear_model.Ridge (alpha = .5)
&&& reg.fit ([[0, 0], [0, 0], [1, 1]], [0, .1, 1])
Ridge(alpha=0.5, copy_X=True, fit_intercept=True, max_iter=None,
normalize=False, random_state=None, solver='auto', tol=0.001)
&&& reg.coef_
array([ 0.,
&&& reg.intercept_
0.13636...
1.1.2.1. Ridge Complexity
This method has the same order of complexity than an
1.1.2.2. Setting the regularization parameter: generalized Cross-Validation
implements ridge regression with built-in
cross-validation of the alpha parameter.
The object works in the same way
as GridSearchCV except that it defaults to Generalized Cross-Validation
(GCV), an efficient form of leave-one-out cross-validation:
&&& from sklearn import linear_model
&&& reg = linear_model.RidgeCV(alphas=[0.1, 1.0, 10.0])
&&& reg.fit([[0, 0], [0, 0], [1, 1]], [0, .1, 1])
RidgeCV(alphas=[0.1, 1.0, 10.0], cv=None, fit_intercept=True, scoring=None,
normalize=False)
&&& reg.alpha_
References
“Notes on Regularized Least Squares”, Rifkin & Lippert (,
1.1.3. Lasso
is a linear model that estimates sparse coefficients.
It is useful in some contexts due to its tendency to prefer solutions
with fewer parameter values, effectively reducing the number of variables
upon which the given solution is dependent. For this reason, the Lasso
and its variants are fundamental to the field of compressed sensing.
Under certain conditions, it can recover the exact set of non-zero
weights (see
Mathematically, it consists of a linear model trained with
as regularizer. The objective function to minimize is:
The lasso estimate thus solves the minimization of the
least-squares penalty with
added, where
is a constant and
is the -norm of
the parameter vector.
The implementation in the class
uses coordinate descent as
the algorithm to fit the coefficients. See
for another implementation:
&&& from sklearn import linear_model
&&& reg = linear_model.Lasso(alpha = 0.1)
&&& reg.fit([[0, 0], [1, 1]], [0, 1])
Lasso(alpha=0.1, copy_X=True, fit_intercept=True, max_iter=1000,
normalize=False, positive=False, precompute=False, random_state=None,
selection='cyclic', tol=0.0001, warm_start=False)
&&& reg.predict([[1, 1]])
array([ 0.8])
Also useful for lower-level tasks is the function
computes the coefficients along the full path of possible values.
Feature selection with Lasso
As the Lasso regression yields sparse models, it can
thus be used to perform feature selection, as detailed in
1.1.3.1. Setting regularization parameter
The alpha parameter controls the degree of sparsity of the coefficients
estimated.
1.1.3.1.1. Using cross-validation
scikit-learn exposes objects that set the Lasso alpha parameter by
cross-validation:
is based on the
explained below.
For high-dimensional datasets with many collinear regressors,
is most often preferable. However,
the advantage of exploring more relevant values of alpha parameter, and
if the number of samples is very small compared to the number of
features, it is often faster than .
1.1.3.1.3. Comparison with the regularization parameter of SVM
The equivalence between alpha and the regularization parameter of SVM,
C is given by alpha = 1 / C or alpha = 1 / (n_samples * C),
depending on the estimator and the exact objective function optimized by the
1.1.4. Multi-task Lasso
is a linear model that estimates sparse
coefficients for multiple regression problems jointly: y is a 2D array,
of shape (n_samples, n_tasks). The constraint is that the selected
features are the same for all the regression problems, also called tasks.
The following figure compares the location of the non-zeros in W obtained
with a simple Lasso or a MultiTaskLasso. The Lasso estimates yields
scattered non-zeros while the non-zeros of the MultiTaskLasso are full
Fitting a time-series model, imposing that any active feature be active at all times.
Mathematically, it consists of a linear model trained with a mixed
prior as regularizer.
The objective function to minimize is:
indicates the Frobenius norm:
The implementation in the class
uses coordinate descent as
the algorithm to fit the coefficients.
1.1.5. Elastic Net
is a linear regression model trained with L1 and L2 prior
as regularizer. This combination allows for learning a sparse model where
few of the weights are non-zero like , while still maintaining
the regularization properties of . We control the convex
combination of L1 and L2 using the l1_ratio parameter.
Elastic-net is useful when there are multiple features which are
correlated with one another. Lasso is likely to pick one of these
at random, while elastic-net is likely to pick both.
A practical advantage of trading-off between Lasso and Ridge is it allows
Elastic-Net to inherit some of Ridge’s stability under rotation.
The objective function to minimize is in this case
can be used to set the parameters
alpha () and l1_ratio () by cross-validation.
1.1.6. Multi-task Elastic Net
is an elastic-net model that estimates sparse
coefficients for multiple regression problems jointly: Y is a 2D array,
of shape (n_samples, n_tasks). The constraint is that the selected
features are the same for all the regression problems, also called tasks.
Mathematically, it consists of a linear model trained with a mixed
prior as regularizer.
The objective function to minimize is:
The implementation in the class
uses coordinate descent as
the algorithm to fit the coefficients.
can be used to set the parameters
alpha () and l1_ratio () by cross-validation.
1.1.7. Least Angle Regression
Least-angle regression (LARS) is a regression algorithm for
high-dimensional data, developed by Bradley Efron, Trevor Hastie, Iain
Johnstone and Robert Tibshirani. LARS is similar to forward stepwise
regression. At each step, it finds the predictor most correlated with the
response. When there are multiple predictors having equal correlation, instead
of continuing along the same predictor, it proceeds in a direction equiangular
between the predictors.
The advantages of LARS are:
It is numerically efficient in contexts where p && n (i.e., when the
number of dimensions is significantly greater than the number of
It is computationally just as fast as forward selection and has
the same order of complexity as an ordinary least squares.
It produces a full piecewise linear solution path, which is
useful in cross-validation or similar attempts to tune the model.
If two variables are almost equally correlated with the response,
then their coefficients should increase at approximately the same
rate. The algorithm thus behaves as intuition would expect, and
also is more stable.
It is easily modified to produce solutions for other estimators,
like the Lasso.
The disadvantages of the LARS method include:
Because LARS is based upon an iterative refitting of the
residuals, it would appear to be especially sensitive to the
effects of noise. This problem is discussed in detail by Weisberg
in the discussion section of the Efron et al. (2004) Annals of
Statistics article.
The LARS model can be used using estimator , or its
low-level implementation .
1.1.8. LARS Lasso
is a lasso model implemented using the LARS
algorithm, and unlike the implementation based on coordinate_descent,
this yields the exact solution, which is piecewise linear as a
function of the norm of its coefficients.
&&& from sklearn import linear_model
&&& reg = linear_model.LassoLars(alpha=.1)
&&& reg.fit([[0, 0], [1, 1]], [0, 1])
LassoLars(alpha=0.1, copy_X=True, eps=..., fit_intercept=True,
fit_path=True, max_iter=500, normalize=True, positive=False,
precompute='auto', verbose=False)
&&& reg.coef_
array([ 0.717157...,
The Lars algorithm provides the full path of the coefficients along
the regularization parameter almost for free, thus a common operation
consist of retrieving the path with function
1.1.8.1. Mathematical formulation
The algorithm is similar to forward stepwise regression, but instead
of including variables at each step, the estimated parameters are
increased in a direction equiangular to each one’s correlations with
the residual.
Instead of giving a vector result, the LARS solution consists of a
curve denoting the solution for each value of the L1 norm of the
parameter vector. The full coefficients path is stored in the array
coef_path_, which has size (n_features, max_features+1). The first
column is always zero.
References:
Original Algorithm is detailed in the paper
by Hastie et al.
1.1.9. Orthogonal Matching Pursuit (OMP)
implements the OMP
algorithm for approximating the fit of a linear model with constraints imposed
on the number of non-zero coefficients (ie. the L 0 pseudo-norm).
Being a forward feature selection method like ,
orthogonal matching pursuit can approximate the optimum solution vector with a
fixed number of non-zero elements:
Alternatively, orthogonal matching pursuit can target a specific error instead
of a specific number of non-zero coefficients. This can be expressed as:
OMP is based on a greedy algorithm that includes at each step the atom most
highly correlated with the current residual. It is similar to the simpler
matching pursuit (MP) method, but better in that at each iteration, the
residual is recomputed using an orthogonal projection on the space of the
previously chosen dictionary elements.
References:
S. G. Mallat, Z. Zhang,
1.1.10. Bayesian Regression
Bayesian regression techniques can be used to include regularization
parameters in the estimation procedure: the regularization parameter is
not set in a hard sense but tuned to the data at hand.
This can be done by introducing
over the hyper parameters of the model.
regularization used in
is equivalent
to finding a maximum a posteriori estimation under a Gaussian prior over the
parameters
with precision .
Instead of setting
lambda manually, it is possible to treat it as a random variable to be
estimated from the data.
To obtain a fully probabilistic model, the output
is assumed
to be Gaussian distributed around :
Alpha is again treated as a random variable that is to be estimated from the
The advantages of Bayesian Regression are:
It adapts to the data at hand.
It can be used to include regularization parameters in the
estimation procedure.
The disadvantages of Bayesian regression include:
Inference of the model can be time consuming.
References
A good introduction to Bayesian methods is given in C. Bishop: Pattern
Recognition and Machine learning
Original Algorithm is detailed in the
book Bayesian learning for neural
networks by Radford M. Neal
1.1.10.1. Bayesian Ridge Regression
estimates a probabilistic model of the
regression problem as described above.
The prior for the parameter
is given by a spherical Gaussian:
The priors over
are chosen to be , the
conjugate prior for the precision of the Gaussian.
The resulting model is called Bayesian Ridge Regression, and is similar to the
classical .
The parameters ,
are estimated jointly during the fit of the model.
remaining hyperparameters are the parameters of the gamma priors over
These are usually chosen to be
non-informative.
The parameters are estimated by maximizing the marginal
log likelihood.
By default .
Bayesian Ridge Regression is used for regression:
&&& from sklearn import linear_model
&&& X = [[0., 0.], [1., 1.], [2., 2.], [3., 3.]]
&&& Y = [0., 1., 2., 3.]
&&& reg = linear_model.BayesianRidge()
&&& reg.fit(X, Y)
BayesianRidge(alpha_1=1e-06, alpha_2=1e-06, compute_score=False, copy_X=True,
fit_intercept=True, lambda_1=1e-06, lambda_2=1e-06, n_iter=300,
normalize=False, tol=0.001, verbose=False)
After being fitted, the model can then be used to predict new values:
&&& reg.predict ([[1, 0.]])
array([ 0.])
The weights
of the model can be access:
&&& reg.coef_
array([ 0.,
Due to the Bayesian framework, the weights found are slightly different to the
ones found by . However, Bayesian Ridge Regression
is more robust to ill-posed problem.
References
More details can be found in the article
by MacKay, David J. C.
1.1.10.2. Automatic Relevance Determination - ARD
is very similar to ,
but can lead to sparser weights
poses a different prior over , by dropping the
assumption of the Gaussian being spherical.
Instead, the distribution over
is assumed to be an axis-parallel,
elliptical Gaussian distribution.
This means each weight
is drawn from a Gaussian distribution,
centered on zero and with a precision :
In contrast to , each coordinate of
has its own standard deviation . The prior over all
is chosen to be the same gamma distribution given by
hyperparameters
ARD is also known in the literature as Sparse Bayesian Learning and
Relevance Vector Machine
References:
1.1.11. Logistic regression
Logistic regression, despite its name, is a linear model for classification
rather than regression. Logistic regression is also known in the literature as
logit regression, maximum-entropy classification (MaxEnt)
or the log-linear classifier. In this model, the probabilities describing the possible outcomes of a single trial are modeled using a .
The implementation of logistic regression in scikit-learn can be accessed from
class . This implementation can fit binary, One-vs-
Rest, or multinomial logistic regression with optional L2 or L1
regularization.
As an optimization problem, binary class L2 penalized logistic regression
minimizes the following cost function:
Similarly, L1 regularized logistic regression solves the following
optimization problem
The solvers implemented in the class
are “liblinear”, “newton-cg”, “lbfgs”, “sag” and “saga”:
The solver “liblinear” uses a coordinate descent (CD) algorithm, and relies
on the excellent C++ , which is shipped with
scikit-learn. However, the CD algorithm implemented in liblinear cannot learn
a true multinomial (multiclass) instead, the optimization problem is
decomposed in a “one-vs-rest” fashion so separate binary classifiers are
trained for all classes. This happens under the hood, so
instances using this solver behave as multiclass
classifiers. For L1 penalization
calculate the lower bound for C in order to get a non “null” (all feature
weights to zero) model.
The “lbfgs”, “sag” and “newton-cg” solvers only support L2 penalization and
are found to converge faster for some high dimensional data. Setting
multi_class to “multinomial” with these solvers learns a true multinomial
logistic regression model , which means that its probability estimates
should be better calibrated than the default “one-vs-rest” setting.
The “sag” solver uses a Stochastic Average Gradient descent . It is faster
than other solvers for large datasets, when both the number of samples and the
number of features are large.
The “saga” solver
is a variant of “sag” that also supports the
non-smooth penalty=”l1” option. This is therefore the solver of choice
for sparse multinomial logistic regression.
In a nutshell, one may choose the solver with the following rules:
L1 penalty
“liblinear” or “saga”
Multinomial loss
“lbfgs”, “sag”, “saga” or “newton-cg”
Very Large dataset (n_samples)
“sag” or “saga”
The “saga” solver is often the best choice. The “liblinear” solver is
used by default for historical reasons.
For large dataset, you may also consider using
with ‘log’ loss.
Differences from liblinear:
There might be a difference in the scores obtained between
with solver=liblinear
or LinearSVC and the external liblinear library directly,
when fit_intercept=False and the fit coef_ (or) the data to
be predicted are zeroes. This is because for the sample(s) with
decision_function zero,
and LinearSVC
predict the negative class, while liblinear predicts the positive class.
Note that a model with fit_intercept=False and having many samples with
decision_function zero, is likely to be a underfit, bad model and you are
advised to set fit_intercept=True and increase the intercept_scaling.
Feature selection with sparse logistic regression
A logistic regression with L1 penalty yields sparse models, and can
thus be used to perform feature selection, as detailed in
implements Logistic Regression with
builtin cross-validation to find out the optimal C parameter.
“newton-cg”, “sag”, “saga” and “lbfgs” solvers are found to be faster
for high-dimensional dense data, due to warm-starting. For the
multiclass case, if multi_class option is set to “ovr”, an optimal C
is obtained for each class and if the multi_class option is set to
“multinomial”, an optimal C is obtained by minimizing the cross-entropy
References:
1.1.12. Stochastic Gradient Descent - SGD
Stochastic gradient descent is a simple yet very efficient approach
to fit linear models. It is particularly useful when the number of samples
(and the number of features) is very large.
The partial_fit method allows only/out-of-core learning.
The classes
functionality to fit linear models for classification and regression
using different (convex) loss functions and different penalties.
E.g., with loss=&log&,
fits a logistic regression model,
while with loss=&hinge& it fits a linear support vector machine (SVM).
References
1.1.13. Perceptron
is another simple algorithm suitable for large scale
learning. By default:
It does not require a learning rate.
It is not regularized (penalized).
It updates its model only on mistakes.
The last characteristic implies that the Perceptron is slightly faster to
train than SGD with the hinge loss and that the resulting models are
1.1.14. Passive Aggressive Algorithms
The passive-aggressive algorithms are a family of algorithms for large-scale
learning. They are similar to the Perceptron in that they do not require a
learning rate. However, contrary to the Perceptron, they include a
regularization parameter C.
For classification,
can be used with
loss='hinge' (PA-I) or loss='squared_hinge' (PA-II).
For regression,
can be used with
loss='epsilon_insensitive' (PA-I) or
loss='squared_epsilon_insensitive' (PA-II).
References:
K. Crammer, O. Dekel, J. Keshat, S. Shalev-Shwartz, Y. Singer - JMLR 7 (2006)
1.1.15. Robustness regression: outliers and modeling errors
Robust regression is interested in fitting a regression model in the
presence of corrupt data: either outliers, or error in the model.
1.1.15.1. Different scenario and useful concepts
There are different things to keep in mind when dealing with data
corrupted by outliers:
Outliers in X or in y?
Fraction of outliers versus amplitude of error
The number of outlying points matters, but also how much they are
An important notion of robust fitting is that of breakdown point: the
fraction of data that can be outlying for the fit to start missing the
inlying data.
Note that in general, robust fitting in high-dimensional setting (large
n_features) is very hard. The robust models here will probably not work
in these settings.
Trade-offs: which estimator?
Scikit-learn provides 3 robust regression estimators:
should be faster than
unless the number of samples are very large, i.e n_samples && n_features.
This is because
fit on smaller subsets of the data. However, both
are unlikely to be as robust as
for the default parameters.
is faster than
and scales much better with the number of samples
will deal better with large
outliers in the y direction (most common situation)
will cope better with
medium-size outliers in the X direction, but this property will
disappear in large dimensional settings.
When in doubt, use
1.1.15.2. RANSAC: RANdom SAmple Consensus
RANSAC (RANdom SAmple Consensus) fits a model from random subsets of
inliers from the complete data set.
RANSAC is a non-deterministic algorithm producing only a reasonable result with
a certain probability, which is dependent on the number of iterations (see
max_trials parameter). It is typically used for linear and non-linear
regression problems and is especially popular in the fields of photogrammetric
computer vision.
The algorithm splits the complete input sample data into a set of inliers,
which may be subject to noise, and outliers, which are e.g. caused by erroneous
measurements or invalid hypotheses about the data. The resulting model is then
estimated only from the determined inliers.
1.1.15.2.1. Details of the algorithm
Each iteration performs the following steps:
Select min_samples random samples from the original data and check
whether the set of data is valid (see is_data_valid).
Fit a model to the random subset (base_estimator.fit) and check
whether the estimated model is valid (see is_model_valid).
Classify all data as inliers or outliers by calculating the residuals
to the estimated model (base_estimator.predict(X) - y) - all data
samples with absolute residuals smaller than the residual_threshold
are considered as inliers.
Save fitted model as best model if number of inlier samples is
maximal. In case the current estimated model has the same number of
inliers, it is only considered as the best model if it has better score.
These steps are performed either a maximum number of times (max_trials) or
until one of the special stop criteria are met (see stop_n_inliers and
stop_score). The final model is estimated using all inlier samples (consensus
set) of the previously determined best model.
The is_data_valid and is_model_valid functions allow to identify and reject
degenerate combinations of random sub-samples. If the estimated model is not
needed for identifying degenerate cases, is_data_valid should be used as it
is called prior to fitting the model and thus leading to better computational
performance.
References:
Martin A. Fischler and Robert C. Bolles - SRI International (1981)
Sunglok Choi, Taemin Kim and Wonpil Yu - BMVC (2009)
1.1.15.3. Theil-Sen estimator: generalized-median-based estimator
estimator uses a generalization of the median in
multiple dimensions. It is thus robust to multivariate outliers. Note however
that the robustness of the estimator decreases quickly with the dimensionality
of the problem. It looses its robustness properties and becomes no
better than an ordinary least squares in high dimension.
References:
The implementation of
in scikit-learn follows a
generalization to a multivariate linear regression model
spatial median which is a generalization of the median to multiple
dimensions .
In terms of time and space complexity, Theil-Sen scales according to
which makes it infeasible to be applied exhaustively to problems with a
large number of samples and features. Therefore, the magnitude of a
subpopulation can be chosen to limit the time and space complexity by
considering only a random subset of all possible combinations.
References:
1.1.15.4. Huber Regression
is different to
because it applies a
linear loss to samples that are classified as outliers.
A sample is classified as an inlier if the absolute error of that sample is
lesser than a certain threshold. It differs from
because it does not ignore the effect of the outliers
but gives a lesser weight to them.
The loss function that
minimizes is given by
It is advised to set the parameter epsilon to 1.35 to achieve 95% statistical efficiency.
1.1.15.5. Notes
differs from using
with loss set to huber
in the following ways.
is scaling invariant. Once epsilon is set, scaling X and y
down or up by different values would produce the same robustness to outliers as before.
as compared to
where epsilon has to be set again when X and y are
should be more efficient to use on data with small number of
samples while
needs a number of passes on the training data to
produce the same robustness.
References:
Peter J. Huber, Elvezio M. Ronchetti: Robust Statistics, Concomitant scale estimates, pg 172
Also, this estimator is different from the R implementation of Robust Regression
() because the R implementation does a weighted least
squares implementation with weights given to each sample on the basis of how much the residual is
greater than a certain threshold.
1.1.16. Polynomial regression: extending linear models with basis functions
One common pattern within machine learning is to use linear models trained
on nonlinear functions of the data.
This approach maintains the generally
fast performance of linear methods, while allowing them to fit a much wider
range of data.
For example, a simple linear regression can be extended by constructing
polynomial features from the coefficients.
In the standard linear
regression case, you might have a model that looks like this for
two-dimensional data:
If we want to fit a paraboloid to the data instead of a plane, we can combine
the features in second-order polynomials, so that the model looks like this:
The (sometimes surprising) observation is that this is still a linear model:
to see this, imagine creating a new variable
With this re-labeling of the data, our problem can be written
We see that the resulting polynomial regression is in the same class of
linear models we’d considered above (i.e. the model is linear in )
and can be solved by the same techniques.
By considering linear fits within
a higher-dimensional space built with these basis functions, the model has the
flexibility to fit a much broader range of data.
Here is an example of applying this idea to one-dimensional data, using
polynomial features of varying degrees:
This figure is created using the
preprocessor.
This preprocessor transforms an input data matrix into a new data matrix
of a given degree.
It can be used as follows:
&&& from sklearn.preprocessing import PolynomialFeatures
&&& import numpy as np
&&& X = np.arange(6).reshape(3, 2)
array([[0, 1],
&&& poly = PolynomialFeatures(degree=2)
&&& poly.fit_transform(X)
The features of X have been transformed from
, and can now be used within
any linear model.
This sort of preprocessing can be streamlined with the
tools. A single object representing a simple
polynomial regression can be created and used as follows:
&&& from sklearn.preprocessing import PolynomialFeatures
&&& from sklearn.linear_model import LinearRegression
&&& from sklearn.pipeline import Pipeline
&&& import numpy as np
&&& model = Pipeline([('poly', PolynomialFeatures(degree=3)),
('linear', LinearRegression(fit_intercept=False))])
&&& # fit to an order-3 polynomial data
&&& x = np.arange(5)
&&& y = 3 - 2 * x + x ** 2 - x ** 3
&&& model = model.fit(x[:, np.newaxis], y)
&&& model.named_steps['linear'].coef_
array([ 3., -2.,
The linear model trained on polynomial features is able to exactly recover
the input polynomial coefficients.
In some cases it’s not necessary to include higher powers of any single feature,
but only the so-called interaction features
that multiply together at most
distinct features.
These can be gotten from
with the setting
interaction_only=True.
For example, when dealing with boolean features,
represents the conjunction of two booleans.
This way, we can solve the XOR problem with a linear classifier:
&&& from sklearn.linear_model import Perceptron
&&& from sklearn.preprocessing import PolynomialFeatures
&&& import numpy as np
&&& X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
&&& y = X[:, 0] ^ X[:, 1]
array([0, 1, 1, 0])
&&& X = PolynomialFeatures(interaction_only=True).fit_transform(X).astype(int)
array([[1, 0, 0, 0],
[1, 0, 1, 0],
[1, 1, 0, 0],
[1, 1, 1, 1]])
&&& clf = Perceptron(fit_intercept=False, max_iter=10, tol=None,
shuffle=False).fit(X, y)
And the classifier “predictions” are perfect:
&&& clf.predict(X)
array([0, 1, 1, 0])
&&& clf.score(X, y)