work/smartcore
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge potential next release v0.4 (#187) Breaking Changes

Browse Source 
* First draft of the new n-dimensional arrays + NB use case
* Improves default implementation of multiple Array methods
* Refactors tree methods
* Adds matrix decomposition routines
* Adds matrix decomposition methods to ndarray and nalgebra bindings
* Refactoring + linear regression now uses array2
* Ridge & Linear regression
* LBFGS optimizer & logistic regression
* LBFGS optimizer & logistic regression
* Changes linear methods, metrics and model selection methods to new n-dimensional arrays
* Switches KNN and clustering algorithms to new n-d array layer
* Refactors distance metrics
* Optimizes knn and clustering methods
* Refactors metrics module
* Switches decomposition methods to n-dimensional arrays
* Linalg refactoring - cleanup rng merge (#172)
* Remove legacy DenseMatrix and BaseMatrix implementation. Port the new Number, FloatNumber and Array implementation into module structure.
* Exclude AUC metrics. Needs reimplementation
* Improve developers walkthrough

New traits system in place at `src/numbers` and `src/linalg`
Co-authored-by: Lorenzo <tunedconsulting@gmail.com>

* Provide SupervisedEstimator with a constructor to avoid explicit dynamical box allocation in 'cross_validate' and 'cross_validate_predict' as required by the use of 'dyn' as per Rust 2021
* Implement getters to use as_ref() in src/neighbors
* Implement getters to use as_ref() in src/naive_bayes
* Implement getters to use as_ref() in src/linear
* Add Clone to src/naive_bayes
* Change signature for cross_validate and other model_selection functions to abide to use of dyn in Rust 2021
* Implement ndarray-bindings. Remove FloatNumber from implementations
* Drop nalgebra-bindings support (as decided in conf-call to go for ndarray)
* Remove benches. Benches will have their own repo at smartcore-benches
* Implement SVC
* Implement SVC serialization. Move search parameters in dedicated module
* Implement SVR. Definitely too slow
* Fix compilation issues for wasm (#202)

Co-authored-by: Luis Moreno <morenol@users.noreply.github.com>
* Fix tests (#203)

* Port linalg/traits/stats.rs
* Improve methods naming
* Improve Display for DenseMatrix

Co-authored-by: Montana Low <montanalow@users.noreply.github.com>
Co-authored-by: VolodymyrOrlov <volodymyr.orlov@gmail.com>
This commit is contained in:
Lorenzo
2022-10-31 10:44:57 +00:00
committed by morenol
parent a32eb66a6a
commit a7fa0585eb
110 changed files with 10327 additions and 9107 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/linear/logistic_regression.rs
+264 -212
View File
@@ -10,7 +10,7 @@
//! Example:
//!
//! ```
//! use smartcore::linalg::naive::dense_matrix::*;
//! use smartcore::linalg::basic::matrix::DenseMatrix;
//! use smartcore::linear::logistic_regression::*;
//!
//! //Iris data
@@ -36,8 +36,8 @@
//! &[6.6, 2.9, 4.6, 1.3],
//! &[5.2, 2.7, 3.9, 1.4],
//! ]);
//! let y: Vec<f64> = vec![
//! 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
//! let y: Vec<i32> = vec![
//! 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
//! ];
//!
//! let lr = LogisticRegression::fit(&x, &y, Default::default()).unwrap();
@@ -54,14 +54,17 @@
//! <script id="MathJax-script" async src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
use std::cmp::Ordering;
use std::fmt::Debug;
use std::marker::PhantomData;
#[cfg(feature = "serde")]
use serde::{Deserialize, Serialize};
use crate::api::{Predictor, SupervisedEstimator};
use crate::error::Failed;
use crate::linalg::Matrix;
use crate::math::num::RealNumber;
use crate::linalg::basic::arrays::{Array1, Array2, MutArrayView1};
use crate::numbers::basenum::Number;
use crate::numbers::floatnum::FloatNumber;
use crate::numbers::realnum::RealNumber;
use crate::optimization::first_order::lbfgs::LBFGS;
use crate::optimization::first_order::{FirstOrderOptimizer, OptimizerResult};
use crate::optimization::line_search::Backtracking;
@@ -84,7 +87,7 @@ impl Default for LogisticRegressionSolverName {
/// Logistic Regression parameters
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
#[derive(Debug, Clone)]
pub struct LogisticRegressionParameters<T: RealNumber> {
pub struct LogisticRegressionParameters<T: Number + FloatNumber> {
#[cfg_attr(feature = "serde", serde(default))]
/// Solver to use for estimation of regression coefficients.
pub solver: LogisticRegressionSolverName,
@@ -96,7 +99,7 @@ pub struct LogisticRegressionParameters<T: RealNumber> {
/// Logistic Regression grid search parameters
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
#[derive(Debug, Clone)]
pub struct LogisticRegressionSearchParameters<T: RealNumber> {
pub struct LogisticRegressionSearchParameters<T: Number> {
#[cfg_attr(feature = "serde", serde(default))]
/// Solver to use for estimation of regression coefficients.
pub solver: Vec<LogisticRegressionSolverName>,
@@ -106,13 +109,13 @@ pub struct LogisticRegressionSearchParameters<T: RealNumber> {
}
/// Logistic Regression grid search iterator
pub struct LogisticRegressionSearchParametersIterator<T: RealNumber> {
pub struct LogisticRegressionSearchParametersIterator<T: Number> {
logistic_regression_search_parameters: LogisticRegressionSearchParameters<T>,
current_solver: usize,
current_alpha: usize,
}
impl<T: RealNumber> IntoIterator for LogisticRegressionSearchParameters<T> {
impl<T: Number + FloatNumber> IntoIterator for LogisticRegressionSearchParameters<T> {
type Item = LogisticRegressionParameters<T>;
type IntoIter = LogisticRegressionSearchParametersIterator<T>;
@@ -125,7 +128,7 @@ impl<T: RealNumber> IntoIterator for LogisticRegressionSearchParameters<T> {
}
}
impl<T: RealNumber> Iterator for LogisticRegressionSearchParametersIterator<T> {
impl<T: Number + FloatNumber> Iterator for LogisticRegressionSearchParametersIterator<T> {
type Item = LogisticRegressionParameters<T>;
fn next(&mut self) -> Option<Self::Item> {
@@ -155,7 +158,7 @@ impl<T: RealNumber> Iterator for LogisticRegressionSearchParametersIterator<T> {
}
}
impl<T: RealNumber> Default for LogisticRegressionSearchParameters<T> {
impl<T: Number + FloatNumber> Default for LogisticRegressionSearchParameters<T> {
fn default() -> Self {
let default_params = LogisticRegressionParameters::default();
@@ -169,36 +172,50 @@ impl<T: RealNumber> Default for LogisticRegressionSearchParameters<T> {
/// Logistic Regression
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
#[derive(Debug)]
pub struct LogisticRegression<T: RealNumber, M: Matrix<T>> {
coefficients: M,
intercept: M,
classes: Vec<T>,
pub struct LogisticRegression<
TX: Number + FloatNumber + RealNumber,
TY: Number + Ord,
X: Array2<TX>,
Y: Array1<TY>,
> {
coefficients: Option<X>,
intercept: Option<X>,
classes: Option<Vec<TY>>,
num_attributes: usize,
num_classes: usize,
_phantom_tx: PhantomData<TX>,
_phantom_y: PhantomData<Y>,
}
trait ObjectiveFunction<T: RealNumber, M: Matrix<T>> {
fn f(&self, w_bias: &M) -> T;
fn df(&self, g: &mut M, w_bias: &M);
trait ObjectiveFunction<T: Number + FloatNumber, X: Array2<T>> {
///
fn f(&self, w_bias: &[T]) -> T;
fn partial_dot(w: &M, x: &M, v_col: usize, m_row: usize) -> T {
///
#[allow(clippy::ptr_arg)]
fn df(&self, g: &mut Vec<T>, w_bias: &Vec<T>);
///
#[allow(clippy::ptr_arg)]
fn partial_dot(w: &[T], x: &X, v_col: usize, m_row: usize) -> T {
let mut sum = T::zero();
let p = x.shape().1;
for i in 0..p {
sum += x.get(m_row, i) * w.get(0, i + v_col);
sum += *x.get((m_row, i)) * w[i + v_col];
}
sum + w.get(0, p + v_col)
sum + w[p + v_col]
}
}
struct BinaryObjectiveFunction<'a, T: RealNumber, M: Matrix<T>> {
x: &'a M,
struct BinaryObjectiveFunction<'a, T: Number + FloatNumber, X: Array2<T>> {
x: &'a X,
y: Vec<usize>,
alpha: T,
_phantom_t: PhantomData<T>,
}
impl<T: RealNumber> LogisticRegressionParameters<T> {
impl<T: Number + FloatNumber> LogisticRegressionParameters<T> {
/// Solver to use for estimation of regression coefficients.
pub fn with_solver(mut self, solver: LogisticRegressionSolverName) -> Self {
self.solver = solver;
@@ -211,7 +228,7 @@ impl<T: RealNumber> LogisticRegressionParameters<T> {
}
}
impl<T: RealNumber> Default for LogisticRegressionParameters<T> {
impl<T: Number + FloatNumber> Default for LogisticRegressionParameters<T> {
fn default() -> Self {
LogisticRegressionParameters {
solver: LogisticRegressionSolverName::default(),
@@ -220,29 +237,39 @@ impl<T: RealNumber> Default for LogisticRegressionParameters<T> {
}
}
impl<T: RealNumber, M: Matrix<T>> PartialEq for LogisticRegression<T, M> {
impl<TX: Number + FloatNumber + RealNumber, TY: Number + Ord, X: Array2<TX>, Y: Array1<TY>>
PartialEq for LogisticRegression<TX, TY, X, Y>
{
fn eq(&self, other: &Self) -> bool {
if self.num_classes != other.num_classes
|| self.num_attributes != other.num_attributes
|| self.classes.len() != other.classes.len()
|| self.classes().len() != other.classes().len()
{
false
} else {
for i in 0..self.classes.len() {
if (self.classes[i] - other.classes[i]).abs() > T::epsilon() {
for i in 0..self.classes().len() {
if self.classes()[i] != other.classes()[i] {
return false;
}
}
self.coefficients == other.coefficients && self.intercept == other.intercept
self.coefficients()
.iterator(0)
.zip(other.coefficients().iterator(0))
.all(|(&a, &b)| (a - b).abs() <= TX::epsilon())
&& self
.intercept()
.iterator(0)
.zip(other.intercept().iterator(0))
.all(|(&a, &b)| (a - b).abs() <= TX::epsilon())
}
}
}
impl<'a, T: RealNumber, M: Matrix<T>> ObjectiveFunction<T, M>
for BinaryObjectiveFunction<'a, T, M>
impl<'a, T: Number + FloatNumber, X: Array2<T>> ObjectiveFunction<T, X>
for BinaryObjectiveFunction<'a, T, X>
{
fn f(&self, w_bias: &M) -> T {
fn f(&self, w_bias: &[T]) -> T {
let mut f = T::zero();
let (n, p) = self.x.shape();
@@ -253,18 +280,17 @@ impl<'a, T: RealNumber, M: Matrix<T>> ObjectiveFunction<T, M>
if self.alpha > T::zero() {
let mut w_squared = T::zero();
for i in 0..p {
let w = w_bias.get(0, i);
w_squared += w * w;
for w_bias_i in w_bias.iter().take(p) {
w_squared += *w_bias_i * *w_bias_i;
}
f += T::half() * self.alpha * w_squared;
f += T::from_f64(0.5).unwrap() * self.alpha * w_squared;
}
f
}
fn df(&self, g: &mut M, w_bias: &M) {
g.copy_from(&M::zeros(1, g.shape().1));
fn df(&self, g: &mut Vec<T>, w_bias: &Vec<T>) {
g.copy_from(&Vec::zeros(g.len()));
let (n, p) = self.x.shape();
@@ -272,86 +298,79 @@ impl<'a, T: RealNumber, M: Matrix<T>> ObjectiveFunction<T, M>
let wx = BinaryObjectiveFunction::partial_dot(w_bias, self.x, 0, i);
let dyi = (T::from(self.y[i]).unwrap()) - wx.sigmoid();
for j in 0..p {
g.set(0, j, g.get(0, j) - dyi * self.x.get(i, j));
for (j, g_j) in g.iter_mut().enumerate().take(p) {
*g_j -= dyi * *self.x.get((i, j));
}
g.set(0, p, g.get(0, p) - dyi);
g[p] -= dyi;
}
if self.alpha > T::zero() {
for i in 0..p {
let w = w_bias.get(0, i);
g.set(0, i, g.get(0, i) + self.alpha * w);
let w = w_bias[i];
g[i] += self.alpha * w;
}
}
}
}
struct MultiClassObjectiveFunction<'a, T: RealNumber, M: Matrix<T>> {
x: &'a M,
struct MultiClassObjectiveFunction<'a, T: Number + FloatNumber, X: Array2<T>> {
x: &'a X,
y: Vec<usize>,
k: usize,
alpha: T,
_phantom_t: PhantomData<T>,
}
impl<'a, T: RealNumber, M: Matrix<T>> ObjectiveFunction<T, M>
for MultiClassObjectiveFunction<'a, T, M>
impl<'a, T: Number + FloatNumber + RealNumber, X: Array2<T>> ObjectiveFunction<T, X>
for MultiClassObjectiveFunction<'a, T, X>
{
fn f(&self, w_bias: &M) -> T {
fn f(&self, w_bias: &[T]) -> T {
let mut f = T::zero();
let mut prob = M::zeros(1, self.k);
let mut prob = vec![T::zero(); self.k];
let (n, p) = self.x.shape();
for i in 0..n {
for j in 0..self.k {
prob.set(
0,
j,
MultiClassObjectiveFunction::partial_dot(w_bias, self.x, j * (p + 1), i),
);
for (j, prob_j) in prob.iter_mut().enumerate().take(self.k) {
*prob_j = MultiClassObjectiveFunction::partial_dot(w_bias, self.x, j * (p + 1), i);
}
prob.softmax_mut();
f -= prob.get(0, self.y[i]).ln();
f -= prob[self.y[i]].ln();
}
if self.alpha > T::zero() {
let mut w_squared = T::zero();
for i in 0..self.k {
for j in 0..p {
let wi = w_bias.get(0, i * (p + 1) + j);
let wi = w_bias[i * (p + 1) + j];
w_squared += wi * wi;
}
}
f += T::half() * self.alpha * w_squared;
f += T::from_f64(0.5).unwrap() * self.alpha * w_squared;
}
f
}
fn df(&self, g: &mut M, w: &M) {
g.copy_from(&M::zeros(1, g.shape().1));
fn df(&self, g: &mut Vec<T>, w: &Vec<T>) {
g.copy_from(&Vec::zeros(g.len()));
let mut prob = M::zeros(1, self.k);
let mut prob = vec![T::zero(); self.k];
let (n, p) = self.x.shape();
for i in 0..n {
for j in 0..self.k {
prob.set(
0,
j,
MultiClassObjectiveFunction::partial_dot(w, self.x, j * (p + 1), i),
);
for (j, prob_j) in prob.iter_mut().enumerate().take(self.k) {
*prob_j = MultiClassObjectiveFunction::partial_dot(w, self.x, j * (p + 1), i);
}
prob.softmax_mut();
for j in 0..self.k {
let yi = (if self.y[i] == j { T::one() } else { T::zero() }) - prob.get(0, j);
let yi = (if self.y[i] == j { T::one() } else { T::zero() }) - prob[j];
for l in 0..p {
let pos = j * (p + 1);
g.set(0, pos + l, g.get(0, pos + l) - yi * self.x.get(i, l));
g[pos + l] -= yi * *self.x.get((i, l));
}
g.set(0, j * (p + 1) + p, g.get(0, j * (p + 1) + p) - yi);
g[j * (p + 1) + p] -= yi;
}
}
@@ -359,46 +378,57 @@ impl<'a, T: RealNumber, M: Matrix<T>> ObjectiveFunction<T, M>
for i in 0..self.k {
for j in 0..p {
let pos = i * (p + 1);
let wi = w.get(0, pos + j);
g.set(0, pos + j, g.get(0, pos + j) + self.alpha * wi);
let wi = w[pos + j];
g[pos + j] += self.alpha * wi;
}
}
}
}
}
impl<T: RealNumber, M: Matrix<T>>
SupervisedEstimator<M, M::RowVector, LogisticRegressionParameters<T>>
for LogisticRegression<T, M>
impl<TX: Number + FloatNumber + RealNumber, TY: Number + Ord, X: Array2<TX>, Y: Array1<TY>>
SupervisedEstimator<X, Y, LogisticRegressionParameters<TX>>
for LogisticRegression<TX, TY, X, Y>
{
fn fit(
x: &M,
y: &M::RowVector,
parameters: LogisticRegressionParameters<T>,
) -> Result<Self, Failed> {
fn new() -> Self {
Self {
coefficients: Option::None,
intercept: Option::None,
classes: Option::None,
num_attributes: 0,
num_classes: 0,
_phantom_tx: PhantomData,
_phantom_y: PhantomData,
}
}
fn fit(x: &X, y: &Y, parameters: LogisticRegressionParameters<TX>) -> Result<Self, Failed> {
LogisticRegression::fit(x, y, parameters)
}
}
impl<T: RealNumber, M: Matrix<T>> Predictor<M, M::RowVector> for LogisticRegression<T, M> {
fn predict(&self, x: &M) -> Result<M::RowVector, Failed> {
impl<TX: Number + FloatNumber + RealNumber, TY: Number + Ord, X: Array2<TX>, Y: Array1<TY>>
Predictor<X, Y> for LogisticRegression<TX, TY, X, Y>
{
fn predict(&self, x: &X) -> Result<Y, Failed> {
self.predict(x)
}
}
impl<T: RealNumber, M: Matrix<T>> LogisticRegression<T, M> {
impl<TX: Number + FloatNumber + RealNumber, TY: Number + Ord, X: Array2<TX>, Y: Array1<TY>>
LogisticRegression<TX, TY, X, Y>
{
/// Fits Logistic Regression to your data.
/// * `x` - _NxM_ matrix with _N_ observations and _M_ features in each observation.
/// * `y` - target class values
/// * `parameters` - other parameters, use `Default::default()` to set parameters to default values.
pub fn fit(
x: &M,
y: &M::RowVector,
parameters: LogisticRegressionParameters<T>,
) -> Result<LogisticRegression<T, M>, Failed> {
let y_m = M::from_row_vector(y.clone());
x: &X,
y: &Y,
parameters: LogisticRegressionParameters<TX>,
) -> Result<LogisticRegression<TX, TY, X, Y>, Failed> {
let (x_nrows, num_attributes) = x.shape();
let (_, y_nrows) = y_m.shape();
let y_nrows = y.shape();
if x_nrows != y_nrows {
return Err(Failed::fit(
@@ -406,15 +436,15 @@ impl<T: RealNumber, M: Matrix<T>> LogisticRegression<T, M> {
));
}
let classes = y_m.unique();
let classes = y.unique();
let k = classes.len();
let mut yi: Vec<usize> = vec![0; y_nrows];
for (i, yi_i) in yi.iter_mut().enumerate().take(y_nrows) {
let yc = y_m.get(0, i);
*yi_i = classes.iter().position(|c| yc == *c).unwrap();
let yc = y.get(i);
*yi_i = classes.iter().position(|c| yc == c).unwrap();
}
match k.cmp(&2) {
@@ -423,45 +453,55 @@ impl<T: RealNumber, M: Matrix<T>> LogisticRegression<T, M> {
k
))),
Ordering::Equal => {
let x0 = M::zeros(1, num_attributes + 1);
let x0 = Vec::zeros(num_attributes + 1);
let objective = BinaryObjectiveFunction {
x,
y: yi,
alpha: parameters.alpha,
_phantom_t: PhantomData,
};
let result = LogisticRegression::minimize(x0, objective);
let result = Self::minimize(x0, objective);
let weights = result.x;
let weights = X::from_iterator(result.x.into_iter(), 1, num_attributes + 1, 0);
let coefficients = weights.slice(0..1, 0..num_attributes);
let intercept = weights.slice(0..1, num_attributes..num_attributes + 1);
Ok(LogisticRegression {
coefficients: weights.slice(0..1, 0..num_attributes),
intercept: weights.slice(0..1, num_attributes..num_attributes + 1),
classes,
coefficients: Some(X::from_slice(coefficients.as_ref())),
intercept: Some(X::from_slice(intercept.as_ref())),
classes: Some(classes),
num_attributes,
num_classes: k,
_phantom_tx: PhantomData,
_phantom_y: PhantomData,
})
}
Ordering::Greater => {
let x0 = M::zeros(1, (num_attributes + 1) * k);
let x0 = Vec::zeros((num_attributes + 1) * k);
let objective = MultiClassObjectiveFunction {
x,
y: yi,
k,
alpha: parameters.alpha,
_phantom_t: PhantomData,
};
let result = LogisticRegression::minimize(x0, objective);
let weights = result.x.reshape(k, num_attributes + 1);
let result = Self::minimize(x0, objective);
let weights = X::from_iterator(result.x.into_iter(), k, num_attributes + 1, 0);
let coefficients = weights.slice(0..k, 0..num_attributes);
let intercept = weights.slice(0..k, num_attributes..num_attributes + 1);
Ok(LogisticRegression {
coefficients: weights.slice(0..k, 0..num_attributes),
intercept: weights.slice(0..k, num_attributes..num_attributes + 1),
classes,
coefficients: Some(X::from_slice(coefficients.as_ref())),
intercept: Some(X::from_slice(intercept.as_ref())),
classes: Some(classes),
num_attributes,
num_classes: k,
_phantom_tx: PhantomData,
_phantom_y: PhantomData,
})
}
}
@@ -469,17 +509,17 @@ impl<T: RealNumber, M: Matrix<T>> LogisticRegression<T, M> {
/// Predict class labels for samples in `x`.
/// * `x` - _KxM_ data where _K_ is number of observations and _M_ is number of features.
pub fn predict(&self, x: &M) -> Result<M::RowVector, Failed> {
pub fn predict(&self, x: &X) -> Result<Y, Failed> {
let n = x.shape().0;
let mut result = M::zeros(1, n);
let mut result = Y::zeros(n);
if self.num_classes == 2 {
let y_hat: Vec<T> = x.ab(false, &self.coefficients, true).get_col_as_vec(0);
let intercept = self.intercept.get(0, 0);
for (i, y_hat_i) in y_hat.iter().enumerate().take(n) {
let y_hat = x.ab(false, self.coefficients(), true);
let intercept = *self.intercept().get((0, 0));
for (i, y_hat_i) in y_hat.iterator(0).enumerate().take(n) {
result.set(
0,
i,
self.classes[if (*y_hat_i + intercept).sigmoid() > T::half() {
self.classes()[if RealNumber::sigmoid(*y_hat_i + intercept) > RealNumber::half()
{
1
} else {
0
@@ -487,40 +527,48 @@ impl<T: RealNumber, M: Matrix<T>> LogisticRegression<T, M> {
);
}
} else {
let mut y_hat = x.matmul(&self.coefficients.transpose());
let mut y_hat = x.matmul(&self.coefficients().transpose());
for r in 0..n {
for c in 0..self.num_classes {
y_hat.set(r, c, y_hat.get(r, c) + self.intercept.get(c, 0));
y_hat.set((r, c), *y_hat.get((r, c)) + *self.intercept().get((c, 0)));
}
}
let class_idxs = y_hat.argmax();
let class_idxs = y_hat.argmax(1);
for (i, class_i) in class_idxs.iter().enumerate().take(n) {
result.set(0, i, self.classes[*class_i]);
result.set(i, self.classes()[*class_i]);
}
}
Ok(result.to_row_vector())
Ok(result)
}
/// Get estimates regression coefficients
pub fn coefficients(&self) -> &M {
&self.coefficients
/// Get estimates regression coefficients, this create a sharable reference
pub fn coefficients(&self) -> &X {
self.coefficients.as_ref().unwrap()
}
/// Get estimate of intercept
pub fn intercept(&self) -> &M {
&self.intercept
/// Get estimate of intercept, this create a sharable reference
pub fn intercept(&self) -> &X {
self.intercept.as_ref().unwrap()
}
fn minimize(x0: M, objective: impl ObjectiveFunction<T, M>) -> OptimizerResult<T, M> {
let f = |w: &M| -> T { objective.f(w) };
/// Get classes, this create a sharable reference
pub fn classes(&self) -> &Vec<TY> {
self.classes.as_ref().unwrap()
}
let df = |g: &mut M, w: &M| objective.df(g, w);
fn minimize(
x0: Vec<TX>,
objective: impl ObjectiveFunction<TX, X>,
) -> OptimizerResult<TX, Vec<TX>> {
let f = |w: &Vec<TX>| -> TX { objective.f(w) };
let ls: Backtracking<T> = Backtracking {
let df = |g: &mut Vec<TX>, w: &Vec<TX>| objective.df(g, w);
let ls: Backtracking<TX> = Backtracking {
order: FunctionOrder::THIRD,
..Default::default()
};
let optimizer: LBFGS<T> = Default::default();
let optimizer: LBFGS = Default::default();
optimizer.optimize(&f, &df, &x0, &ls)
}
@@ -530,8 +578,8 @@ impl<T: RealNumber, M: Matrix<T>> LogisticRegression<T, M> {
mod tests {
use super::*;
use crate::dataset::generator::make_blobs;
use crate::linalg::naive::dense_matrix::*;
use crate::metrics::accuracy;
use crate::linalg::basic::arrays::Array;
use crate::linalg::basic::matrix::DenseMatrix;
#[test]
fn search_parameters() {
@@ -576,24 +624,17 @@ mod tests {
y: y.clone(),
k: 3,
alpha: 0.0,
_phantom_t: PhantomData,
};
let mut g: DenseMatrix<f64> = DenseMatrix::zeros(1, 9);
let mut g = vec![0f64; 9];
objective.df(
&mut g,
&DenseMatrix::row_vector_from_array(&[1., 2., 3., 4., 5., 6., 7., 8., 9.]),
);
objective.df(
&mut g,
&DenseMatrix::row_vector_from_array(&[1., 2., 3., 4., 5., 6., 7., 8., 9.]),
);
objective.df(&mut g, &vec![1., 2., 3., 4., 5., 6., 7., 8., 9.]);
objective.df(&mut g, &vec![1., 2., 3., 4., 5., 6., 7., 8., 9.]);
assert!((g.get(0, 0) + 33.000068218163484).abs() < std::f64::EPSILON);
assert!((g[0] + 33.000068218163484).abs() < std::f64::EPSILON);
let f = objective.f(&DenseMatrix::row_vector_from_array(&[
1., 2., 3., 4., 5., 6., 7., 8., 9.,
]));
let f = objective.f(&vec![1., 2., 3., 4., 5., 6., 7., 8., 9.]);
assert!((f - 408.0052230582765).abs() < std::f64::EPSILON);
@@ -602,18 +643,14 @@ mod tests {
y: y.clone(),
k: 3,
alpha: 1.0,
_phantom_t: PhantomData,
};
let f = objective_reg.f(&DenseMatrix::row_vector_from_array(&[
1., 2., 3., 4., 5., 6., 7., 8., 9.,
]));
let f = objective_reg.f(&vec![1., 2., 3., 4., 5., 6., 7., 8., 9.]);
assert!((f - 487.5052).abs() < 1e-4);
objective_reg.df(
&mut g,
&DenseMatrix::row_vector_from_array(&[1., 2., 3., 4., 5., 6., 7., 8., 9.]),
);
assert!((g.get(0, 0).abs() - 32.0).abs() < 1e-4);
objective_reg.df(&mut g, &vec![1., 2., 3., 4., 5., 6., 7., 8., 9.]);
assert!((g[0].abs() - 32.0).abs() < 1e-4);
}
#[cfg_attr(target_arch = "wasm32", wasm_bindgen_test::wasm_bindgen_test)]
@@ -643,18 +680,19 @@ mod tests {
x: &x,
y: y.clone(),
alpha: 0.0,
_phantom_t: PhantomData,
};
let mut g: DenseMatrix<f64> = DenseMatrix::zeros(1, 3);
let mut g = vec![0f64; 3];
objective.df(&mut g, &DenseMatrix::row_vector_from_array(&[1., 2., 3.]));
objective.df(&mut g, &DenseMatrix::row_vector_from_array(&[1., 2., 3.]));
objective.df(&mut g, &vec![1., 2., 3.]);
objective.df(&mut g, &vec![1., 2., 3.]);
assert!((g.get(0, 0) - 26.051064349381285).abs() < std::f64::EPSILON);
assert!((g.get(0, 1) - 10.239000702928523).abs() < std::f64::EPSILON);
assert!((g.get(0, 2) - 3.869294270156324).abs() < std::f64::EPSILON);
assert!((g[0] - 26.051064349381285).abs() < std::f64::EPSILON);
assert!((g[1] - 10.239000702928523).abs() < std::f64::EPSILON);
assert!((g[2] - 3.869294270156324).abs() < std::f64::EPSILON);
let f = objective.f(&DenseMatrix::row_vector_from_array(&[1., 2., 3.]));
let f = objective.f(&vec![1., 2., 3.]);
assert!((f - 59.76994756647412).abs() < std::f64::EPSILON);
@@ -662,21 +700,22 @@ mod tests {
x: &x,
y: y.clone(),
alpha: 1.0,
_phantom_t: PhantomData,
};
let f = objective_reg.f(&DenseMatrix::row_vector_from_array(&[1., 2., 3.]));
let f = objective_reg.f(&vec![1., 2., 3.]);
assert!((f - 62.2699).abs() < 1e-4);
objective_reg.df(&mut g, &DenseMatrix::row_vector_from_array(&[1., 2., 3.]));
assert!((g.get(0, 0) - 27.0511).abs() < 1e-4);
assert!((g.get(0, 1) - 12.239).abs() < 1e-4);
assert!((g.get(0, 2) - 3.8693).abs() < 1e-4);
objective_reg.df(&mut g, &vec![1., 2., 3.]);
assert!((g[0] - 27.0511).abs() < 1e-4);
assert!((g[1] - 12.239).abs() < 1e-4);
assert!((g[2] - 3.8693).abs() < 1e-4);
}
#[cfg_attr(target_arch = "wasm32", wasm_bindgen_test::wasm_bindgen_test)]
#[test]
fn lr_fit_predict() {
let x = DenseMatrix::from_2d_array(&[
let x: DenseMatrix<f64> = DenseMatrix::from_2d_array(&[
&[1., -5.],
&[2., 5.],
&[3., -2.],
@@ -693,22 +732,23 @@ mod tests {
&[8., 2.],
&[9., 0.],
]);
let y: Vec<f64> = vec![0., 0., 1., 1., 2., 1., 1., 0., 0., 2., 1., 1., 0., 0., 1.];
let y: Vec<i32> = vec![0, 0, 1, 1, 2, 1, 1, 0, 0, 2, 1, 1, 0, 0, 1];
let lr = LogisticRegression::fit(&x, &y, Default::default()).unwrap();
assert_eq!(lr.coefficients().shape(), (3, 2));
assert_eq!(lr.intercept().shape(), (3, 1));
assert!((lr.coefficients().get(0, 0) - 0.0435).abs() < 1e-4);
assert!((lr.intercept().get(0, 0) - 0.1250).abs() < 1e-4);
assert!((*lr.coefficients().get((0, 0)) - 0.0435).abs() < 1e-4);
assert!(
(*lr.intercept().get((0, 0)) - 0.1250).abs() < 1e-4,
"expected to be least than 1e-4, got {}",
(*lr.intercept().get((0, 0)) - 0.1250).abs()
);
let y_hat = lr.predict(&x).unwrap();
assert_eq!(
y_hat,
vec![0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
);
assert_eq!(y_hat, vec![0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]);
}
#[cfg_attr(target_arch = "wasm32", wasm_bindgen_test::wasm_bindgen_test)]
@@ -716,14 +756,14 @@ mod tests {
fn lr_fit_predict_multiclass() {
let blobs = make_blobs(15, 4, 3);
let x = DenseMatrix::from_vec(15, 4, &blobs.data);
let y = blobs.target;
let x: DenseMatrix<f32> = DenseMatrix::from_iterator(blobs.data.into_iter(), 15, 4, 0);
let y: Vec<i32> = blobs.target.into_iter().map(|v| v as i32).collect();
let lr = LogisticRegression::fit(&x, &y, Default::default()).unwrap();
let y_hat = lr.predict(&x).unwrap();
assert!(accuracy(&y_hat, &y) > 0.9);
assert_eq!(y_hat, vec![0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2]);
let lr_reg = LogisticRegression::fit(
&x,
@@ -732,7 +772,10 @@ mod tests {
)
.unwrap();
assert!(lr_reg.coefficients().abs().sum() < lr.coefficients().abs().sum());
let reg_coeff_sum: f32 = lr_reg.coefficients().abs().iter().sum();
let coeff: f32 = lr.coefficients().abs().iter().sum();
assert!(reg_coeff_sum < coeff);
}
#[cfg_attr(target_arch = "wasm32", wasm_bindgen_test::wasm_bindgen_test)]
@@ -740,14 +783,17 @@ mod tests {
fn lr_fit_predict_binary() {
let blobs = make_blobs(20, 4, 2);
let x = DenseMatrix::from_vec(20, 4, &blobs.data);
let y = blobs.target;
let x = DenseMatrix::from_iterator(blobs.data.into_iter(), 20, 4, 0);
let y: Vec<i32> = blobs.target.into_iter().map(|v| v as i32).collect();
let lr = LogisticRegression::fit(&x, &y, Default::default()).unwrap();
let y_hat = lr.predict(&x).unwrap();
assert!(accuracy(&y_hat, &y) > 0.9);
assert_eq!(
y_hat,
vec![0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1]
);
let lr_reg = LogisticRegression::fit(
&x,
@@ -756,39 +802,43 @@ mod tests {
)
.unwrap();
assert!(lr_reg.coefficients().abs().sum() < lr.coefficients().abs().sum());
let reg_coeff_sum: f32 = lr_reg.coefficients().abs().iter().sum();
let coeff: f32 = lr.coefficients().abs().iter().sum();
assert!(reg_coeff_sum < coeff);
}
#[cfg_attr(target_arch = "wasm32", wasm_bindgen_test::wasm_bindgen_test)]
#[test]
#[cfg(feature = "serde")]
fn serde() {
let x = DenseMatrix::from_2d_array(&[
&[1., -5.],
&[2., 5.],
&[3., -2.],
&[1., 2.],
&[2., 0.],
&[6., -5.],
&[7., 5.],
&[6., -2.],
&[7., 2.],
&[6., 0.],
&[8., -5.],
&[9., 5.],
&[10., -2.],
&[8., 2.],
&[9., 0.],
]);
let y: Vec<f64> = vec![0., 0., 1., 1., 2., 1., 1., 0., 0., 2., 1., 1., 0., 0., 1.];
// TODO: serialization for the new DenseMatrix needs to be implemented
// #[cfg_attr(target_arch = "wasm32", wasm_bindgen_test::wasm_bindgen_test)]
// #[test]
// #[cfg(feature = "serde")]
// fn serde() {
// let x = DenseMatrix::from_2d_array(&[
// &[1., -5.],
// &[2., 5.],
// &[3., -2.],
// &[1., 2.],
// &[2., 0.],
// &[6., -5.],
// &[7., 5.],
// &[6., -2.],
// &[7., 2.],
// &[6., 0.],
// &[8., -5.],
// &[9., 5.],
// &[10., -2.],
// &[8., 2.],
// &[9., 0.],
// ]);
// let y: Vec<i32> = vec![0, 0, 1, 1, 2, 1, 1, 0, 0, 2, 1, 1, 0, 0, 1];
let lr = LogisticRegression::fit(&x, &y, Default::default()).unwrap();
// let lr = LogisticRegression::fit(&x, &y, Default::default()).unwrap();
let deserialized_lr: LogisticRegression<f64, DenseMatrix<f64>> =
serde_json::from_str(&serde_json::to_string(&lr).unwrap()).unwrap();
// let deserialized_lr: LogisticRegression<f64, i32, DenseMatrix<f64>, Vec<i32>> =
// serde_json::from_str(&serde_json::to_string(&lr).unwrap()).unwrap();
assert_eq!(lr, deserialized_lr);
}
// assert_eq!(lr, deserialized_lr);
// }
#[cfg_attr(target_arch = "wasm32", wasm_bindgen_test::wasm_bindgen_test)]
#[test]
@@ -815,9 +865,7 @@ mod tests {
&[6.6, 2.9, 4.6, 1.3],
&[5.2, 2.7, 3.9, 1.4],
]);
let y: Vec<f64> = vec![
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
];
let y: Vec<i32> = vec![0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1];
let lr = LogisticRegression::fit(&x, &y, Default::default()).unwrap();
let lr_reg = LogisticRegression::fit(
@@ -829,13 +877,17 @@ mod tests {
let y_hat = lr.predict(&x).unwrap();
let error: f64 = y
let error: i32 = y
.into_iter()
.zip(y_hat.into_iter())
.map(|(a, b)| (a - b).abs())
.sum();
assert!(error <= 1.0);
assert!(lr_reg.coefficients().abs().sum() < lr.coefficients().abs().sum());
assert!(error <= 1);
let reg_coeff_sum: f32 = lr_reg.coefficients().abs().iter().sum();
let coeff: f32 = lr.coefficients().abs().iter().sum();
assert!(reg_coeff_sum < coeff);
}
}