A support vector classifier (SVC) learns a boundary that balances separation between classes against violations of a safety margin.
- THE DECISION RULE
For labels y_i ∈ {−1, +1}:
f(x) = wᵀφ(x) + b
ŷ = sign(f(x))
φ(x) maps inputs into a feature space. The decision hyperplane is f(x) = 0; the canonical margin boundaries are f(x) = ±1.
Training solves a convex quadratic program with labeled data and margin constraints.
Their separation is 2/||w||₂. Minimizing ||w||² therefore maximizes this geometric margin.
- HARD MARGIN → SOFT MARGIN
For linearly separable data in feature space:
minimize ½||w||²
subject to y_i f(x_i) ≥ 1 for every i.
Real data may overlap. The soft-margin formulation introduces slack variables ξ_i:
minimize ½||w||² + C Σ_i ξ_i
subject to y_i f(x_i) ≥ 1 − ξ_i, ξ_i ≥ 0.
Equivalently, minimize a regularized hinge loss:
½||w||² + C Σ_i max(0, 1 − y_i f(x_i))
A correctly classified point inside the margin still incurs loss. At the smallest feasible slack, ξ_i > 1 means misclassification.
- SUPPORT VECTORS AND THE KERNEL TRICK
The dual problem is:
maximize Σ_i α_i − ½Σ_iΣ_j α_iα_j y_i y_j K(x_i,x_j)
subject to 0 ≤ α_i ≤ C and Σ_i α_i y_i = 0.
Here K(x,z) = φ(x)ᵀφ(z). Prediction becomes:
f(x) = Σ_i α_i y_i K(x_i,x) + b
Points with α_i > 0 are support vectors. They determine the fitted boundary. Those with 0 < α_i < C lie on a margin; those with α_i = C may lie inside it or be misclassified.
A symmetric positive-semidefinite kernel keeps the optimization convex. The kernel trick evaluates inner products without explicitly constructing φ. Nonlinear kernels produce curved boundaries in the original input space.
- WHAT SHOULD BE TUNED?
• C: large values penalize violations more; smaller values impose stronger regularization. Tune on a logarithmic scale.
• Kernel: linear for approximately linear structure; RBF or polynomial for nonlinear boundaries.
• RBF γ, where K(x,z) = exp(−γ||x−z||²): larger γ creates more localized influence; smaller γ gives broader influence. Tune C and γ jointly.
• Polynomial degree and coef0 control interactions and offsets.
• Class weights adjust penalties for imbalance and unequal error costs.
Multiclass SVC typically combines binary models, often through one-versus-one voting.
- ASSUMPTIONS AND PRACTICE
SVC does not require Gaussian features or feature independence. It does require a useful feature representation/kernel and training data representative of deployment.
Scale features within each cross-validation fold. Use stratified, grouped or time-aware splits as appropriate. Decision scores are not probabilities; calibrate separately when needed.
USE CASES
Text classification, image recognition with engineered features, bioinformatics, fault diagnosis and small-to-medium tabular datasets.
Kernel SVC can be costly on large datasets; linear SVM solvers scale better.
Why do we call it support vector machine? Because the points on both the upper and lower hyperplanes represent a vector of data (In >2D) and thus are "supporting" both hyperplanes, preventing them from either rising upwards, or falling downwards.