machine-learningmathematical-optimization
In the regularization,why we use θ^2 rather than θ？
The regularization is lambda*sum(θ^2)
Solution
I've already answered this in your previous question (see last paragraph), but I'll try again.
The problem regularizing with sum(θ) is that you may have θ parameters that cancel each other
Example:
θ_1 = +1000000
θ_2 = -1000001
The sum(θ) here is +1000000 -1000001 = -1 which is small
The sum(θ²) is 1000000² + (-1000001)² which is very big.
If you use sum(θ) you may end up without regularization (which was the goal) because of large θ values that escaped the regularization because the terms cancel each other out.
You may use sum(|θ|) depending on your search/optimisation algorithm. But I know θ² (L2 norm) to be popular and works well with gradient descent.
- StratifiedKFold vs KFold in scikit-learn
- Training a Keras model to identify leap years
- Ideas for Extracting Blade Tip Coordinates from masked Wind Turbine Image
- Macro VS Micro VS Weighted VS Samples F1 Score
- Doing PyWavelets calculation on GPU
- Training loss increases instead of decrease with epochs
- How to save a Dataset in multiple shards using `tf.data.Dataset.save`
- why explain logit as 'unscaled log probabililty' in sotfmax_cross_entropy_with_logits?
- What is the loss function used in Trainer from the Transformers library of Hugging Face?
- Using features extracted using a pretrained CNN as new features for an CNN/NN
- InvalidArgumentError: No DNN in stream executor while training a TensorFlow RetinaNet model on Google Colab
- ALS (Alternating Least Square) algorithm in multiple rankings for a user
- How does one set the pad token correctly (not to eos) during fine-tuning to avoid model not predicting EOS?
- How to create image of confusion matrix in Python
- Cross-validation with nb method
- GPU utilization almost always 0 during training Hugging Face Transformer
- The “Forward/Backward Passage Size” is too large for the pytorch model (Yolov3)
- How many images(minimum) should be there in each classes for training YOLO?
- Why do neural networks work so well?
- Will larger batch size make computation time less in machine learning?
- Why KL divergence is negative in Pytorch?
- Creating a voice identification system using machine learning
- Forward pass with all samples
- override pytorch Dataset efficiently
- fit method in sklearn
- Calibrating Probabilities in lightgbm or XGBoost
- Implementation of F1-score, IOU and Dice Score
- Is it ok to have the training history very similar to the validation history?
- How to understand Shapley value for binary classification problem?
- Stochastic Gradient Descent for Logistic Regression always returns a cost of Inf and weight vector never gets any closer