I am very new to ISTIO and would like to get clarified with my following doubts.
Details
Current AKS version 1.18.14
planning upgrade to AKS 1.19.11
Current istio version 1.7
Planning upgrade to 1.8
We are planning to upgrade Istio version from 1.7 to 1.8 in our AKS cluster 1.18.14 in our production.
But I am not sure about the proper method of upgrade to follow in production since there are multiple methods are given by Istio.
I don't have any clue about the how the current Istio setup is done in my environment and what profile settings we used as it was done long before. Could understand below are the steps followed to install istio..
Istio was installed following way:
Created manifest:
istioctl manifest generate --set profile=default -f /manifests/overlay/overlay.yaml > $HOME/generated-manifest.yaml
Installed istio:
istioctl install --set profile=default -f /manifests/overlay/overlay.yaml
Verified istio against the deployed manifest:
istioctl verify-install -f $HOME/generated-manifest.yaml
Is there any method to export all the existing settings (the one currently running) and do the upgrade?
So I am looking for a production ready approach to upgrade Istio with all existing settings in placed.
Consider replicating the environment and performing the upgrade on dev/stage first to make sure it works for you and your infrastructure.
Can be done by getting installed state custom resource and all settings:
kubectl -n istio-system get IstioOperator installed-state -o yaml > installed-state.yaml
istioctlFrom 1.7.3 to 1.8.6, this will be similar for other versions, however upgrades should be no more than 1 minor version of difference e.g. 1.5 to 1.6.
Available versions and releases can be checked in Istio Github.
1 - install istioctl version 1.8.6:
Get required binaries:
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.8.6 TARGET_ARCH=x86_64 sh -
and copy istiolctl bin:
sudo cp bin/istioctl /usr/local/bin/
2 - run istioctl version to confirm istioctl version and control/data plane versions:
client version: 1.8.6
control plane version: 1.7.3
data plane version: 1.7.3 (3 proxies)
3 - run istioctl x precheck to see if revision was set (it may be different if set - see warning at the end of section)
Istio Revision "" already installed in namespace "istio-system"
There are two main upgrade strategies:
Vendor suggests to go with canary as it's more safe and can be tested before final migration.
4 - Create a backup:
kubectl get istio-io --all-namespaces -oyaml > "$HOME"/istio_resource_backup.yaml
Can be restored with:
kubectl apply -f "$HOME"/istio_resource_backup.yaml
5 - Control plane - install Canary version
istioctl install --set profile=default --set revision=1-8-6
Check it installed successfully by running following commands:
kubectl get pods -n istio-system -l app=istiod
NAME READY STATUS RESTARTS AGE
istiod-1-8-6-b855c557b-qq4qd 1/1 Running 0 44s
istiod-54b46bbc58-wzklh 1/1 Running 0 14m
kubectl get svc -n istio-system -l app=istiod
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
istiod ClusterIP 10.109.24.78 <none> 15010/TCP,15012/TCP,443/TCP,15014/TCP,853/TCP 15m
istiod-1-8-6 ClusterIP 10.101.241.85 <none> 15010/TCP,15012/TCP,443/TCP,15014/TCP 2m12s
kubectl get mutatingwebhookconfigurations
NAME WEBHOOKS AGE
istio-sidecar-injector 1 15m
istio-sidecar-injector-1-8-6 1 98s
6 - Data plane
Unlike istiod, Istio gateways do not run revision-specific instances, but are instead in-place upgraded to use the new control plane revision. You can verify that the
istio-ingressgateway is using thecanaryrevision by running the following command:
istioctl proxy-status | grep $(kubectl -n istio-system get pod -l app=istio-ingressgateway -o jsonpath='{.items..metadata.name}') | awk '{print $7}'
To upgrade the namespace
NAME_SPACE, remove theistio-injectionlabel, and add theistio.io/revlabel to point to thecanaryrevision. Theistio-injectionlabel must be removed because it takes precedence over theistio.io/revlabel for backward compatibility:
kubectl label namespace NAME_SPACE istio-injection- istio.io/rev=1-8-6
Once namespace(s) updated, pods need to be re-injected. This can be done by restarting them, e.g. with:
kubectl rollout restart deployment -n NAME_SPACE
Verify pods are now using canary istiod:
istioctl proxy-status
NAME CDS LDS EDS RDS ISTIOD VERSION
istio-ingressgateway-6664d4b478-j7vhh.istio-system SYNCED SYNCED SYNCED NOT SENT istiod-1-8-6-b855c557b-qq4qd 1.8.6
nginx-68748d7f8c-82x8k.default SYNCED SYNCED SYNCED SYNCED istiod-1-8-6-b855c557b-qq4qd 1.8.6
nginx-68748d7f8c-fnhgz.default SYNCED SYNCED SYNCED SYNCED istiod-1-8-6-b855c557b-qq4qd 1.8.6
7 - Uninstall old control-plane
Run:
istioctl x uninstall -f manifests/profiles/default.yaml
Check only canary control-plane is running:
kubectl get pods -n istio-system -l app=istiod
NAME READY STATUS RESTARTS AGE
istiod-1-8-6-b855c557b-qq4qd 1/1 Running 0 17m
Please get familiar with istio installation methods' pros and cons.
Moving this from comments. There are more challenges with updating from 1.7.3 to 1.8.6 istio versions.
For removing current control-plane -f with previous manifest should be used.
When applying for the same manifest to 1.8.6 versions, there are errors about policy and telemetry components:
Error: failed to get profile and enabled components: failed to read profile: unknown field "telemetry" in v1alpha1.IstioComponentSetSpec
After digging, it appeared, even though api version is used the same - v1alpha1, newer version of istioctl operator can't validate manifest from 1.7.3.
I took installed-state.yaml as it's described at the beginning of the asnwer from 1.7.3 and 1.8.6 istio installations and got diff between them: policy and telemetry components are completely missing in 1.8.6 which explains the errors. Also there are some changes as well. Github link to the diff file, left is 1.7.3, right is 1.8.6.
In that case it's probably impossible to upgrade without manual work with manifests:
1 - check if manifest which was applied is default or has changes. Get a default profile (note! istioctl should be used 1.7.3):
istioctl profile dump default > default-profile.yaml
2 - If manifest is default, then safely proceed to install canary with --set profile=default.
3 - Manifest is not default and has customization. Using istioctl 1.8.6 get a dump of default profile:
istioctl profile dump default > default-profile-186.yaml
"Adapt" it to current existing manifest and then proceed to install canary with -f option and adapted manifest.