Error: rpc error: code = Unknown desc = release orc8r failed: timed out waiting for the condition on ../../main.tf line 22, in resource "helm_release" "orc8r":
22: resource "helm_release" "orc8r" {
Resolution steps
Check if there is an issue with Helm repo URL, password, or container image tag
Ensure container image registry and Terraform values are correct
Run following kubectl commands to get more details on the error encountered
kubectl --namespace orc8r get pods
kubectl --namespace orc8r describe pods
# pod status ImagePullBackOff, indicates that Image wasn't found# pod can also be crash looping, pod status will provide this information,# in this case, looking at the logs of individual container will help debug this further
Get Helm's view of images
helm version # provides current version of Helm
helm -n orc8r list # provides the list of releases which are currently deployed under Orc8r namespace
helm -n orc8r get values orc8r # gets the values file for the Orc8r release
NMS pod wasn't showing up properly
$ kubectl exec -it $(kubectl get pod -l app.kubernetes.io/component=magmalte -o jsonpath='{.items[0].metadata.name}') -- yarn setAdminPassword master xxxx@xxxx.com 1234
error: error executing jsonpath "{.items[0].metadata.name}": Error executing template: array index out of bounds: index 0, length 0. Printing more information for debugging the template:
template was:
{.items[0].metadata.name}
object given to jsonpath engine was:
map[string]interface {}{"apiVersion":"v1", "items":[]interface {}{}, "kind":"List", "metadata":map[string]interface {}{"resourceVersion":"", "selfLink":""}}
error: pod, type/name or --filename must be specified
Resolution steps
The problem was that deploy_nms wasn't set to true in variables.tf. Set deploy_nms to true and rerun terraform apply
No resources found in default namespace after Terraform apply
No resources found in default namespace after terraform apply
Resolution steps
Terraform seems to reset the kubectl namespace — fix using kubens to select the orc8r namespace
Errors creating Secrets Manager
error creating Secrets Manager Secret: InvalidRequestException:
You can't create this secret because a secret with this name is already
scheduled for deletion
Resolution steps
Change secretsmanager_orc8r_secret variable to new value
AWS doesn't fully delete secrets for safety reasons, so deleted secrets need a name change before successful regeneration
If you want to fully, "immediately" delete the secret (danger zone!), consider aws secretsmanager delete-secret --secret-id your_secret_id --force-delete-without-recovery(ref)
Error: ValidationException: Domain is being deleted for aws_elasticsearch_domain_policy
`Error: ValidationException: Domain is being deleted` for `aws_elasticsearch_domain_policy`
Resolution steps
Sometimes ES domains aren't deleted immediately. Change ES domain to new value and stop tracking old domain
Ensure ES domain is actually marked for deletion, e.g. via AWS console
Change elasticsearch_domain_name variable to new value
terraform state rm module.orc8r.aws_elasticsearch_domain.es[0] stop tracking old ES domain
Terraform apply failed due to some Python module error
Warning: Applied changes may be incompleteThe plan was created with the -target option in effect, so some changes
requested in the configuration may have been ignored and the output values may
not be fully updated. Run the following command to verify that no other
changes are pending:
terraform planNote that the -target option is not suitable for routine use, and is provided
only for exceptional situations such as recovering from errors or mistakes, or
when Terraform specifically suggests to use it as part of an error message.Error: Error running command ' ../../scripts/create_orc8r_secrets.py \
'orc8r-secrets' 'us-west-2' \
"~/secrets/certs"
': exit status 1. Output: Traceback (most recent call last):
File "../../scripts/create_orc8r_secrets.py", line 12, in
import boto3
ModuleNotFoundError: No module named 'boto3'
Resolution steps
Identify and install the relevant Python module - pip3 install boto3
Specifically ensure if deployment specific tooling has been installed
