Skip to main content

Troubleshooting

No ingress IP available​

When you try to deploy an application to a site and see no-ingress-ip-available, there are several possible reasons:

supctl show system sites target-site --fields ingress-allocation-method
  • you have configured ingress allocation on the site, but the pool or DHCP server is out of IP addresses. If it is a pool, please verify using the following command, as a site provider:
supctl show -s target-site system cluster ingress-allocation
  • if the pool of IP addresses is configured and has unallocated IP addresses, but a subtenant still gets a No ingress IP available error, this may be an indication of an access rights issue. Verify that the subtenant has access to the desired ingress IP block, see predictable ingress IP addresses :
supctl show tenants subtenant --field resource-profile
supctl show resource-profiles target-profile --field ingress-ipv4-ranges

No volume match​

When you try to deploy an application and see no-volume-match, you have either set match-volume-labels in the application, and the system was not able to match any volume labels on the site. The other option is that no volumes are configured on the particular site, see storage configuration.

Missing declared volumes​

Let's say you get an error like container 'database': volumes declared in the image must be mounted: '/var/lib/mysql'. This tells you that you will need to mount a volume at path /var/lib/mysql, see volumes.

No host match​

When you get an error like no-host-label-match, it means an application service has a match-host-labels placement rule, but no host matched this rule. The solution is to add such host labels to the site.

Calls to API returns 401 after a while​

When authenticating towards the Edge Enforcer APIs, the token received will timeout. If the token is not refreshed, you will get 401 HTTP codes after expiry. See maintaining a token.

Application​

This section contains some general tips and tricks to troubleshoot an application.

In the example below we have deployed an application to four sites, but in one of the sites, the application doesn't start.

Deployment State​

First, we will check the deployment state of the application, to make sure it has been scheduled to run on the expected sites.

supctl show application-deployments my-dep
name: my-dep
application: my-app
application-version: *
placement:
  match-site-labels: |
    system/type = edge
status:
  oper-status: deploy-failure
  images:
    - name: registry-1.docker.io/alpine
      status: present
      digest: sha256:69704ef328d05a9f806b
  selected-sites:
    - gothenburg-bergakungen
    - helsingborg-roda-kvarn
    - stockholm-sergel
    - stockholm-sture
  application-versions:
    - version: undefined
      deployed-to:
        - gothenburg-bergakungen
        - helsingborg-roda-kvarn
        - stockholm-sture
      deploy-failed:
        - site: stockholm-sergel

From the highlighted info we can see

  1. Image is present
  2. Application has been deployed to the correct sites
  3. Application has failed in the stockholm-sergel site.

Application State in Site​

Since we now know it has failed in stockholm-sergel, let's look at the application's service instance in that site.

supctl show --site stockholm-sergel applications my-app service-instances
- name: my-svc-1
  oper-status: error
  host: stockholm-sergel-001
  application-network:
    ips:
      - 172.27.0.1/16
  gateway-network:
    ips:
      - 172.25.255.2/24
  ingress:
    ips: []
  containers:
    - name: alpine
      id: e4a156bb880e
      oper-status: failed
      error-message: container movie-theater-owner.my-app.my-svc-1.alpine exited
      start-time: 2021-10-26T11:19:39.022Z
      current-restarts: 9
      total-restarts: 9

Here we can again see we have a problem and that the application has restarted multiple times, let's try to find out what it is by looking at the logs.

Distributed logging​

Here we see that we have an error in the stockholm-sergel site.

supctl do volga query-topics --all-sites --topics match-topic-labels='application=my-app' output-payload-only=true filter-re-match=Error
stockholm-sergel-001 : Error: Can't open database
stockholm-sergel-001 : Error: Can't open database
stockholm-sergel-001 : Error: Can't open database
stockholm-sergel-001 : Error: Can't open database
stockholm-sergel-001 : Error: Can't open database
stockholm-sergel-001 : Error: Can't open database
stockholm-sergel-001 : Error: Can't open database
stockholm-sergel-001 : Error: Can't open database

Or if we want to get all logs from a specific site

supctl do volga query-topics --site-names=sthlm --topics match-topic-labels='application=my-app' output-payload-only=true
stockholm-sergel-003 : Error: Can't open database
stockholm-sergel-003 : Error: Can't open database
stockholm-sergel-003 : Error: Can't open database
stockholm-sergel-003 : Error: Can't open database
stockholm-sergel-003 : Error: Can't open database

Hopefully this provides enough information to bring the appliction up in all sites.

Remote access to the container​

In some cases, even though the container doesn't exit, it can be helpful to get an interactive shell in a running container.

If we need to "shell" into a container in a site, we can use the exec-interactive command.

supctl do --site helsingborg-roda-kvarn applications my-app service-instances my-svc-1 containers alpine exec-interactive sh
/ # ps
PID   USER     TIME  COMMAND
    1 root      0:00 /sbin/docker-init -- sh /cfg/run.sh
    6 root      0:00 sh /cfg/run.sh
    7 root      0:00 sleep infinity
    8 root      0:00 sh
   13 root      0:00 ps