Troubleshooting Deployments

Understand deployment failures and unresponsive applications

Your app doesn't respond to the public URL or your service is in an abnormal state and you're wondering why? Below you will find a few common errors and failures which might happen.

When you investigate failures, you should pay attention to the state of the instances composing your service. When you deploy a service, Koyeb will start one or more instances depending on your configuration. All these instances will be health-checked before routing production traffic to them.

Unhealthy instance

Instances are detected as unhealthy when health checks fail.

More precisely, every time an instance is booted to support your service, all the network ports you decided to open will be health-checked every 10 secs. If any of these ports do not respond after the boot of the instance, the instance will be declared as unhealthy.

Unresponsive network ports usually come from one of these reasons:

  1. Network port configuration issue: the network port you selected is not the one your service is listening on
  2. Binding on localhost instead of 0.0.0.0: if your application is bound to localhost or 127.0.0.1, nobody outside your container can reach your application. Ensure your application is bound to 0.0.0.0 to make it reachable
  3. Service configuration issue: your service is not starting due to an application-specific issue like a missing environment variable or a connection issue to another service
  4. Service failure: your service might fail at startup or when it's processing requests due to a lack of memory or temporary storage

If you don't understand why an instance is unhealthy, check the logs of the instance.

Metrics

To investigate and diagnose performance issues, you can use Koyeb Metrics, which provides you a high-level overview of your services running on Koyeb and comes natively integrated into the platform. With Koyeb Metrics, you can access to your service's CPU usage, memory usage, average response time, request throughput, and public data transfer.

A common source of performance issues is caused by not enough CPU or memory resources allocated to the service. If your service is using a high value of CPU or memory, like around 70-80%, this may be an indication you need to scale up your service or switch to a bigger instance size.

You can view your service metric's by going to the console and looking under the Metrics tab in your service view. Read the full Koyeb Metrics documentation to learn more.

Logs

Runtime logs allow you to troubleshoot deployments and follow logs written by your Services on standard streams (stdout/stderr) in real-time.

By default, the logs of all the instances composing the service are displayed in a single log stream. For each log entry, we display the:

  • instance_id: The instance on which the entry was written
  • stream: The standard stream (stdout/stderr) which was used to write the log
  • message: The message

For instance, if "Hey there, I'm a service running on Koyeb" was printed on stdout of your instance-131869ab, you would get a line looking like this:

instance-131869ab stdout Hey there, I'm a service running on Koyeb

The default runtime logs view shows logs for both currently running instances and past, destroyed, instances. You can filter logs per instance.

Each time you apply a filter, we will display the last 1000 log lines available. Runtime logs are kept 30 days.

Slow service

Slow service response time is usually related to a lack of resources or a contention for external resources like a database. Check the service metrics to see if it is resource related.

Instances shell access

For advanced troubleshooting, you can execute commands directly on the instances supporting your service.

For image-based deployments, the user is the user defined in your image specification. In Dockerfiles, it is specified with the USER instruction. If no user is defined, root is chosen.

For git-driven deployments, the user is the user assigned by the buildpack. In most cases, it will be koyeb.

$ koyeb instances list ID SERVICE STATUS DEPLOYMENT ID DATACENTER CREATED AT 1bbb7ac7 my-demo-app/main HEALTHY a6615e32 par1 16 Dec 21 10:49 UTC $ koyeb instances exec 1bbb7ac7 -- netstat -tulpn Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:3000 0.0.0.0:* LISTEN 27/node

You may also run short-lived debugging sessions and get access to a shell on the running instance.

$ koyeb instances list ID SERVICE STATUS DEPLOYMENT ID DATACENTER CREATED AT 1bbb7ac7 my-demo-app/main HEALTHY a6615e32 par1 16 Dec 21 10:49 UTC $ koyeb instances exec 1bbb7ac7 /bin/sh /app $ ls next.config.js node_modules package.json public