AWS Reference Implementation
This project contains an Internal Developer Platform (IDP) reference implementation for AWS. This project can bring up an IDP on EKS with all the tools configured and ready to use in production. It will install addons on an EKS cluster as Argo CD apps using GitOps Bridge App of ApplicationSets pattern. Check out the Getting Started guide for installing this solution on an EKS cluster.
Applications deployed in this repository are a starting point to get environment into production.
Architecture Overview
Addons
All the addons are helm charts with static values configured in packages/<addon-name>/values.yaml and dynamic values based on Argo CD cluster secret label/annotations values in packages/addons/values.yaml.
| Name | Namespace | Purpose | Chart Version | Chart |
|---|---|---|---|---|
| Argo CD | argocd | Installation and management of addon Argo CD application | 8.0.14 | Link |
| Argo Workflows | argo | Workflow tool for continuous integration tasks | 0.45.18 | Link |
| Backstage | backstage | Self-Service Web UI (Developer Portal) for developers | 2.6.0 | Link |
| Cert Manager | cert-manager | Certificate manager for addons and developer applications using Let's Encrypt | 1.17.2 | Link |
| Crossplane | crossplane-system | IaC controller for provisioning infrastructure | 1.20.0 | Link |
| ACK | ack-system | IaC controller for provisioning infrastructure | TBD | Coming soon check #54 |
| External DNS | external-dns | Domain management using Route 53 | 1.16.1 | Link |
| External Secrets | external-secrets | Secret Management using AWS Secret Manager and AWS Systems Manager Parameter Store | Version | Link |
| Ingress NGINX | ingress-nginx | Ingress controller for L7 network traffic routing | 4.7.0 | Link |
| Keycloak | keycloak | Identity provider for User Authentication | 24.7.3 | Link |
Check out more details about the installation flow.
Installation Flow Diagram
This diagram illustrates the high-level installation flow for the CNOE AWS Reference Implementation. It shows how the local environment interacts with AWS resources to deploy and configure the platform on an EKS cluster.
Getting Started
Step 1. ⚙️ Prepare Environment for Installation
📦 Install Binaries
The installation requires the following binaries in the local environment:
🔐 Configure AWS Credentials
Configure the AWS CLI with credentials of an IAM role which has access to the EKS cluster. Follow the instructions in the AWS documentation to configure the AWS CLI.
If the installation steps are being executed on an EC2 instance, ensure that the EC2 IAM instance role has permissions to access the EKS cluster or the AWS CLI is configured as mentioned above.
Step 2. 🏢 Create GitHub Organization
Backstage and Argo CD in this reference implementation are integrated with GitHub. Both Backstage and ArgoCD use Github Apps for authenticating with Github.
Therefore, a GitHub Organization should be created in order to create GitHub Apps for these integrations. Follow the instructions in GitHub documentation to create new organization or visit here.
It is recommended to use a Github Organization instead of a personal github ID as Backstage has certain limitations for using personal account Github Apps for authenticating to Github. Also, the Github Organization is FREE.
Step 3. 🍴 Fork the Repository
Once the organization is created, fork this repository to the new GitHub Organization by following instructions in GitHub documentation.
Step 4. 💻 Create GitHub Apps
There are two ways to create a GitHub App. You can use the Backstage CLI npx @backstage/cli create-github-app <github-org> as per instructions in the Backstage documentation, or create it manually per these instructions in the GitHub documentation.
Create the following apps and store them in the corresponding file path.
| App Name | Purpose | Required Permissions | File Path | Expected Content |
|---|---|---|---|---|
| Backstage | Used for automatically importing Backstage configuration such as Organization information, templates and creating new repositories for developer applications. | For All Repositories - Read access to members, metadata, and organization administration - Read and write access to administration and code | private/backstage-github.yaml |  |
| Argo CD | Used for deploying resources to cluster specified by Argo CD applications. | For All Repositories - Read access to checks, code, members, and metadata | private/argocd-github.yaml |  |
The template files for both these Github Apps are available in private directory. Copy these template files to above mentioned file path by running following command:
cp private/argocd-github.yaml.template private/argocd-github.yaml
cp private/backstage-github.yaml.template private/backstage-github.yaml
After this, update the values in these files by getting them from files created by backstage-cli (if used) or get the values from Github page of App Overview.
Argo CD requires url and installationId of the GitHub app. The url is the GitHub URL of the organization. The installationId can be captured by navigating to the app installation page with URL https://github.com/organizations/<Organization-name>/settings/installations/<ID>. You can find more information on this page.
If the app is created using the Backstage CLI, it creates files in the current working directory. These files contain credentials. Handle them with care. It is recommended to remove these files after copying the content over to files in the private directory
The rest of the installation process assumes the GitHub apps credentials are available in private/backstage-github.yaml and private/argocd-github.yaml
Step 5. ⚙️ Configure Reference Implementation
The reference implementation uses config.yaml file in the repository root directory to configure the installation values. The config.yaml should be updated with appropriate values before proceeding. Refer to the following table and update all the values appropriately. All the values are required.
| Parameter | Description | Type |
|---|---|---|
cluster_name | Name of the EKS cluster for reference implementation (The name should satisfy criteria of a valid kubernetes resource name) | string |
auto_mode | Set to "true" if EKS cluster is Auto Mode, otherwise "false" | string |
repo.url | GitHub URL of the fork in the Github Org | string |
repo.revision | Branch or tag which should be used for Argo CD Apps | string |
repo.basepath | Directory in which the configuration of addons is stored | string |
region | AWS Region of the EKS cluster and config secret | string |
domain | Base Domain name for exposing services (This should be base domain or sub domain of the Route53 Hosted Zone) | string |
route53_hosted_zone_id | Route53 hosted zone ID for configuring external-dns | string |
path_routing | Enable path routing ("true") vs domain-based routing ("false") | string |
tags | Arbitrary key-value pairs for AWS resource tagging | object |
If these values are updated after installation, ensure to run the command in the next step to update the values in AWS Secret Manager. Otherwise, the updated values will not reflect in the live installation.
Step 6. 🔒 Create Secrets in AWS Secret Manager
The values required for the installation are stored in AWS Secret Manager in two secrets:
- cnoe-ref-impl/config: Stores values from
config.yamlin JSON - cnoe-ref-impl/github-app: Stores GitHub App credentials with file name as key and content of the file as value from private directory.
Run the command below to create new secrets or update the existing secrets if they already exist.
./scripts/create-config-secrets.sh
DO NOT move to next steps without completing all the instructions in this step
Step 7. ☸️ Create EKS Cluster
The reference implementation can be installed on a new EKS cluster which can be created like this:
export REPO_ROOT=$(git rev-parse --show-toplevel)$REPO_ROOT/scripts/create-cluster.sh
You will be prompted to select eksctl or terraform
For more details on each type of tools check the corresponding guides:
- eksctl: Follow the instructions
- terraform: Follow the instructions
This will create all the prerequisite AWS Resources required for the reference implementation, which includes:
- EKS cluster with Auto Mode or Without Auto Mode (Managed Node Group with 4 nodes)
- Pod Identity Associations for following Addons:
| Name | Namespace | Service Account Name | Permissions |
|---|---|---|---|
| Crossplane | crossplane-system | provider-aws | Admin Permissions but with permission boundary |
| External Secrets | external-secrets | external-secrets | Permissions |
| External DNS | external-dns | external-dns | Permissions |
| AWS Load Balancer Controller (When not using Auto Mode) | kube-system | aws-load-balancer-controller | Permissions |
| AWS EBS CSI Controller (When not using Auto Mode) | kube-system | ebs-csi-controller-sa | Permissions |
Using Existing EKS Cluster
The reference implementation can be installed on an existing EKS Cluster only if the above prerequisites are completed.
Step 8. 🚀 Deployment
Before moving forward, ensure that the kubectl context is set to the EKS cluster and the configured AWS IAM role has access to the cluster.
▶️ Start the Installation Process
All the addons are installed as Argo CD apps. At the start of the installation, Argo CD and External Secret Operator are installed on the EKS cluster as a helm chart. Once Argo CD on EKS is up, other addons are installed through it and finally the Argo CD on EKS also manages itself and External Secret Operator. Check out more details about the installation flow. Run the following command to start the installation.
scripts/install.sh
Step 9. 🌐 Accessing the Platform
The addons with Web UI are exposed using the base domain configured in Step 5. The URLs can be retrieved by running the following command:
scripts/get-urls.sh
The URL depends on the setting for path_routing. Refer to following table for URLs:
| App Name | URL (w/ Path Routing) | URL (w/o Path Routing) |
|---|---|---|
| Backstage | https://[domain] | https://backstage.[domain] |
| Argo CD | https://[domain]/argocd | https://argocd.[domain] |
| Argo Workflows | https://[domain]/argo-workflows | https://argo-workflows.[domain] |
📊 Monitor Deployment Process
The installation script will continue to run until all the Argo CD apps for addons are healthy. To monitor the process, use the instructions below to access the instance of Argo CD running on EKS.
Check if the kubectl context is set to the EKS cluster and it can access the EKS cluster.
You can use kubectl to check the status of the Argo CD applications
kubectl get applications -n argocd --watch
Get the credentials for Argo CD and start a port-forward with this command:
kubectl get secrets -n argocd argocd-initial-admin-secret -oyaml | yq '.data.password' | base64 -d && echokubectl port-forward -n argocd svc/argocd-server 8080:80
Depending upon the configuration, Argo CD will be accessible at http://localhost:8080 or http://localhost:8080/argocd.
All the addons are configured with Keycloak SSO user1 and the user password for it can be retrieved using the following command:
kubectl get secret -n keycloak keycloak-config -o jsonpath='{.data.USER1_PASSWORD}' | base64 -d && echo
Once all the Argo CD apps on the EKS cluster are reporting healthy status, try out the examples to create a new application through Backstage. For troubleshooting, refer to the troubleshooting guide.
Cleanup
Before proceeding with the cleanup, ensure any Kubernetes resources created outside of the installation process such as Argo CD Apps, deployments, volumes etc. are deleted.
Run the following command to remove all the addons created by this installation:
scripts/uninstall.sh
This script will only remove resources other than CRDs from the EKS cluster so that the same cluster can be used for re-installation which is useful during development. To remove CRDs, use the following command:
scripts/cleanup-crds.sh