Kubernetes Multi-Juicer

Cloud
Page content

If you ever wanna run a Multiplayer OWASP Juice Shop CTF on your own, here are some Notes and Info for bloody beginners

References

Prerequisite

you’ve got

btw. all this commands should run on macOS. linux may needs some adjustments …

Set Environment

set a few variables as we need them later in the scripts

test $domain || domain='your.domain'
host='ctf'
fqdn="${host}.${domain}"

echo "*** $(date)***"   >>  info.md
echo "FQDN: ${fqdn}"    >>  info.md
echo "domain=${domain}" >   .env
echo "host=${host}"     >>  .env
echo "fqdn=${fqdn}"     >>  .env

Check Connection

are we connected and authorized to DO ?

doctl account get
doctl auth list

$ doctl account get
User Email        Team       Droplet Limit    Email Verified    User UUID                                   Status
mail@your.domain  YourTeam   25               true              xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx    active

$ doctl auth list
default (current)

Connect to Digital Ocean if needed

if your not connected to DO, generate a Token on the API Menu and add it like this:

doctl auth init --context TOKENID

Create Cluster / 1 CPU, 2GB RAM

this needs a few minutes. just be patient :)

#time doctl kubernetes cluster create juicy-k8s --region fra1 --node-pool "auto-scale=true;min-nodes=3;max-nodes=5"

Create Cluster / 2 CPU, 2GB RAM

needed to update to a stronger box due to k8s issues!

time doctl kubernetes cluster create juicy-k8s --region fra1 --node-pool "size=s-2vcpu-2gb;auto-scale=true;min-nodes=3;max-nodes=5"

$ time doctl kubernetes cluster create juicy-k8s
Notice: Cluster is provisioning, waiting for cluster to be running
.....................................................
Notice: Cluster created, fetching credentials
Notice: Adding cluster credentials to kubeconfig file found in "/Users/stoege/.kube/config"
Notice: Setting current-context to do-nyc1-juicy-k8s
ID                                      Name         Region    Version         Auto Upgrade    Status     Node Pools
c3564b4f-501d-450b-xxxx-xxxxxxxxxxxx    juicy-k8s    nyc1      1.22.11-do.0    false           running    juicy-k8s-default-pool

real	4m57.624s
user	0m0.129s
sys	0m0.063s

List Clusters

doctl kubernetes cluster list

$ doctl kubernetes cluster list
ID                                      Name         Region    Version         Auto Upgrade    Status     Node Pools
5bd4401c-7a82-4240-xxxx-xxxxxxxxxxxx    juicy-k8s    fra1      1.22.11-do.0    false           running    juicy-k8s-pool-1

Get the Current Context

kubectl config current-context

$ kubectl config current-context
do-fra1-juicy-k8s

Install MultiJuicer via Helm

helm repo add multi-juicer https://iteratec.github.io/multi-juicer/
time helm install multi-juicer multi-juicer/multi-juicer

helm install multi-juicer multi-juicer/multi-juicer
NAME: multi-juicer
LAST DEPLOYED: Fri Jul  8 23:36:45 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
MultiJuicer deployed! 🎉🥳

To administrate the cluster you can log into the JuiceBalancer with the admin account:
Username: admin
Password: ${kubectl get secrets juice-balancer-secret -o=jsonpath='{.data.adminPassword}' | base64 --decode}

Extract Admin Password

adminpw=$(kubectl get secrets juice-balancer-secret -o=jsonpath='{.data.adminPassword}' | base64 --decode)
echo "Admin Password: ${adminpw}" |tee -a info.md
unset adminpw

Show Nodes and Pods

this may needs a few moments …

while true; do echo; kubectl get nodes; kubectl get pods; echo; sleep 5; done

kubectl get nodes
NAME                           STATUS   ROLES    AGE     VERSION
juicy-k8s-default-pool-cicp5   Ready    <none>   2m59s   v1.22.11
juicy-k8s-default-pool-cictf   Ready    <none>   15m     v1.22.11
juicy-k8s-default-pool-cictx   Ready    <none>   14m     v1.22.11
juicy-k8s-default-pool-cicty   Ready    <none>   15m     v1.22.11

kubectl get pods
NAME                                 READY   STATUS    RESTARTS   AGE
juice-balancer-846449bd74-g4ktk      1/1     Running   0          5m27s
progress-watchdog-6586786884-w76vh   1/1     Running   0          5m27s

wait until all nodes and pods are status “Ready/Running”

Add Loadbalancer and Expose to Inet

List Certificates

doctl compute certificate list

ID    Name    DNS Names    SHA-1 Fingerprint    Expiration Date    Created At    Type    State

Create Certificate

doctl compute certificate create --type lets_encrypt --name ctf --dns-names ${fqdn}

$ doctl compute certificate create --type lets_encrypt --name ctf --dns-names ctf.8192.ch
ID                                      Name    DNS Names      SHA-1 Fingerprint    Expiration Date         Created At              Type            State
a18460c3-554a-4bd7-xxxx-xxxxxxxxxxxx    ctf     ctf.your.domain                     0001-01-01T00:00:00Z    2022-07-08T21:47:00Z    lets_encrypt    pending

and wait a Moment until State switches to verified

doctl compute certificate list

build loadbalancer config

extract certificate id and put in config

certid=$(doctl compute certificate list |awk -v pat="$fqdn" '$0 ~ pat{print $1}')

cat << EOF > do-lb.yaml
kind: Service
apiVersion: v1
metadata:
  name: multi-juicer-loadbalancer
  annotations:
    service.beta.kubernetes.io/do-loadbalancer-protocol: 'http2'
    service.beta.kubernetes.io/do-loadbalancer-certificate-id: '${certid}'
    service.beta.kubernetes.io/do-loadbalancer-redirect-http-to-https: 'true'
    service.beta.kubernetes.io/do-loadbalancer-algorithm: 'round_robin'
    service.beta.kubernetes.io/do-loadbalancer-healthcheck-protocol: 'http'
    service.beta.kubernetes.io/do-loadbalancer-healthcheck-path: '/balancer/'
spec:
  type: LoadBalancer
  selector:
    app.kubernetes.io/instance: multi-juicer
    app.kubernetes.io/name: multi-juicer
  ports:
    - name: http
      protocol: TCP
      port: 443
      targetPort: 3000
EOF

kubectl create -f do-lb.yaml

Wait for LBL IP

this needs 2-3min

while true; do kubectl describe services multi-juicer-loadbalancer |awk '/LoadBalancer Ingress/{print $3}'; echo "."; sleep 5; done

Get LBL IP

lbip=$(kubectl describe services multi-juicer-loadbalancer |awk '/LoadBalancer Ingress/{print $3}')
echo "Loadbalancer IP: ${lbip}" |tee -a info.md

-> Loadbalancer IP: 178.128.xxx.yyy

Update DNS

set A record ctf.your.domain to 178.128.xxx.yyy

doctl compute domain records create ${domain} --record-type A --record-data ${lbip} --record-ttl 3600 --record-name ${host}
doctl compute domain records list ${domain}

Check DNS

doctl compute domain records list ${domain} |grep ${host}

dig +short @9.9.9.9 ${fqdn}
dig +short @ns1.digitalocean.com ${fqdn}

host ${fqdn}

Check Page

echo "https://${fqdn}"
echo "https://${lbip}"

you should get the Welcome Page !

Now, the Teams can register themself …

… and start the Challenge !

CleanUp

as the Machines costs Money (-> up to 60 CHF/Month), you wanna delete and remove all the stuff after the Challange)

# Delete Multi Juicer
helm delete multi-juicer

# Delete the loadbalancer
kubectl delete -f do-lb.yaml

# Delete the kubernetes cluster
doctl kubernetes cluster delete juicy-k8s -f

# Delete Certificate
certid=$(doctl compute certificate list |grep ${fqdn} |awk '{print $1}')
doctl compute certificate delete ${certid} -f

# Delete A records
arecordid=$(doctl compute domain records list ${domain} |grep ${host} |awk '{print $1}')
doctl compute domain records delete ${domain} ${arecordid} -f

# Droplets killed as well ?
doctl compute droplet list

# in Digital Ocean Page:
$-> delete the Hosts if still running (should not be ...)

Kubernets Commands

kubectl config get-contexts
kubectl cluster-info
kubectl version
kubectl get nodes
kubectl help

doctl kubernetes cluster kubeconfig
show <cluster-id|cluster-name>
doctl kubernetes

Setup Script / Part 1

copy / paste following Script. Should run under MACOS without Problems ;)

ETA: around 10min

export project="juicy-k8s"
export domain="mydomain.com"
export host="shop"

cat << 'EOS' > install_k8s.sh
#!/usr/bin/env bash

# MultiJuicer Setup Script

test -d game || mkdir game
cd game

if [[ -f .env ]]; then

  source .env

else

  test $project || domain='PROJECT-NOT-SET'
  test $domain  || domain='DOMAIN-NOT-SET'
  test $host    || host='HOST-NOT-SET'
  fqdn="${host}.${domain}"
  echo $project, $domain, $host, $fqdn

  export project="$project"
  export domain="$domain"
  export host="$host"
  export fqdn="$fqdn"

  echo "project=${project}"   >   .env
  echo "domain=${domain}"     >>  .env
  echo "host=${host}"         >>  .env
  echo "fqdn=${fqdn}"         >>  .env

fi

echo
echo "*** $(date)***"       |tee info.md
echo "Project: ${project}"  |tee -a info.md
echo "FQDN: ${fqdn}"        |tee -a info.md
echo

read -p "Press enter to continue (or CTRL-C to break)"

doctl kubernetes cluster \
  create ${project} \
  --region fra1 \
  --node-pool "size=s-2vcpu-2gb;auto-scale=true;min-nodes=3;max-nodes=5"
echo "sleep 15"; sleep 15

kubectl config current-context
helm repo add multi-juicer https://iteratec.github.io/multi-juicer/
helm install multi-juicer multi-juicer/multi-juicer
echo "sleep 15"; sleep 15

adminpw=$(kubectl get secrets juice-balancer-secret -o=jsonpath='{.data.adminPassword}' | base64 --decode)
echo "Admin Password: ${adminpw}" |tee -a info.md
unset adminpw

kubectl get nodes
kubectl get pods

doctl compute certificate list

doctl compute certificate create --type lets_encrypt --name ${host} --dns-names ${fqdn},${host}1.${domain},${host}-1.${domain},${host}2.${domain},${host}-2.${domain},${host}3.${domain},${host}-3.${domain}
echo "sleep 15"; sleep 15

certid=$(doctl compute certificate list |awk -v pat="$fqdn" '$0 ~ pat{print $1}')

cat << EOF > do-lb.yaml
kind: Service
apiVersion: v1
metadata:
  name: multi-juicer-loadbalancer
  annotations:
    service.beta.kubernetes.io/do-loadbalancer-protocol: 'http2'
    service.beta.kubernetes.io/do-loadbalancer-certificate-id: '${certid}'
    service.beta.kubernetes.io/do-loadbalancer-redirect-http-to-https: 'true'
    service.beta.kubernetes.io/do-loadbalancer-algorithm: 'round_robin'
    service.beta.kubernetes.io/do-loadbalancer-healthcheck-protocol: 'http'
    service.beta.kubernetes.io/do-loadbalancer-healthcheck-path: '/balancer/'
spec:
  type: LoadBalancer
  selector:
    app.kubernetes.io/instance: multi-juicer
    app.kubernetes.io/name: multi-juicer
  ports:
    - name: http
      protocol: TCP
      port: 443
      targetPort: 3000
EOF

kubectl create -f do-lb.yaml
EOS

chmod 700 install_k8s.sh

Wait for Loadbalancer

let this commands running until you got an ip address … takes up to 3min

while true; do \
  lbip=$(kubectl describe services multi-juicer-loadbalancer |awk '/LoadBalancer Ingress/{print $3}'); \
  echo $lbip; \
  echo .; \
  sleep 5; \
done

Setup Script / Part 2

get IP for Loadbalancer, set DNS Record, done

cat << 'EOS' > install_dns.sh
# Env
test -f game/.env && source game/.env || exit 1

lbip=$(kubectl describe services multi-juicer-loadbalancer |awk '/LoadBalancer Ingress/{print $3}')
echo "Loadbalancer IP: ${lbip}" |tee -a game/info.md

# A Record
doctl compute domain records create ${domain} \
  --record-type A \
  --record-data ${lbip} \
  --record-name ${host}

# More A Records
for i in {1..3}; do
doctl compute domain records create ${domain} \
  --record-type A \
    --record-ttl 3600 \
  --record-data ${lbip} \
  --record-name ${host}${i}
done

# some CNAME
for i in {1..3}; do
  doctl compute domain records create ${domain} \
    --record-type CNAME \
    --record-ttl 3600 \
    --record-name ${host}-${i} \
    --record-data ${fqdn}.
done

doctl compute domain records list ${domain}

echo; echo "Done! Check Page:"
echo "https://${fqdn}"
echo "https://${lbip}"
echo "$(cat game/info.md |grep -i password)"; echo
EOS

chmod 700 install_dns.sh

Setup Script / Uninstall

cat << 'EOF' > uninstall.sh
# Env
test -f game/.env && source game/.env || exit 1

# Delete Multi Juicer
helm delete multi-juicer

# Delete the loadbalancer
kubectl delete -f game/do-lb.yaml

# Delete the kubernetes cluster
doctl kubernetes cluster delete juicy-k8s -f

# Delete Certificate
certid=$(doctl compute certificate list |grep ${fqdn} |awk '{print $1}')
doctl compute certificate delete ${certid} -f

# Delete A records
arecordid=$(doctl compute domain records list ${domain} |grep ${host} |awk '{print $1}')
doctl compute domain records delete ${domain} ${arecordid} -f

# Cleanup
rm -rf game

# Wait until gone
echo "press CTLR-C when all nodes are gone ..."
while true; do doctl compute droplet list; sleep 5; done
EOF

chmod 700 uninstall.sh

Project Management

Create Project

doctl projects create --name "Test123" --purpose "Learning"

List Projects

$ doctl projects list -o json
[
  {
    "id": "aa444047-7eee-4548-94db-5485b705b905",
    "owner_uuid": "8a0d816ec1a780bfb985147b10157563d1c0ff4a",
    "owner_id": 1883170,
    "name": "Test123",
    "description": "",
    "purpose": "Other: Learning",
    "environment": "",
    "is_default": false,
    "created_at": "2022-08-10T12:04:03Z",
    "updated_at": "2022-08-10T12:04:03Z"
    "updated_at": "2022-07-06T20:14:51Z"
  }

List all Records

set Variable for all following Snippets

mydomain="bliblablu.com"

List all Records

doctl compute domain records list ${mydomain}

List all ID’s for A Records

doctl compute domain records list ${mydomain} --format Type,ID |\
  awk '/^A /{ print $2 }'

Delete all “A” Records

-f will enforce without asking !

doctl compute domain records delete ${mydomain} -f $(\
  doctl compute domain records list ${mydomain} --format Type,ID |\
  awk '/^A /{ print $2 }' \
  )

List all Records except NS/SOA

doctl compute domain records list ${mydomain} --no-header --format Type,ID |\
  awk '!/^(NS|SOA) /{ print $2 }'

Delete all Records except NS/SOA

-f will enforce without asking !

doctl compute domain records delete ${mydomain} -f $(\
  doctl compute domain records list ${mydomain} --no-header --format Type,ID |\
  awk '!/^(NS|SOA) /{ print $2 }' \
  )

or

doctl compute domain records list ${mydomain} --no-header --format Type,ID |\
  awk '!/^(NS|SOA) /{ print $2 }' |\
  doctl compute domain records delete ${mydomain} -f $(cat -)
Any Comments ?

sha256: d8c59a70d9b6c07e289a9395b2f38cfa0b32cc27abf323fc4f32c407283d4a3c