Ingress
Let's talk about Nginx Ingress, but there are several different ingress options. We'll set up Nginx and ensure it uses TLS.
Review reading:
NodePort opens the same port on all cluster nodes and redirects that port to a specific cluster IP. It doesn't matter which node IP we use; what differentiates is the port. But in this case, we could always use the same node IP, and that's where the load balancer comes in, to have a set of node IPs so requests can be balanced with a single IP address and domain.
Before starting, delete all previously created network policies.
root@cks-master:~# kubectl get networkpolicies
NAME POD-SELECTOR AGE
backend run=backend 21h
default-deny <none> 21h
frontend run=frontend 21h
root@cks-master:~# kubectl delete networkpolicies backend default-deny frontend
networkpolicy.networking.k8s.io "backend" deleted
networkpolicy.networking.k8s.io "default-deny" deleted
networkpolicy.networking.k8s.io "frontend" deleted
Let's install Nginx Ingress.
root@cks-master:~# kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.11.2/deploy/static/provider/cloud/deploy.yaml
namespace/ingress-nginx created #<<< Namespace created
serviceaccount/ingress-nginx created
serviceaccount/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
configmap/ingress-nginx-controller created # Config
service/ingress-nginx-controller created
service/ingress-nginx-controller-admission created
deployment.apps/ingress-nginx-controller created # Deployment
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created
ingressclass.networking.k8s.io/nginx created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
# If we observe carefully, we have here a LoadBalancer type service that is pending
# This happens because we don't have an external IP yet, but we already have nodeports 32383 and 32617
root@cks-master:~# k -n ingress-nginx get svc,pod
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.101.64.57 <pending> 80:32383/TCP,443:32617/TCP 88s
service/ingress-nginx-controller-admission ClusterIP 10.102.213.88 <none> 443/TCP 88s
NAME READY STATUS RESTARTS AGE
pod/ingress-nginx-admission-create-tmn2q 0/1 Completed 0 88s
pod/ingress-nginx-admission-patch-5dv4z 0/1 Completed 0 88s
pod/ingress-nginx-controller-7d4db76476-np7x6 1/1 Running 0 88s # Pod running nginx
## We can observe in the deployment that we have some argument configurations we pass to nginx
root@cks-master:~# k describe deployments.apps -n ingress-nginx ingress-nginx-controller
Name: ingress-nginx-controller
Namespace: ingress-nginx
CreationTimestamp: Sun, 18 Aug 2024 14:55:41 +0000
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/part-of=ingress-nginx
app.kubernetes.io/version=1.11.2
Annotations: deployment.kubernetes.io/revision: 1
Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 1 max unavailable, 25% max surge
Pod Template:
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/part-of=ingress-nginx
app.kubernetes.io/version=1.11.2
Service Account: ingress-nginx
Containers:
controller:
Image: registry.k8s.io/ingress-nginx/controller:v1.11.2@sha256:d5f8217feeac4887cb1ed21f27c2674e58be06bd8f5184cacea2a69abaf78dce
Ports: 80/TCP, 443/TCP, 8443/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
SeccompProfile: RuntimeDefault
Args:
######### Configuration block ###############
/nginx-ingress-controller
--publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
--election-id=ingress-nginx-leader
--controller-class=k8s.io/ingress-nginx
--ingress-class=nginx
--configmap=$(POD_NAMESPACE)/ingress-nginx-controller
--validating-webhook=:8443
--validating-webhook-certificate=/usr/local/certificates/cert
--validating-webhook-key=/usr/local/certificates/key
--enable-metrics=false
#########################################
Requests:
cpu: 100m
memory: 90Mi
Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
Environment:
POD_NAME: (v1:metadata.name)
POD_NAMESPACE: (v1:metadata.namespace)
LD_PRELOAD: /usr/local/lib/libmimalloc.so
Mounts:
/usr/local/certificates/ from webhook-cert (ro)
Volumes:
webhook-cert:
Type: Secret (a volume populated by a Secret)
SecretName: ingress-nginx-admission
Optional: false
Node-Selectors: kubernetes.io/os=linux
Tolerations: <none>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: ingress-nginx-controller-7d4db76476 (1/1 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 2m50s deployment-controller Scaled up replica set ingress-nginx-controller-7d4db76476 to 1
Get the public IP of any of your machines on gcloud and let's check the specific ports we have.
We reached nginx, but it doesn't know what to do, it has nowhere to point.
Let's create two pods and define the rules.
root@cks-master:~# kubectl run app1 --image=nginx
pod/app1 created
root@cks-master:~# kubectl run app2 --image=nginx
pod/app2 created
root@cks-master:~# kubectl expose pod/app1 --port 80
service/app1 exposed
root@cks-master:~# kubectl expose pod/app2 --port 80
service/app2 exposed
root@cks-master:~# k get pod
NAME READY STATUS RESTARTS AGE
app1 1/1 Running 0 6m41s
app2 1/1 Running 0 6m35s
root@cks-master:~# k get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
app1 ClusterIP 10.105.1.235 <none> 80/TCP 5m34s
app2 ClusterIP 10.98.16.26 <none> 80/TCP 5m29s
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 2d19h
root@cks-master:~#
Now let's create the ingress rule to point to these two services.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: secure-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
ingressClassName: nginx
rules:
- http:
paths:
- path: /app1
pathType: Prefix
backend:
service:
name: app1
port:
number: 80
- path: /app2
pathType: Prefix
backend:
service:
name: app2
port:
number: 80
Note that the host doesn't matter to us at this moment.
root@cks-master:~# vim ingress.yaml
root@cks-master:~# k apply -f ingress.yaml
ingress.networking.k8s.io/secure-ingress created
root@cks-master:~# k get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
secure-ingress nginx * 80 21s
root@cks-master:~#
If we switch to the HTTPS port and try to curl, for example, we have a problem.
❯ curl https://34.71.254.120:32617/app1
curl: (60) SSL certificate problem: self-signed certificate
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
# The -k accepts self-signed certificates.
❯ curl https://34.71.254.120:32617/app1 -k
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
# Let's improve verbose to see what happened
❯ curl https://34.71.254.120:32617/app1 -kv
* Trying 34.71.254.120:32617...
* Connected to 34.71.254.120 (34.71.254.120) port 32617 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
* start date: Aug 18 14:55:52 2024 GMT
* expire date: Aug 18 14:55:52 2025 GMT
## If we don't define a certificate, nginx uses one created by it by default.
* issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
* SSL certificate verify result: self-signed certificate (18), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x6480f130beb0)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /app1 HTTP/2
> Host: 34.71.254.120:32617
> user-agent: curl/7.81.0
> accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200
< date: Sun, 18 Aug 2024 16:11:36 GMT
< content-type: text/html
< content-length: 615
< last-modified: Mon, 12 Aug 2024 14:21:01 GMT
< etag: "66ba1a4d-267"
< accept-ranges: bytes
< strict-transport-security: max-age=31536000; includeSubDomains
<
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection #0 to host 34.71.254.120 left intact
Documentation to create a TLS certificate.
Let's create a certificate for a domain and then use it in our nginx and then extend our rule to use the correct TLS.
The -nodes option in the openssl command is used to specify that the private key generated by the command should not be encrypted. When you create a private key using openssl, by default, you can choose to protect it with a password. This protection encrypts the key, requiring a password to be provided whenever the key is used.
We don't need to fill in anything, just the domain we're going to use.
root@cks-master:~# openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes
Generating a RSA private key
.....................................................................++++
................................................................................................................................................................++++
writing new private key to 'key.pem'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:studycks.com ## Here
Email Address []:
root@cks-master:~# ls *.pem
cert.pem key.pem
# Let's create our secret in kubernetes with tls.
# Let's make it clear that everything here was applied in the default namespace, so this key and default ingress rules are namespace resources.
root@cks-master:~# k create secret tls studycks --cert=cert.pem --key=key.pem
secret/studycks created
root@cks-master:~# k get ingress,secrets
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/secure-ingress nginx * 80 30m
NAME TYPE DATA AGE
secret/studycks kubernetes.io/tls 2 27s
Now we need to edit our ingress and add a tls section that will tell which certificate we're going to use.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: secure-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
ingressClassName: nginx
tls:
- hosts:
- studycks.com
secretName: studycks
rules:
# Now we're required to specify the host
- host: studycks.com
http:
paths:
- path: /app1
pathType: Prefix
backend:
service:
name: app1
port:
number: 80
- path: /app2
pathType: Prefix
backend:
service:
name: app2
port:
number: 80
root@cks-master:~# vim ingress.yaml
root@cks-master:~# k apply -f ingress.yaml
ingress.networking.k8s.io/secure-ingress configured
Returning to our local terminal, let's redo the test.
❯ curl https://34.71.254.120:32617/app1 -kv
* Trying 34.71.254.120:32617...
* Connected to 34.71.254.120 (34.71.254.120) port 32617 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
# We still have the same problem because we didn't use the domain. The ingress is filtering by domain, so if we use the IP directly we don't have a request initiated by studycks.com.
* subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
* start date: Aug 18 14:55:52 2024 GMT
* expire date: Aug 18 14:55:52 2025 GMT
* issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
* SSL certificate verify result: self-signed certificate (18), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x5d5f442d5eb0)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /app1 HTTP/2
> Host: 34.71.254.120:32617
> user-agent: curl/7.81.0
> accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 404
< date: Sun, 18 Aug 2024 16:35:29 GMT
< content-type: text/html
< content-length: 146
< strict-transport-security: max-age=31536000; includeSubDomains
<
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection #0 to host 34.71.254.120 left intact
Since this domain doesn't belong to us and is not a domain that can be resolved, we'll resolve it within our machine locally by saying that studycks.com:32617 will go to the machine's IP.
❯ curl https://studycks.com:32617/app1 -kv --resolve studycks.com:32617:34.71.254.120
* Added studycks.com:32617:34.71.254.120 to DNS cache
* Hostname studycks.com was found in DNS cache
* Trying 34.71.254.120:32617...
* Connected to studycks.com (34.71.254.120) port 32617 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd; CN=studycks.com # Now we're using the correct certificate.
* start date: Aug 18 16:26:42 2024 GMT
* expire date: Aug 18 16:26:42 2025 GMT
* issuer: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd; CN=studycks.com
* SSL certificate verify result: self-signed certificate (18), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x5c6e3ab84eb0)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /app1 HTTP/2
> Host: studycks.com:32617
> user-agent: curl/7.81.0
> accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200
< date: Sun, 18 Aug 2024 16:42:16 GMT
< content-type: text/html
< content-length: 615
< last-modified: Mon, 12 Aug 2024 14:21:01 GMT
< etag: "66ba1a4d-267"
< accept-ranges: bytes
< strict-transport-security: max-age=31536000; includeSubDomains
<
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection #0 to host studycks.com left intact