Setting up Zero in Docker environment with https #9712

8 Answer(s)

• 

  1

  zony created 3 years ago

  Support Team

  Hi ips-ad, In response to these issues, my suggestions are as follows:

  1. For different mirror versions, you can use tag tags to distinguish between versions. I suggest pushing to your private mirror warehouse after the CI/CD pipeline is built.
  2. In our practice, we used environment variables instead of appsettings. json. The best way is to use the configuration center, here I recommend using Apollo, its related documents are: https://github.com/ctripcorp/apollo
  3. Our practice is to use http for communication at the back end, NGINX for reverse proxy at the front end, and SSL formal configuration on NGINX.

  Save Cancel
• 

  0

  ipservant created 3 years ago

  Hi zony,

  thank you very much for the information! We followed your suggestion and have set up a reverse proxy environment based on an OPNsense firewall with NGINX plugin acting as reverse proxy with SSL offloading, but having issues with occasional 504 errors now.

  The setup looks like this: WAN (public IP, port 4443/https) - OPNsense with NGINX - ZeroHost (private IP, port 8081/http on Docker container)

  At first glance, the ZeroHost seems to be reachable fine, the certificate is also accepted by the browsers and Swagger UI is working. But then, after a couple of minutes and/or calls to the API (I didn't find a 100% reliable way to reproduce yet) the page is giving 504 errors by the NGINX proxy. If I open a second browser window (in incognito mode to make sure it has no "past"), then the page is reacting fine again until, after more or less the same time as described above, I'm getting 504s again. The ZeroHost was always replying fine directly on the Docker port though (tested with curl on the OPNsense's CLI), so I don't believe in "real" timeout of the app, but a proxy issue.

  This is what NGINX logs say about it: *777 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 5.171.xx.78, server: subdomain.our-domain.com, request: "GET /api/services/app/Objekte/GetDistinctAkteForObjekte HTTP/2.0", upstream: "http://172.20.xx.2:8081/api/services/app/Objekte/GetDistinctAkteForObjekte", host: "subdomain.our-domain.com:4443", referrer: "https://subdomain.our-domain.com/"

  Before NGINX I was trying the same setup with pfSense and HAProxy, which showed the exact same behavior, with the following log of HAProxy, where the sH-- flag is stating something like "server-side timeout" and "proxy was waiting for complete, valid response HEADERS from the server":

  [2020-10-11T11:16:14.857080] Incoming log entry; line='<134>Oct 11 13:16:14 haproxy[78394]: 116.202.xxx:54896 [11/Oct/2020:13:15:14.839] frontend-host~ backend-host_ipvANY/172.20.x.2 0/0/1/-1/60018 504 194 - - sH-- 1/1/0/0/0 0/0 "GET /swagger/index.html HTTP/1.1"\x0a'

  What are we missing?! Followed really basic howtos on the setups using the GUI tools of pfSense/HAproxy and OPNsense/NGINX to create the proxy configs and played around with various settings, but with no luck so far. Do you have a working NGINX template for SSL offloading as reference? I'm not posting mine (yet) as it is quite long (and this post already is..), but I can of course do that if necessary.

  Any help is appreciated!

  Save Cancel
• 

  0

  zony created 3 years ago

  Support Team

  Hi ips-ad, It seems that NGINX thinks that your backend cannot be connected. I am not sure about your network environment. I have not used OPNsense, I suggest you start NGINX directly through the Docker image, and put the NGINX container and dotnet system under the same docker network. You can open ports 80 and 443 of the NGINX container, so you can forward through the firewall. The following are the specific steps and related configuration files:

  1. Shell run docker network create internal-network.
  2. Download Docker-Compose and install it.
  3. Write Docker-Compose file and run it with docker-compose up -d.
  4. Write nginx configuration file for reverse proxy.

  Warning

  1. The dotnet system must be mounted on the internal-system network.
  2. The /etc/nginx/conf.d/ of the NGINX container maps to the path you specify.
  3. After the configuration is changed successfully, you need to restart the nginx container, or execute the docker exec -ti nginx /etc/init.d/nginx reload command.

  Docker Compose File

  version: '3'
  services:
    nginx:
      container_name: "nginx"
      image: nginx
      ports:
        - "80:80"
        - "443:443"
      volumes:
        # ~/Volumes/Nginx/Configs needs to be created in advance to map NGINX configuration files.
        - ~/Volumes/Nginx/Configs:/etc/nginx/conf.d
      restart: always
  
  networks:
    internal-network:
      external: true
  

  NGINX Config File

  server {
      listen 443 ssl http2;
  
      server_name your.domain.com;
  
      ssl on;
  
      # your ssl files.
      ssl_certificate   /etc/nginx/conf.d/your_ssl.pem;
      ssl_certificate_key  /etc/nginx/conf.d/your_ssl.key;
      
      ssl_session_timeout 5m;
      ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE:ECDH:AES:HIGH:!NULL:!aNULL:!MD5:!ADH:!RC4;
      ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
      ssl_prefer_server_ciphers on;
      underscores_in_headers on;
  
      location / {
          proxy_redirect off;
          proxy_set_header Host $host;
          proxy_set_header X-real-IP $remote_addr;
          proxy_set_header X-NginX-Proxy true;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto $scheme;
          
          # your backend service(docker container name).
          proxy_pass http://core-service:80;
      }
  }
  

  Save Cancel
• 

  0

  ipservant created 3 years ago

  Thanks, I followed that and set up a separate NGINX docker instance for frontend and host each. I had to add proxy_http_version 1.1; to make SignalR/Websocket work, but there is still an issue left that I can't switch language or tenant at the very first login page (it just reloads with the previous language and always as host, no tenant). Am I missing any config on NGINX? This happens only if I route host throgh NGINX and regardless of http or https.

  Here is my NGINX config for host:

      upstream myapp1 {
          server 10.0.0.3:9901;
      }
  
  server {
      listen 443 ssl http2;
  
      server_name _;
  
      ssl on;
  
      # your ssl files.
      ssl_certificate   /etc/nginx/conf.d/xxx.crt;
      ssl_certificate_key  /etc/nginx/conf.d/xxx.key;
  
      ssl_session_timeout 5m;
      ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE:ECDH:AES:HIGH:!NULL:!aNULL:!MD5:!ADH:!RC4;
      ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
      ssl_prefer_server_ciphers on;
      underscores_in_headers on;
  
      location / {
  proxy_http_version      1.1;
          proxy_redirect off;
          proxy_set_header Host $host;
          proxy_set_header X-real-IP $remote_addr;
          proxy_set_header X-NginX-Proxy true;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto $scheme;
  
          # your backend service(docker container name).
          proxy_pass http://myapp1;
      }
  }
  

  ...and the part that came with the docker image:

  user  nginx;
  worker_processes  1;
  
  error_log  /var/log/nginx/error.log warn;
  pid        /var/run/nginx.pid;
  
  
  events {
      worker_connections  1024;
  }
  
  
  http {
      include       /etc/nginx/mime.types;
      default_type  application/octet-stream;
  
      log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                        '$status $body_bytes_sent "$http_referer" '
                        '"$http_user_agent" "$http_x_forwarded_for"';
  
      access_log  /var/log/nginx/access.log  main;
  
      sendfile        on;
      #tcp_nopush     on;
  
      keepalive_timeout  65;
  
      #gzip  on;
  
      include /etc/nginx/conf.d/*.conf;
  }
  

  Again, thanks for your advice!

  Save Cancel
• 

  1

  ismcagdas created 3 years ago

  Support Team

  Hi,

  You can try this for tenant and language switch problem, see http://nginx.org/en/docs/http/ngx_http_core_module.html#ignore_invalid_headers

  Save Cancel
• 

  0

  ipservant created 3 years ago

  Cool, we found the same approach yesterday and it seems to work fine so far. These are now our host and ng configs for the NGINX proxies, maybe they will be useful as referencs for others, too. Do you mind cross-checking them once more in regards of system availability and safety? Thanks!

  upstream zero_hosts {
      server 10.0.0.3:9901;
      #more servers for load balancing/redundancy later
  }
  
  server {
      listen 443 ssl http2;
      server_name _;
      underscores_in_headers on;
      ignore_invalid_headers off;
  
      # your ssl files.
      ssl_certificate   /etc/nginx/conf.d/ip-servant.com_bundle.crt;
      ssl_certificate_key  /etc/nginx/conf.d/ip-servant.com.key;
      ssl_session_timeout 5m;
      ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE:ECDH:AES:HIGH:!NULL:!aNULL:!MD5:!ADH:!RC4;
      ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
      ssl_prefer_server_ciphers on;
  
      location / {
          proxy_redirect off;
          proxy_http_version 1.1;
          proxy_set_header Upgrade $http_upgrade;
          proxy_set_header Connection $http_upgrade;
          proxy_set_header Host $host;
          proxy_set_header X-real-IP $remote_addr;
          proxy_set_header X-NginX-Proxy true;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto $scheme;
  
          # your backend service(docker container name).
          proxy_pass http://zero_hosts;
      }
  }
  

  upstream zero_ngs {
      server 10.0.0.3:9902;
  }
  
  server {
      listen 80 default_server;
      server_name _;
      return 301 https://$host$request_uri;
  }
  
  server {
      listen 443 ssl http2;
      server_name _;
      underscores_in_headers on;
  
      # your ssl files.
      ssl_certificate   /etc/nginx/conf.d/ip-servant.com_bundle.crt;
      ssl_certificate_key  /etc/nginx/conf.d/ip-servant.com.key;
      ssl_session_timeout 5m;
      ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE:ECDH:AES:HIGH:!NULL:!aNULL:!MD5:!ADH:!RC4;
      ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
      ssl_prefer_server_ciphers on;
  
      location / {
          proxy_redirect off;
          proxy_set_header Host $host;
          proxy_set_header X-real-IP $remote_addr;
          proxy_set_header X-NginX-Proxy true;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto $scheme;
  
          # your backend service(docker container name).
          proxy_pass http://zero_ngs;
      }
  }
  

  Save Cancel
• 

  0

  zony created 3 years ago

  Support Team

  Hi ips-ad, If you need to be compatible with http, you can add the following configuration to the CONFIG file.

  server {
      listen 80;
      server_name your_domain;
      return 301 https://your_domain$request_uri;
  }
  

  Save Cancel
• 

  0

  ipservant created 3 years ago

  Yes, we have done that for the frontend already under the same IP, and there is currently no need to have the API redirect to https automatically as all the appsettings/appconfigs point towards https directly.

  Thanks guys for the support, I think we're fine here to close the request :-)

  Save Cancel