I'm running the latest version of Gitlab Community Edition within a Docker container behind a jwilder nginx Proxy. The Rack Attack Gem is disabled by configuration but when a Gitlab Runner job failed to authenticate against the registry (which is on the same domain but with port 4567) a red banner occurs within Gitlab:
An error occurred while fetching the job
When I refresh the Gitlab instance I just see 403 HTTP status which tells me, that I'm no longer allowed to access Gitlab. Even when I login with a different user in another Browser window (even a private window).
I'm not sure what's going on here and are willing to share a much information as possible. But I don't really know where to start because the only thing which makes sense was the Rack Attack plugin which is disabled.
I tried to build a new version of the docker container by throwing the current one away. I also cleared the bound redis instance just to ensure rack attack has really no affect on the current behaviour.
docker info:
Containers: 58
Running: 21
Paused: 0
Stopped: 37
Images: 615
Server Version: 18.09.0
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 1313
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: active
NodeID: ...
Is Manager: true
ClusterID: ...
Managers: 1
Nodes: 1
Default Address Pool: 10.0.0.0/8
SubnetSize: 24
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: ...
Manager Addresses:
...:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
runc version: 96ec2177ae841256168fcf76954f7177af9446eb
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-42-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.28GiB
Name: myserver.example.com
ID: JB7Q:3MNW:GJ6Q:...:EI5K:NKU3
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
location=cloud
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
WARNING: No swap limit support
docker-compose.yml
version: '3'
services:
gitlab:
image: 'gitlab/gitlab-ce:latest'
container_name: gitlab
environment:
- VIRTUAL_HOST=${REPOSITORY_HOST},${REGISTRY_HOST}
- VIRTUAL_PORT=80
- LETSENCRYPT_HOST=${REPOSITORY_HOST},${REGISTRY_HOST}
- LETSENCRYPT_EMAIL=${LETSENCRYPT_EMAIL}
- GITLAB_REGISTRY_ENABLED=true
- GITLAB_REGISTRY_HOST=${REPOSITORY_HOST}:4567
- GITLAB_REGISTRY_CERT_PATH=/certs/registry.example.com.chain.pem
- GITLAB_REGISTRY_KEY_PATH=/certs/registry.example.com.key
- REDIS_HOST=redis
- REDIS_PORT=6379
expose:
- 80
- 22
ports:
- '2222:22'
- '4567:4567'
restart: always
volumes:
- '/srv/gitlab/config:/etc/gitlab'
- '/srv/gitlab/logs:/var/log/gitlab'
- '/srv/gitlab/data:/var/opt/gitlab'
- '/srv/www/nginx-proxy/certs:/certs:ro'
logging:
driver: gelf
options:
gelf-address: ${GELF_HOST}
redis:
restart: always
image: sameersbn/redis:latest
command:
- --loglevel warning
volumes:
- /srv/gitlab/redis:/var/lib/redis:Z
logging:
driver: gelf
options:
gelf-address: ${GELF_HOST}
networks:
default:
external:
name: nginx-proxy
gitlab.rb configuration:
external_url 'https://repository.example.com'
gitlab_rails['gitlab_email_enabled'] = true
gitlab_rails['gitlab_email_from'] = '[email protected]'
gitlab_rails['gitlab_email_display_name'] = 'Gitlab'
gitlab_rails['gitlab_email_reply_to'] = '[email protected]'
gitlab_rails['lfs_enabled'] = true
gitlab_rails['omniauth_enabled'] = true
gitlab_rails['omniauth_allow_single_sign_on'] = false
gitlab_rails['omniauth_block_auto_created_users'] = true
gitlab_rails['redis_host'] = "redis"
gitlab_rails['redis_port'] = 6379
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp"
gitlab_rails['smtp_port'] = 25
gitlab_rails['smtp_user_name'] = "changeme"
gitlab_rails['smtp_password'] = "changeme"
gitlab_rails['smtp_domain'] = "example.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = false
gitlab_rails['smtp_tls'] = false
gitlab_rails['smtp_openssl_verify_mode'] = 'none'
registry_external_url 'https://repository.example.com:4567'
gitlab_rails['registry_enabled'] = true
registry['enable'] = true
redis['enable'] = true
nginx['listen_port'] = 80
nginx['listen_https'] = false
registry_nginx['enable'] = true
registry_nginx['listen_port'] = 4567
registry_nginx['listen_https'] = true
registry_nginx['ssl_certificate'] = "/certs/registry.example.com.crt"
registry_nginx['ssl_certificate_key'] = "/certs/registry.example.com.key"
registry_nginx['listen_addresses'] = ['*', '[::]']
registry_nginx['proxy_set_headers'] = {
"Host" => "$http_host",
"X-Real-IP" => "$remote_addr",
"X-Forwarded-For" => "$proxy_add_x_forwarded_for",
"X-Forwarded-Proto" => "https",
"X-Forwarded-Ssl" => "on"
}
.gitlab-ci.yml
image: php:7.3-fpm-alpine
cache:
paths:
- vendor/
- node_modules/
stages:
- build
- test
- release
variables:
DOCKER_DRIVER: overlay
NIGHTLY_IMAGE: $CI_REGISTRY/$CI_PROJECT_PATH:$CI_COMMIT_REF_NAME
LATEST_IMAGE: $CI_REGISTRY/$CI_PROJECT_PATH:$CI_BUILD_REF_NAME:latest
before_script:
- sh .gitlab/docker_install.sh > /dev/null
- docker info
- cp .env.example .env
- sed -i "s/CACHE_DRIVER=.*/CACHE_DRIVER=array/" .env
- sed -i "s/SESSION_DRIVER=.*/SESSION_DRIVER=array/" .env
- php composer.phar install
- php artisan key:generate
- npm install
- npm run prod
build:
stage: build
script:
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
- docker build --pull -t $NIGHTLY_IMAGE .
- docker build -t $NIGHTLY_IMAGE -f ./docker/php/Dockerfile.production --pull .
- docker push $NIGHTLY_IMAGE
- docker push $LATEST_IMAGE
test:7.2:
stage: test
image: php:7.2-fpm-alpine
script:
- ./vendor/bin/phpunit
- npm run test
test:7.3:
stage: test
image: php:7.3-fpm-alpine
script:
- ./vendor/bin/phpunit
- npm run test
release:
stage: release
script:
- docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
- docker pull $TEST_IMAGE
- docker tag $TEST_IMAGE $RELEASE_IMAGE
- docker push $RELEASE_IMAGE
only:
- master
.gitlab/docker_install.sh
#!/bin/sh
[[ ! -e /.dockerenv ]] && exit 0
set -xe
apk add --update \
docker \
libpq \
libpng-dev \
jpeg-dev \
freetype-dev \
libxrender \
fontconfig \
libxext-dev \
openldap-dev \
git \
nodejs nodejs-npm \
libzip-dev \
zip \
curl \
&& docker-php-ext-configure gd --with-freetype-dir=/usr/include/ --with-jpeg-dir=/usr/include/ \
&& docker-php-ext-configure zip --with-libzip \
&& docker-php-ext-install -j$(nproc) pdo_mysql mysqli gd zip ldap pcntl
php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
php composer-setup.php
php -r "unlink('composer-setup.php');"
curl --location --output /usr/local/bin/phpunit https://phar.phpunit.de/phpunit.phar
chmod +x /usr/local/bin/phpunit
curl -L https://github.com/docker/compose/releases/download/1.23.2/docker-compose-Linux-x86_64 > /usr/local/bin/docker-compose \
&& chmod +x /usr/local/bin/docker-compose
So currently I'm not able to login to Gitlab with ANY user. It must be possible to login again and I would like to understand what's going on here and why Gitlab blocks every login/request with a forbidden response.
Looks like it was a problem with Redis. Some postings mentioned a rack attack ban and I was pretty sure this wasn't the case because it's disabled by default, but after deleting flushing all Redis keys, I was able to use Gitlab again without 403 Forbidden errors.
https://gitlab.com/gitlab-org/gitlab-ce/issues/37090
So i did the following in my case:
$ docker exec -it redis bash
$ redis-cli
$ FLUSHALL