Contents
Last update: 11.06.2020
DIY BigBlueButton Server
The FD Seminar runs on a cloud server with the following specifications:
We found these to be optimal for running a single BigBlueButton session with close to 100 participants.
Below I outline the steps followed to set up our BigBlueButton instance.
Happy hacking!
Gustavo
P.S. FWIW Help the World by Healing Your NGINX Configuration.
Disclaimer: I am not an expert in server administration (not even close). The information below is provided for reference only and under no warranty of applicability or suitability. Use at your own risk!
BigBlueButton
The FD Seminar uses BigBlueButton (BBB)—an open-source web conferencing software—installed following the BBB official installation guide.
Greenlight
Greenlight is the standard end-user interface for the BBB server (needed to manage users, rooms, and recordings). It can be installed following the Greenlight official installation guide. For customising Greenlight to fit the needs of the seminar, I followed the Greenlight official customisation guide.
TURN server
Some users experienced problems when trying to join audio sessions in our BigBlueButton instance (error 1007). We set up a TURN server following the BBB official guide for setting up a TURN server to solve this issue.
These are the specifications of our TURN server:
Live streaming
We use OBS Studio to stream the talks to an RTMP server which we set up following this guide at the NGINX blog, see also this guide for configuring the CORS headers. We stream using MPEG-DASH) and indigo-player.
These are the specifications of our RTMP server:
Firewall
We use UFW to protect our servers by allowing incoming traffic only through a custom SSH port as well as the ports required by BBB to function, see for example this guide provided by Linode.
Server monitoring
We use Grafana and Prometheus to monitor our BBB instance. We the following Docker container provided by a third-party developer.
21.05.2020
The FD Seminar ran on a cloud server with the following specifications:
We found these to be sub-optimal for running a single BBB session with 77 participants and 8 shared webcams (some users reported low audio quality).
Automation
We manage our servers with Ansible; the playbooks below are run in the following order: TURN → BBB → Greenlight.
Updating BigBlueButton
# https://docs.bigbluebutton.org/2.2/install.html
# https://docs.bigbluebutton.org/2.2/customize.html
- hosts: BBB server's hostname
become: yes
vars:
fqdn: FQDN of your BBB server
fqdn_turn: FQDN of your turn server
custom_ssh_port: 22
tasks:
- name: register cronjob for updating let's encrypt certificates
cron:
name: "renew letsencrypt certs"
minute: "30"
hour: "2"
weekday: "1"
job: "/usr/bin/certbot renew >> /var/log/le-renew.log"
- name: register cronjob for restarting nginx updating let's encrypt certificates
cron:
name: "restart nginx after renewing letsencrypt certs"
minute: "35"
hour: "2"
weekday: "1"
job: "/bin/systemctl reload nginx"
- name: deny all incoming connections
ufw:
default: deny
direction: incoming
- name: allow all outgoing connections
ufw:
default: allow
direction: outgoing
- name: allow connections to required port (SSH)
ufw:
rule: allow
port: '{{ custom_ssh_port }}'
- name: allow connections to port 80 (HTTP)
ufw:
rule: allow
port: '80'
proto: tcp
- name: allow connections to port 433 (TLS)
ufw:
rule: allow
port: '443'
proto: tcp
- name: allow connections to port 16384:32768 (BBB)
ufw:
rule: allow
port: '16384:32768'
proto: udp
- name: enable uncomplicated firewall (UFW)
ufw:
state: enabled
- name: update all packages to the latest version
apt:
upgrade: dist
update_cache: yes
- name: restore FQDN
command: bbb-conf --setip {{ fqdn }}
- name: configure FreeSWITCH to support ipv6
copy:
src: files/nginx/bigbluebutton_sip_addr_map.conf
dest: /etc/nginx/conf.d/bigbluebutton_sip_addr_map.conf
- name: replace static ip (v4) by $freeswitch_addr (ipv4+ipv6) in nginx configuration file for use with FreeSWITCH
lineinfile:
path: /etc/bigbluebutton/nginx/sip.nginx
regexp: 'proxy_pass'
line: 'proxy_pass https://$freeswitch_addr:7443;'
state: present
- name: enable 3pcc in FreeSWITCHipv6 sip profile
lineinfile:
path: /opt/freeswitch/etc/freeswitch/sip_profiles/external-ipv6.xml
regexp: ' <!--<param name="enable-3pcc" value="true"/>-->'
line: ' <param name="enable-3pcc" value="true"/>'
state: present
- name: configure TURN server
copy:
src: files/coturn/turn-stun-servers.xml
dest: /usr/share/bbb-web/WEB-INF/classes/spring/turn-stun-servers.xml
owner: bigbluebutton
group: bigbluebutton
mode: u=rw,g=r,o=r
- name: configure FreeSWITCH to use own TURN server (rtp)
lineinfile:
path: /opt/freeswitch/etc/freeswitch/vars.xml
regexp: 'external_rtp_ip=stun:'
line: ' <X-PRE-PROCESS cmd="set" data="external_rtp_ip=stun:{{ fqdn_turn }}"/>'
state: present
- name: configure FreeSWITCH to use own TURN server (sip)
lineinfile:
path: /opt/freeswitch/etc/freeswitch/vars.xml
regexp: 'external_sip_ip=stun:'
line: ' <X-PRE-PROCESS cmd="set" data="external_sip_ip=stun:{{ fqdn_turn }}"/>'
state: present
- name: get local username
local_action: command whoami
become: no
register: local_user
- name: retrieve current TURN static auth secret
shell: ssh {{ local_user.stdout }}@{{ fqdn_turn }} cat /etc/turnserver.conf | grep static-auth-secret= | awk -F '=' '{print $NF}'
register: current_secret
delegate_to: 127.0.0.1
become_user: "{{ local_user.stdout }}"
- name: update TURN static auth secret
replace:
path: /usr/share/bbb-web/WEB-INF/classes/spring/turn-stun-servers.xml
regexp: 'STATIC_AUTH_SECRET'
replace: "{{ current_secret.stdout }}"
- name: set URL for default presentation
replace:
path: /usr/share/bbb-web/WEB-INF/classes/bigbluebutton.properties
regexp: 'beans.presentationService.defaultUploadedPresentation=.*'
replace: 'beans.presentationService.defaultUploadedPresentation=https://www.fd-seminar.xyz/pdf/bbb-welcome.pdf'
- name: set default welcome message
replace:
path: /usr/share/bbb-web/WEB-INF/classes/bigbluebutton.properties
regexp: 'defaultWelcomeMessage=.*'
replace: 'defaultWelcomeMessage=Welcome to <b>%%CONFNAME%%</b>!<br/><br/>A live stream of the talk is also available at <a href="https://www.fd-seminar.xyz/live" target="_blank"><u>https://www.fd-seminar.xyz/live</u></a><br/><br/>Username: fd-seminar<br/>Password: access code for this meeting<br/><br/>Minimal instructions on how to use BigBlueButton are available <a href="https://www.fd-seminar.xyz/pdf/bbb-howto.pdf">here</a>'
- name: set default welcome message (footer)
replace:
path: /usr/share/bbb-web/WEB-INF/classes/bigbluebutton.properties
regexp: 'defaultWelcomeMessageFooter=.*'
replace: 'defaultWelcomeMessageFooter=Thank you for joining!'
- name: set maximum presentation page number
replace:
path: /usr/share/bbb-web/WEB-INF/classes/bigbluebutton.properties
regexp: 'maxNumPages=.*'
replace: 'maxNumPages=1000'
- name: set maximum presentation file size
replace:
path: /usr/share/bbb-web/WEB-INF/classes/bigbluebutton.properties
regexp: 'maxFileSizeUpload=.*'
replace: 'maxFileSizeUpload=30000000'
- name: mute meetings on start by default
replace:
path: /usr/share/bbb-web/WEB-INF/classes/bigbluebutton.properties
regexp: 'muteOnStart=.*'
replace: 'muteOnStart=true'
- name: force HTML5 client (participants)
replace:
path: /usr/share/bbb-web/WEB-INF/classes/bigbluebutton.properties
regexp: 'attendeesJoinViaHTML5Client=.*'
replace: 'attendeesJoinViaHTML5Client=true'
- name: force HTML5 client (moderators)
replace:
path: /usr/share/bbb-web/WEB-INF/classes/bigbluebutton.properties
regexp: 'moderatorsJoinViaHTML5Client=.*'
replace: 'moderatorsJoinViaHTML5Client=true'
- name: disable "you are now muted" feedback sound
replace:
path: /opt/freeswitch/etc/freeswitch/autoload_configs/conference.conf.xml
regexp: ' <param name="muted-sound" value="conference/conf-muted.wav"/>'
replace: ' <!-- <param name="muted-sound" value="conference/conf-muted.wav"/> -->'
- name: disable "you are now unmuted" feedback sound
replace:
path: /opt/freeswitch/etc/freeswitch/autoload_configs/conference.conf.xml
regexp: ' <param name="unmuted-sound" value="conference/conf-unmuted.wav"/>'
replace: ' <!-- <param name="unmuted-sound" value="conference/conf-unmuted.wav"/> -->'
- name: set camera defaults
shell: >
yq w -i /usr/share/meteor/bundle/programs/server/assets/app/config/settings.yml public.kurento.cameraProfiles.[0].bitrate 50
yq w -i /usr/share/meteor/bundle/programs/server/assets/app/config/settings.yml public.kurento.cameraProfiles.[1].bitrate 100
yq w -i /usr/share/meteor/bundle/programs/server/assets/app/config/settings.yml public.kurento.cameraProfiles.[2].bitrate 200
yq w -i /usr/share/meteor/bundle/programs/server/assets/app/config/settings.yml public.kurento.cameraProfiles.[3].bitrate 300
yq w -i /usr/share/meteor/bundle/programs/server/assets/app/config/settings.yml public.kurento.cameraProfiles.[0].default true
yq w -i /usr/share/meteor/bundle/programs/server/assets/app/config/settings.yml public.kurento.cameraProfiles.[1].default false
yq w -i /usr/share/meteor/bundle/programs/server/assets/app/config/settings.yml public.kurento.cameraProfiles.[2].default false
yq w -i /usr/share/meteor/bundle/programs/server/assets/app/config/settings.yml public.kurento.cameraProfiles.[3].default false
- name: delete raw data from unpublished recordings after 3 days
replace:
path: /etc/cron.daily/bigbluebutton
regexp: 'recorded_days=.*'
replace: 'recorded_days=3'
- name: delete raw data from published recordings after 3 days
replace:
path: /etc/cron.daily/bigbluebutton
regexp: 'published_days=.*'
replace: 'published_days=3'
- name: disable echo test
replace:
path: /usr/share/meteor/bundle/programs/server/assets/app/config/settings.yml
regexp: ' skipCheck:.*'
replace: ' skipCheck: true'
- name: restart BigBlueButton
command: bbb-conf --restart
- name: restart nginx
service:
name: nginx
state: restarted
enabled: yes
Updating Greenlight
# https://docs.bigbluebutton.org/greenlight/gl-customize.html
- hosts: bbb.fd-seminar.xyz
tasks:
- name: update greenlight sources
command: git pull upstream master
args:
chdir: /var/docker/greenlight
become: yes
- name: push greenlight sources to sourcehut
command: git push origin fd-seminar
args:
chdir: /var/docker/greenlight
become: yes
- name: stop greenlight
command: docker-compose down
args:
chdir: /var/docker/greenlight
become: yes
- name: rebuild greenlight
command: /var/docker/greenlight/scripts/image_build.sh fd-seminar-bbb release-v2
args:
chdir: /var/docker/greenlight
become: yes
- name: start greenlight
command: docker-compose up -d
args:
chdir: /var/docker/greenlight
become: yes
Updating the TURN server
# https://docs.bigbluebutton.org/2.2/setup-turn-server.html
- hosts: TURN server's hostname
become: yes
vars:
custom_ssh_port: 22
tasks:
- name: deny all incoming connections
ufw:
default: deny
direction: incoming
- name: allow all outgoing connections
ufw:
default: allow
direction: outgoing
- name: allow connections to required port (SSH)
ufw:
rule: allow
port: '{{ custom_ssh_port }}'
- name: allow connections to port 443/tcp (TLS)
ufw:
rule: allow
port: '443'
proto: tcp
- name: allow connections to port 433/udp (TLS)
ufw:
rule: allow
port: '443'
proto: udp
- name: allow connections to port 3478/tcp (COTURN)
ufw:
rule: allow
port: '3478'
proto: tcp
- name: allow connections to port 3478/udp (COTURN)
ufw:
rule: allow
port: '3478'
proto: udp
- name: allow connections to port 49152:65535/udp (relay ports)
ufw:
rule: allow
port: '49152:65535'
proto: udp
- name: enable uncomplicated firewall (UFW)
ufw:
state: enabled
- name: update all packages to the latest version
apt:
upgrade: dist
update_cache: yes
- name: full system update
apt:
update_cache: yes
upgrade: dist
- name: install coturn
apt:
name: coturn
state: present
- name: set TLS listening port to port 443 (to bypass firewalls)
lineinfile:
path: /etc/turnserver.conf
regexp: 'tls-listening-port=.*'
line: 'tls-listening-port=443'
- name: enable fingerprints in TURN messages
lineinfile:
path: /etc/turnserver.conf
regexp: '#fingerprint'
line: 'fingerprint'
state: present
- name: enable long-term credential mechanism
lineinfile:
path: /etc/turnserver.conf
regexp: '#lt-cred-mech'
line: 'lt-cred-mech'
state: present
- name: enable secret-based authentication
lineinfile:
path: /etc/turnserver.conf
regexp: '#use-auth-secret'
line: 'use-auth-secret'
state: present
- name: generated static auth secret
shell: openssl rand -hex 16
register: new_secret
- name: update static auth secret
lineinfile:
path: /etc/turnserver.conf
regexp: 'static-auth-secret=.*'
line: 'static-auth-secret={{ new_secret.stdout }}'
state: present
- name: set default realm
lineinfile:
path: /etc/turnserver.conf
regexp: 'realm=.*'
line: 'realm=fd-seminar.xyz'
state: present
- name: set path to let's encrypt certificate file
lineinfile:
path: /etc/turnserver.conf
regexp: 'cert=.*'
line: 'cert=/etc/letsencrypt/live/TURN-SERVER-URL/fullchain.pem'
state: present
- name: set path to let's encrypt private key
lineinfile:
path: /etc/turnserver.conf
regexp: 'pkey=.*'
line: 'pkey=/etc/letsencrypt/live/TURN-SERVER-URL/privkey.pem'
state: present
- name: limit the allowed ciphers
lineinfile:
path: /etc/turnserver.conf
regexp: 'cipher-list=.*'
line: 'cipher-list="ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS"'
state: present
- name: enable longer DH TLS key
lineinfile:
path: /etc/turnserver.conf
regexp: '#dh2066'
line: 'dh2066'
state: present
- name: set path to log file
lineinfile:
path: /etc/turnserver.conf
regexp: 'log-file=.*'
line: 'log-file=/var/log/coturn.log'
state: present
- name: enable logging into a single file
lineinfile:
path: /etc/turnserver.conf
regexp: '#simple-log'
line: 'simple-log'
state: present
- name: disable TLS/DTLS protocols v1
lineinfile:
path: /etc/turnserver.conf
regexp: '#no-tlsv1'
line: 'no-tlsv1'
state: present
- name: disable TLS/DTLS protocols v1.1
lineinfile:
path: /etc/turnserver.conf
regexp: '#no-tlsv1_1'
line: 'no-tlsv1_1'
state: present
- name: log to a single file
lineinfile:
path: /etc/turnserver.conf
regexp: 'log-file=.*'
line: 'log-file=/var/log/coturn.log'
state: present
- name: copy log rotation conf
copy:
src: files/coturn/coturn
dest: /etc/logrotate.d/coturn
owner: root
group: root
mode: u=rw,g=r,o=r
- name: enable coturn service in conf file
lineinfile:
path: /etc/default/coturn
regexp: '#TURNSERVER_ENABLED=.*'
line: 'TURNSERVER_ENABLED=1'
state: present
- name: enable coturn service
service:
name: coturn
state: restarted
enabled: yes