redis-server on AnyStack

How to install a Mastodon instance on Ubuntu 16.04 LTS

Thu, 06 Apr 2017 10:06:00 +0000

Have you already the new open-source social network Mastodon ? I have launched few days ago my own instance to contribute to this project.

If you want to launch your own instance, here a quick tutorial to install mastodon on a Ubuntu 16.04 LTS server with Nginx as a reverse-proxy and Cloudflare CDN/SSL.

To follow the steps of this tutorial, a user with sudo rights is enough, you don’t need to login as root.

Install the dependencies

sudo apt-get update

Install nodejs

curl -sL https://deb.nodesource.com/setup_6.x | sudo bash -
sudo apt-get install imagemagick ffmpeg libpq-dev libxml2-dev libxslt1-dev nodejs
sudo npm install -g yarn

Install redis-server

sudo apt-get install redis-server redis-tools

Install Postgresql

sudo apt-get install postgresql postgresql-contrib

sudo su - postgres
psql
CREATE USER mastodon CREATEDB;
\q

You will also have to enable ident authentication so users can login without password

sudo sed -i '/^local.*postgres.*peer$/a host all all 127.0.0.1/32 ident' \
/etc/postgresql/9.?/main/pg_hba.conf

And to install the ident daemon :

sudo apt-get install pidentd
sudo systemctl enable pidentd
sudo systemctl start pidentd
sudo systemctl restart postgresql

Install Ruby

Install the dependencies

sudo apt-get install autoconf bison build-essential libssl-dev libyaml-dev libreadline6-dev zlib1g-dev libncurses5-dev libffi-dev libgdbm3 libgdbm-dev

Create a user mastodon without password :

adduser --disabled-password --disabled-login mastodon

Then login as mastodon :

su - mastodon

Install rbenv and rbenv-build

git clone https://github.com/rbenv/rbenv.git ~/.rbenv
echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(rbenv init -)"' >> ~/.bashrc
source ~/.bashrc

We have to logout to apply the bash modification, and we install ruby-build.

exit
su - mastodon
git clone https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build

Then install ruby 2.4.1 for mastodon :

rbenv install 2.4.1
rbenv global 2.4.1

You can check the ruby version installed with the command :

ruby -v

Mastodon installation

We have installed all the dependencies required to setup Mastodon, so we can now clone the github repository. We use git checkout to select the last release instead of the master branch.

cd ~
git clone https://github.com/tootsuite/mastodon.git live
cd live
git checkout $(git tag | tail -n 1)

Install bundler to manage the dependencies and disable the gem documentation

echo "gem: --no-document" > ~/.gemrc
gem install bundler --no-ri

Then launch the installation

bundle install --deployment --without development test
yarn install

You can now copy the configuration sample file and edit with your own informations

cp .env.production.sample .env.production
nano .env.production

You have to add the following informations :

# Service dependencies
REDIS_HOST=localhost
REDIS_PORT=6379
DB_HOST=/var/run/postgresql
DB_USER=mastodon
DB_NAME=mastodon_production
DB_PASS=
DB_PORT=5432

# Federation
LOCAL_DOMAIN=yourdomain.com
LOCAL_HTTPS=true

Ainsi que la partie SMTP qui permettra aux utilisateurs de confirmer leur inscription :

# E-mail configuration
SMTP_SERVER=mail.yourdomain.com
SMTP_PORT=587
SMTP_LOGIN=noreply@yourdomain.com
SMTP_PASSWORD=YourPassword
SMTP_FROM_ADDRESS=noreply@yourdomain.com

For the application secret part, you can use the command bundle exec rake secret to generate the 3 secret keys, then you have just to copy them into the configuration file.

To setup the database and the assets :

RAILS_ENV=production bundle exec rails db:setup
RAILS_ENV=production bundle exec rails assets:precompile

Adding systemd services

Web service

nano /etc/systemd/system/mastodon-web.service

[Unit]
 Description=mastodon-web
 After=network.target

[Service]
 Type=simple
 User=mastodon
 WorkingDirectory=/home/mastodon/live
 Environment="RAILS_ENV=production"
 Environment="PORT=3000"
 ExecStart=/home/mastodon/.rbenv/shims/bundle exec puma -C config/puma.rb
 TimeoutSec=15
 Restart=always

[Install]
 WantedBy=multi-user.target

Background service

nano /etc/systemd/system/mastodon-sidekiq.service

[Unit]
 Description=mastodon-sidekiq
 After=network.target

[Service]
 Type=simple
 User=mastodon
 WorkingDirectory=/home/mastodon/live
 Environment="RAILS_ENV=production"
 Environment="DB_POOL=5"
 ExecStart=/home/mastodon/.rbenv/shims/bundle exec sidekiq -c 5 -q default -q mailers -q pull -q push
 TimeoutSec=15
 Restart=always

[Install]
 WantedBy=multi-user.target

API service

nano /etc/systemd/system/mastodon-streaming.service

[Unit]
 Description=mastodon-streaming
 After=network.target

[Service]
 Type=simple
 User=mastodon
 WorkingDirectory=/home/mastodon/live
 Environment="NODE_ENV=production"
 Environment="PORT=4000"
 ExecStart=/usr/bin/npm run start
 TimeoutSec=15
 Restart=always

[Install]
 WantedBy=multi-user.target

Then we can enable our systemd services :

systemctl enable /etc/systemd/system/mastodon-*.service

And we can start our mastodon instance :

sudo systemctl start mastodon-web.service mastodon-sidekiq.service mastodon-streaming.service

Add the required crons for mastodon :

crontab -u mastodon -e

RAILS_ENV=production
@daily cd /home/mastodon/live && RAILS_ENV=production /home/mastodon/.rbenv/shims/bundle exec rails mastodon:media:remove_remote

Nginx reverse-proxy setup :

So we have setup mastodon, but to access to our instance directly under https and with our domain, we need to setup a reverse-proxy using Nginx.

At first, install Nginx :

wget -O - https://nginx.org/keys/nginx_signing.key | sudo apt-key add -
sudo echo "deb http://nginx.org/packages/ubuntu/ $(lsb_release -sc) nginx" > /etc/apt/sources.list.d/nginx.list
sudo apt update
sudo apt install nginx

We have now to create our nginx configuration file for ou domain, we will use the configuration based on the Angristan’s model :

nano /etc/nginx/sites-enabled/yourdomain.com

map $http_upgrade $connection_upgrade {
 default upgrade;
 '' close;
}
server {
 listen 80;
 listen [::]:80;
 server_name www.yourdomain.com yourdomain.com;
 return 301 https://votredomaine.com$request_uri;

 access_log /dev/null;
 error_log /dev/null;
}

server {
 listen 443 ssl http2;
 listen [::]:443 ssl http2;
 server_name www.yourdomain.com yourdomain.com;

 access_log /var/log/nginx/yourdomain.com-access.log;
 error_log /var/log/nginx/yourdomain.com-error.log;

 ssl_certificate /etc/letsencrypt/live/fullchain.pem;
 ssl_certificate_key /etc/letsencrypt/live/privkey.pem;
 ssl_protocols TLSv1.2;
 ssl_ciphers EECDH+AESGCM:EECDH+AES;
 ssl_prefer_server_ciphers on;
 add_header Strict-Transport-Security "max-age=15552000; preload";

 keepalive_timeout 70;
 sendfile on;
 client_max_body_size 0;
 gzip off;

 root /home/mastodon/live/public;

 location / {
 try_files $uri @proxy;
 }

 location @proxy {
 proxy_set_header Host $host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto https;
 proxy_pass_header Server;
 proxy_pass http://127.0.0.1:3000;
 proxy_buffering off;
 proxy_redirect off;
 proxy_http_version 1.1;
 proxy_set_header Upgrade $http_upgrade;
 proxy_set_header Connection $connection_upgrade;
 tcp_nodelay on;
 }

 location /api/v1/streaming {
 proxy_set_header Host $host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto https;
 proxy_pass http://127.0.0.1:4000;
 proxy_buffering off;
 proxy_redirect off;
 proxy_http_version 1.1;
 proxy_set_header Upgrade $http_upgrade;
 proxy_set_header Connection $connection_upgrade;
 tcp_nodelay on;
 }

 error_page 500 501 502 503 504 /500.html;
}

You can start nginx with the command :

service nginx start

Then we will use Let’s Encrypt to generate a SSL certificate.

cd /opt
git clone https://github.com/letsencrypt/letsencrypt
cd letsencrypt
./letsencrypt-auto certonly --webroot -w /home/mastodon/live -d www.yourdomain.com -d yourdomain.com --email vous@yourdomain.com --text --rsa-key-size 4096

Updating Mastodon

Mastodon is updated very often, so if you want to keep your instance up-to-date you will have to follow the following steps :

su - mastodon
cd live
gem install bundler --no-ri
git fetch
git checkout v1.X.X
bundle install --deployment --without development test
NODE_ENV=production npm upgrade --global yarn
yarn install
RAILS_ENV=production bundle exec rails assets:clean
RAILS_ENV=production bundle exec rails assets:precompile
RAILS_ENV=production bundle exec rails db:migrate
exit

Then you just have to restart the instance :

sudo systemctl restart mastodon-web.service mastodon-sidekiq.service mastodon-streaming.service

If you need to update ruby with a new release use :

rbenv install 2.4.1
gem install bundler --no-ri

Create your administrator account

You instance is now running properly but you have to register to create your account and then you will be able to set this account as administrator with the following command :

RAILS_ENV=production bundle exec rails mastodon:make_admin USERNAME=votre-utilisateur

My instance mastodon.top

![mastodon.top/images/2017/08/Screenshot_44.png) With VirtuBox, I have launched a Mastodon instance and if you are not registered on Mastodon yet, you can register at mastodon.top. We are already more than 200 users on my instance and enough resources for more users. You can find more informations about the instance on the page about and the real-time monitoring of the server at instance.mastodon.top.

Don’t hesitate to give your opinion on Mastodon, to tell me if there are errors in the tutorial or to ask for help if you want to deploy your own instance.

You can also follow me on Mastodon : thomas_virtubox@mastodon.top

How to setup Magento 2.1.7 with EasyEngine - Nginx & Redis-cache

Mon, 07 Nov 2016 20:49:00 +0000

If you have already setup redis-server for object-caching with WordPress, you are maybe looking for something more powerful to setup ? Magento 2 will be the perfect benchmark for your server. It’s an open source e-commerce software and content management system for e-commerce websites based on the PHP Zend Framework. Magento is compatible with the most advanced caching systems including Varnish, Memcached or Redis.

So in this tutorial, we will see how to setup Magento 2.1.7 with Nginx, php7, Mariadb and Redis-server using EasyEngine.

At first, you need a VPS with ubuntu/debian and at least 2GB RAM. In production, 4GB will be the minimum, when 8GB will be better.

Server setup & configuration

## install easyengine
wget -qO ee rt.cx/ee && bash ee

## install Nginx - php5.6 + php7 - Mariadb - phpmyadmin - redis - memcached
ee stack install --web --admin --php7 --redis

## create your domain vhost (--letsencrypt optional)
ee site create yourdomain.com --mysql --php7

## install php-intl for magento and unzip
apt-get update && apt-get install php7.0-intl unzip

Change the php configuration according to magento documentation :

## change php.ini settings for magento
nano /etc/php/7.0/fpm/php.ini

max_execution_time = 1800
max_input_time = 1800
memory_limit = 4096M
post_max_size = 512M

## add the following lines after the extension declaration : [opcache]
opcache.enable=1
opcache.enable_cli=0
opcache.memory_consumption=128
opcache.interned_strings_buffer=4
opcache.max_accelerated_files=20000
opcache.max_wasted_percentage=5
opcache.validate_timestamps=1
opcache.revalidate_freq=2
opcache.optimization_level=0xffffffff

## restart php7.0-fpm
service php7.0-fpm restart

Magento installation

We have already done almost all the server configuration. Now we have to download magento and to launch the installer.

Download it from the Magento website : https://magento.com/tech-resources/download

To upload it via sftp, set a password for the user www-data with the command

passwd www-data

Then allow shell access for www-data with the command :

usermod -s /bin/bash www-data

After that you will be able to use the sftp client of your choice and to upload the magento archive in your website folder with the correct permissions. If you are looking for a sftp client for Windows, I recommend you Cyberduck.

Then go into your website folder and unzip the magento archive.

cd /var/www/yourdomain.com/htdocs
unzip Magento-CE-2.1.6_sample_data-2017-03-29-04-34-51.zip

## set www-data as the owner of the directory and all magento files
cd .. 
chown -R www-data:www-data htdocs

But to be able to launch the installer with nginx, we have to edit the site configuration, for that we will use the nginx.conf.sample file included with magento.

I have edit this file with the correct settings for EasyEngine and you can download it here : magento.conf

You have just to upload this file in /var/www/yourdomain.com/conf/nginx, because with easyengine, all configurations files in this folder are automatically include in your website configuration.

Then edit your website configuration in /etc/nginx/sites-available/yourdomain.com to look like :

server {

 server_name yourdomain.com www.yourdomain.com;
 set $MAGE_ROOT /var/www/yourdomain.com/htdocs;

 access_log /var/log/nginx/yourdomain.com.access.log rt_cache;
 error_log /var/log/nginx/yourdomain.com.error.log;

 include /var/www/yourdomain.com/conf/nginx/*.conf;
}

Check your nginx configuration with the command nginx -t and apply it with service nginx reload

We can now launch the installation by reaching the domain with our browser

Just follow the step, and use the databases user credentials available in the file /var/www/yourdomain.com/ee-config.php

To run properly, Magento require to add some cronjobs. If you are already logged as www-data, you have just to use the command : crontab -e and to add the following lines. Just replace yourdomain.com with your real domain.

* * * * * php /var/www/yourdomain.com/htdocs/bin/magento cron:run | grep -v "Ran jobs by schedule" >> /var/www/yourdomain.com/htdocs/var/log/magento.cron.log
* * * * * php /var/www/yourdomain.com/htdocs/update/cron.php >> /var/www/yourdomain.com/htdocs/var/log/update.cron.log
* * * * * php /var/www/yourdomain.com/htdocs/bin/magento setup:cron:run >> /var/www/yourdomain.com/htdocs/var/log/setup.cron.log

Some other useful command to get started with the magento cli

## deploy the magento sample data
php bin/magento sampledata:deploy

## upgrade your magento install
php bin/magento setup:upgrade

## Generates DI configuration after setup:upgrade
php bin/magento setup:di:compile

## clean magento cache
php bin/magento cache:clean

## backup magento
php bin/magento setup:backup

Your Magento instance is now running properly, but we want to make it faster. For that we will use redis for session storage and full-page caching. I let you read the Magento DevDocs to understand why Redis is better.

Use redis for session and page cache

So, to use Redis-server, there are no plugins like for WordPress, but we have just to edit the file app/etc/env.php and to set the correct settings to use redis for the session storage and the full page caching instead of files. You can use the following example : env.php

There are two blocks to add to use redis-server, it require to use a database for each block :

session caching

'session' => 
 array (
 'save' => 'redis',
 'redis' => 
 array (
	'host' => '127.0.0.1',
	'port' => '6379',
	'password' => '',
	'timeout' => '2.5',
	'persistent_identifier' => 'magento2:',
	'database' => '1',
	'compression_threshold' => '2048',
	'compression_library' => 'gzip',
	'log_level' => '1',
	'max_concurrency' => '6',
	'break_after_frontend' => '5',
	'break_after_adminhtml' => '30',
	'first_lifetime' => '600',
	'bot_first_lifetime' => '60',
	'bot_lifetime' => '7200',
	'disable_locking' => '0',
	'min_lifetime' => '60',
	'max_lifetime' => '2592000',
 ),

Full-page caching

'cache' =>
array(
 'frontend' =>
 array(
 'default' =>
 array(
 'backend' => 'Cm_Cache_Backend_Redis',
 'backend_options' =>
 array(
 'server' => '127.0.0.1',
 'port' => '6379',
 ),
 ),
 'page_cache' =>
 array(
 'backend' => 'Cm_Cache_Backend_Redis',
 'backend_options' =>
 array(
 'server' => '127.0.0.1',
 'port' => '6379',
 'database' => '3',
 'compress_data' => '0',
 ),
 ),
 ),
),

You can now refresh the Magento cache to make sure you are using Redis. You can also use EasyEngine to install phpRedisAdmin to see the redis databases with ee stack install --phpredisadmin

Multiple stores configuration

That’s one of the main feature of Magento, the ability to host several stores/websites on the same magento instance. At first you have to create a new website, store and store view in the Admin panel. For this part just follow the Magento Guide. Then create your additional nginx vhost with :

ee site create your2domain.com

And edit it to look like :

server {

 server_name your2domain.com www.your2domain.com;
 access_log /var/log/nginx/your2domain.com.access.log ;
 error_log /var/log/nginx/your2domain.com.error.log;

 set $MAGE_ROOT /var/www/yourdomain.com/htdocs;

 index index.html index.htm;

 include /var/www/yourdomain.com/conf/nginx/*.conf;
}

You have to make sure you have included the same magento.conf file than for the main website. After that we have to use the Nginx map directive to pass the values of the Magento variables to nginx, so we will create a file map.conf in /etc/nginx/conf.d with the following content :

map $http_host $MAGE_RUN_CODE {
 yourdomain.com yourdomain-code;
 your2domain.com your2domain-code;
}

So if you want to add another store later, you will just have to create another vhost and to add a line in this file with the new domain and the associated website code. Then edit the file /etc/nginx/fastcgi_params and add the following line at the end :

fastcgi_param MAGE_RUN_TYPE website;
fastcgi_param MAGE_RUN_CODE $MAGE_RUN_CODE;

Reload nginx with nginx -t && service nginx reload and you can now check if your two stores are working properly.

If you have any issue to setup Magento with EasyEngine, post a reply on the Magento thread already available. I will answer to your questions.

New VPS for the Blog - Setup Guide

Fri, 05 Aug 2016 18:59:00 +0000

I was using a VPS with Plesk Onyx to host the Blog, but even with Redis in a Docker container to make it faster, it wasn’t enough …

So I have setup a small VPS on my dedicated server with Ubuntu 16.04 LTS using VMware vSphere 6. I will make another post about virtualization but in this post, we will talk only about the VPS setup with easyengine, which is really my favorite tool for WordPress setup. You can also read my first post about easyengine : Here

First step, easyengine install :

 wget -qO ee rt.cx/ee && bash ee

Then WordPress setup :

 ee site create anystack.xyz --wpredis --php7 --letsencrypt

And it’s already good for the server setup with nginx, php7, mariadb, redis-cache and let’s encrypt for the SSL. Check your DNS are pointing to your server before launching let’s encrypt or it will fail to setup the SSL certificate.

But the work is not finished yet, because I have to upload my database and my files to get my content back. For that i will need to have access to the database, so let’s do that with phpmyadmin :

 ee stack install --phpmyadmin

phpmyadmin is now available at http://my-server-ip:22222

Butt protected by an ACL, and to change the user and the password, you just have to use :

 ee secure --auth

And to have your login details for phpmyadmin :

 ee site info anystack.xyz

So I have now uploaded my database, and I need my files. But it’s not needed to install ftp. The easiest way to do, is to add a password to www-data and to allow sftp access :

 passwd www-data

And you will just have to replace /usr/sbin/nologin by /bin/bash in the file /etc/passwd. As FTP/SFTP client, I use Cyberduck which is the fastest client I have found, and you can use it for almost all protocol, including S3 storage at AWS or Google Drive.

After uploading my files, just have to purge the redis cache, and that’s already over.

Ubuntu - Blazing fast WordPress with Nginx php7 redis

Sun, 26 Jun 2016 00:37:00 +0000

You have buy a VPS to speed up WordPress, but you are using a control panel and it's not faster than before ? In this tutorial, you will learn how to setup a Blazing fast WordPress with Nginx php7 and redis cache.

This will require to use the the command linux via SSH. Yes I know, it's less user friendly than a control panel. But it will not require any resources after the install and we have an amazing tools to help us : EasyEngine

So, first we will talk about what we are going to install and why it will make WordPress faster.

Nginx : To pronounce "Engine X", it's the web server created in 2002, created to handle high traffic with a low memory footprint. And this is the reason which explain why Nginx is so popular at the moment. It can also be used as a reverse proxy or as a load-balancer, which means if you have few small VPS, you will be able to create a stable and reliable platform to host your website.

php7 : I will not explain what is the php but more about the difference between the 5.6 and the 7. It's the performance and there is nothing better than a small graphic to compare them. And WordPress is fully compatible with the latest version of php.

MariaDB : If you haven't already used it, MariaDB is a fork of Mysql from his creator, and it's an open-source solution which is currently faster than Mysql with a lower memory and processor footprint. So no need to hesitate, if you are still using msql, even in 5.7, it's time to move !

Redis : What is Redis ? It's a remote dictionary server, which will manage your database and store the values in the memory. It's a "NoSQL" solution as Redis can be used as a database by writing data on your ~~hard drive~~ SSD. For WordPress there is no faster solution as it will eliminate the queries to your database, and even if you are using Mariadb, it will be faster and using less resources.

But you are maybe thinking it will be very hard to configure all that stuff properly. No, you will be able to do it in less than 10 minutes ! Because we are going to use EasyEngine, the most powerful tools to create an optimized WordPress website.

It's a command line tool to create install and configure a complete web server with Nginx, PHP, MariaDB and Postfix in a single command. It also provide you the ability to manage properly your virtualhosts with Nginx and make it easy with simple command to use.

So to start you need a VPS with Ubuntu 14.04. EasyEngine is also compatible with debian 7/8 and Ubuntu 16.04. But as EE has been build at first for ubuntu, it's the most stable solution.

At first we need to install EasyEngine.

wget -qO ee rt.cx/ee && sudo bash ee

Done, EasyEngine is now ready to install what we need. So we are looking for WordPress with nginx, php7, redis and as it will not be harder, a SSL with let's encrypt. But you have the choice with EE :

A simple website for html :

ee site create example.com --html

A website with php :

ee site create example.com --php

A website with mysql :

ee site create example.com --mysql

A website with SSL by let's encrypt

ee site create example.com --letsencrypt

And for WordPress ? You just have to choose :

ee site create example.com --wp # install wordpress without any page caching

ee site create example.com --w3tc # install wordpress with w3-total-cache plugin

ee site create example.com --wpsc # install wordpress with whisp-super-cache plugin

ee site create example.com --wpfc # install wordpress + nginx fastcgi_cache

ee site create example.com --wpredis # install wordpress + nginx redis_cache

So for my example, I will use :

ee site create yourwebsite.com --wpredis --letsencrypt --php7

You will have to enter Yes to the EE questions, and yes your wordpress with redis php7 and SSL is already installed and configured. You also have two plugins in WordPress :

Nginx Helper to clear the cache
Redis Cache : To connect WordPress with Redis and to clear the cache

But you will maybe ask how you can connect to the database, what is the user of mysql and wordpress ? You can use a custom user during the install process by using

--user=admin and -pass=password to set WordPress login details.

But you can also look at the install log to find them as EE display it in the console :

nano /var/log/ee/install.log

To see all the commands available with the websites :

ee site

If you want to edit some default configuration (database, user etc) :

nano /etc/ee/ee.conf

By default EasyEngine create a backend access with all tools you can install with it (phpmyadmin, cache, php infos) at the address : http://yourserverIP:22222

To secure it properly just use :

ee secure --auth

It will ask you for a user and a password. You can edit it at anytime.

To edit the default port :

ee secure --port

And to get the mysql user of a website :

ee site info yoursite.com

So I recommend you to change the port of admin tools, and to install what you need in the following :

Adminer (phpmyadmin alternative )

ee stack install --adminer

phpMyAdmin :

ee stack install --phpmyadmin

phpAdminRedis (to manage data in cache) :

ee stack install --phpredisadmin

All the Admin tools (Adminer, phpMyAdmin, phpMemcachedAdmin, FastCGI cleanup script, OPcache, Webgrind, Anemometer) :

ee stack install --admin

You have to agree this tool is really awesome. And if you look to your memory usage using htop, you will see the difference with a control panel. Your website will be faster and you have the ability to edit your configuration or to add module when you want.

So to conclude this tutorial here the links you will need to understand perfectly EasyEngine and to use it for all your setup :

Leave a comment if you have any question !