1. Provisioning

This document provides a step-by-step introduction to the process of installing the full stack for Carto open-source onto a self-hosted server running CentOS 7. We've tried to provide enough narration along the way to make this guide useful to anyone with a cursory level of knowledge of linux and web servers and a high level of persistence. This guide was authored by Jeremy Kidwell at the University of Birmingham, but is the product of a collaboration with a range of persons who helped along the way, see the footnotes for credits and enjoy!

1. Provisioning

CartoDB can use a range of resources. For our purposes, which involve installing an instance of CartoDB on a virtual server hosted at the University of Birmingham for the Mapping Community project we plan to host a range of POI datasets, shapefiles, and demographic data pertaining to community groups in the UK, with the eventual goal of hosting data for the whole of Europe and whoever wants to join in the fun (Austrailiasia anyone?).

We set up a local PostgreSQL server with PostGIS extensions in order to test the size of these datasets and projected a maximum use for year 1 of this server of 500GB. For a whole host of best-practice reasons, we've provisioned this setup with a web server and PostGIS server hosted separately (but on the same LAN). We will run a parallel set of servers in a development platform which will mirror the production setup.

Firewall Configuration

If you're working with virtual servers that will be behind a firewall, this installation guide will require the following configuration to work. For the sake of this example, "Server01" is the web server and "Server02" is the postgresql server.

Server01 should be open to inbound (http) traffic on ports 80, 443, 9090 (SQL API), 9191 (Maps API) for any IP address.
Server02 should be open to inbound (postgresql) traffic on port 5432 from Server01

You will also likely want to have ports open for management, so leaving open port 22 for ssh etc.

2. Installation

A Few Notes on Converting "CartoDB Ubuntu" to Centos...

The CartoDB installation process is well-documented in the official documentation, however this process is explicitly linked to Ubuntu 12.04 x64. Though relatively similar to Ubuntu, CentOS has a different package manager and dependencies will be handled in a slightly different way, so the purpose of this documentation is to indicate departures from that official documentation for our platform.

The basic components included in CartoDB include the following:

PostgreSQL (note, as of May 2017 CartoDB requires PostgreSQL 9.5.x and PostGIS 2.2.x - check the official docs at http://cartodb.readthedocs.io/en/latest/components/postgresql.html to see if this has changed)
Redis (note, CartoDB requires redis 3.x version - check here to see if this has changed
NodeJS 6.9 (the Carto platform is on the way towards transitioning from the former requirement of 0.10 towards 6.x the latter of which we've installed with and tested here)\ and npm 3.10.9 (same as before, so upgrading from official carto doc recommendation of 2.14.16)\
Ruby (recommended Ruby 2.2.3)
GEOS 3.5.0, GDAL 1.11 (though note that CartoDB uses ogr2ogr2 version 2.1.2 in parallel for some features), Mapnik, ImageMagick
unp, zip, JSON-C, PROJ4
CartoDB SQL API (found at: git://github.com/CartoDB/CartoDB-SQL-API.git)
CartoDB MAPS API (found at git://github.com/CartoDB/Windshaft-cartodb.git)
Editor (found at [https://github.com/CartoDB/cartodb.git](https://github.com/CartoDB/cartodb.git\)

Packages are not directly equivalent in the CentOS package manager "yum" to Ubuntu's "apt-get", so we need to deviate sightly from those packages named in the Ubuntu-based CartoDB install. Best practice is to first check for an easy equivalent which uses the same name in yum. If this doesn't exist, you have two options. You can look for an error message in your compile process and use the command yum whatprovides "/[filename here]* or you can have a look at the actual contents of the Ubuntu package specified using their website here: [http://packages.ubuntu.com](http://packages.ubuntu.com%29%29 and try to find a package that bundles the same components %28or individually!%29 using yum. You can read a bit more about [how to install software on CentOS here]%28https://www.centos.org/forums/viewtopic.php?f=12&t=871) and then use yum search [pattern to match in package name]. It is also worth noting that the CartoDB Ubuntu install guide includes several instances where the guide needs packages which are beyond the standard Ubuntu repositories as well. In this case, they provide access to a Carto-specific Ubuntu PPA, PPA being an acronym for Personal Package Archive and essentially just a bundle of precompiled software known to be compatible with a specific system type. You can view the contents of the CartoDB/GIS PPA by clicking here. If you're running into compatibility issues or are wondering which version of a component is getting installed, check out the PPA which provides a very specific list!

Here's an example of how we've done this:

One of the first install commands in Ubuntu is to install the following using apt-get:

sudo apt-get install autoconf binutils-doc bison build-essential flex

A quick search using the command line reveals that there are some obvious yum equivalents:

sudo yum install autoconf bison flex

This is fine, and we should install these straight-away. But with the above example, we can see that "binutils-doc" and "build-essential" don't have exact name equivalents in yum. With some careful sleuthing we've found that the proper packages are¹:

sudo yum install binutils
sudo yum install gcc gcc-c++ make openssl-devel tcl

You won't need to go through this process here as we've already identified all the equivalent packages in this guide (or recommended installation from source code in some cases), but in future cases, there may be additional dependencies that need to be resolved, so hopefully enterprising users will be able to conduct appropriate reconnaissance.

Two other things that are worth noting about the translation from Ubuntu to CentOS/RHEL - paths are specified differently, and often CentOS is more conservative about not passing paths to the shell, this means that you may get file not found errors even when you know that you've already installed the proper package. We address this in a few ways below, but usually good to confirm first when you hit an erorr that it isn't just a matter of a path missing, especially if you're using sudo. The other matter to be aware of is that CentOS is tremendously conservative in terms of adopting new frameworks, so we'll just install straight from source below in some cases rather than bothering with the process of trying to get the code installed via a package.

That's it for now let's get started...

2.1 PostgresSQL Server

Note, for our purposes, this guide makes use of two virtual servers . CartoDB can be run on a single server (which is how the Ubuntu instructions go), so this is just our approach here. We begin by configuring our second virtual server which will host PostgreSQL. This order is necessary because the database is needed when we do the configuration on the Web Server:

a. Set System Locale²

localedef -c -f UTF-8 -i en_GB en_GB.UTF-8
export LC_ALL=en_GB.UTF-8

b. Install Basic Requirements:

Begin by installing some of the basic required components of your server:

sudo yum install autoconf bison flex binutils
sudo yum install gcc gcc-c++ make openssl-devel tcl nano wget
sudo yum install dejavu-sans-fonts lato-fonts open-sans-fonts

In contrast to Ubuntu, the main CentOS repository lacks many server packages, so it is typical to use the EPEL ([Extra Packages for Enterprise Linux](https://fedoraproject.org/wiki/EPEL/FAQ#howtouse%29\) repository and "voila" some of the packages noted below, including "redis" and "postgis" become available:

sudo rpm -Uvh http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-9.noarch.rpm

Note: these versions change, so you may want to browse to http://dl.fedoraproject.org/pub/epel/7/x86_64/e/ to see what the latest version is if this doesn't work.

c. Install Git

You will need git commands in order to handle some repositories and install some dependencies:

sudo yum install git

Note: in the Ubuntu guide, there is a series of packages listed under "APT Tools", in case you're wondering, we've integrated these elsewhere already in this guide, so no need to worry!

d. PostgreSQL

Get postgresql 9.5 repository installed

Note: The default in CentOS at the time this guide was put together was 9.2, so you'll want to be sure that you specify the proper repository. We've also altered all the below to be sure they are specific to 9.5.

Here's how you add the additional repository for postgresql 9.5:

sudo rpm -Uvh https://download.postgresql.org/pub/repos/yum/9.5/redhat/rhel-7-x86_64/pgdg-centos95-9.5-3.noarch.rpm
sudo yum update

Install client packages:

sudo yum install postgresql95 postgresql95-odbc PyGreSQL

Note: we were just fine skipping these ubuntu packages from the official Carto install doc: libpq5 libpq-dev postgresql-client-common (many bits are already dependencies or specified in other ways above)

Install server packages³:

sudo yum install postgresql95-server postgresql95-contrib postgresql95-devel postgresql95-plpython

Now we need to initialize our database:

sudo /usr/pgsql-9.5/bin/postgresql95-setup initdb

Start the service:

sudo systemctl start postgresql-9.5

Enable the service to run on startup:

sudo systemctl enable postgresql-9.5.service

Modify the pg_hba.conf file (this is sort of like the firewall configuration for the database server) to specify whether it should expect connections from outside, which is quite likely when working with PostGIS. We like to use nano as our text editor: sudo nano /var/lib/pgsql/9.5/data/pg_hba.conf

The contexts of your pg_hba.conf should be something like the following (with much of the documentation/comments from the actual file excluded here), you should probably make it the following (for now):

# TYPE  DATABASE        USER            ADDRESS                 METHOD
# "local" is for Unix domain socket connections only, modified from stock to "trust" as per CartoDB docs
local   all             all                                     trust
# this addition from CartoDB docs
local   all             postgres                                trust
# IPv4 local connections:
host    all             all             127.0.0.1/32            trust
# Allow Web Server connection:
host    all             Postgres        _webserver IP address_  trust
# IPv6 local connections:
host    all             all             ::1/128                 ident
# Allow replication connections from localhost, by a user with the
# replication privilege.
#local   replication     postgres                                peer
#host    replication     postgres        127.0.0.1/32            ident
#host    replication     postgres        ::1/128                 ident

Note: Be sure you substitute an ip address for your web server as indicated above "webserver IP address" and don't forget to include a "/32" after the address (this is the "netmask").

Restart server for changes to take effect:

sudo systemctl restart postgresql-9.5

Now create your Postgres users:

sudo createuser publicuser --no-createrole --no-createdb --no-superuser -U postgres
sudo createuser tileuser --no-createrole --no-createdb --no-superuser -U postgres

Install the CartoDB postgresql extension.

This extension contains functions that are used by different parts of the CartoDB platform, included the Editor and the SQL and Maps API (as below).

cd /opt
git clone https://github.com/CartoDB/cartodb-postgresql.git
cd cartodb-postgresql/

Note: for the next line, have a look at https://github.com/CartoDB/cartodb-postgresql/tags to see what the latest tag is, and then substitute in the line below - this will detach your clone from the branch which may continue to develop! For this install, we've used 0.18.5. It is also worth noting that environment variables aren't passed by default in some linux distributions, including CentOS, so we need to explicitly pass the path variable or the command will fail.⁴

git checkout <LATEST cartodb-postgresql tag>
export PATH=/usr/pgsql-9.5/bin:$PATH
sudo env "PATH=$PATH" make all install

Note: tags for an existing install can be discovered using the command git describe --tags

e. GIS dependencies

Install the following dependencies:

sudo yum install proj proj-devel json-c json-c-devel python-simplejson geos geos-devel gdal gdal-devel gdal-libs gdal-devel 
sudo yum install libjpeg-turbo libjpeg-turbo-devel ImageMagick-devel giflib-devel pango-devel

f. PostGIS

Install PostGIS

sudo yum install libxml2-devel
sudo yum install postgis2_95 postgis2_95-devel libxml2-devel

Initialize template postgis database. We create a template database in postgresql that will contain the postgis extension. This way, every time CartoDB creates a new user database it just clones this template database

sudo createdb -T template0 -O postgres -U postgres -E UTF8 template_postgis
sudo createlang plpgsql -U postgres -d template_postgis
psql -U postgres template_postgis -c 'CREATE EXTENSION postgis;CREATE EXTENSION postgis_topology;'
sudo ldconfig

Run an installcheck to verify the database has been installed properly

sudo PGUSER=postgres make installcheck # to run tests

Note: check https://github.com/cartodb/cartodb-postgresql for further reference

Restart PostgreSQL after all this process

sudo systemctl restart postgresql-9.5

If you are working with a 2 server configuration, you may also want to edit the postgresql.conf file to tell the server to listen to hosts beyond "localhost".

sudo nano  /var/lib/pgsql/9.5/data/postgresql.conf

Edit the line beginning with # listen_addresses = 'localhost' # what IP address(es) to listen on;

At this point your PostgresSQL server installation should be all done, now let's move on to the web server installation:

2.2 Web Server

Login to Cartodb Web server ("Server01").

a. Set System Locale²

localedef -c -f UTF-8 -i en_GB en_GB.UTF-8
export LC_ALL=en_GB.UTF-8

b. Install Basic Requirements:

sudo yum install autoconf bison flex binutils
sudo yum install gcc gcc-c++ make openssl-devel tcl

c. Install Git

You will need git commands in order to handle some repositories and install some dependencies:

sudo yum install git

d. PostgreSQL

Even though we're not running PostgreSQL on this server, the web server will require many of the same libraries, so we'll go through a similar process to what we began above for this server here. Add the additional repository for postgresql 9.5:

sudo rpm -Uvh https://download.postgresql.org/pub/repos/yum/9.5/redhat/rhel-7-x86_64/pgdg-centos95-9.5-3.noarch.rpm
sudo yum update

Install following packages:

sudo yum install postgresql95 postgresql95-odbc PyGreSQL postgresql95-devel postgresql95-plpython

e. GIS dependencies

Install the following dependencies:

sudo yum install proj proj-devel json-c json-c-devel python-simplejson geos geos-devel gdal gdal-devel gdal-libs gdal-devel 
sudo yum install libjpeg-turbo libjpeg-turbo-devel ImageMagick-devel giflib-devel pango-devel

f. PostGIS

Install PostGIS

sudo yum install libxml2-devel
sudo yum install postgis2_95 postgis2_95-devel libxml2-devel

g. Redis

Now that we've finished the basics, and replicated essential postgresql components, we'll move on to installation of the web server and the Carto stack.

Centos is too conservative to provide redis 3.x in a package, so we must install from source. At the time of this writing, 3.2.8 was the latest "stable" release, but it's worth clicking here to check first in case things have moved along. Get and install redis:

cd ~/
wget http://download.redis.io/releases/redis-3.2.8.tar.gz
tar xzf redis-3.2.8.tar.gz
cd redis-3.2.8
make

Note: if you follow the documentation on redis and attempt a "make test", make sure you first check for running "redis" processes and kill any that are still running you can find these processes using the following command: ps -ef|grep redis then just run the command: kill <process id number>

Now install redis as a production server:

sudo make install
cd utils
chmod +x install_server.sh
sudo ./install_server.sh

Note, the dialogue which will ensue when you run install _server.sh, will ask you to specify the path to CLI executable. This should be "/usr/local/bin/redis-server"

Check the status of your redis server:

service redis_6379 status

If you want to allow outside access to the redis server (which is not necessary, and I can't imagine why one might do so!), you should definitely set a password for redis. Open the file for editing using: nano /etc/redis/6379.conf then look for the line beginning with # requirepass foobared and replace "foobared" with a strong alternative. Finish by restarting the redis server with service redis_6379 restart

h. NodeJS⁵

As above, it is easier just to install Node.js from source so you can be sure you have the right version installed. On your server, use wget and paste the link that you copied in order to download the archive file:

Note: at the time of writing, we will need nodeJS v6.9.2 to work with Windshaft-cartodb.

cd ~/
wget http://nodejs.org/dist/v6.9.2/node-v6.9.2.tar.gz

Extract the archive and move into the new directory by typing:

tar xzvf node-v* && cd node-v*

Configure and compile the software:

./configure --prefix=/usr
make

The compilation will take a wee while. When it is finished, you can install the software onto your system by typing:

sudo make install

To check that the installation was successful, you can ask Node and NPM (the node package manager) to display its version number:

node --version
npm -v

i. Create "Carto" User With Sudo Privilege

Now let's create a user account which the carto services can run under so that they aren't using your linux user account.

sudo adduser carto

Use the passwd command to update the new user's password (choose a strong option here, as you've already done for redis above. Do write it down, don't use the same password you've used for reds.):

sudo passwd carto

Use the usermod command to add the user to the wheel group.

sudo usermod -aG wheel carto

You're going to need to login to this new user account so that file permissions and ownership are set properly when you install the CartoDB stack in a moment.

su carto

j. Ruby

Download ruby-install. Ruby-install is a script that makes ruby install easier. It’s not needed to get ruby installed but it helps in the process.

cd ~/
wget -O ruby-install-0.5.0.tar.gz https://github.com/postmodern/ruby-install/archive/v0.5.0.tar.gz
tar -xzvf ruby-install-0.5.0.tar.gz
cd ruby-install-0.5.0/
sudo make install

Install some ruby dependencies

sudo yum install readline-devel bzip2

Install ruby 2.2.3. CartoDB has been deeply tested with Ruby 2.2.

sudo env "PATH=$PATH" ruby-install ruby 2.2.3

Ruby-install will leave everything in /opt/rubies/ruby-2.2.3/bin. To be able to run ruby and gem later on, you’ll need to add the Ruby 2.2.3 bin folder to your PATH variable. It’s also a good idea to include this line in your bashrc so that it gets loaded on restart

export PATH=$PATH:/opt/rubies/ruby-2.2.3/bin

Install bundler. Bundler is an app used to manage ruby dependencies. It is needed by CartoDB’s editor

sudo env "PATH=$PATH" gem install bundler

Install compass. It will be needed later on by CartoDB’s editor

sudo env "PATH=$PATH" gem install compass

k. GCC Library

The system wide GCC library from Centos 7 is incompatible with CartoDB MAP API. Therefore we will need to manually compile the library and make it available for later use. The minimum version of GCC is v5.1.0.

First install prerequisites:

sudo yum install gmp gmp-devel mpfr mpft-devel libmpc libmpc-devel zip unzip gdal-devel

The install GCC, from which we'll extract the library:

cd ~/
wget http://gnu.uberglobalmirror.com/gcc/gcc-5.1.0/gcc-5.1.0.tar.bz2
tar -xjf gcc-5.1.0.tar.bz2
cd gcc-5.1.0
./configure --disable-multilib
make

Note: the compile will take a while as there are huge number of objects to run through.

At this point, we do not need to "make install" as we only need to copy the library file (libstdc++.so.6.0.21) to the system /lib64 folder

sudo cp ./prev-x86_64-unknown-linux-gnu/libstdc++-v3/src/.libs/libstdc++.so.6.0.21 /lib64/

Update current symbolic link of the library

sudo rm /lib64/libstdc++.so.6
sudo ln -s /lib64/libstdc++.so.6.0.21 /lib64/libstdc++.so.6

l. GDAL 2.1.2

The Centos repository only provides GDAL v1.11.4 so we will need to install the latest GDAL v2.1.2 from source. It was originally the case (and is still documented) that Carto uses two versions of GDAL in parallel, in order to borrow some features that were reintroduced in GDAL 2.x, but this use seems to have been deprecated. Our installation here will install GDAL v2.1.2 without any other side loaded versions of GDAL.

cd ~/
wget http://download.osgeo.org/gdal/2.1.2/gdal-2.1.2.tar.gz
tar -xzf gdal-2.1.2.tar.gz
cd gdal-2.1.2
./configure --with-geos=yes --with-pg=/usr/pgsql-9.5/bin/pg_config --prefix=/usr
make
sudo make install

Note: the two flags included above for 'configure' are very important. Make sure that GEOS support shows "yes" and that the install script is able to find pg_config and PostgresQL.

m. Add unp (decompression tool) to Centos

CartoDB requires the unp package to decompress any zip files. But unp is not available as standard RPM package for Centos, as it was designed for Debian and ported to Ubuntu. Luckily for us, because it is a perl script, the process of porting to Centos is nonexistent. One only needs to download the script straight from the Debian repository and make it executable:

cd /usr/bin
sudo wget https://sources.debian.net/data/main/u/unp/2.0~pre7+nmu1/unp
chmod 755 /usr/bin/unp

All done. Let's move on to begin installing CartoDB components:

n. CartoDB Components

Editor

Download the editor code:

cd /opt
sudo git clone --recursive https://github.com/CartoDB/cartodb.git
sudo chown -R carto cartodb
cd cartodb

Install pip:

sudo wget  -O /tmp/get-pip.py https://bootstrap.pypa.io/get-pip.py
sudo python /tmp/get-pip.py

Install a necessary package for python dependencies:

sudo yum install python-devel

Install dependencies

sudo yum install ImageMagick patch
export PATH=$PATH:/usr/pgsql-9.5/bin/:/opt/rubies/ruby-2.2.3/bin
RAILS_ENV=production bundle install --deployment --without development test
npm install
sudo env "PATH=$PATH" pip install --no-use-wheel -r python_requirements.txt --global-option=build_ext --global-option="-I/usr/include/gdal"

Reminder: environment name is used above. Also note that "bundle install" above will fail if "/usr/pgsql-9.5/bin and "/opt/rubies/ruby-2.2.3/bin" is not included in your $PATH statement. Best practices on CentOS suggest that your path should be modified using a script in /etc/profile.d.

Add the grunt command to the PATH:

export PATH=$PATH:$PWD/node_modules/grunt-cli/bin

Install all necessary gems:

bundle install

Precompile assets. Note that the last parameter is the environment used to run the application. It must be the same used in the Maps and SQL APIs

cp config/grunt_development.json config/grunt_production.json
ulimit -n 2560
bundle exec grunt --environment production

Reminder: environment name here, but note that we're using the template for development to make our "grunt_production.json". You can omit this first line above if working with a "development" environment as there is (obviously) already a config file for grunt_development.

Create configuration files:

cp config/app_config.yml.sample config/app_config.yml
cp config/database.yml.sample config/database.yml

Note: modify database.yml to include details of your database (in our case using the details of Server02) if you're not using localhost.

Initialise the metadata database:

RAILS_ENV=production bundle exec rake db:create
RAILS_ENV=production bundle exec rake db:migrate

Reminder: above should be modified if using "development" environment.

Now modify configuration files for the carto web application:

nano /opt/cartodb/config/app_config.yml

Change the line starting with: session_domain: '.localhost.lan' to: session_domain: 'carto.mapping.community'

Change the line starting with: subdomainless_urls: false to: subdomainless_urls: true

Change the line starting with: vizjson_cache_domains: ['.localhost.lan'] to # vizjson_cache_domains: ['.localhost.lan'] (it should be commented out)

(JK Note: next line unsure - may need to revert this to a side loaded configuration, cf. original Readme.md from May, content under "Now finally edit the hosts file to include localhost.lan and whatever your server URL will be")

Change the line starting with: account_host: 'localhost.lan:3000' to: account_host: 'carto.mapping.community'

Make the following changes to the sql_api section of this file as below:

  sql_api:
    private:
      protocol:   'https'
      domain:     'carto.mapping.community'
      endpoint:   '/api/v1/sql'
      port:       9090
    public:
      protocol:   'https'
      domain:     'carto.mapping.community'
      endpoint:   '/api/v2/sql'
      port:       9090

Make the following changes to the tiler section of this file as below:

  tiler:
    filter: 'mapnik'
    internal:
      protocol:      'https'
      domain:        'carto.mapping.community'
      port:          '9191'
      host:          'carto.mapping.community'
      verifycert:     false
    private:
      protocol:      'https'
      domain:        'carto.mapping.community'
      port:          '9191'
      verifycert:     false
    public:
      protocol:      'https'
      domain:        'carto.mapping.community'
      port:          '9191'
      verifycert:     false

Save the app_config.yml file and then edit the hosts file to include localhost.lan and whatever your server URL will be:

sudo nano /etc/hosts

Add the following line:

127.0.0.1   localhost.lan carto.mapping.community

Create /etc/carto so we can edit all the config files in here

sudo mkdir /etc/carto
cd /etc/carto
sudo ln -s /opt/cartodb/config

Create /var/log/carto so we can put all the logs files in here

sudo mkdir /var/log/carto
sudo chmod 777 /var/log/carto

o. CartoDB APIs

All the hard work above has paid off, as you have a working server environment that will provide the basis for an installation of the Carto platform:

SQL API:

Download API:

Note, here and below we need to copy all the core cartodb files into the system-wise web server paths so that they aren't being served out of our home directory. We're going with /opt for our files here (to see why, read here):

cd /opt
sudo git clone git://github.com/CartoDB/CartoDB-SQL-API.git
sudo chown -R carto CartoDB-SQL-API
cd /opt/CartoDB-SQL-API
git checkout master

Note: if you want to install a different version of this API, be sure to change "master" in the above to reflect either the tags/[name of tag] or the exact version name.

Install npm dependencies

npm install

Create your configuration files from the templates provided in the config directory. Note, the name of the filename of the configuration must be the same than the environment you are going to use to start the service.

Note: departing from the standard Carto instruction set, we've specified the environment as production, because we are running two parallel environments (yep, 4 virtual servers). Do make a note of the fact that the name of the environment will crop up increasingly often further below, so best to do a word search for "production" and substitute as necessary if you want to run a "development" environment.

cp config/environments/production.js.example config/environments/production.js

Note: As you'll already know, our project is hosted at carto.mapping.community. You'll want to substitute every instance of "mapping.community" in the prescribed configuration changes here and below for your own DNS name otherwise the server will definitely not work.

We need to modify the SQL-API configuration file so that the server will work with our DNS name:

nano /opt/CartoDB-SQL-API/config/environments/production.js

Change the setting for module.exports.node_host to:

module.exports.node_host    = '';

Change the line starting with: module.exports.user_from_host to:

module.exports.user_from_host = '^(.*)\\.carto\\.mapping\\.community$';

Note: hostname to change above!

Change the setting for module.exports.db_host to your PostgreSQL server IP:

module.exports.db_host      = 'PostgresSQL server IP address';

Change the setting for module.exports.db_port to:

module.exports.db_port      = '5432';

Change allowedHosts to use your domain name:

module.exports.oauth = {
    allowedHosts: ['carto.mapping.community']

Note: hostname to change above!

Change module.exports.log_filename to a central log location:

module.exports.log_filename = '/var/log/carto/log/SQL-API_log'

Now, try to start the service to confirm that it is installed correctly (note, again the second parameter is always the environment if the service. Remember to use the same you used in the configuration).

node app.js production

Reminder: see note a few lines above re: "production" here which is "development" in carto documentation.

MAPS API:

Download the API:

cd /opt
sudo git clone git://github.com/CartoDB/Windshaft-cartodb.git
sudo chown -R carto Windshaft-cartodb
cd Windshaft-cartodb
git checkout master

Install npm dependencies:

npm install

Create configuration. The name of the filename of the configuration must be the same than the environment you are going to use to start the service. Let’s assume it’s production

Reminder: environnment name below (as noted above)!

cp config/environments/production.js.example config/environments/production.js

You need to make some crucial modifications to the windshaft configuration file:

Reminder: specific DNS settings below!

nano /opt/Windshaft-cartodb/config/environments/production.js

Change the setting for ,host: to be empty, as below:

var config = {
    environment: 'production'
    ,port: 8181
    ,host: ''

Change the setting for ,user_from_host to reflect your hostname, formatted as follows:

,user_from_host: 'carto\\.mapping\\.community'

Change the setting under ,postgres for host and port to reflect your PostgreSQL settings (as follows), substituting your IP for 'postgres server IP' below and 5432 for port:

,postgres: {
        // Parameters to pass to datasource plugin of mapnik
        // See http://github.com/mapnik/mapnik/wiki/PostGIS
        user: "publicuser",
        password: "public",
        host: 'postgres server IP',
        port: 5432,

Under ,analysis change the endpoint url to match your server url, using the following convention:

            endpoint: 'http://carto.mapping.community:8080/api/v2/sql/job'

and change the hostHeaderTemplate in the same way:

            hostHeaderTemplate: '{{=it.username}}.carto.mapping.community'

Under ,millstone change the setting for cache_basedir to

        cache_basedir: '/opt/cartodb/tile_assets/'

Change log_filename to a central log location:

log_filename: '/var/log/carto/log/node-windshaft.log'

Finally, make sure you change the options for cnd_url to be null as follows:

  cdn_url: {
    http: '',
    https: '',

Save the changes you've made above to the windshaft configuration file and proceed.

Now, create the cache_basedir folder if it doesn't exist:

mkdir /opt/cartodb/tile_assets/

Start the service to check if it is installed correctly. The second parameter is always the environment of the service. Remember to use the same you used in the configuration.

node app.js production

Reminder environnment name above (as noted above)!

If this step fails saying log folder not found, you may need to manually create it, e.g. mkdir /opt/Windshaft-cartodb/logs

p. Install Apache and Passenger

Now we need a web server which we'll use with passenger to connect to the Carto rails server. NGINX is what Carto uses for their production servers, but we've gone with Apache for the sake of convenience. There are some indications that performance is better for high-volume installations using NGINX, so that's the main reason to defer to the alternative.

Because Carto is fundamentally a rails web server, we will need to use Passenger to connect the rails web server to Apache. By default, web applications running behind rails use non-standard ports. We will use the passenger web proxy along with the apache web server to redirect traffic to default web server ports 80/443.

For this section, you may want to brief the very helpful guides located on the Passenger website (for a non-rvm setup into NGINX on CentOS) and Passenger App install guide (starting from step 3).

Install Apache:

sudo yum install httpd

Install Passenger:

sudo env "PATH=$PATH" gem install passenger

Set SELinux to permissive:

sudo setenforce 0

Install some dependencies:

sudo yum install libcurl-devel httpd-devel

Run the Passenger Apache module installer:

sudo env "PATH=$PATH" passenger-install-apache2-module

The module installer will take you through a brief dialogue. You can just hit the enter key after each prompt to confirm that you are happy with the default selection (a "Ruby" focussed install), it will then compile and install passenger for apache.

q. Generate self-signed SSL certificate

Because we're using https, we need to assign some secure certificates to our web server. Note, you should strongly resist any suggestions that you need to pay for https certificates. There is absolutely no reason to pay a Certificate authority for your certificates when perfectly secure and respected services like letsencrypt.com exist! We'll start with self-signed certificates and then go on to install letsencrypt certificates which will be automatically renewed using certbot below:

sudo mkdir /etc/httpd/certs/ 
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/httpd/certs/OurKey.key -out /etc/httpd/certs/OurCert.crt

r. Configure Apache/Passenger with virtualhost and reverse proxy

Add a new config file passenger.conf

cd /etc/httpd/conf.d/
sudo nano passenger.conf

Paste the following into your new passenger.conf file and save:

Note: be sure you change DNS below to your own hostname; also IMPORTANT - check to see what version of passenger was installed by the above command and adjust the lines below to reflect the proper path name, which includes the version ("5.1.3" here, but 5.1.7 as of the last revision of this document).

   LoadModule passenger_module /opt/rubies/ruby-2.2.3/lib/ruby/gems/2.2.0/gems/passenger-5.1.3/buildout/apache2/mod_passenger.so
   <IfModule mod_passenger.c>
     PassengerRoot /opt/rubies/ruby-2.2.3/lib/ruby/gems/2.2.0/gems/passenger-5.1.3
     PassengerDefaultRuby /opt/rubies/ruby-2.2.3/bin/ruby
   </IfModule>
<VirtualHost *:80>
    ServerName carto.mapping.community

    # Tell Apache and Passenger where your app's 'public' directory is
    DocumentRoot /opt/cartodb/public
    RailsEnv production
    SetEnv RAILS_LOG_BASE_PATH /var/log/carto
    SetEnv RAILS_CONFIG_BASE_PATH /etc/carto
    # Relax Apache security settings
    <Directory /opt/cartodb/public>
      AllowOverride all
      Allow from all
      Options -MultiViews
      # Uncomment this if you're on Apache >= 2.4:
      Require all granted
    </Directory>
</VirtualHost>

<VirtualHost *:443>
    ServerName carto.mapping.community
    ServerAlias carto.mapping.community
    # !!! Be sure to point DocumentRoot to 'public'!
    DocumentRoot /opt/cartodb/public
    RailsEnv production
    # This will let CartoDB write all logs to /var/log/carto
    SetEnv RAILS_LOG_BASE_PATH /var/log/carto
    # This will change CartoDB to read all configs from /etc/carto
    SetEnv RAILS_CONFIG_BASE_PATH /etc/carto
    PassengerSpawnMethod direct

    SSLEngine on

    SSLCertificateFile /etc/httpd/certs/OurCert.crt
    SSLCertificateKeyFile /etc/httpd/certs/OurKey.key

    <Directory /opt/cartodb/public>
        # This relaxes Apache security settings.
        AllowOverride all
        Allow from all
        Require all granted
        # MultiViews must be turned off.
        Options -MultiViews
    </Directory>
</VirtualHost>

Note: if you are using certbot (see below) you must manually edit these two lines (from above) to use the following instead (YOUR_DIRECTORY should be changed to the domain name based directory assigned by certbot):

SSLCertificateFile /etc/letsencrypt/live/YOUR_DIRECTORY/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/YOUR_DIRECTORY/privkey.pem

We'll need to create a separate configuration file to enable reverse proxy for the Carto SQL_API:

sudo nano sqlapi.conf

Paste the following into your new sqlapi.conf file and save:

Note: be sure you change DNS below to your own hostname

Listen 9090

NameVirtualHost *:9090

<VirtualHost *:9090>
    SSLProxyEngine On 
    ServerName carto.mapping.community

    SSLEngine on

    SSLCertificateFile /etc/httpd/certs/OurCert.crt
    SSLCertificateKeyFile /etc/httpd/certs/OurKey.key

    # this preserves original header and domain
    ProxyPreserveHost On

    ProxyPass / http://localhost:8080/
    ProxyPassReverse / http://localhost:8080/

</VirtualHost>

Note: if you are using certbot (see below) you must manually edit these two lines (from above) to use the following instead (YOUR_DIRECTORY should be changed to the domain name based directory assigned by certbot):

SSLCertificateFile /etc/letsencrypt/live/YOUR_DIRECTORY/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/YOUR_DIRECTORY/privkey.pem

Create a separate configuration file to enable reverse proxy for the Windshaft-carto:

sudo nano mapapi.conf

Note: be sure you change DNS below to your own hostname Paste the following into your new mapapi.conf file and save:

Listen 9191

NameVirtualHost *:9191

<VirtualHost *:9191>
    SSLProxyEngine On 
    ServerName carto.mapping.community


    SSLEngine on

    SSLCertificateFile /etc/httpd/certs/OurCert.crt
    SSLCertificateKeyFile /etc/httpd/certs/OurKey.key

    # this preserves original header and domain
    ProxyPreserveHost On

    # hardcoded for now, need to fix
    ProxyPass / http://localhost:8181/
    ProxyPassReverse / http://localhost:8181/


</VirtualHost>

Note: if you are using certbot (see below) you must manually edit these two lines (from above) to use the following instead (YOUR_DIRECTORY should be changed to the domain name based directory assigned by certbot):

SSLCertificateFile /etc/letsencrypt/live/YOUR_DIRECTORY/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/YOUR_DIRECTORY/privkey.pem

Now start apache:

sudo systemctl start httpd

s. Install Certbot to Use letsencrypt SSL Certificate (Optional)

The EFF has set up free hosting for secure certificates with a large federation of top-tier web-hosting firms via [http://letsencrypt.org]. You can read more on their website about the service. We'll be using the apache instance of "certbot" to keep our ssl certificates fresh:

sudo yum -y install yum-utils
sudo yum-config-manager --enable rhui-REGION-rhel-server-extras rhui-REGION-rhel-server-optional
sudo yum install python-certbot-apache

Once Certbot is installed, run it and let it replace the existing self-signed certificate we created above:

sudo certbot --apache

Make sure you have modified the directives SSLCertificateFile and SSLCertificateKeyFile across all relevant Apache configuration files as noted in the previous section. You can check whether there are lines you've missed using the following command:

grep -R SSLCertificate * /etc/httpd/conf.d/*

Make sure you restart the apache server after all these changes have been made:

sudo systemctl restart httpd

Start the Server (to Test Out Functionality)

We're nearly done now, so it's time to start up all the services and see how they work!

First, start the redis-server that allows access to the SQL and Maps APIs:

Note, because we've run sudo ./install_server.sh as part of the redis install above, there's no need to run redis-server & here as is specified in the carto install documentation. Worth double checking, but if you've followed this guide redis should already be running.

Start the two import services using node:

cd /opt/CartoDB-SQL-API && node app.js production &
cd /opt/Windshaft-cartodb && node app.js production &

Start Apache/Passenger

sudo service httpd start

Reminder: environment name above!

In a different process/console start the resque process (this serves map tiles and responds to requests)

export PATH=$PATH:/opt/rubies/ruby-2.2.3/bin
RAILS_ENV=production bundle exec ./script/resque

Reminder: environment name above!

Test that your server is up:

curl localhost

This should return:

You are being redirected.

You can also navigate in a web browser to your server URL and confirm that you see a pretty dark blue Carto login screen.

The most likely culprit at this point if you're unable to connect to your new server is a firewall policy. To test whether your server is accessible try the following command from another desktop that should have access to the server:

telnet <server domain name> 443
telnet <server domain name> 80

This command will try to set up an (unsuccessful) telnet session to your web server on the server ports http (80) and https (443). If you see something like the following, albeit with your server's IP address, your server is accessible:

$ telnet 147.188.126.125 443
Trying 147.188.126.125...
Connected to 147.188.126.125.
Escape character is '^]'.

Your attempts to connect to the server should also register in the apache access logs, so you can use the command sudo tail /var/log/httpd/access_log to show the last few lines of your apache log. There should be a time stamped indication of your access attempts, something like:

<workstation IP address omitted> - - [09/May/2017:11:36:12 +0100] "GET /login HTTP/1.1" 200 3997 "https://carto.mapping.community/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/602.4.8 (KHTML, like Gecko) Version/10.0.3 Safari/602.4.8"

To look for problems, use the following commands to check relevant log files:

sudo tail /var/log/httpd/error_log
sudo tail /var/log/carto/log/production.log

You may also want to turn off software firewalls to see if they're causing issues. On Centos this will be either:

systemctl stop firewalld
systemctl stop iptables

You can take snippets of text from error messages to try and find some help with your configuration on stack exchange, a slack group, or the carto google group.

Now hopefully you're up and running, there are a few steps still required to get user accounts created and open up features on your new server:

3. Additional Configuration to do after the Carto stack is running

3.1 Carto User and Group Configuration

a. Create users and organisations

With a rails web app you use bundle exec rake to perform modifications to the database. To get a complete list of your options available, run the following command: RAILS_ENV=production bundle exec rake -T

There are a few things that can cause problems for running this: (1) make sure you are in the proper path /opt/cartodb (2) make sure your path variable still includes ruby, the following command will sort things for you export PATH=$PATH:/opt/rubies/ruby-2.2.3/bin (3) make sure you precede your bundle command by setting the rails environment to "production" (unless you've gone with a development environment!) (4) Pay attention to where you are using single and double quotes. And make sure your word processor hasn't change them to "smart quotes"

If these things are in place, you should be able to run a whole range of system tasks to create users, organisations, and set user settings. You should begin by setting up the publicuser account and then creating one for yourself:

cd /opt/cartodb
RAILS_ENV=production bundle exec rake cartodb:db:create_publicuser
RAILS_ENV=production bundle exec rake cartodb:db:create_user SUBDOMAIN='username' EMAIL='[email protected]' PASSWORD='password'

note: replace "username", "email address", and "password" above with your preferences.

Now let's elevate your account privileges. To save time later, we're going to rely on an environment variable to set the username ("Subdomain" in cartodb vernacular) so that you can actually cut and paste most of the commands from this guide:

export SUBDOMAIN=YOUR_USERNAME
RAILS_ENV=production bundle exec rake cartodb:db:set_user_quota["${SUBDOMAIN}",10240]
RAILS_ENV=production bundle exec rake cartodb:db:set_unlimited_table_quota["${SUBDOMAIN}"]
RAILS_ENV=production bundle exec rake cartodb:db:set_user_private_tables_enabled["${SUBDOMAIN}",'true']
RAILS_ENV=production bundle exec rake cartodb:db:set_user_account_type["${SUBDOMAIN}",'[DEDICATED]']
RAILS_ENV=production bundle exec rake cartodb:set_custom_limits_for_user["${SUBDOMAIN}",import_fil‌e_size,table_row_cou‌nt,concurrent_import‌s]
RAILS_ENV=production bundle exec rake user:change:max_layers["${SUBDOMAIN}",25]

Now your account has a 10gb storage limit, unlimited tables, the ability to create private tables, and a "dedicated" class account.

You can re-use the command listed above to create more user accounts. You may also want to create organisations. The following command will do this for you:

RAILS_ENV=production bundle exec rake cartodb:db:create_new_organization_with_owner ORGANIZATION_NAME='birmingham-uni' ORGANIZATION_DISPLAY_NAME='University of Birmingham Researchers' ORGANIZATION_SEATS='25' ORGANIZATION_QUOTA='1073741824' USERNAME='ADMIN_USER'

Note: You need to change USERNAME above to reflect a user account to serve as "owner" for the organisation. Make sure you've already created this account before running this command. You should also change the ORGANIZATION_NAME, ORGANIZATION_DISPLAY_NAME to something that matches your preferred settings.

b. Activate "Sync Tables"

This feature enables you to regularly synchronise a table in carto with another external source. It's a fantastic feature and very useful if you work with other data sources. To enable it:

As above begin by logging into the PostgresSQL server

sudo su
su postgres
psql carto_db_production

UPDATE users SET sync_tables_enabled = 't';

\q
exit
exit

sync tables also require a script to run every 15 minutes, which will enqueue pending synchronizations (run this in the cartodb/ directory):

RAILS_ENV=production bundle exec rake cartodb:sync_tables

This command will need to be scheduled to run at a regular interval, i.e. every 15 minutes or so. Also, the environment should be changed in the command as necessary. Add to crontab like this:

*/15 * * * *    cd /opt/cartodb && RAILS_ENV=production /opt/rubies/ruby-2.2.3/bin/bundle exec rake cartodb:sync_tables

You may also want to enable specific external data sources, like arcgis. To do this, edit /opt/cartodb/config/app_config.yml and modify the following lines:

  datasources:
    arcgis_enabled: true

While you're in there, you may also want to edit a few other lines in /opt/cartodb/config/app_config.yml for instance, the email address to be used for outbound messages:

  mailer:
    from: 'from name <[email protected]>'

3.2 CentOS Server Configuration for Production Carto

a. systemd service for CartoDB-SQL-API

Note: system services that you will create below are also located in "config" in this repository.

Create a systemd service which we can use to load CartoDB-SQL-API:

cd /etc/systemd/system
sudo nano cartodb-sql.service

Edit (or create) the file and save:

[Service]
ExecStart=/usr/bin/node /opt/CartoDB-SQL-API/app.js production
Restart=always
StandardOutput="/var/log/carto/log/SQL-API_log"
StandardError="/var/log/carto/log/SQL-API_log"
SyslogIdentifier=CartoDB-SQL-API
WorkingDirectory=/opt/CartoDB-SQL-API
User=carto
Group=carto
Environment='NODE_ENV=production'

[Install]
WantedBy=multi-user.target

Create the empty log file manually

touch /var/log/carto/log/SQL-API_log

Start CartoDB-SQL-API

systemctl start cartodb-sql

Enable it to auto-startup at boot

systemctl enable cartodb-sql

b. systemd service for Windshaft-cartodb

Add Windshaft-cartodb to systemd service:

cd /etc/systemd/system
sudo nano windshaft-cartodb.service

Edit the file and save:

[Service]
ExecStart=/usr/bin/node /opt/Windshaft-cartodb/app.js production
Restart=always
StandardOutput="/var/log/carto/log/Windshaft_log"
StandardError="/var/log/carto/log/Windshaft_log"
SyslogIdentifier=Windshaft-cartodb
WorkingDirectory=/opt/Windshaft-cartodb
User=carto
Group=carto
Environment='NODE_ENV=production'

[Install]
WantedBy=multi-user.target

Create the empty log file manually

touch /var/log/carto/log/Windshaft_log

Start CartoDB-SQL-API

systemctl start windshaft-cartodb

Enable it to auto-startup at boot

systemctl enable windshaft-cartodb

c. systemd service for CartoDB Resque process

Create a bash script

cd /opt/cartodb/script
nano run_resque.sh

Edit and save the file as below:

#!/bin/sh

export PATH=$PATH:/opt/rubies/ruby-2.2.3/bin
cd /opt/cartodb
RAILS_ENV=production /opt/rubies/ruby-2.2.3/bin/bundle exec /opt/cartodb/script/resque &> /var/log/carto/log/resque_log

Make the script executable

chmod 775 run_resque.sh

Create the empty log file manually

touch /var/log/carto/log/resque_log

Add cartodb-resque to systemd service:

cd /etc/systemd/system
sudo nano cartodb-resque.service

Edit the file and save:

[Service]
ExecStart=/bin/sh /opt/cartodb/script/run_resque.sh
Restart=always
StandardOutput="/var/log/carto/log/resque_log"
StandardError="/var/log/carto/log/resque_log"
SyslogIdentifier=cartodb-resque
WorkingDirectory=/opt/cartodb
User=carto
Group=carto
Environment='NODE_ENV=production'

[Install]
WantedBy=multi-user.target

Start cartodb-resque

systemctl start cartodb-resque

Enable it to auto-startup at boot

systemctl enable cartodb-resque

d. Configure Redis Persistence

We will need to enable AOF Persistence so Redis will store information that need to be persisted.

sudo nano /etc/redis/6379.conf

Change appendonly no to appendonly yes

Restart redis

sudo systemctl restart redis_6379

3.3 Install Additional Carto Services (data services, observatory, etc.)

a. Carto Data Services API

Login to DB server

Install server and client extensions:

cd /opt
git clone https://github.com/CartoDB/dataservices-api.git
export PATH=$PATH:/usr/pgsql-9.5/bin/
cd dataservices-api
cd client && sudo make install
cd -
cd server/extension && sudo make install

Install python library:

# in dataservices-api repo root path:
cd server/lib/python/cartodb_services && pip install -r requirements.txt && sudo pip install . --upgrade

Create a database to hold all the server part and a user for it:

CREATE DATABASE dataservices_db ENCODING = 'UTF8' LC_COLLATE = 'en_US.UTF-8' LC_CTYPE = 'en_US.UTF-8';
CREATE USER dataservices_user;

Install needed extensions in dataservices_db database:

psql -U postgres -d dataservices_db -c "BEGIN;CREATE EXTENSION IF NOT EXISTS plproxy; COMMIT" -e
psql -U postgres -d dataservices_db -c "BEGIN;CREATE EXTENSION IF NOT EXISTS plpythonu; COMMIT" -e
psql -U postgres -d dataservices_db -c "BEGIN;CREATE EXTENSION IF NOT EXISTS cdb_dataservices_server; COMMIT" -e

b. Carto Data Services

Login to DB server

Make the extension available in postgres:

cd /opt
git clone https://github.com/CartoDB/data-services.git
cd data-services/geocoder/extension
sudo make install

Download the internal geocoder data:

cd /opt/data-services/geocoder
./geocoder_dowload_dumps

Once the data is downloaded, execute this command:

./geocoder_restore_dump postgres dataservices_db db_dumps/*.sql

Install geocoder extension:

cd /opt/data-services/geocoder
sudo make all install

Install onto a CARTO user's database (it is mandatory to install it into a CARTO user's database)

psql -U development_cartodb_user_fe3b850a-01c0-48f9-8a26-a82f09e9b53f cartodb_dev_user_fe3b850a-01c0-48f9-8a26-a82f09e9b53f_db
CREATE EXTENSION cdb_geocoder;

c. Install data observatory extension

Begin by logging out of the web server and login to your db server (if you're working with a 2 server configuration)

Make the extension available in postgresql to be installed:

cd /opt
git clone https://github.com/CartoDB/observatory-extension.git
cd observatory
sudo make install

This extension needs data, dumps are not available so we're going to use the test fixtures to make it work.

psql -U postgres -d dataservices_db -f src/pg/test/fixtures/load_fixtures.sql

Give permission to execute and select to the dataservices_user user:

psql -U postgres -d dataservices_db -c "BEGIN;CREATE EXTENSION IF NOT EXISTS observatory VERSION 'dev'; COMMIT" -e
psql -U postgres -d dataservices_db -c "BEGIN;GRANT SELECT ON ALL TABLES IN SCHEMA cdb_observatory TO dataservices_user; COMMIT" -e
psql -U postgres -d dataservices_db -c "BEGIN;GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA cdb_observatory TO dataservices_user; COMMIT" -e
psql -U postgres -d dataservices_db -c "BEGIN;GRANT SELECT ON ALL TABLES IN SCHEMA observatory TO dataservices_user; COMMIT" -e
psql -U postgres -d dataservices_db -c "BEGIN;GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA observatory TO dataservices_user; COMMIT" -e

d. Configure Carto Server to use Data Services / Observatory

Login to DB Server

Enter psql CLI for carto_db_production

psql -U postgres carto_db_production

Redis configuration

SELECT CDB_Conf_SetConf(
    'redis_metadata_config',
    '{"redis_host": "localhost", "redis_port": 6379, "sentinel_master_id": "", "timeout": 0.1, "redis_db": 5}'
);
SELECT CDB_Conf_SetConf(
    'redis_metrics_config',
    '{"redis_host": "localhost", "redis_port": 6379, "sentinel_master_id": "", "timeout": 0.1, "redis_db": 5}'
);

Users/Organizations

SELECT CDB_Conf_SetConf(
    'user_config',
    '{"is_organization": false, "entity_name": "<YOUR_USERNAME>"}'
);

Data Observatory

SELECT CDB_Conf_SetConf(
    'data_observatory_conf',
    '{"connection": {"whitelist": [], "production": "host=localhost port=5432 dbname=dataservices_db user=geocoder_api", "staging": "host=localhost port=5432 dbname=dataservices_db user=geocoder_api"}}'
);
\q

User database configuration

User (client) databases need also some configuration so that the client extension can access the server:

psql -U postgres cartodb_user_7747c61a-23a1-4ccc-a738-747e8ee24821_db
SELECT CDB_Conf_SetConf('user_config', '{"is_organization": false, "entity_name": "<YOUR_USERNAME>"}');

The geocoder_server_config (the name is not accurate for historical reasons) entry points to the dataservices server DB (you can use a specific database for the server or your same user's):

SELECT CDB_Conf_SetConf('geocoder_server_config', '{ "connection_str": "host=localhost port=5432 dbname=<SERVER_DB_NAME> user=postgres"}');

e. Update Cartodb configuration

Make sure you are logged into the Web Server (if on a 2 server config)

Edit the following session of /opt/cartodb/config/app_config.yml as below

  dataservices:
    enabled:
        geocoder_internal: true
        hires_geocoder: false
        isolines: false
        routing: false
        data_observatory: true

Add the following entry to the geocoder entry of the cartodb/config/app_config.yml file:

  geocoder:
    #force_batch: true
    #disable_cache: true
    api:
      host: '10.128.0.7'
      port: 5432
      user: 'dataservices_user'
      dbname: 'dataservices_db'

Execute the rake tasks to update all the users and organizations:

cd /opt/cartodb
RAILS_ENV=production bundle exec rake cartodb:db:configure_geocoder_extension_for_non_org_users[username,all_users]
RAILS_ENV=production bundle exec rake cartodb:db:configure_geocoder_extension_for_organizations[organization_name,all_organizations]

Restart Apache to make it effective.

sudo service httpd restart

4. Upgrade process for Carto components

In this next section, we will document the process of upgrading your Carto server components to new versions. The Carto team works on a "continuous development" basis, so updates are very frequently made to the platform. However, major releases are given tags in github, and important changes (and steps required for upgrade) are detailed in the release note history. Especially because versions change so often , you should take the following as an example of how to set up an upgrade and not strictly as a de facto process for your server.

4.1 Check versions for currently installed Carto components:

You will need to know what versions of each component you installed originally. To test this, do the following:

On your web server:

cd /opt/cartodb
git describe --tags

cd /opt/CartoDB-SQL-API
git describe --tags

cd /opt/Windshaft-cartodb
git describe --tags

cd /opt/dataservices-api
git describe --tags

On your database server, check:

cd /opt/cartodb-postgresql
git describe --tags

cd /opt/dataservices
git describe --tags

cd /opt/observatory
git describe --tags

Now you have a collection of "tags" you can use to generate a list of updates required. Now make your way over to each github page for these components and check the current "tag" (note that some of these have multiple branches, especially the postgresql components, so be sure you're not accidentally downgrading).

The steps you need to take will be an aggregation of all details from each version increment detailed in the documentation for those components, also with the caveat that you're working with CentOS and not Ubuntu (see above for our suggestios on how to translate ubuntu packages into Centos etc.).

For the CartoDB application, see here: https://github.com/CartoDB/cartodb/blob/master/NEWS.md
For the Carto SQL API, see here: https://github.com/CartoDB/CartoDB-SQL-API/blob/master/NEWS.md
For Windshaft-CartoDB (the tiler and MAPS API), see here: https://github.com/CartoDB/Windshaft-cartodb/blob/master/NEWS.md
For Data Services API: https://github.com/CartoDB/dataservices-api/blob/master/NEWS.md

And on your database server, components to check are:

4.2 On Web server

a. Upgrade CartoDB application

Stop Apache first

sudo service httpd stop

Make a backup of your current cartodb folder

cp -Rp /opt/cartodb /opt/cartodb.backup

Note. make sure you have enough space under /opt or you can choose a different location

Login as carto user: su carto

Check out the latest version from Office Github source

cd /opt/cartodb
git checkout master
git pull

Install dependencies

export PATH=$PATH:/usr/pgsql-9.5/bin/:/opt/rubies/ruby-2.2.3/bin
RAILS_ENV=production bundle install --deployment --without development test

Edit package.json file and change the required version of grunt from 1.0.1 to 0.4.5

nano package.json
Change "grunt": "1.0.1", to "grunt": "0.4.5",

Note: At the time of writing, in order to install grunt for Production without errors, we have to keep grunt at version 0.4.5. If you are using CartoDB in Development environment, you can skip this step.

Run npm to install other dependencies

npm install

Install all necessary gems:

export PATH=$PATH:$PWD/node_modules/grunt-cli/bin
bundle exec grunt --environment production

Run following rake tasks

RAILS_ENV=production bundle exec rake db:migrate
RAILS_ENV=production bundle exec rake cartodb:db:reset_trigger_check_quota
RAILS_ENV=production bundle exec rake cartodb:db:load_functions

Update the postgresql extension:

cd /opt/cartodb/lib/sql/
PGUSER=postgres make all install

Restart Apache

sudo service httpd start

Note: if you do a clean re-install for your upgrade process, you will need to re-create /opt/cartodb/script/run_resque.sh in order for the Windshaft application service to function properly (as per step 3.2.c above).

b. Upgrade Carto SQL API application

Stop SQL API service first

sudo service cartodb-sql stop

Make a backup of your current cartodb folder

cp -Rp /opt/CartoDB-SQL-API /opt/CartoDB-SQL-API.backup

Note. make sure you have enough space under /opt or you can choose a different location

Preserve your carto-sql-api configuration file:

cp /opt/CartoDB-SQL-API/config/environments/production.js /tmp

Option 1: we have found in testing that it may be better to perform a fresh reinstall of the new version rather than upgrade in-line. If you'd like to do this, here are the steps:

sudo rm -fr CartoDB-SQL-API
cd /opt
sudo git clone https://github.com/CartoDB/CartoDB-SQL-API.git
sudo chown -R carto Windshaft-cartodb
cd /opt/CartoDB-SQL-API
npm install
sudo cp /tmp/production.js /opt/CartoDB-SQL-API/config/environments/
sudo chown carto:carto production.js

Option 2: Alternately, check out the latest version from Office Github source from within your existing direction:

cd /opt/CartoDB-SQL-API
git checkout master
git pull

Start SQL API service:

sudo service cartodb-sql start

c. Upgrade Windshaft-CartoDB application

Stop windshaft service first

sudo service windshaft-cartodb stop

Make a backup of your current cartodb folder

cp -Rp /opt/Windshaft-cartodb /opt/Windshaft-cartodb.backup

Note: make sure you have enough space under /opt or you can choose a different location

Preserve your windshaft configuration file:

cp /opt/Windshaft-cartodb/config/environments/production.js /tmp

Option 1: we have found in testing that it may be better to perform a fresh reinstall of the new version rather than upgrade in-line. If you'd like to do this, here are the steps:

sudo rm -fr /opt/Windshaft-cartodb
cd /opt
git clone https://github.com/CartoDB/Windshaft-cartodb.git
sudo chown -R carto Windshaft-cartodb
cd Windshaft-cartodb
npm install
sudo cp /tmp/production.js /opt/Windshaft-cartodb/config/environments/
sudo chown carto:carto production.js
mkdir /opt/cartodb/tile_assets/
touch /opt/Windshaft-cartodb/logs

Option 2: Alternately, check out the latest version from Office Github source from within your existing direction:

cd /opt/Windshaft-cartodb
git checkout master
git pull

Start Windshaft service

sudo service windshaft-cartodb start

4.3 On DB server

a. Upgrade Cartodb-postgresql

Check out the latest version from Office Github source

cd /opt/cartodb-postgresql
git checkout master
git pull

Install the new extension

export PATH=$PATH:/usr/pgsql-9.5/bin/:/opt/rubies/ruby-2.2.3/bin
sudo env "PATH=$PATH" make all install

Make sure the new exensions are created

sudo su -l postgres
psql -d carto_db_production
CREATE EXTENSION IF NOT EXISTS plpythonu;
CREATE EXTENSION IF NOT EXISTS postgis;
CREATE EXTENSION IF NOT EXISTS cartodb;
\q

b. Upgrade Observatory extension

Check out the latest version from Office Github source

cd /opt/observatory-extension
git checkout master
git pull

Install the new extension

PGUSER=postgres sudo make install

c. Upgrade dataservices-api

Check out the latest version from Office Github source

cd /opt/dataservices-api
git checkout master
git pull

Install the new API

cd client && sudo make install
cd -
cd server/extension && sudo make install

d. Upgrade data-services

Check out the latest version from Office Github source

cd /opt/data-services
git checkout master
git pull

Install the new geocode extension

sudo make all install

Notes

Thanks to http://unix.stackexchange.com/a/63068/94615 for help with the section below. ↩
Thanks to http://serverfault.com/a/275411 for this answer. Though note that we've modified this for Britain. :) ↩ ↩²
Note: some of the details below have been drawn (with gratitude) from http://www.unixmen.com/postgresql-9-4-released-install-centos-7 ↩
For more on this fix I've used here, see http://unix.stackexchange.com/questions/83191/how-to-make-sudo-preserve-path. ↩
Thanks to https://www.digitalocean.com/community/tutorials/how-to-install-node-js-on-a-centos-7-server for most of the below details for installing node.js on centos. ↩

Name		Name	Last commit message	Last commit date
Latest commit History 86 Commits
config		config
README.md		README.md

mapcomm/mapping.community-carto_install_doc

Folders and files

Latest commit

History

Repository files navigation

1. Provisioning

Firewall Configuration

2. Installation

A Few Notes on Converting "CartoDB Ubuntu" to Centos...

2.1 PostgresSQL Server

a. Set System Locale2

b. Install Basic Requirements:

c. Install Git

d. PostgreSQL

Get postgresql 9.5 repository installed

Install client packages:

Install server packages3:

e. GIS dependencies

f. PostGIS

2.2 Web Server

a. Set System Locale2

b. Install Basic Requirements:

c. Install Git

d. PostgreSQL

e. GIS dependencies

f. PostGIS

g. Redis

h. NodeJS5

i. Create "Carto" User With Sudo Privilege

j. Ruby

k. GCC Library

l. GDAL 2.1.2

m. Add unp (decompression tool) to Centos

n. CartoDB Components

Editor

o. CartoDB APIs

SQL API:

MAPS API:

p. Install Apache and Passenger

q. Generate self-signed SSL certificate

r. Configure Apache/Passenger with virtualhost and reverse proxy

s. Install Certbot to Use letsencrypt SSL Certificate (Optional)

Start the Server (to Test Out Functionality)

3. Additional Configuration to do after the Carto stack is running

3.1 Carto User and Group Configuration

a. Create users and organisations

b. Activate "Sync Tables"

3.2 CentOS Server Configuration for Production Carto

a. systemd service for CartoDB-SQL-API

b. systemd service for Windshaft-cartodb

c. systemd service for CartoDB Resque process

d. Configure Redis Persistence

3.3 Install Additional Carto Services (data services, observatory, etc.)

a. Carto Data Services API

b. Carto Data Services

c. Install data observatory extension

d. Configure Carto Server to use Data Services / Observatory

e. Update Cartodb configuration

4. Upgrade process for Carto components

4.1 Check versions for currently installed Carto components:

4.2 On Web server

a. Upgrade CartoDB application

b. Upgrade Carto SQL API application

c. Upgrade Windshaft-CartoDB application

4.3 On DB server

a. Upgrade Cartodb-postgresql

b. Upgrade Observatory extension

c. Upgrade dataservices-api

d. Upgrade data-services

Notes

Footnotes

About

Topics

Resources

Stars

Watchers

Forks

Releases 1

Packages 0

Contributors 2

a. Set System Locale²

Install server packages³:

a. Set System Locale²

h. NodeJS⁵

Packages