Skip to main content

Upgrade Hubble Application Server

To upgrade a Hubble Application Server, you need to apply a new HubbleApplicationServer-<version>.tar.gz package to your Linux server, as described in this section.

Running commands

Some of the following commands require you to be a root / privileged user. If you are running as a unprivileged user, you can avoid problems by running these steps as root (i.e. using the "sudo -i" command). If you really want to run them as a normal user, please prefix the commands with "sudo" as necessary.

Backup

It is strongly recommended that a backup is performed prior to an upgrade. Refer to the backup instructions in the Hubble Supplementary Deployment Topics guide for details.

Before starting the upgrade

It is recommended that you update your Linux operating system to get all the latest security patches.

Alerts

If you want to use the same alerts agents you had prior to 20.1, then copy the IP addresses/DNS names for your web servers into the Alerts Agents field. If you had alerts and no longer want them, leave the Alerts Agent field blank.

Before Deployment Day

Listed below are the actions that should be performed before the deployment date.

Extract the Hubble Application Server package

  1. Obtain the deployment package (HubbleApplicationServer-*.tar.gz) and copy it to the Linux machine at the location where you want to deploy your Hubble Application Server.
  2. Extract the contents of the package to the /etc/hubble folder by running this command:

    # This command assumes you are in the same folder as the

    HubbleApplicationServer-< version>.tar.gz file

    mkdir -p /etc/hubble && tar zxvf HubbleApplicationServer- <version>.tar.gz -C ./hubble

Verify that all the Prerequisites are met

  1. To verify that the prerequisites are all met, go to the directory into which you extracted the package and type the following:

    /etc/hubble/pre-req-tests.sh

    This will run some checks, and if they pass, each will be followed by an "OK" in green color if they pass, or "ERROR" in red if they fail.

  2. The result of those checks are stored in log files in:

    /var/log/hubble

    The files are named with the format hubble-install-date_and_time.log.

  3. The most recent log file (the date and time are in the file name), with the current prerequisite check, should be sent to Hubble Support team before the deployment day.

Important:

All the tests must pass before the deployment day, otherwise the deployment cannot take place.

Note: If the docker images from the previous installation are still running, the prerequisite tests will fail with a message similar to the following:

ERROR - Port 5432 is not free, docker-proxy process is listening to the port That is not a problem, as the port is already in use by the previous version.

On Deployment Day

Before Installing Hubble

Restore run-list.json:

/etc/hubble/store_run_list.sh

Install Hubble

Install the files by navigating to the directory in which the package was extracted and running the following commands:

cd ./hubble && ./install.sh

Update the Hubble Configurations

You will always need to re-submit your configurations using these steps, even if there are no changes to apply.
  1. Start the Hubble Configuration UI by running:

    /etc/hubble/Configuration/start.sh

  2. Go to the Hubble Configuration form by accessing http:// <application_server_ip_address>:3000/ in a browser.
  3. Fill in the form with the configuration details of your server(s).

    Note:

    • If you are upgrading from any Hubble release, older than 20.1, you will need to configure the alerts agents for your system in the configuration UI.
    • If you want the same alerts agents as you had prior to the upgrade, then copy the IP addresses/DNS names for your web servers into the alerts agents field (see the image below).
    • If you had alerts and no longer want them, leave the alerts agent field blank. The same goes for if you want no alerts on a new install.
  4. Once finished, press the "Submit and Download" button.
  5. You can now the stop the Hubble Configuration UI by running:

    /etc/hubble/Configuration/stop.sh

    Important: If you are upgrading from any Hubble release older than 19.2, you will need to change the docker storage driver to overlay2:

    /etc/hubble/configure-overlay2.sh

    <name_of_device_for_docker_use>

    This command will format the drive, so please ensure you select the correct device. As an example, if device sdb has been provisioned for docker use, run:

        /etc/hubble/configure-overlay2.sh sdb    

Restart the Hubble Application Server Services

  1. Submitting your server information in the Hubble Configuration UI will automatically generate all the files needed to run the Hubble Application Server, so you are now ready to start the services.
  2. (Optional) If you selected the use of HTTPS (under "Web Protocol" in the Configuration UI), you will need to provide a valid certificate at this point by following the instructions in How to setup HTTPS in the Hubble Supplementary Deployment Topics guide. If you do not have a valid certificate yet and want to use HTTP temporarily, repeat the steps in "Generate the Hubble Configurations" above, and select HTTP as the Web Protocol.
  3. To start the Hubble service, run the following command:

    /etc/hubble/start.sh

    Note:

    This script is the only way to start the docker containers.

    The script will always check the prerequisites and will not start the Hubble services until all the prerequisites are met.

    If the start.sh script failed due to the prerequisites not being met, you can carry out corrective measures and re-test them by running the following command:

    /etc/hubble/pre-req-tests.sh

  4. Now that the Hubble services are up and running, verify that all is working correctly by viewing the results of the deployment verification tests (DVTs).

    The Deployment Verification Tests will run automatically once the Hubble Services containers have been downloaded. During first time installs, it may take a while as the tests are set up:

    When the tests are complete you will see output similar to the following:

    We expect all tests to pass for a successful Application Server install.

  5. Once the DVTs have been run, the configurations will be stored in the storage container. If no problems arise, an output similar to this will be displayed:

    6. Note: Circumstances may arise in which the storage container is slow to initialize, in which case the configurations upload may fail.

    If this does happen, as a workaround, you should wait a few minutes and then run the following script to upload the configurations directly:

        /etc/hubble/store_run_list.sh    

  6. If the DVTs are passed successful, you can now proceed with the Web Server(s) deployment.

After the Upgrade has Completed

If you are upgrading from 2016.1 to an 18.x or later major release, you will need to manually increment the repository version in the database.

  1. Execute the following command to start the 'psql' command within the repository docker container:
    docker exec -it -u postgres hubble_hubble_repository_1 psql hubble_repository
  2. Now execute the following SQL code to update the version to the correct value:
    UPDATE public.repositories 
    SET settingvalue= '12' WHERE repository = 'hubble'
     AND section = 'Versions' 
    AND item = 'MinorVersion' 
    AND settingvalue = '11'
    \g
  3. After executing this you should see the following output to the console, which suggests that the update has taken effect:
    UPDATE 1

Post-Installation

After the Web Server deployment we remove the file run-list.json from the S3 service:

/etc/hubble/remove_run_list.sh

Optional Configurations

Harden Application Server installation

From 20.4 onwards, Hashicorp Vault is installed and running on the application server. Its purpose is to store and manage Hubble secrets in a secure manner. When installed and configured, Vault by design creates an unseal key and root token. These can either be saved on the disk or noted down by the sysadmin. By default, for ease of use on system reboots and configurations, the keys are left on disk. In case you want to secure the installation further, you can run the following in the app server:

Important: Be sure to note both keys down before proceeding as there is no way to recover them.

/etc/hubble/harden-vault.sh

Once hardened, on every system reboot a sysadmin has to ssh into the app server. Execute the following and fill in the secrets when prompted. Start also needs to be run every time containers were stopped:

/etc/hubble/start.sh

User Namespace Isolation for Docker Containers

Hubble Web deploys docker containers which run services necessary to its functioning. By default, the processes running inside those containers run as root in the host machine (although a user inside the container may be different than root, it maps to root in the host). To improve security, our docker container can map users to a non-root user in the host. To do so docker, namespaces need to be configured:

  1. Stop Hubble containers by running:
    /etc/hubble/stop.sh
  2. Create the user in the host machine that will become the container’s user (in this example we will call it myuser). Alternatively, you can use default Docker user (dockremap). To do this, leave the user field blank in the steps below.
  3. Set the following permissions to the user on Hubble data directory and sub-directories:
    chmod +x -R "/mnt/data/containers/hubble_repository/init" chown -R myuser:mygroup "/etc/hubble" "/mnt/data/"
  4. Next, run:
    start.sh --configure-namespaces myuser

    This will configure Docker daemon to use namespace remap.

  5. To disable namespace remap, edit /etc/docker/daemon.json file to remove the line containing:
    “userns”: “myuser”
  6. After this, you should change application and data folders owner to root:
    chown -R root:root /etc/hubble /mnt/data

  7. Restart docker service:

    • For systemd:

      systemctl restart docker

    • For sysV:

      service docker restart

  8. Start hubble containers by running:

    /etc/hubble/start.sh

The start script will run health checks after starting the services. When docker namespaces are enabled, those checks will fail (see screenshot below). The failures can be ignored if Hubble Web functionality is working as expected.


From 20.4 onwards, Hashicorp Vault is installed on the application server and running as a docker container. The Docker namespaces feature has been disabled for it, and so hubble_vault will be running as root.

For more information read the official docker documentation on user namespaces at: https://docs.docker.com/engine/security/userns-remap/

Was this article helpful?

We're sorry to hear that.