search by tags

for the user

adventures into the land of the command line

upgrading an elk stack and installing x-pack

These steps are for upgrading ELK 5.5.1 to 5.6.9 on ubuntu 16.04 and then installing the X-Pack addon. Repeat for each node in the cluster one by one.

Upgrading ELK 5.5.1 to 5.6.9

Disable shard allocation

# curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
  "transient": {
    "cluster.routing.allocation.enable": "none"
  }
}
'

{"acknowledged":true,"persistent":{},"transient":{"cluster":{"routing":{"allocation":{"enable":"all"}}}}}

Stop non-essential indexing and perform a synced flush (Optional)

# curl -X POST "localhost:9200/_flush/synced"

Stop and upgrade a single node

# service elasticsearch stop
# ps -ef | grep [e]lastic

# apt-cache madison elasticsearch
elasticsearch |      5.6.9 | https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
elasticsearch |      5.6.8 | https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
elasticsearch |      5.6.7 | https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
elasticsearch |      5.6.6 | https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
elasticsearch |      5.6.5 | https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
elasticsearch |      5.6.4 | https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
elasticsearch |      5.6.3 | https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
elasticsearch |      5.6.2 | https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
elasticsearch |      5.6.1 | https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
elasticsearch |      5.6.0 | https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
elasticsearch |      5.5.3 | https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
elasticsearch |      5.5.2 | https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
elasticsearch |      5.5.1 | https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
elasticsearch |      5.5.0 | https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
elasticsearch |      5.4.3 | https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
elasticsearch |      5.4.2 | https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
elasticsearch |      5.4.1 | https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
elasticsearch |      5.4.0 | https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
elasticsearch |      5.3.3 | https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
elasticsearch |      5.3.2 | https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
elasticsearch |      5.3.1 | https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
elasticsearch |      5.3.0 | https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
elasticsearch |      5.2.2 | https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
elasticsearch |      5.2.1 | https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
elasticsearch |      5.2.0 | https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
elasticsearch |      5.1.2 | https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
elasticsearch |      5.1.1 | https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
elasticsearch |      5.0.2 | https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
elasticsearch |      5.0.1 | https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
elasticsearch |      5.0.0 | https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
elasticsearch | 1.7.3+dfsg-3 | http://azure.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
elasticsearch | 1.7.3+dfsg-3 | http://azure.archive.ubuntu.com/ubuntu xenial/universe Sources


# apt-get upgrade elasticsearch=5.6.9


# dpkg -l | grep elasticsearch
ii  elasticsearch                       5.6.9                                      all          Elasticsearch is a distributed RESTful search engine built for the cloud. Reference documentation can be found at https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html and the 'Elasticsearch: The Definitive Guide' book can be found at https://www.elastic.co/guide/en/elasticsearch/guide/current/index.html
ii  elasticsearch-curator               5.5.3                                      amd64        Have indices in Elasticsearch? This is the tool for you!

Like a museum curator manages the exhibits and collections on display,
Elasticsearch Curator helps you curate, or manage your indices.

# dpkg -l | grep logstash
ii  logstash                            1:5.6.9-1                                  all          An extensible logging pipeline

# dpkg -l | grep kibana
ii  kibana                              5.6.9                                      amd64        Explore and visualize your Elasticsearch data

# service elasticsearch start

# ps -ef | grep [e]lastic
elastic+  47382      1 99 12:39 ?        00:00:32 /usr/bin/java -Xms8g -Xmx8g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch

wait a bit….
.
.
.
Start the upgraded node

# curl -X GET "localhost:9200/_cat/nodes"
123.456.78.90  2 98 15 0.66 1.32 1.97 mdi - mygroovyelk-1
123.456.78.91    92 44 1.65 1.90 2.27 mdi * mygroovyelk-2

Re-enable shard allocation

# curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
  "transient": {
    "cluster.routing.allocation.enable": "all"
  }
}
'

{"acknowledged":true,"persistent":{},"transient":{"cluster":{"routing":{"allocation":{"enable":"all"}}}}}

Wait for the node to recover

# curl -X GET "localhost:9200/_cat/health"
1527079301 12:41:41 milogging1 yellow 2 2 257 247 0 2 235 1 - 52.0%

# curl -X GET "localhost:9200/_cat/health"
1527079312 12:41:52 milogging1 yellow 2 2 266 247 0 2 226 0 - 53.8%

# curl -X GET "localhost:9200/_cat/health"
1527079346 12:42:26 milogging1 yellow 2 2 287 247 0 2 205 0 - 58.1%

# curl -X GET "localhost:9200/_cat/health"
1527079845 12:50:45 milogging1 yellow 2 2 488 247 0 2 4 0 - 98.8%

# curl -X GET "localhost:9200/_cat/health"
1527079994 12:53:14 milogging1 yellow 2 2 490 247 0 2 2 0 - 99.2%

# curl -X GET "localhost:9200/_cat/health"
1527080383 12:59:43 milogging1 green 2 2 494 247 0 0 0 0 - 100.0%

# curl -X GET "localhost:9200/_cat/health?v"
epoch      timestamp cluster    status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1527080380 12:59:40  milogging1 green           2         2    494 247    0    0        0             0                  -                100.0%

Restart kibana and logstash

# service kibana restart
# service logstash restart

Repeat for the next node in the cluster. Once all nodes have been upgraded, you can install x-pack on them.

=========================================================================================================================================================

Once again, repeat for each node in the cluster one by one.

Install X-Pack 5.6.9 for Elasticsearch

# curl "localhost:9200/_nodes/settings?pretty=true" | grep home
"home" : "/usr/share/elasticsearch"

# cd /usr/share/elasticsearch/
# bin/elasticsearch-plugin install x-pack --batch

-> Downloading x-pack from elastic
[=================================================] 100%
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.io.FilePermission \\.\pipe\* read,write
* java.lang.RuntimePermission accessClassInPackage.com.sun.activation.registries
* java.lang.RuntimePermission getClassLoader
* java.lang.RuntimePermission setContextClassLoader
* java.lang.RuntimePermission setFactory
* java.net.SocketPermission * connect,accept,resolve
* java.security.SecurityPermission createPolicy.JavaPolicy
* java.security.SecurityPermission getPolicy
* java.security.SecurityPermission putProviderProperty.BC
* java.security.SecurityPermission setPolicy
* java.util.PropertyPermission * read,write
* javax.net.ssl.SSLPermission setHostnameVerifier
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@        WARNING: plugin forks a native controller        @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
This plugin launches a native controller that is not subject to the Java
security manager nor to system call filters.
-> Installed x-pack

# service elasticsearch restart

You should now get a 401 response when using the elastic api

# curl -X GET "localhost:9200/_cat/health?v"

{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication token for REST request [/_cat/health?v]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication token for REST request [/_cat/health?v]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}

Install X-Pack 5.6.9 for Kibana

# cd /usr/share/kibana/
# sudo -u kibana bin/kibana-plugin install x-pack

Attempting to transfer from x-pack
Attempting to transfer from https://artifacts.elastic.co/downloads/kibana-plugins/x-pack/x-pack-5.6.9.zip
Transferring 119595626 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Optimizing and caching browser bundles...    

Install X-Pack 5.6.9 for Logstash

# cd /usr/share/logstash/
# bin/logstash-plugin install x-pack

Downloading file: https://artifacts.elastic.co/downloads/logstash-plugins/x-pack/x-pack-5.6.9.zip
Downloading [=============================================================] 100%
Installing file: /tmp/studtmp-9676d592993622aeb4bb6942519f37e19c147f9e9f745ee32dffd155c161/x-pack-5.6.9.zip
Install successful

Reset the logstash_system user’s password so that you can set it in the config

curl -XPUT -u elastic 'localhost:9200/_xpack/security/user/logstash_system/_password?pretty' -H 'Content-Type: application/json' -d'
{
  "password": "mygroovypassword"
}
'

Enter host password for user 'elastic':
{ }

By default the elastic user’s password is changeme, but you should change this, just like changing the logstash_user’s password to something else.

Add the elasticsearch basic auth to the logstash config, with the logstash_system user here…

# vim /etc/logstash/logstash.yml

xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: http://localhost:9200
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: mygroovypassword

…and the elastic user here

# vim /etc/logstash/conf.d/99_output_elasticsearch.conf

output {
  if "beats" in [tags] {
    elasticsearch {
      hosts => ["http://localhost:9200"]
      user => "elastic"
      password => "mygroovypassword"
      index => "ls-%{[fields][program]}-%{+YYYY.MM.dd}"
      document_type => "%{[@metadata][type]}"
    }
  }
  if "healthcheck" in [tags] {
    elasticsearch {
      hosts => ["http://localhost:9200"]
      user => "elastic"
      password => "mygroovypassword"
      index => "ls-healtcheck-%{+YYYY.MM.dd}"
    }
  }
}

Restart Logstash

# service logstash restart

Go to kibana in your browser and you’ll be shown a login page (if you didn’t have auth enabled before), and some new menus granted by X-Pack. Happy elking.