Adoptable Cookbooks List

Looking for a cookbook to adopt? You can now see a list of cookbooks available for adoption!
List of Adoptable Cookbooks

Supermarket Belongs to the Community

Supermarket belongs to the community. While Chef has the responsibility to keep it running and be stewards of its functionality, what it does and how it works is driven by the community. The chef/supermarket repository will continue to be where development of the Supermarket application takes place. Come be part of shaping the direction of Supermarket by opening issues and pull requests or by joining us on the Chef Mailing List.

Select Badges

Select Supported Platforms

Select Status

RSS

elkstack (29) Versions 5.0.2

Installs/Configures elkstack

Policyfile
Berkshelf
Knife
cookbook 'elkstack', '= 5.0.2', :supermarket
cookbook 'elkstack', '= 5.0.2'
knife supermarket install elkstack
knife supermarket download elkstack
README
Dependencies
Changelog
Quality 0%

elkstack

Elasticsearch, Logstash, and Kibana stack. Due to the recommendations of the
community, we are not using the embedded elasticsearch functionality of logstash
at this point. This cookbook provides recipes for all three components, along
with wrapper recipes such as single or cluster to facilitate different use
cases.

This stack's design is intended for one or many standalone nodes, with a full
stack of elasticsearch, logstash, and kibana. The only difference between one
and many nodes is that elasticsearch is clustered together. Data dispatched to
Logstash on a particular node will use the local elasticsearch transport
interface to index those logs to the node (and thus, the cluster). HTTP traffic
dispatched to Kibana on port 80 on any node will also use the local
elasticsearch HTTP interface to fetch and manipulate data.

Please read the individual recipe summaries to understand what each recipe does,
as well as what each wrapper recipe is actually wrapping. As much as possible,
upstream attributes have been exposed/overriden for our needs.

Things you should know

  • This cookbook requires java. Because not everyone has the same desires for
    java versions, concurrently installed versions, or particular vendor versions,
    this cookbook simply assumes you have already satisfied this requirement. This
    cookbook does ship with default attributes to make the community cookbook use
    Java 7 over the default of Java 6.

  • You must update your Berksfile to use this cookbook. Due to the upstream
    changes constantly occuring, you should consult the Berksfile in this cookbook
    and use its sources for kibana, logstash, and elasticsearch cookbooks.
    Eventually, as PRs get merged, this may no longer be a hard requirement. But the
    hardest thing will be that kibana in supermarket is currently a different
    actual cookbook.

  • You should probably disable the nginx virtualhost that comes with the kibana
    cookbook and create your own configuration, securing it as appropriate for your
    own requirements. See the kibana_web LWRP documentation for more on what
    attributes should be set to accomplish this.

  • If you'd like to disable backups using cloud files, set
    node['elkstack']['config']['backups']['enabled'] = false (it defaults to
    true). If you'd like to override the backup schedule/behavior for ES, simply
    disable the backup crontab entry by setting
    node['elkstack']['config']['backups']['cron']=false. This cookbook will still
    configure everything except the cronjob, and then you may create another one
    with your own schedule using the cron_d LWRP.

  • Please note that this cookbook does not restart elasticsearch automatically,
    in order to avoid causing an outage of the cluster. It does restart nginx and
    logstash, however. You will have to restart elasticsearch after the initial
    bootstrap. You may also need to bounce logstash if it seems confused about
    losing a connection to eleasticsearch (unusual, but happens).

  • You may want to consider adjusting node['elasticsearch']['discovery']['search_query']
    if you are sharing one cluster among multiple environments. Just put a chef
    search in that attribute and this will use that search instead of one scoped to
    chef environments.

  • You may want to consider adjusting node['elasticsearch']['allocated_memory']
    if you are seeing an initial convergence failure (see #50).
    The chef client has been known to take up to 500mb or more on initial
    convergence. Combined with an initial allocation of 40% memory for ES, and 20%
    for logstash, that only leaves about 40% for the OS and chef. On a 2gb server,
    that ends up being 800mb for ES, about 400mb for logstash, leaving 800mb for
    the OS and the initial chef client run. After the initial run, memory footprint
    for the chef-client tends to be much, much lower, and ES is able to start.

  • The agent and logstash recipes requires a pre-generated SSL key and
    certificate due to the requirements of the lumberjack protocol. This cookbook
    will consult node['elkstack']['config']['lumberjack_data_bag'] in order to
    locate and load a database that stores this key. It will first try an encrypted
    data bag, and if that doesn't work, will try an unencrypted data bag of the same
    name. If no data bag is found, it will autogenerate one and save it as an
    encrypted data bag. This means you must already have a 'secret file' on the node
    for an encryption key, as this is a require to use any encrypted data bags.
    To generate a key of your own, use something like:

    openssl req -x509 -newkey rsa:2048 -keyout lumberjack.key -out lumberjack.crt -nodes -days 1000

    This key and certificate data should be placed in data bag with name
    node['elkstack']['config']['lumberjack_data_bag'] under key and
    certificate keys, and base64 encoded into a single line string. You may also
    supply these secrets with some other method and populate the appropriate
    node.run_state values (see _secrets.rb for more details). Note that this is
    not a PKI trust model, but an explicit trust model. You may also set the data bag key to false to disable lumberjack entirely.

There exists a make-lumberjack-key.sh to help you make this. For Go 1.3+, you may be required
by the standard libraries to create a SAN cert as described here.

[Changelog](CHANGELOG.md)

See CHANGELOG.md for additional information about changes to this stack over time.

Supported Platforms

Ubuntu 12.04

Ubuntu 14.04

CentOS 6.5

Attributes

<table>
<tr>
<th>Key</th>
<th>Type</th>
<th>Description</th>
<th>Default</th>
</tr>
<tr>
<td><tt>['elkstack']['config']['logstash']['instance_name']</tt></td>
<td>String</td>
<td>Default logstash instance name</td>
<td><tt>server</tt></td>
</tr>
<tr>
<td><tt>['elasticsearch']['discovery']['search_query']</tt></td>
<td>String</td>
<td>A query to search for and connect Elasticsearch to cluster nodes</td>
<td>(see attributes/elasticsearch.rb)</td>
</tr>
<tr>
<td><tt>['logstash_forwarder']['config']['files']</tt></td>
<td>Hash</td>
<td>See customizing the stack section below.</td>
<td>Most logs in /var/log</td>
</tr>
<tr>
<td><tt>['elkstack']['config']['data_disk']['disk_config_type']</tt></td>
<td>Boolean or String</td>
<td>See customizing the stack section below.</td>
<td><tt>false</tt></td>
</tr>
<tr>
<td><tt>['elkstack']['config']['iptables']</tt></td>
<td>Boolean</td>
<td>Enable/Disable iptables functionality</td>
<td><tt>true</tt></td>
</tr>
<tr>
<td><tt>['elkstack']['config']['site_name']</tt></td>
<td>String</td>
<td>Control the name of the self-signed SSL key and cert in /etc/nginx/ssl</td>
<td><tt>kibana</tt></td>
</tr>
<tr>
<td><tt>['elkstack']['config']['kibana']['redirect']</tt></td>
<td>Boolean</td>
<td>Enable/Disable nginx redirect for kibana from port 80 to port 443</td>
<td><tt>true</tt></td>
</tr>
<tr>
<td><tt>node.run_state['elkstack_kibana_username']</tt> and <tt>['elkstack']['config']['kibana']['username']</tt></td>
<td>String</td>
<td>Default username for basic auth for kibana, run_state used first</td>
<td><tt>kibana</tt></td>
</tr>
<tr>
<td><tt>node.run_state['elkstack_kibana_password']</tt></td>
<td>String</td>
<td>Password for basic auth for kibana</td>
<td>random from <tt>Opscode::OpenSSL::Password</tt></td>
</tr>
<tr>
<td><tt>['elkstack']['config']['lumberjack_data_bag']</tt></td>
<td>String</td>
<td>Data bag name for lumberjack key and certificate</td>
<td><tt>lumberjack</tt></td>
</tr>
<tr>
<td><tt>['elkstack']['config']['custom_logstash']['name']</tt></td>
<td>Array of strings</td>
<td>See attributes/logstash.rb for an explanation of how to use this attribute to populate additional logstash configuration file templates</td>
<td><tt>[]</tt></td>
</tr>
<tr>
<td><tt>['elkstack']['config']['restart_logstash_service']</tt></td>
<td>Boolean</td>
<td>Restart logstash if we deploy a custom config file</td>
<td><tt>true</tt></td>
</tr>
</table>

Customizing the stack

To override local storage for elasticsearch nodes (the stack will format and mount, as well as configure elasticsearch), set ['elkstack']['config']['data_disk']['disk_config_type'] to custom and provide each storage device and mount point in the following way:
```ruby
disk_config = {
'file_system' => 'ext4',
'mount_options' => 'rw,user',
'mount_path' => '/usr/local/var/data/elasticsearch/disk1',
'format_command' => 'mkfs -t ext4 ',
'fs_check_command' => 'dumpe2fs'
}

node.override['elasticsearch']['data']['devices']['/dev/xvde1'] = disk_config
node.override['elasticsearch']['path']['data'] = disk_config['mount_path']
```

To add additional logstash configuration to this stack, simply add additional
templates in your wrapper cookbook. They should be placed in
"#{@basedir}/#{@instance}/etc/conf.d" (see the config provider in the logstash
cookbook). If you choose to use logstash-forwarder instead of the regular agent,
please see the hash structure in attributes/forwarder.rb for adding additional
files for the forwarder to watch and forward, node['logstash_forwarder']['config']['files'].

To override the nginx configuration, simply supply a new template and specify
your cookbook using ['kibana']['nginx']['template_cookbook'] and
['kibana']['nginx']['template']. You can also override just the password for
the reverse proxy using node.run_state['elkstack_kibana_password'].

By default, platformstack will call the `::agent recipe here. If you have a
need for the forwarder recipe instead, just please note that you should turn off
the platformstack flag for logging, and include the elkstack::agent recipe
directly.

To override anything else, set the appropriate node hash (logstash, kibana, or elasticsearch).

Usage

elkstack::default

A simple wrapper recipe that sets up Elasticsearch, Logstash, and Kibana. Also
configures an rsyslog sink into logstash on the local box. Everything except
Logstash and Kibana is locked down to listen only on localhost.

elkstack::agent

A simple wrapper recipe that sets up a logstash agent on the local box. Also
configures an rsyslog sink into logstash on the local box.

elkstack::forwarder

A go-based alternative to the normal
agent, configured simply to watch logs forward them directly on to the cluster. This
project is in heavy development, and is not publishing releases very often, so the
packaged versions may be quite old or buggy. As of the addition of the recipe, the
package was almost a year behind current development, but only because there also
had been no releases either.

elkstack::elasticsearch

Leans on the upstream elasticsearch/cookbook-elasticsearch cookbook for much
of its work. We do override the default set of plugins to be installed, as well
as the amount of JVM heap. See attributes/default.rb for those settings.

This recipe also tags the node so that other nodes that run this recipe can
discover it, and configure Elasticsearch appropriately to join their cluster.
It uses a tag, the current chef environment, and the cluster name as the default
search criteria.

Most of this is configurable using the upstream Elasticsearch cookbook's
attributes, including the chef search itself. There is not an easy toggle to
turn off the search, however.
Enables iptables rules if node['elkstack']['iptables']['enabled'] is not nil.

elkstack::logstash

Leans on the upstream lusis/chef-logstash cookbook for much
of its work. We do override the default set of plugins to be installed, as well
as the amount of JVM heap. See attributes/default.rb for those settings.

elkstack::kibana

Leans on the upstream lusis/chef-kibana cookbook for most of its work. Sets up
an nginx site for kibana by default. By default, it also does not pass through
most of the http paths directly to elasticsearch (whitelist).

elkstack::newrelic

Validates if there is a newrelic license set and based on that, see if the node
is tagged as 'elkstack' and creates a file with elasticsearch details. Installs
python, pip and setuptools packages in order to support newrelic_meetme_plugin

elkstack::acl

Adds cluster node basic iptables rules and cluster iptables rules if appropriate attributes
are set.

elkstack::agent_acl

Adds agent node basic iptables rules if appropriate attributes are set.

elkstack::disk_setup

Look for node['elkstack']['config']['data_disk']['disk_config_type'] to be truthy, and configure the upstream elasticsearch cookbook to format, mount, and use devices appropriately.

elkstack::*_monitoring

These correspond with the recipes above, and just provide a way to pull out the
monitoring work to make the original recipes cleaner.

Miscellaneous

The wrapper recipes are single and cluster. These change attributes and then
invoke elasticsearch, logstash, kibana, and rsyslog. Finally, there are
utility recipes like java and newrelic (not invoked otherwise), as well as
acl which is called by _base if node['elkstack']['iptables']['enabled'].

Contributing

See CONTRIBUTING.

Authors

Author:: Rackspace (devops-chef@rackspace.com)

License

# Copyright 2014, Rackspace Hosting
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

5.0.2

  • Make elkstack more chef-solo friendly. We now check for solo before doing includes of elasticsearch::search_discovery, as well as better error checking on empty values when search has not been used, RE: #144.

5.0.1

  • Move default lumberjack certs/keys to /etc from /opt/logstash, RE: #145.

5.0.0

  • Remove the kibana.yml Kibana 4 workaround, now that kibana works again out of the box.
  • Stop shipping Java wrappers. You must include java::default or get Java some other way, fixes #138.
  • Clean up Berksfile. Many dependencies have since been fixed and/or released to Supermarket.
  • Move back to logstash-forwarder pkg repos and removes golang deps. Fixes #139.
  • Stop making a distinction between a cluster install vs. a single all-in-one install without any agent, fixes #135.
  • Merge default and cluster and single recipes. Default is now a clustered elkstack.
  • Tags and search/discovery all operate off 'elkstack' tag. There is no more 'elkstack_cluster' tag.
  • Fixed some of the rspec/chefspec unit tests as well.
  • The lumberjack protocol will no longer be the default for Logstash to communicate, due to hosted chef changes and golang runtime SSL changes (fixes #56, #14).
  • Don't test the lumberjack protocol with the server install by default, removes lumberjack test suite in .kitchen.yml
  • Default to tcp/udp for communication between logstash instances, added flag: node['elkstack']['config']['agent_protocol'] = 'tcp_udp' # could also be lumberjack
  • Don't try to load lumberjack secrets by default for agent, don't fail if they aren't present (rename recipes/_secrets.rb -> recipes/_lumberjack_secrets.rb)
  • Remove tests for lumberjack.key/crt now from default test suites, remove extra data bags with lumberjack keypairs

4.2.3

  • Update kibana configuration file to support more kibana 4 parameter names
  • Allow lumberjack functionality to be disabled
  • Replace git:// with https:// to get logstash_forwarder

4.2.2

  • Bump Elasticsearch to 1.4.4
  • Update kibana configuration file to support more kibana 4 parameter names
  • Disable Kibana 4 workarounds

4.2.1

Miscellaneous fixes

  • Array for logstash configs needed a default_unless
  • Go ahead and add a restart of the logstash agent before testing, to avoid CI timing weirdness

4.2.0

Workarounds and more support for Kibana 4

  • Use 'kibana' as basic auth password for kibana, in test-kitchen
  • Update to latest Elasticsearch (1.3.4 to 1.4.3) as Kibana 4 requires 1.4.x or greater (fixes #108)
  • Enable dynamic scripting in Elasticsearch by default, as Kibana 4 requires it
  • Add kibana4_workarounds recipe which deploys a working Kibana 4 configuration file (fixes #103)
  • Add node['elkstack']['kibana4_workaround'] that is enabled by by default, guards execution of recipe kibana4_workarounds
  • Tests for elasticsearch health now drop replicas first, as kibana 4 seems to create an immediately orphaned one on a single node
  • Remove 'no keypair' test for servers and agents in integration suites, add unit tests for the same

4.1.0

  • Kibana password can now be a node attribute or run_state entry, is now respected (#99)
  • Add missing runit dependency (#107)
  • More workarounds and fixes for kibana 4 (#104)
  • Change logstash forwarder to be installed from Git since packages were removed (#109)
  • Update to newer upstream logstash cookbook to get fix for https://github.com/lusis/chef-logstash/issues/387 (#111)

4.0.0

  • Use the latest lusis/chef-kibana cookbook, now installs Kibana 4
  • Server name now defaults to node['fqdn']
  • Depend on kibana_lwrp from Supermarket now, no more Berksfile entries

3.2.7

  • Unpin and go back to chef 12
  • Fix python builds on centos with setuptools manual run

3.2.6

  • Fixed xmx and xms error in agent

3.2.5

  • Bump for dev. Update poor comment.

3.2.4

  • Ensure to pass the instance name to any custom templates, to be sure they end up in the correct directory

3.2.3

  • Exit the agent recipe if no servers are found, don't try to proceed
  • Add warnings about what the discovery logic is doing for the agent
  • Don't search for server/cluster nodes on chef-solo, allow overrides
  • Fixup tests to handle the chef-solo case but beef up check logic for fall through

3.2.2

  • Add newrelic user to system, so agent will start.
  • Cleanup attributes & recipe for backup based on testing, ES plugin changes
  • Fix bug in unencrypted data bag access

3.2.1

  • Fix a bug where the agent recipe was not installing custom configuration files with the right instance name (was using 'default' instead of 'agent'). Added a test fixture/wrapper cookbook in order to verify the correct behavior.

  • Gave existing agent serverspec tests some easier to read description groups, to help read output easier.

3.2.0

  • Allow wrappers and other cookbooks to supply additional logstash_config template files.

3.1.4

  • Don't just raise an error, actually disable backups when cloud account credentials aren't present

3.1.3

  • Clean up attributes so they don't error out when no cloud account is found

3.1.2

  • Adds support for backups via snapshot API

3.1.1

  • Update to latest Elasticsearch (v1.3.4)

3.1.0

  • Remove attribute for additional templates. You should now call logstash_ LWRPs directly to get this functionality back.
  • Remove automatic inclusion of platformstack cookbooks. These should be called by a downstream wrapper.
  • Better documentation for java requirement, Berkshelf requirements, and keypair requirements for Lumberjack.

3.0.2

  • Check logstash base and error nicely if node['logstash']['instance_default']['basedir'] isn't availablea

3.0.1

  • Cleanup logging flags for agent

2.1.3

  • Add attribute for additional templates to populate when this cookbook configures a logstash agent
  • Correct platformstack/logging-enabled check to be more accurate
  • Move lumberjack back to using JSON for exchanging data

2.1.2

  • Fix an upstream logstash cookbook issue, contribute it back upstream (#360). Once that is merged, we can go back to upstream.
  • Move agent attributes out more, to be their own explicit settings.

2.1.1

  • Split out agent attributes into new attribute file.
  • Clamp down agent memory usage from 256M to 10% of system at most.
  • Fix logstash version typo, 1.4.1 vs. 1.4.2.

2.1.0

  • Bump elasticsearch to version 1.3.3. This should improve memory consumption and has a host of other bug fixes. See release notes.

2.0.1

  • Add log warning for when we are automatically generating lumberjack keypairs
  • Updates to README about memory consumption and how to optimize it
  • Moved variables around in agent to make intent more clear (no logic changes)

2.0.0

  • Add a forwarder.rb recipe that installs logstash-forwarder as an alternative to logstash as an agent, including unit and integration tests.
  • Add additional tests for existing test-kitchen suites to ensure new lumberjack keypair is written to disk.
  • Fix a chefspec test issue where tests were checking for something that didn't make sense, didn't pass.

1.1.0

  • Add chef_environment fields to be shipped to central cluster
  • Add ACL recipe for agents to open up to :9300 on the ELK cluster nodes
  • Fix protocol from node to transport on logstash agents (to avoid needing more ACLs)

1.0.0

  • Bump logstash to version 1.4.2.
  • Added top-level agent recipe and test suite, intended for running logstash-agent on a regular server. Uses elasticsearch::search_discovery to find and list elk cluster nodes (can override this as well, see elasticsearch cookbook for how). Agent uses lumberjack protocol by default and requires a keypair before converging.
  • Switch nodes to communicate using node protocol on eslocal:9300 between logstash and elasticsearch on the same box.
  • Write chefspec tests for default (server nodes) and agent to 100% coverage. Add chef-sugar to Gemfile to be able to converge with chefspec and fauxhai.
  • Drop dependency on logstash_stack (didn't need to be there any longer, wasn't used).
  • Nodes no longer forward directly to logstash on the remote side, they forward to a local logstash listening for syslog.
  • Bugfix: /etc/hosts is now correctly populated for all classes of elkstack, not just multi-node.
  • Bugfix: The newrelic plugin now monitors against eslocal, not localhost.

0.3.0

  • Bumped default Elasticsearch version to 1.3.2. Java 7 is now required, and some tests had to be adjusted.
  • Fix bug in username not being used in tests because it was wrong in /etc/nginx/htpassword.curl.
  • Fix bug where port 443 was not open in iptables.

0.2.0

  • Added the ability to disable redirects on kibana

0.1.3

  • Sheppy Reno - Convert process monitors to platformstack

0.1.2

  • Add more options for kibana username and password fields under basic auth over SSL on nginx.

0.1.1

  • Seperate recipes per service, add searching and tests. Major workarounds for logstash cookbook.

0.1.0

  • Initial release of elkstack

Foodcritic Metric
            

5.0.2 failed this metric

FC015: Consider converting definition to a LWRP: /tmp/cook/85178fe2b659a904e91fcc57/elkstack/definitions/logstash_custom_config.rb:1
FC031: Cookbook without metadata file: /tmp/cook/85178fe2b659a904e91fcc57/elkstack/metadata.rb:1
FC045: Consider setting cookbook name in metadata: /tmp/cook/85178fe2b659a904e91fcc57/elkstack/metadata.rb:1