Skip to content

Latest commit

 

History

History
1811 lines (1152 loc) · 63.2 KB

CHANGELOG.asciidoc

File metadata and controls

1811 lines (1152 loc) · 63.2 KB
Raw

Beats version 6.0.0-alpha2

Breaking changes

Filebeat

  • Rename input_type field to prospector.type 4294

  • The @metadata.type field, added by the Logstash output, is now hardcoded to doc and will be removed in future versions. 4331.

Bugfixes

Affecting all Beats

  • Fix importing the dashboards when the limit for max open files is too low. 4244

  • Fix configuration documentation for kubernetes processor 4313

  • Fix misspelling in add_locale configuration option for abbreviation.

Filebeat

  • Fix race condition on harvester stopping with reloading enabled. 3779

  • Fix recursive glob config parsing and resolution across restarts. 4269

  • Allow string characters in user agent patch version (NGINX and Apache) 4415

  • Fix grok pattern in filebeat module system/auth without hostname. 4224

Metricbeat

  • Fix a debug statement that said a module wrapper had stopped when it hadn’t. 4264

  • Use MemAvailable value from /proc/meminfo on Linux 3.14. 4316

  • Fix panic when events were dropped by filters. 4327

  • Add filtering to system filesystem metricset to remove relative mountpoints like those from Linux network namespaces. 4370

  • Remove unnecessary print statement in schema apis. 4355

  • Fix type of field haproxy.stat.check.health.last. 4407

Packetbeat - Enable memcache filtering only if a port is specified in the config file. 4335 - Enable memcache filtering only if a port is specified in the config file. 4335

Added

Affecting all Beats

  • Upgraded to Golang 1.8.3. 4401

  • Added the possibility to set Elasticsearch mapping template settings from the Beat configuration file. 4284 4317

  • Add a variable to the SysV init scripts to make it easier to change the user. 4340

  • Add the option to write the generated Elasticsearch mapping template into a file. 4323

  • Add instance_name in GCE add_cloud_metadata processor. 4414

  • Add add_docker_metadata processor. 4352

  • Add logging.files permissions option. 4295

Filebeat

  • Add experimental Redis slow log prospector type. 4180

Metricbeat

  • Add macOS implementation of the system diskio metricset. 4144

  • Add process_summary metricset that records high level metrics about processes. 4231

  • Add kube-state-metrics based metrics to kubernetes module 4253

  • Add debug logging to Jolokia JMX metricset. 4341

  • Add events metricset for kubernetes metricbeat module 4315

  • Change Metricbeat default configuration file to be better optimized for most users. 4329

  • Add experimental RabbitMQ module. 4394

  • Add Kibana dashboard for the Kubernetes modules. 4138

Deprecated

Affecting all Beats

  • The @metadata.type field, added by the Logstash output, is deprecated, hardcoded to doc and will be removed in future versions. 4331.

Filebeat

  • Deprecate input_type prospector config. Use type config option instead. 4294

Known Issue

  • If the Elasticsearch output is not enabled, but setup.template options are present (like it’s the case in the default Metricbeat configuration), the Beat stops with an error: "Template loading requested but the Elasticsearch output is not configured/enabled". To avoid this error, disable the template loading explicitly setup.template.enabled: false.

Beats version 6.0.0-alpha1

Breaking changes

Affecting all Beats

  • Introduce beat version in the Elasticsearch index and mapping template 3527

  • Usage of field _type is now ignored and hardcoded to doc. 3757

  • Change vendor manager from glide to govendor. 3851

  • Rename error field to error.message. 3987

  • Change dashboards. config options to setup.dashboards.. 3921

  • Change outputs.elasticsearch.template.* to `setup.template.* 4080

Filebeat

  • Remove code to convert states from 1.x. 3767

  • Remove deprecated config options force_close_files and close_older. 3768

  • Change clean_removed behaviour to also remove states for files which cannot be found anymore under the same name. 3827

  • Remove document_type config option. Use fields instead. 4204

  • Move json_error under error.message and error.key. 4167

Packetbeat

  • Remove deprecated geoip. 3766

  • Replace waitstop command line argument by shutdown_timeout in configuration file. 3588

Winlogbeat

  • Remove metrics endpoint. Replaced by http endpoint in libbeat (see #3717). 3901

Bugfixes

Affecting all Beats

  • Add _id, _type, _index and _score fields in the generated index pattern. 3282

Filebeat

  • Fix the Mysql slowlog parsing of IP addresses. 4183

  • Fix issue that new prospector was not reloaded on conflict 4128

Heartbeat

  • Use IP type of elasticsearch for ip field. 3926

Metricbeat

  • Support common.Time in mapstriface.toTime() 3812

  • Fix MongoDB dbstats fields mapping. 4025

  • Fixing prometheus collector to aggregate metrics based on metric family. 4075

  • Fixing multiEventFetch error reporting when no events are returned 4153

Added

Affecting all Beats

  • Initialize a beats UUID from file on startup. 3615

  • Add new add_locale processor to export the local timezone with an event. 3902

  • Add http endpoint. 3717

  • Updated to Go 1.8.1. 4033

  • Add kubernetes processor 3888

  • Add support for include_labels and include_annotations in kubernetes processor 4043

  • Support new index_patterns field when loading templates for Elasticsearch >= 6.0 4056

  • Adding goimports support to make check and fmt 4114

  • Make kubernetes indexers/matchers pluggable 4151

  • Abstracting pod interface in kubernetes plugin to enable easier vendoring 4152

Filebeat

  • Restructure input.Event to be inline with outputs.Data 3823

  • Add base for supporting prospector level processors 3853

  • Add filebeat.config.path as replacement for config_dir. 4051

  • Add a recursive_glob.enabled setting to expand ** in patterns. 3980

  • Add Icinga module. 3904

  • Add ability to parse nginx logs exposing the X-Forwarded-For header instead of the remote address.

Heartbeat

  • Event format and field naming changes in Heartbeat and sample Dashboard. 4091

Metricbeat

  • Add experimental metricset perfmon to Windows module. 3758

  • Add memcached module with stats metricset. 3693

  • Add the process.cmdline.cache.enabled config option to the System Process Metricset. 3891

  • Add new MetricSet interfaces for developers (Closer, ReportingFetcher, and PushMetricSet). 3908

  • Add kubelet module 3916

  • Add dropwizard module 4022

  • Adding query APIs for metricsets and modules from metricbeat registry 4102

  • Fixing nil pointer on prometheus collector when http response is nil 4119

  • Add http module with json metricset. 4092

  • Add the option to the system module to include only the first top N processes by CPU and memory. 4127.

  • Add experimental Vsphere module. 4028

  • Add experimental Elasticsearch module. 3903

  • Add experimental Kibana module. 3895

  • Move elasticsearch metricset node_stats under node.stats namespace. 4142

  • Make IP port indexer constructor public 4434

Packetbeat

  • Add fields and fields_under_root to Packetbeat protocols configurations. 3518

  • Add list style Packetbeat protocols configurations. This change supports specifying multiple configurations of the same protocol analyzer. 3518

Winlogbeat

Deprecated

Affecting all Beats

  • Usage of field _type is deprecated. It should not be used in queries or dashboards. 3409

Packetbeat

  • Deprecate dictionary style protocols configuration. 3518

Winlogbeat

Known Issue

Filebeat

  • Prospector reloading only works properly with new files. 3546

Beats version 5.4.1

Bugfixes

Affecting all Beats

  • Fix importing the dashboards when the limit for max open files is too low. 4244

  • Fix console output. 4045

Filebeat

  • Fix issue that new prospector was not reloaded on conflict. 4128

  • Fix grok pattern in filebeat module system/auth without hostname. 4224

  • Fix the Mysql slowlog parsing of IP addresses. 4183

Added

Affecting all Beats

  • Binaries upgraded to Go 1.7.6 which contains security fixes. 4400

Winlogbeat

  • Add the ability to use LevelRaw if Level isn’t populated in the event XML. 4257

Beats version 5.4.0

Bugfixes

Affecting all Beats

  • Improve error message when downloading the dashboards fails. 3805

  • Fix potential Elasticsearch output URL parsing error if protocol scheme is missing. 3671

  • Downgrade Elasticsearch per batch item failure log to debug level. 3953

  • Make @timestamp accessible from format strings. 3721

Filebeat

  • Allow log lines without a program name in the Syslog fileset. 3944

  • Don’t stop Filebeat when modules are used with the Logstash output. 3929

Metricbeat

  • Fixing panic on the Prometheus collector when label has a comma. 3947

  • Make system process metricset honor the cpu_ticks config option. 3590

Winlogbeat

  • Fix null terminators include in raw XML string when include_xml is enabled. 3943

Added

Affecting all Beats

  • Update index mappings to support future Elasticsearch 6.X. 3778

Filebeat

  • Add auditd module for reading audit logs on Linux. 3750 3941

  • Add fileset for the Linux authorization logs. 3669

Heartbeat

  • Add default ports in HTTP monitor. 3924

Metricbeat

  • Add beta Jolokia module. 3844

  • Add dashboard for the MySQL module. 3716

  • Module configuration reloading is now beta instead of experimental. 3841

  • Marked http fields from the HAProxy module optional to improve compatibility with 1.5. 3788

  • Add support for custom HTTP headers and TLS for the Metricbeat modules. 3945

Packetbeat

  • Add DNS dashboard for an overview the DNS traffic. 3883

  • Add DNS Tunneling dashboard to highlight domains with large numbers of subdomains or high data volume. 3884

Beats version 5.3.2

Bugfixes

Filebeat

  • Properly shut down crawler in case one prospector is misconfigured. 4037

  • Fix panic in JSON decoding code if the input line is "null". 4042

Beats version 5.3.1

Bugfixes

Affecting all Beats

  • Fix panic when testing regex-AST to match against date patterns. 3889

  • Fix panic due to race condition in kafka output. 4098

Filebeat

  • Fix modules default file permissions. 3879

  • Allow - in Apache access log byte count. 3863

Metricbeat

  • Avoid errors when some Apache status fields are missing. 3074

Beats version 5.3.0

Breaking changes

Affecting all Beats

  • Configuration files must be owned by the user running the Beat or by root, and they must not be writable by others. 3544 3689

  • Change Beat generator. Use $GOPATH/src/github.com/elastic/beats/script/generate.py to generate a beat. 3452

Filebeat

  • Always use absolute path for event and registry. This can lead to issues when relative paths were used before. 3328

Metricbeat

  • Linux cgroup metrics are now enabled by default for the system process metricset. The configuration option for the feature was renamed from cgroups to process.cgroups.enabled. 3519

  • Change field names couchbase.node.couch..actual_disk_size. to couchbase.node.couch..disk_size. 3545

Bugfixes

Affecting all Beats

  • Add _id, _type, _index and _score fields in the generated index pattern. 3282

Filebeat - Always use absolute path for event and registry. 3328 - Raise an exception in case there is a syntax error in one of the configuration files available under filebeat.config_dir. 3573 - Fix empty registry file on machine crash. 3537

Metricbeat

  • Add error handling to system process metricset for when Linux cgroups are missing from the kernel. 3692

  • Add labels to the Docker healthcheck metricset output. 3707

Winlogbeat

  • Fix handling of empty strings in event_data. 3705

Added

Affecting all Beats

  • Files created by Beats (logs, registry, file output) will have 0600 permissions. 3387.

  • RPM/deb packages will now install the config file with 0600 permissions. 3382

  • Add the option to pass custom HTTP headers to the Elasticsearch output. 3400

  • Unify regexp and contains conditionals, for both to support array of strings and convert numbers to strings if required. 3469

  • Add the option to load the sample dashboards during the Beat startup phase. 3506

  • Disabled date detection in Elasticsearch index templates. Date fields must be explicitly defined in index templates. 3528

  • Using environment variables in the configuration file is now GA, instead of experimental. 3525

Filebeat

  • Add Filebeat modules for system, apache2, mysql, and nginx. 3159

  • Add the pipeline config option at the prospector level, for configuring the Ingest Node pipeline ID. 3433

  • Update regular expressions used for matching file names or lines (multiline, include/exclude functionality) to new matchers improving performance of simple string matches. 3469

  • The symlinks and harverster_limit settings are now GA, instead of experimental. 3525

  • close_timeout is also applied when the output is blocking. 3511

  • Improve handling of different path variants on Windows. 3781

  • Add multiline.flush_pattern option, for specifying the 'end' of a multiline pattern 4019

Metricbeat

  • Add experimental dbstats metricset to MongoDB module. 3228

  • Use persistent, direct connections to the configured nodes for MongoDB module. 3228

  • Add dynamic configuration reloading for modules. 3281

  • Add docker health metricset 3357

  • Add docker image metricset 3467

  • System module uses new matchers for white-listing processes. 3469

  • Add Beta CEPH module with health metricset. 3311

  • Add Beta php_fpm module with pool metricset. 3415

  • The Docker, Kafka, and Prometheus modules are now Beta, instead of experimental. 3525

  • The HAProxy module is now GA, instead of experimental. 3525

  • Add the ability to collect the environment variables from system processes. 3337

Deprecated

Affecting all Beats

  • Usage of field _type is deprecated. It should not be used in queries or dashboards. 3409

Filebeat

  • The experimental publish_async option is now deprecated and is planned to be removed in 6.0. 3525

Beats version 5.2.2

Metricbeat

  • Fix bug docker module hanging when docker container killed. 3610

  • Set timeout to period instead of 1s by default as documented. 3612

Beats version 5.2.1

Bugfixes

Metricbeat

  • Fix go routine leak in docker module. 3492

Packetbeat

  • Fix error in the NFS sample dashboard. 3548

Winlogbeat

  • Fix error in the Winlogbeat sample dashboard. 3548

Beats version 5.2.0

Bugfixes

Affecting all Beats

  • Fix overwriting explicit empty config sections. 2918

Filebeat

  • Fix alignment issue were Filebeat compiled with Go 1.7.4 was crashing on 32 bits system. 3273

Metricbeat

  • Fix service times-out at startup. 3056

  • Kafka module case sensitive host name matching. 3193

  • Fix interface conversion panic in couchbase module 3272

Packetbeat

  • Fix issue where some Cassandra visualizations were showing data from all protocols. 3314

Added

Affecting all Beats

  • Add support for passing list and dictionary settings via -E flag.

  • Support for parsing list and dictionary setting from environment variables.

  • Added new flags to import_dashboards (-cacert, -cert, -key, -insecure). 3139 3163

  • The limit for the number of fields is increased via the mapping template. 3275

  • Updated to Go 1.7.4. 3277

  • Added a NOTICE file containing the notices and licenses of the dependencies. 3334.

Heartbeat

  • First release, containing monitors for ICMP, TCP, and HTTP.

Filebeat

  • Add enabled config option to prospectors. 3157

  • Add target option for decoded_json_field. 3169

Metricbeat

  • Kafka module broker matching enhancements. 3129

  • Add a couchbase module with metricsets for node, cluster and bucket. 3081

  • Export number of cores for CPU module. 3192

  • Experimental Prometheus module. 3202

  • Add system socket module that reports all TCP sockets. 3246

  • Kafka consumer groups metricset. 3240

  • Add jolokia module with dynamic jmx metricset. 3570

Winlogbeat

  • Reduced amount of memory allocated while reading event log records. 3113 3118

Beats version 5.1.2

Bugfixes

Filebeat

  • Fix registry migration issue from old states where files were only harvested after second restart. 3322

Packetbeat

  • Fix error on importing dashboards due to colons in the Cassandra dashboard. 3140

  • Fix error on importing dashboards due to the wrong type for the geo_point fields. 3147

Winlogbeat

  • Fix for "The array bounds are invalid" error when reading large events. 3076

Beats version 5.1.1

Breaking changes

Metricbeat

  • Change data structure of experimental haproxy module. 3003

Filebeat

  • If a file is falling under ignore_older during startup, offset is now set to end of file instead of 0. With the previous logic the whole file was sent in case a line was added and it was inconsistent with files which were harvested previously. 2907

  • tail_files is now only applied on the first scan and not for all new files. 2932

Bugfixes

Affecting all Beats

  • Fix empty benign errors logged by processor actions. 3046

Metricbeat

  • Calculate the fsstat values per mounting point, and not filesystem. 2777

Added

Affecting all Beats

  • Add add_cloud_metadata processor for collecting cloud provider metadata. 2728

  • Added decode_json_fields processor for decoding fields containing JSON strings. 2605

  • Add Tencent Cloud provider for add_cloud_metadata processor. 4023

  • Add Alibaba Cloud provider for add_cloud_metadata processor. 4111

Metricbeat

  • Add experimental Docker module. Provided by Ingensi and @douaejeouit based on dockbeat.

  • Add a sample Redis Kibana dashboard. 2916

  • Add support for MongoDB 3.4 and WiredTiger metrics. 2999

  • Add experimental kafka module with partition metricset. 2969

  • Add raw config option for mysql/status metricset. 3001

  • Add command fields for mysql/status metricset. 3251

Filebeat

  • Add command line option -once to run Filebeat only once and then close. 2456

  • Only load matching states into prospector to improve state handling 2840

  • Reset all states ttl on startup to make sure it is overwritten by new config 2840

  • Persist all states for files which fall under ignore_older to have consistent behaviour 2859

  • Improve shutdown behaviour with large number of files. 3035

Winlogbeat

  • Add event_logs.batch_read_size configuration option. 2641

Beats version 5.1.0 (skipped)

Version 5.1.0 doesn’t exist because, for a short period of time, the Elastic Yum and Apt repositories included unreleased binaries labeled 5.1.0. To avoid confusion and upgrade issues for the people that have installed these without realizing, we decided to skip the 5.1.0 version and release 5.1.1 instead.

Beats version 5.0.2

Bugfixes

Metricbeat

  • Fix the password option in the MongoDB module. 2995

Beats version 5.0.1

Bugfixes

Metricbeat

  • Fix system.process.start_time on Windows. 2848

  • Fix system.process.ppid on Windows. 2860

  • Fix system process metricset for Windows XP and 2003. cmdline will be unavailable. 1704

  • Fix access denied issues in system process metricset by enabling SeDebugPrivilege on Windows. 1897

  • Fix system diskio metricset for Windows XP and 2003. 2885

Packetbeat

  • Fix 'index out of bounds' bug in Packetbeat DNS protocol plugin. 2872

Filebeat

  • Fix registry cleanup issue when files falling under ignore_older after restart. 2818

Added

Metricbeat

  • Add username and password config options to the PostgreSQL module. 2890

  • Add username and password config options to the MongoDB module. 2889

  • Add system core metricset for Windows. 2883

Packetbeat

  • Define client_geoip.location as geo_point in the mappings to be used by the GeoIP processor in the Ingest Node pipeline. 2795

Filebeat

  • Stop Filebeat on registrar loading error. 2868

Beats version 5.0.0-GA

The list below covers the changes between 5.0.0-rc1 and 5.0.0 GA only.

Bugfixes

Affecting all Beats

  • Fix kafka output re-trying batches with too large events. 2735

  • Fix kafka output protocol error if version: 0.10 is configured. 2651

  • Fix kafka output connection closed by broker on SASL/PLAIN. 2717

Metricbeat

  • Fix high CPU usage on macOS when encountering processes with long command lines. 2747

  • Fix high value of system.memory.actual.free and system.memory.actual.used. 2653

  • Change several OpenProcess calls on Windows to request the lowest possible access provilege. 1897

  • Fix system.memory.actual.free high value on Windows. 2653

Filebeat

  • Fix issue when clean_removed and clean_inactive were used together that states were not directly removed from the registry.

  • Fix issue where upgrading a 1.x registry file resulted in duplicate state entries. 2792

Added

Affecting all Beats

  • Add beat.version fields to all events.

Beats version 5.0.0-rc1

Breaking changes

Affecting all Beats

  • A dynamic mapping rule is added to the default Elasticsearch template to treat strings as keywords by default. 2688

Bugfixes

Affecting all Beats

  • Make sure Beats sent always float values when they are defined as float by sending 5.00000 instead of 5. 2627

  • Fix ignoring all fields from drop_fields in case the first field is unknown. 2685

  • Fix dynamic configuration int/uint to float type conversion. 2698

  • Fix primitive types conversion if values are read from environment variables. 2698

Metricbeat

  • Fix default configuration file on Windows to not enabled the load metricset. 2632

Packetbeat

  • Fix the bpf_filter setting. 2660

Filebeat

  • Fix input buffer on encoding problem. 2416

Deprecated

Affecting all Beats

  • Setting port has been deprecated in Redis and Logstash outputs. 2620

Beats version 5.0.0-beta1

Breaking changes

Affecting all Beats

  • Change Elasticsearch output index configuration to be based on format strings. If index has been configured, no date will be appended anymore to the index name. 2119

  • Replace output.kafka.use_type by output.kafka.topic accepting a format string. 2188

  • If the path specified by the -c flag is not absolute and -path.config is not specified, it is considered relative to the current working directory. 2245

  • rename tls configurations section to ssl. 2330

  • rename certificate_key configuration to key. 2330

  • replace tls.insecure with ssl.verification_mode setting. 2330

  • replace tls.min/max_version with ssl.supported_protocols setting requiring full protocol name. 2330

Metricbeat

  • Change field type system.process.cpu.start_time from keyword to date. 1565

  • redis/info metricset fields were renamed up according to the naming conventions.

Packetbeat

  • Group HTTP fields under http.request and http.response 2167

  • Export http.request.body and http.response.body when configured under include_body_for 2167

  • Move ignore_outgoing config to packetbeat.ignore_outgoing 2393

Filebeat

  • Set close_inactive default to 5 minutes (was 1 hour before)

  • Set clean_removed and close_removed to true by default

Bugfixes

Affecting all Beats

  • Fix logstash output handles error twice when asynchronous sending fails. 2441

  • Fix Elasticsearch structured error response parsing error. 2229

  • Fixed the run script to allow the overriding of the configuration file. 2171

  • Fix logstash output crash if no hosts are configured. 2325

  • Fix array value support in -E CLI flag. 2521

  • Fix merging array values if -c CLI flag is used multiple times. 2521

  • Fix beats failing to start due to invalid duplicate key error in configuration file. 2521

  • Fix panic on non writable logging directory. 2571

Metricbeat

  • Fix module filters to work properly with drop_event filter. 2249

Packetbeat

  • Fix mapping for some Packetbeat flow metrics that were not marked as being longs. 2177

  • Fix handling of messages larger than the maximum message size (10MB). 2470

Filebeat

  • Fix processor failure in Filebeat when using regex, contain, or equals with the message field. 2178

  • Fix async publisher sending empty events 2455

  • Fix potential issue with multiple harvester per file on large file numbers or slow output 2541

Winlogbeat

  • Fix corrupt registry file that occurs on power loss by disabling file write caching. 2313

Added

Affecting all Beats

  • Add script to generate the Kibana index-pattern from fields.yml. 2122

  • Enhance Redis output key selection based on format string. 2169

  • Configurable Redis keys using filters and format strings. 2169

  • Add format string support to output.kafka.topic. 2188

  • Add output.kafka.topics for more advanced kafka topic selection per event. 2188

  • Add support for Kafka 0.10. 2190

  • Add SASL/PLAIN authentication support to kafka output. 2190

  • Make Kafka metadata update configurable. 2190

  • Add Kafka version setting (optional) enabling kafka broker version support. 2190

  • Add Kafka message timestamp if at least version 0.10 is configured. 2190

  • Add configurable Kafka event key setting. 2284

  • Add settings for configuring the kafka partitioning strategy. 2284

  • Add partitioner settings reachable_only to ignore partitions not reachable by network. 2284

  • Enhance contains condition to work on fields that are arrays of strings. 2237

  • Lookup the configuration file relative to the -path.config CLI flag. 2245

  • Re-write import_dashboards.sh in Golang. 2155

  • Update to Go 1.7. 2306

  • Log total non-zero internal metrics on shutdown. 2349

  • Add support for encrypted private key files by introducing ssl.key_passphrase setting. 2330

  • Add experimental symlink support with symlinks config 2478

  • Improve validation of registry file on startup.

Metricbeat

  • Use the new scaled_float Elasticsearch type for the percentage values. 2156

  • Add experimental cgroup metrics to the system/process MetricSet. 2184

  • Added a PostgreSQL module. 2253

  • Improve mapping by converting half_float to scaled_float and integers to long. 2430

  • Add experimental haproxy module. 2384

  • Add Kibana dashboard for cgroups data 2555

Packetbeat

  • Add Cassandra protocol analyzer to Packetbeat. 1959

  • Match connections with IPv6 addresses to processes 2254

  • Add IP address to -devices command output 2327

  • Add configuration option for the maximum message size. Used to be hard-coded to 10 MB. 2470

Filebeat

  • Introduce close_timeout harvester options 1926

  • Strip BOM from first message in case of BOM files 2351

  • Add harvester_limit option 2417

Deprecated

Affecting all Beats

  • Topology map is deprecated. This applies to the settings: refresh_topology_freq, topology_expire, save_topology, host_topology, password_topology, db_topology.

Beats version 5.0.0-alpha5

Breaking changes

Affecting all Beats

  • Rename the filters section to processors. 1944

  • Introduce the condition with when in the processor configuration. 1949

  • The Elasticsearch template is now loaded by default. 1993

  • The Redis output index setting is renamed to key. index still works but it’s deprecated. 2077

  • The undocumented file output index setting was removed. Use filename instead. 2077

Metricbeat

  • Create a separate metricSet for load under the system module and remove load information from CPU stats. 2101

  • Add system.load.norm.1, system.load.norm.5 and system.load.norm.15. 2101

  • Add threads fields to mysql module. 2484

Packetbeat

  • Set enabled ` in packetbeat.protocols.icmp configuration to true by default. 1988

Bugfixes

Affecting all Beats

  • Fix sync publisher PublishEvents return value if client is closed concurrently. 2046

Metricbeat

  • Do not send zero values when no value was present in the source. 1972

Filebeat

  • Fix potential data loss between Filebeat restarts, reporting unpublished lines as published. 2041

  • Fix open file handler issue. 2028 2020

  • Fix filtering of JSON events when using integers in conditions. 2038

Winlogbeat

  • Fix potential data loss between Winlogbeat restarts, reporting unpublished lines as published. 2041

Added

Affecting all Beats

  • Periodically log internal metrics. 1955

  • Add enabled setting to all output modules. 1987

  • Command line flag -c can be used multiple times. 1985

  • Add OR/AND/NOT to the condition associated with the processors. 1983

  • Add -E CLI flag for overwriting single config options via command line. 1986

  • Choose the mapping template file based on the Elasticsearch version. 1993

  • Check stdout being available when console output is configured. 2035

Metricbeat

Packetbeat

  • Add enabled setting to Packetbeat protocols. 1988

  • Add enabled setting to Packetbeat network flows configuration. 1988

Filebeat

  • Introduce close_removed and close_renamed harvester options. 1600

  • Introduce close_eof harvester option. 1600

  • Add clean_removed and clean_inactive config option. 1600

Deprecated

Filebeat

  • Deprecate close_older option and replace it with close_inactive. 2051

  • Deprecate force_close_files option and replace it with close_removed and close_renamed. 1600

Beats version 5.0.0-alpha4

Breaking changes

Affecting all Beats

  • The topology_expire option of the Elasticserach output was removed. 1907

Filebeat

  • Stop following symlink. Symlinks are now ignored: 1686

Bugfixes

Affecting all Beats

  • Reset backoff factor on partial ACK. 1803

  • Fix beats load balancer deadlock if max_retries: -1 or publish_async is enabled in filebeat. 1829

  • Fix logstash output with pipelining mode enabled not reconnecting. 1876

  • Empty configuration sections become merge-able with variables containing full path. 1900

  • Fix error message about required fields missing not printing the missing field name. 1900

Metricbeat

  • Fix the CPU values returned for each core. 1863

Packetbeat

  • Add missing nil-check to memcached GapInStream handler. 1162

  • Fix NFSv4 Operation returning the first found first-class operation available in compound requests. 1821

  • Fix TCP overlapping segments not being handled correctly. 1898

Winlogbeat

  • Fix issue with rendering forwarded event log records. 1891

Added

Affecting all Beats

  • Improve error message if compiling regular expression from config files fails. 1900

  • Compression support in the Elasticsearch output. 1835

Metricbeat

  • Add MongoDB module. 1837

Beats version 5.0.0-alpha3

Breaking changes

Affecting all Beats

  • All configuration settings under shipper: are moved to be top level configuration settings. I.e. shipper.name: becomes name: in the configuration file. 1570

Topbeat

  • Topbeat is replaced by Metricbeat.

Filebeat

  • The state for files which fall under ignore_older is not stored anymore. This has the consequence, that if a file which fell under ignore_older is updated, the whole file will be crawled.

Bugfixes

Winlogbeat

  • Adding missing argument to the "Stop processing" log message. 1590

Added

Affecting all Beats

  • Add conditions to generic filtering. 1623

Metricbeat

  • First public release, containing the following modules: apache, mysql, nginx, redis, system, and zookeeper.

Filebeat

  • The registry format was changed to an array instead of dict. The migration to the new format will happen automatically at the first startup. 1703

Deprecated

Affecting all Beats

  • The support for doing GeoIP lookups is deprecated and will be removed in version 6.0. 1601

Beats version 5.0.0-alpha2

Breaking changes

Affecting all Beats

  • On DEB/RPM installations, the binary files are now found under /usr/share/{{beat_name}}/bin, not in /usr/bin. 1385

  • The logs are written by default to self rotating files, instead of syslog. 1371

  • Remove deprecated host option from elasticsearch, logstash and redis outputs. 1474

Packetbeat

  • Configuration of redis topology support changed. 1353

  • Move all Packetbeat configuration options under the packetbeat namespace 1417

Filebeat

  • Default location for the registry file was changed to be data/registry from the binary directory, rather than .filebeat in the current working directory. This affects installations for zip/tar.gz/source, the location for DEB and RPM packages stays the same. 1373

Bugfixes

Affecting all Beats

  • Drain response buffers when pipelining is used by Redis output. 1353

  • Unterminated environment variable expressions in config files will now cause an error 1389

  • Fix issue with the automatic template loading when Elasticsearch is not available on Beat start. 1321

  • Fix bug affecting -cpuprofile, -memprofile, and -httpprof CLI flags 1415

  • Fix race when multiple outputs access the same event with logstash output manipulating event 1410 1428

  • Seed random number generator using crypto.rand package. https://github.com/elastic/beats/pull/1503{1503]

  • Fix beats hanging in -configtest 1213

  • Fix kafka log message output 1516

Filebeat

  • Improvements in registrar dealing with file rotation. 1281

  • Fix issue with JSON decoding where @timestamp or type keys with the wrong type could cause Filebeat to crash. 1378

  • Fix issue with JSON decoding where values having null as values could crash Filebeat. 1466

  • Multiline reader normalizing newline to use \n. 1552

Winlogbeat

  • Fix panic when reading messages larger than 32K characters on Windows XP and 2003. 1498

  • Fix panic that occurs when reading a large events on Windows Vista and newer. 1499

Added

Affecting all Beats

  • Add support for TLS to Redis output. 1353

  • Add SOCKS5 proxy support to Redis output. 1353

  • Failover and load balancing support in redis output. 1353

  • Multiple-worker per host support for redis output. 1353

  • Added ability to escape ${x} in config files to avoid environment variable expansion 1389

  • Configuration options and CLI flags for setting the home, data and config paths. 1373

  • Configuration options and CLI flags for setting the default logs path. 1437

  • Update to Go 1.6.2 1447

  • Add Elasticsearch template files compatible with Elasticsearch 2.x. 1501

  • Add scripts for managing the dashboards of a single Beat 1359

Packetbeat

  • Fix compile issues for OpenBSD. 1347

Topbeat

  • Updated elastic/gosigar version so Topbeat can compile on OpenBSD. 1403

Beats version 5.0.0-alpha1

Breaking changes

libbeat

  • Run function to start a Beat now returns an error instead of directly exiting. 771

  • The method signature of HandleFlags() was changed to allow returning an error 1249

  • Require braces for environment variable expansion in config files 1304

Packetbeat

  • Rename output fields in the dns package. Former flag recursion_allowed becomes recursion_available. 803 Former SOA field ttl becomes minimum. 803

  • The fully qualified domain names which are part of output fields values of the dns package now terminate with a dot. 803

  • Remove the count field from the exported event 1210

Topbeat

  • Rename proc.cpu.user_p with proc.cpu.total_p as it includes CPU time spent in kernel space 631

  • Remove count field from the exported fields 1207

  • Rename input top level config option to topbeat

Filebeat

  • Scalar values in used in the fields configuration setting are no longer automatically converted to strings. 1092

  • Count field was removed from event as not used in filebeat 778

Winlogbeat

  • The message_inserts field was replaced with the event_data field 1053

  • The category field was renamed to task to better align with the Windows Event Log API naming 1053

  • Remove the count field from the exported event 1218

Bugfixes

Affecting all Beats

  • Logstash output will not retry events that are not JSON-encodable 927

Packetbeat

  • Create a proper BPF filter when ICMP is the only enabled protocol 757

  • Check column length in pgsql parser. 565

  • Harden pgsql parser. 565

Topbeat

  • Fix issue with cpu.system_p being greater than 1 on Windows 1128

Filebeat

  • Stop filebeat if started without any prospectors defined or empty prospectors 644 647

  • Improve shutdown of crawler and prospector to wait for clean completion 720

  • Omit fields from Filebeat events when null 899

Winlogbeat

Added

Affecting all Beats

  • Update builds to Golang version 1.6

  • Add option to Elasticsearch output to pass http parameters in index operations 805

  • Improve Logstash and Elasticsearch backoff behavior. 927

  • Add experimental Kafka output. 942

  • Add config file option to configure GOMAXPROCS. 969

  • Improve shutdown handling in libbeat. 1075

  • Add fields and fields_under_root options under the shipper configuration 1092

  • Add the ability to use a SOCKS5 proxy with the Logstash output 823

  • The -configtest flag will now print "Config OK" to stdout on success 1249

Packetbeat

  • Change the DNS library used throughout the dns package to github.com/miekg/dns. 803

  • Add support for NFS v3 and v4. 1231

  • Add support for EDNS and DNSSEC. 1292

Topbeat

  • Add username to processes 845

Filebeat

  • Add the ability to set a list of tags for each prospector 1092

  • Add JSON decoding support 1143

Winlogbeat

  • Add caching of event metadata handles and the system render context for the wineventlog API 888

  • Improve config validation by checking for unknown top-level YAML keys. 1100

  • Add the ability to set tags, fields, and fields_under_root as options for each event log 1092

  • Add additional data to the events published by Winlogbeat. The new fields are activity_id, event_data, keywords, opcode, process_id, provider_guid, related_activity_id, task, thread_id, user_data, and version. 1053

  • Add event_id, level, and provider configuration options for filtering events 1218

  • Add include_xml configuration option for including the raw XML with the event 1218

Known issues

  • All Beats can hang or panic on shutdown if the next server in the pipeline (e.g. Elasticsearch or Logstash) is not reachable. 1319

  • When running the Beats as a service on Windows, you need to manually load the Elasticsearch mapping template. 1315

  • The ES template automatic load doesn’t work if Elasticsearch is not available when the Beat is starting. 1321

Beats version 1.3.1

Bugfixes

Filebeat

  • Fix a concurrent bug on filebeat startup with a large number of prospectors defined. 2509

Packetbeat

  • Fix description for the -I CLI flag. 2480

Winlogbeat

  • Fix corrupt registry file that occurs on power loss by disabling file write caching. 2313

Beats version 1.3.0

Deprecated

Filebeat

  • Undocumented support for following symlinks is deprecated. Filebeat will not follow symlinks in version 5.0. 1767

Bugfixes

Affecting all Beats

  • Fix beats load balancer deadlock if max_retries: -1 or publish_async is enabled in filebeat. 1829

  • Fix output modes backoff counter reset. 1803 1814 1818

  • Set logstash output default bulk_max_size to 2048. 1662

  • Seed random number generator using crypto.rand package. 1503

  • Check stdout being available when console output is configured. 2063

Packetbeat

  • Add missing nil-check to memcached GapInStream handler. 1162

  • Fix NFSv4 Operation returning the first found first-class operation available in compound requests. 1821

  • Fix TCP overlapping segments not being handled correctly. 1917

Added

Affecting all Beats

  • Updated to Go 1.7

Beats version 1.2.3

Bugfixes

Topbeat

  • Fix high CPU usage when using filtering under Windows. 1598

Filebeat

  • Fix rotation issue with ignore_older. 1528

Winlogbeat

  • Fix panic when reading messages larger than 32K characters on Windows XP and 2003. 1498

Added

Filebeat

  • Prevent file opening for files which reached ignore_older. 1649

Beats version 1.2.2

Bugfixes

Affecting all Beats

  • Fix race when multiple outputs access the same event with Logstash output manipulating event. 1410

  • Fix go-daemon (supervisor used in init scripts) hanging when executed over SSH. 1394

Filebeat

  • Improvements in registrar dealing with file rotation. 1281

Beats version 1.2.1

Breaking changes

Affecting all Beats

  • Require braces for environment variable expansion in config files 1304

  • Removed deprecation warning for the Redis output. 1282

Topbeat

  • Fixed name of the setting stats.proc to stats.process in the default configuration file. 1343

  • Fix issue with cpu.system_p being greater than 1 on Windows 1128

Added

Topbeat

  • Add username to processes 845

Beats version 1.2.0

Breaking changes

Filebeat

  • Default config for ignore_older is now infinite instead of 24h, means ignore_older is disabled by default. Use close_older to only close file handlers.

Bugfixes

Packetbeat

  • Split real_ip_header value when it contains multiple IPs 1241

Winlogbeat

  • Fix invalid event_id on Windows XP and Windows 2003 1227

Added

Affecting all Beats

  • Add ability to override configuration settings using environment variables 114

  • Libbeat now always exits through a single exit method for proper cleanup and control 736

  • Add ability to create Elasticsearch mapping on startup 639

Topbeat

  • Add the command line used to start processes 533

Filebeat

  • Add close_older configuration option to complete ignore_older 181

Beats version 1.1.2

Bugfixes

Filebeat

  • Fix registrar bug for rotated files 1010

Beats version 1.1.1

Bugfixes

Affecting all Beats

  • Fix logstash output loop hanging in infinite loop on too many output errors. 944

  • Fix critical bug in filebeat and winlogbeat potentially dropping events. 953

Beats version 1.1.0

Bugfixes

Affecting all Beats

  • Fix logging issue with file based output where newlines could be misplaced during concurrent logging 650

  • Reduce memory usage by separate queue sizes for single events and bulk events. 649 516

  • Set default default bulk_max_size value to 2048 628

Packetbeat

  • Fix setting direction to out and use its value to decide when dropping events if ignore_outgoing is enabled 557

  • Fix logging issue with file-based output where newlines could be misplaced during concurrent logging 650

  • Reduce memory usage by having separate queue sizes for single events and bulk events. 649 516

  • Set default bulk_max_size value to 2048 628

  • Fix logstash window size of 1 not increasing. 598

Packetbeat

  • Fix the condition that determines whether the direction of the transaction is set to "outgoing". Packetbeat uses the direction field to determine which transactions to drop when dropping outgoing transactions. 557

  • Allow PF_RING sniffer type to be configured using pf_ring or pfring 671

Filebeat

  • Set spool_size default value to 2048 628

Added

Affecting all Beats

  • Add include_fields and drop_fields as part of generic filtering 1120

  • Make logstash output compression level configurable. 630

  • Some publisher options refactoring in libbeat 684

  • Move event preprocessor applying GeoIP to packetbeat 772

Packetbeat

  • Add support for capturing DNS over TCP network traffic. 486 554

Topbeat

  • Group all CPU usage per core statistics and export them optionally if cpu_per_core is configured 496

Filebeat

  • Add multiline support for combining multiple related lines into one event. 461

  • Add exclude_lines and include_lines options for regexp based line filtering. 430

  • Add exclude_files configuration option. 563

  • Add experimental option to enable filebeat publisher pipeline to operate asynchonrously 782

Winlogbeat

  • First public release of Winlogbeat

Beats version 1.0.1

Bugfixes

Filebeat

  • Fix force_close_files in case renamed file appeared very fast. 302

Packetbeat

  • Improve MongoDB message correlation. 377

  • Improve redis parser performance. 422

  • Fix panic on nil in redis protocol parser. 384

  • Fix errors redis parser when messages are split in multiple TCP segments. 402

  • Fix errors in redis parser when length prefixed strings contain sequences of CRLF. 402

  • Fix errors in redis parser when dealing with nested arrays. 402

Beats version 1.0.0

Breaking changes

Topbeat

  • Change proc type to process #138

Bugfixes

Affecting all Beats

  • Fix random panic on shutdown by calling shutdown handler only once. elastic/filebeat#204

  • Fix credentials are not send when pinging an elasticsearch host. elastic/fileabeat#287

Filebeat

  • Fix problem that harvesters stopped reading after some time and filebeat stopped processing events #257

  • Fix line truncating by internal buffers being reused by accident #258

  • Set default ignore_older to 24 hours #282

Beats version 1.0.0-rc2

Breaking changes

Affecting all Beats

  • The shipper output field is renamed to beat.name. #285

  • Use of enabled as a configuration option for outputs (elasticsearch, logstash, etc.) has been removed. #264

  • Use of disabled as a configuration option for tls has been removed. #264

  • The -test command line flag was renamed to -configtest. #264

  • Disable geoip by default. To enable it uncomment in config file. #305

Filebeat

  • Removed utf-16be-bom encoding support. Support will be added with fix for #205

  • Rename force_close_windows_files to force_close_files and make it available for all platforms.

Bugfixes

Affecting all Beats

  • Disable logging to stderr after configuration phase. #276

  • Set the default file logging path when not set in config. #275

  • Fix bug silently dropping records based on current window size. elastic/filebeat#226

  • Fix direction field in published events. #300

  • Fix elasticsearch structured errors breaking error handling. #309

Packetbeat

  • Packetbeat will now exit if a configuration error is detected. #357

  • Fixed an issue handling DNS requests containing no questions. #369

Topbeat

  • Fix leak of Windows handles. #98

  • Fix memory leak of process information. #104

Filebeat

  • Filebeat will now exit if a configuration error is detected. #198

  • Fix to enable prospector to harvest existing files that are modified. #199

  • Improve line reading and encoding to better keep track of file offsets based on encoding. #224

  • Set input_type by default to "log"

Added

Affecting all Beats

  • Added beat.hostname to contain the hostname where the Beat is running on as returned by the operating system. #285

  • Added timestamp for file logging. #291

Filebeat

  • Handling end of line under windows was improved #233

Beats version 1.0.0-rc1

Breaking changes

Affecting all Beats

  • Rename timestamp field with @timestamp. #237

Packetbeat

  • Rename timestamp field with @timestamp. #343

Topbeat

  • Rename timestamp field with @timestamp for a better integration with Logstash. #80

Filebeat

  • Rename the timestamp field with @timestamp #168

  • Rename tail_on_rotate prospector config to tail_files

  • Removal of line field in event. Line number was not correct and does not add value. #217

Bugfixes

Affecting all Beats

  • Use stderr for console log output. #219

  • Handle empty event array in publisher. #207

  • Respect '*' debug selector in IsDebug. #226 (elastic#339)

  • Limit number of workers for Elasticsearch output. elastic#226

  • On Windows, remove service related error message when running in the console. #242

  • Fix waitRetry no configured in single output mode configuration. elastic/filebeat#144

  • Use http as the default scheme in the elasticsearch hosts #253

  • Respect max bulk size if bulk publisher (collector) is disabled or sync flag is set.

  • Always evaluate status code from Elasticsearch responses when indexing events. #192

  • Use bulk_max_size configuration option instead of bulk_size. #256

  • Fix max_retries=0 (no retries) configuration option. #266

  • Filename used for file based logging now defaults to beat name. #267

Packetbeat

  • Close file descriptors used to monitor processes. #337

  • Remove old RPM spec file. It moved to elastic/beats-packer. #334

Topbeat

  • Don’t wait for one period until shutdown #75

Filebeat

  • Omit 'fields' from event JSON when null. #126

  • Make offset and line value of type long in elasticsearch template to prevent overflow. #140

  • Fix locking files for writing behaviour. #156

  • Introduce 'document_type' config option per prospector to define document type for event stored in elasticsearch. #133

  • Add 'input_type' field to published events reporting the prospector type being used. #133

  • Fix high CPU usage when not connected to Elasticsearch or Logstash. #144

  • Fix issue that files were not crawled anymore when encoding was set to something other then plain. #182

Added

Affecting all Beats

  • Add Console output plugin. #218

  • Add timestamp to log messages #245

  • Send @metadata.beat to Logstash instead of @metadata.index to prevent possible name clashes and give user full control over index name used for Elasticsearch

  • Add logging messages for bulk publishing in case of error #229

  • Add option to configure number of parallel workers publishing to Elasticsearch or Logstash.

  • Set default bulk size for Elasticsearch output to 50.

  • Set default http timeout for Elasticsearch to 90s.

  • Improve publish retry if sync flag is set by retrying only up to max bulk size events instead of all events to be published.

Filebeat

  • Introduction of backoff, backoff_factor, max_backoff, partial_line_waiting, force_close_windows_files config variables to make crawling more configurable.

  • All Godeps dependencies were updated to master on 2015-10-21 [#122]

  • Set default value for ignore_older config to 10 minutes. #164

  • Added the fields_under_root setting to optionally store the custom fields top level in the output dictionary. #188

  • Add more encodings by using x/text/encodings/htmlindex package to select encoding by name.

Beats version 1.0.0-beta4

Breaking changes

Affecting all Beats

  • Update tls config options naming from dash to underline #162

  • Feature/output modes: Introduction of PublishEvent(s) to be used by beats #118 #115

Packetbeat

  • Renamed http module config file option 'strip_authorization' to 'redact_authorization'

  • Save_topology is set to false by default

  • Rename elasticsearch index to [packetbeat-]YYYY.MM.DD

Topbeat

  • Percentage fields (e.g user_p) are exported as a float between 0 and 1 #34

Bugfixes

Affecting all Beats

  • Determine Elasticsearch index for an event based on UTC time #81

  • Fixing ES output’s defaultDeadTimeout so that it is 60 seconds #103

  • ES outputer: fix timestamp conversion #91

  • Fix TLS insecure config option #239

  • ES outputer: check bulk API per item status code for retransmit on failure.

Packetbeat

  • Support for lower-case header names when redacting http authorization headers

  • Redact proxy-authorization if redact-authorization is set

  • Fix some multithreading issues #203

  • Fix negative response time #216

  • Fix memcache TCP connection being nil after dropping stream data. #299

  • Add missing DNS protocol configuration to documentation #269

Topbeat

  • Don’t divide the reported memory by an extra 1024 #60

Added

Affecting all Beats

  • Add logstash output plugin #151

  • Integration tests for Beat → Logstash → Elasticsearch added #195 #188 #168 #137 #128 #112

  • Large updates and improvements to the documentation

  • Add direction field to publisher output to indicate inbound/outbound transactions #150

  • Add tls configuration support to elasticsearch and logstash outputers #139

  • All external dependencies were updated to the latest version. Update to Golang 1.5.1 #162

  • Guarantee ES index is based in UTC time zone #164

  • Cache: optional per element timeout #144

  • Make it possible to set hosts in different ways. #135

  • Expose more TLS config options #124

  • Use the Beat name in the default configuration file path #99

Packetbeat

  • add [.editorconfig file](http://editorconfig.org/)

  • add (experimental/unsupported?) saltstack files

  • Sample config file cleanup

  • Moved common documentation to [libbeat repository](https://github.com/elastic/libbeat)

  • Update build to go 1.5.1

  • Adding device descriptions to the -device output.

  • Generate coverage for system tests

  • Move go-daemon dependency to beats-packer

  • Rename integration tests to system tests

  • Made the -devices option more user friendly in case sudo is not used. Issue #296.

  • Publish expired DNS transactions #301

  • Update protocol guide to libbeat changes

  • Add protocol registration to new protocol guide

  • Make transaction timeouts configurable #300

  • Add direction field to the exported fields #317

Topbeat

  • Document fields in a standardized format (etc/fields.yml) #34

  • Updated to use new libbeat Publisher #37 #41

  • Update to go 1.5.1 #43

  • Updated configuration files with comments for all options #65

  • Documentation improvements

Deprecated

Affecting all Beats

  • Redis output was deprecated #169 #145

  • Host and port configuration options are deprecated. They are replaced by the hosts configuration option. #141