[[breaking-changes]]
== Breaking Changes

We strive to maintain backward compatibility between minor versions (7.x to 7.y,
for example) so that you can upgrade without changing any configuration files.
Breaking changes are usually introduced only between major versions (such as 6.x
to 7.y). On occasion, we are forced to break compatibility within a given major release
to ensure correctness of operation.

This section covers the changes that you need to be aware of when migrating to
Logstash 7.0.0 and later.

NOTE: Migrating directly between non-consecutive major versions (5.x to
7.x) is not recommended.

See these topics for a description of breaking changes:

* <<breaking-7.0>>
* <<breaking-pq>>
* <<breaking-6.0>>

See also <<releasenotes>>. 

//NOTE: The notable-breaking-changes tagged regions are re-used in the
//Installation and Upgrade Guide

[[breaking-7.0]]
=== Breaking changes in 7.0

Here are the breaking changes for {ls} 7.0. 

// tag::notable-breaking-changes[]
[float]
==== Changes in Logstash Core

These changes can affect any instance of Logstash that uses impacted features.
Changes to Logstash Core are plugin agnostic.

[float]
[[java-exec-default]]
===== Java execution engine enabled by default

The new Java execution engine is enabled by default. It features faster
performance, reduced memory usage, and lower config startup and reload times.

For more information, see the blog post about the
https://www.elastic.co/blog/meet-the-new-logstash-java-execution-engine[initial
release of the Java execution engine].  

We went to considerable lengths to make this change seamless. Still, it's a big
change. If you notice different behaviors that might be related, please 
https://github.com/elastic/logstash/issues[open a GitHub issue] to let us
know.

[float]
[[beats-ecs]]
===== Beats conform to the Elastic Common Schema (ECS)

As of 7.0, Beats fields conform to the {ecs-ref}/index.html[Elastic Common
Schema (ECS)].

If you upgrade Logstash before you upgrade Beats, the payloads continue to use
the pre-ECS schema. If you upgrade your Beats before you upgrade Logstash, then
you'll get payloads with the ECS schema in advance of any Logstash upgrade.

If you see mapping conflicts after upgrade, that is an indication that the
Beats/ECS change is influencing the data reaching existing indices. 

See the *{beats} Platform Reference* for more information on
{beats-ref}/upgrading-6-to-7.html#enable-ecs-compatibility[Beats and ECS].

[float]
[[field-ref-strict]]
===== Field Reference parser is more strict

The Field Reference parser, which is used to interpret references to fields in
your pipelines and plugins, was made to be more strict and will now reject
inputs that are either ambiguous or illegal. Since 6.4, Logstash has emitted
warnings when encountering input that is ambiguous, and allowed an early opt-in
of strict-mode parsing either by providing the command-line flag
`--field-reference-parser STRICT` or by adding `config.field_reference.parser:
STRICT` to `logstash.yml`.

Here's an example. 

*Before*

[source,txt]
-----
logstash-6.7.0 % echo "hello"| bin/logstash -e 'filter { mutate { replace => { "message" => "%{[[]]message]} you" } } }'
[2019-04-05T16:52:18,691][WARN ][org.logstash.FieldReference] Detected ambiguous Field Reference `[[]]message]`, which we expanded to the path `[message]`; in a future release of Logstash, ambiguous Field References will not be expanded.
{
       "message" => "hello you",
      "@version" => "1",
    "@timestamp" => 2019-04-05T15:52:18.546Z,
          "type" => "stdin",
          "host" => "overcraft.lan"
}
-----

*After* 

[source,txt]
-----
logstash-7.0.0 % echo "hello"| bin/logstash -e 'filter { mutate { replace => { "message" => "%{[[]]message]} you" } } }'
[2019-04-05T16:48:09,135][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>java.lang.IllegalStateException: org.logstash.FieldReference$IllegalSyntaxException: Invalid FieldReference: `[[]]message]`
[2019-04-05T16:48:09,167][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
-----

  
[float]
==== Changes in Logstash Plugins

With 7.0.0, we have taken the opportunity to upgrade a number of bundled plugins
to their newest major version, absorbing their breaking changes into the
Logstash distribution.

While these upgrades included new features and important fixes, only the
breaking changes are called out below.

NOTE: The majority of the changes to plugins are the removal of previously-deprecated
and now-obsolete options. Please ensure that your pipeline
configurations do not use these removed options before upgrading.

[float]
===== Codec Plugins

Here are the breaking changes for codec plugins.

*CEF Codec*

* Removed obsolete `sev` option
* Removed obsolete `deprecated_v1_fields` option

*Netflow Codec*

* Changed decoding of application_id to implement RFC6759; the format changes from a pair of colon-separated ids (e.g. `0:40567`) to a variable number of double-dot-separated ids (e.g. `0..12356..40567`).

[float]
===== Filter Plugins

Here are the breaking changes for filter plugins.

*Clone Filter*

* Make `clones` a required option

*Geoip Filter*

* Removed obsolete `lru_cache_size` option

*HTTP Filter*

* Removed obsolete `ssl_certificate_verify` option

[float]
===== Input Plugins

Here are the breaking changes for  input plugins.

*Beats Input*

* Removed obsolete `congestion_threshold` option
* Removed obsolete `target_field_for_codec` option
* Changed default value of `add_hostname` to false

NOTE: In Beats 7.0.0, the fields exported by Beats _to_ the Logstash Beats Input
conform to the {ecs-ref}/index.html[Elastic Common Schema (ECS)]. Many of the
exported fields have been renamed, so you may need to modify your pipeline
configurations to access them at their new locations prior to upgrading your
Beats. See {beats-ref}/breaking-changes-7.0.html[Beats Breaking changes in 7.0]
for the full list of changed names.

*HTTP Input*

* Removed obsolete `ssl_certificate_verify` option

*HTTP Poller Input*

* Removed obsolete `interval` option
* Removed obsolete `ssl_certificate_verify` option

*Tcp Input*

* Removed obsolete `data_timeout` option
* Removed obsolete `ssl_cacert` option

[float]
===== Output Plugins

Here are the breaking changes for output plugins.

*Elasticsearch Output*

* {es} {ref}/index-lifecycle-management.html[Index lifecycle management (ILM)] is
auto-detected and enabled by default if your {es} cluster supports it.
* Remove support for parent/child (still support join data type) since we don't
support multiple document types any more
* Removed obsolete `flush_size` option
* Removed obsolete `idle_flush_time` option

*HTTP Output*

* Removed obsolete `ssl_certificate_verify` option

*Kafka Output*

* Removed obsolete `block_on_buffer_full` option
* Removed obsolete `ssl` option
* Removed obsolete `timeout_ms` option

*Redis Output*

* Removed obsolete `queue` option
* Removed obsolete `name` option

*Sqs Output*

* Removed obsolete `batch` option
* Removed obsolete `batch_timeout` option

*Tcp Output*

* Removed obsolete `message_format` option

// end::notable-breaking-changes[]

[[breaking-pq]]
=== Breaking change across PQ versions prior to Logstash 6.3.0

If you are upgrading from Logstash 6.2.x or any earlier version (including 5.x)
and have the persistent queue enabled, we strongly recommend that you drain or
delete the persistent queue before you upgrade. See <<upgrading-logstash-pqs>>
for information and instructions.

[[breaking-6.0]]
=== Breaking changes in 6.0

Here are the breaking changes for 6.0. 

[float]
==== Changes in Logstash Core

These changes can affect any instance of Logstash that uses impacted features.
Changes to Logstash Core are plugin agnostic.

[float]
===== Application Settings

* The setting `config.reload.interval` has been changed to use time value strings such as `5m`, `10s` etc.
  Previously, users had to convert this to a millisecond time value themselves.
  Note that the unit qualifier (`s`) is required.

[float]
===== RPM/Deb package changes

* For `rpm` and `deb` release artifacts, config files that match the `*.conf` glob pattern must be in the conf.d folder,
  or the files will not be loaded.

[float]
===== Command Line Interface behavior

* The `-e` and `-f` CLI options are now mutually exclusive. This also applies to the corresponding long form options `config.string` and
  `path.config`. This means any configurations  provided via `-e` will no longer be appended to the configurations provided via `-f`.
* Configurations provided with `-f` or `config.path` will not be appended with `stdin` input and `stdout` output automatically.

[float]
===== List of plugins bundled with Logstash

The following plugins were removed from the default bundle based on usage data. You can still install these plugins manually:

* logstash-codec-oldlogstashjson
* logstash-input-couchdb_changes
* logstash-input-irc
* logstash-input-log4j
* logstash-input-lumberjack
* logstash-filter-uuid
* logstash-output-xmpp
* logstash-output-irc
* logstash-output-statsd
