In a 14.x and above Chef Infra Server, it would be great if erchef checked to see whether the /chef index has a read-only block set on it and warned about that in one of the opscode-erchef logfiles, or in the https://FQDN/_status output, something like that.
This way, customers could find out about the issue without requiring a run of chef-server-ctl gather-logs, sending the gather-logs bundle, Support parsing the logfile, and then informing the customer about whatever happened.
I tried the following config and process. I set TRACE level logging on only messages about cluster blocks, removed the data from the index, then set a read-only flag, then tried a reindex to demonstrate the failure, but this must not be the way to set extensive logging on this part of the system. In the end, it was not obvious that the system has the read_only_allow_delete flag set when the disk used by Elasticsearch is not also full
#check the log first. Just restarted, nothing going on.
2021-07-23_03:20:34.60776 [2021-07-22T23:20:34,607][INFO ][o.e.p.PluginsService ] [chef-14.test.lxc] no plugins loaded
2021-07-23_03:20:36.55768 [2021-07-22T23:20:36,557][INFO ][o.e.d.DiscoveryModule ] [chef-14.test.lxc] using discovery type [zen] and host providers [settings]
2021-07-23_03:20:36.91038 [2021-07-22T23:20:36,910][INFO ][o.e.n.Node ] [chef-14.test.lxc] initialized
2021-07-23_03:20:36.91053 [2021-07-22T23:20:36,910][INFO ][o.e.n.Node ] [chef-14.test.lxc] starting ...
2021-07-23_03:20:37.02923 [2021-07-22T23:20:37,024][INFO ][o.e.t.TransportService ] [chef-14.test.lxc] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
2021-07-23_03:20:37.04199 [2021-07-22T23:20:37,041][WARN ][o.e.b.BootstrapChecks ] [chef-14.test.lxc] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
2021-07-23_03:20:40.08664 [2021-07-22T23:20:40,086][INFO ][o.e.c.s.MasterService ] [chef-14.test.lxc] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {chef-14.test.lxc}{CVAAQJfPRCyHNLxSAqNuTw}{0oXMDd33Tr6hhcHb-pR2NQ}{127.0.0.1}{127.0.0.1:9300}
2021-07-23_03:20:40.08991 [2021-07-22T23:20:40,089][INFO ][o.e.c.s.ClusterApplierService] [chef-14.test.lxc] new_master {chef-14.test.lxc}{CVAAQJfPRCyHNLxSAqNuTw}{0oXMDd33Tr6hhcHb-pR2NQ}{127.0.0.1}{127.0.0.1:9300}, reason: apply cluster state (from master [master {chef-14.test.lxc}{CVAAQJfPRCyHNLxSAqNuTw}{0oXMDd33Tr6hhcHb-pR2NQ}{127.0.0.1}{127.0.0.1:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
2021-07-23_03:20:40.10070 [2021-07-22T23:20:40,100][INFO ][o.e.h.n.Netty4HttpServerTransport] [chef-14.test.lxc] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
2021-07-23_03:20:40.10072 [2021-07-22T23:20:40,100][INFO ][o.e.n.Node ] [chef-14.test.lxc] started
2021-07-23_03:20:40.17271 [2021-07-22T23:20:40,172][WARN ][o.e.d.i.m.AllFieldMapper ] [chef-14.test.lxc] [_all] is deprecated in 6.0+ and will be removed in 7.0. As a replacement, you can use [copy_to] on mapping fields to create your own catch all field.
2021-07-23_03:20:40.23951 [2021-07-22T23:20:40,239][INFO ][o.e.g.GatewayService ] [chef-14.test.lxc] recovered [1] indices into cluster_state
2021-07-23_03:20:40.25682 [2021-07-22T23:20:40,256][WARN ][o.e.d.i.m.AllFieldMapper ] [chef-14.test.lxc] [_all] is deprecated in 6.0+ and will be removed in 7.0. As a replacement, you can use [copy_to] on mapping fields to create your own catch all field.
2021-07-23_03:20:40.57460 [2021-07-22T23:20:40,574][INFO ][o.e.c.r.a.AllocationService] [chef-14.test.lxc] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[chef][1]] ...]).
2021-07-23_03:22:14.39501 [2021-07-22T23:22:14,394][WARN ][o.e.d.i.m.AllFieldMapper ] [chef-14.test.lxc] [_all] is deprecated in 6.0+ and will be removed in 7.0. As a replacement, you can use [copy_to] on mapping fields to create your own catch all field.
#Set the TRACE logging hopefully for the cluster block
curl -X PUT "localhost:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'
{
"transient": {
"logger.org.elasticsearch.cluster.block": "TRACE"
}
}
'
{
"acknowledged" : true,
"persistent" : { },
"transient" : {
"logger" : {
"org" : {
"elasticsearch" : {
"cluster" : {
"block" : "TRACE"
}
}
}
}
}
}
#Notice the docs.count is at 16
curl -sS 'http://localhost:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open chef K4_K9mBPQSaoNHOuvqFT4Q 5 1 16 0 1.1mb 1.1mb
#empty the index, but leave it in place
curl -X POST localhost:9200/chef/_delete_by_query?conflicts=proceed -H 'Content-type: application/json' -d '
{
"query": {
"match_all": {}
}
}'
{"took":81,"timed_out":false,"total":16,"deleted":16,"batches":1,"version_conflicts":0,"noops":0,"retries":{"bulk":0,"search":0},"throttled_millis":0,"requests_per_second":-1.0,"throttled_until_millis":0,"failures":[]}
#check the index again. It is now empty
curl -sS 'http://localhost:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open chef K4_K9mBPQSaoNHOuvqFT4Q 5 1 0 0 1.1mb 1.1mb
#Set the read_only_allow_delete flag
curl -sS -X PUT http://localhost:9200/chef/_settings -H 'Content-Type: application/json' -d '{
"index.blocks.read_only_allow_delete": true
}'
{"acknowledged":true}
# Now, try to do a reindex like this
chef-server-ctl reindex delivery -d -t
Reindexing orgs: ["delivery"]
- Disabling the Chef API.
- Reindexing.
Removing all index entries for organization 'delivery'...
Sending all data for organization 'delivery' to be indexed again. It may take some time before everything is available via search.
- Re-enabling the Chef API
0.379376896 seconds to reindex.
# Then check the index docs.count. Nothing. The read-only flag stopped the writes required by the reindex
# curl -sS 'http://localhost:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open chef K4_K9mBPQSaoNHOuvqFT4Q 5 1 0 0 1.1mb 1.1mb
# No useful info in the /var/log/opscode/elasticsearch/current logfile output below. No complaints there. Now, if the disk used by Elasticsearch was full, the logfile would have at least one message about the /chef index being read-only in it like this: 2021-07-23_02:44:33.29728 org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
2021-07-23_03:20:40.10072 [2021-07-22T23:20:40,100][INFO ][o.e.n.Node ] [chef-14.test.lxc] started
2021-07-23_03:20:40.17271 [2021-07-22T23:20:40,172][WARN ][o.e.d.i.m.AllFieldMapper ] [chef-14.test.lxc] [_all] is deprecated in 6.0+ and will be removed in 7.0. As a replacement, you can use [copy_to] on mapping fields to create your own catch all field.
2021-07-23_03:20:40.23951 [2021-07-22T23:20:40,239][INFO ][o.e.g.GatewayService ] [chef-14.test.lxc] recovered [1] indices into cluster_state
2021-07-23_03:20:40.25682 [2021-07-22T23:20:40,256][WARN ][o.e.d.i.m.AllFieldMapper ] [chef-14.test.lxc] [_all] is deprecated in 6.0+ and will be removed in 7.0. As a replacement, you can use [copy_to] on mapping fields to create your own catch all field.
2021-07-23_03:20:40.57460 [2021-07-22T23:20:40,574][INFO ][o.e.c.r.a.AllocationService] [chef-14.test.lxc] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[chef][1]] ...]).
2021-07-23_03:22:14.39501 [2021-07-22T23:22:14,394][WARN ][o.e.d.i.m.AllFieldMapper ] [chef-14.test.lxc] [_all] is deprecated in 6.0+ and will be removed in 7.0. As a replacement, you can use [copy_to] on mapping fields to create your own catch all field.
2021-07-23_03:30:57.42751 [2021-07-22T23:30:57,427][WARN ][o.e.d.i.m.AllFieldMapper ] [chef-14.test.lxc] [_all] is deprecated in 6.0+ and will be removed in 7.0. As a replacement, you can use [copy_to] on mapping fields to create your own catch all field.
2021-07-23_03:33:02.72124 [2021-07-22T23:33:02,720][WARN ][o.e.d.c.ParseField ] [chef-14.test.lxc] Deprecated field [lowercase_expanded_terms] used, replaced by [Decision is now made by the analyzer]
2021-07-23_03:34:07.94356 [2021-07-22T23:34:07,943][WARN ][o.e.d.i.m.AllFieldMapper ] [chef-14.test.lxc] [_all] is deprecated in 6.0+ and will be removed in 7.0. As a replacement, you can use [copy_to] on mapping fields to create your own catch all field.
2021-07-23_03:34:27.00541 [2021-07-22T23:34:27,005][WARN ][o.e.d.c.ParseField ] [chef-14.test.lxc] Deprecated field [lowercase_expanded_terms] used, replaced by [Decision is now made by the analyzer]
