Known issues - Splunk Documentation (2023)

Workaround:
Permissions are now properly applied to the following REST endpoints:

•/services/server/introspection/search/dispatch•/services/server/introspection/search/distributed

Splunk software now displays errors when users with insufficient privileges access these endpoints using the 'rest' command or the HTTP API. To resolve these errors, add the list_introspection capability to the authorize.conf file for the role of the user that requires the endpoint information. Adding this capability to your roles ensures that searches and integrations continue to work properly.

Workaround:
Set limits.conf back to default, by removing any override of max_searches_per_process.

For example:

[search]max_searches_per_process=1

to

[search]

Workaround:
Create a $SPLUNK_HOME/etc/system/local/user-seed.conf and restart Splunk

[user_info]
PASSWORD = <yourpassword>

Workaround:
Through the command line, you can open notepad and modify the password file to regain 'Admin' status.

Workaround:
1. Do not upgrade past Splunk 8.0.5 on Solaris 11.3

OR2. Upgrade to Solaris 11.4

Workaround:
Apps may experience this issue if they: implement a custom search command using the Splunk Enterprise SDK for Python between versions 1.6.5 and 1.6.13; are executed by Splunk Enterprise or Splunk Cloud using Python 3; and are sent events with multi-byte characters.

App developers whose apps implement a custom search command using a version of the Splunk Enterprise SDK for Python must update to version 1.6.14 or higher and release new versions of their apps.

Splunk Enterprise and Splunk Cloud administrators who are using apps impacted by this issue must update to app versions that use the Splunk Enterprise SDK for Python version 1.6.14 or higher. If this is not possible, administrators are encouraged to either: allow these apps to be executed using Python 2; or cease usage of impacted apps until updated versions are available.

Workaround:
Update dpkg to version 1.17.6 or later.

Workaround:
Create a $SPLUNK_HOME/etc/system/local/user-seed.conf and restart Splunk

[user_info]
PASSWORD = <yourpassword>

Workaround:
If security is not a significant concern, simply revert back to the 6.5.x SSL/TLS defaults, e.g. for e-mail, add to $SPLUNK_HOME/etc/system/local/alert_actions.conf

[email]
sslVersions = *,-ssl2
cipherSuite = TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH

To configure LDAP with the same settings used by e-mail alerts:$SPLUNK_HOME/etc/openldap/ldap.conf

TLS_PROTOCOL_MIN 3.1
TLS_CIPHER_SUITE TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH

To completely revert the LDAP configuration to the 6.5.x SSL/TLS defaults, comment out TLS_PROTOCOL_MIN and TLS_CIPHER_SUITE

The example below is for a Postfix SMTP server:

 Protocol: TLSv1 Cipher: DHE-RSA-AES256-SHA Verify return code: 19 (self signed certificate in certificate chain)

2. Check the OpenSSL output for Protocol and Cipher. In the example above, Protocol = TLSv1 and Cipher = DHE-RSA-AES256-SHA

$SPLUNK_HOME/etc/system/local/alert_actions.conf

Workaround:
Add

allow_caching=f to the lookup command:

| lookup <name> allow_caching=f ... 

On 7.3+:Add allow_caching=f to the lookup definition on the search head

transforms.conf:[<lookup name>]allow_caching = f

To check if you might be running into this issue, you'll need to enable debug on the search in question by adding:

| noop log_DEBUG=CachedProvider<pre>If you have hits for the cached lookup, like in the sample log below, you can hit this issue.<pre>DEBUG CachedProvider - Cached provider metrics: lookup=<lookup name> hits=67064 misses=321 total=67385

Workaround:
In Splunk Enterprise 8.1.7, 8.2.4, and higher change the job_default_auto_cancel setting in $SPLUNK_HOME/etc/system/local/web.conf from the default value of 30 to 62.

Details
This issue is caused by power saving settings in recent browser versions, where Javascript timers may be throttled. The user typically sees the following message in the search window on foreground searches:

DAG Execution Exception: Search has been cancelled
Search auto-canceled
The search job has failed due to an error. You may be able to view the job in the Job Inspector

Workaround:
none

Workaround:
This is limited to standalone Splunk instances or when returning data from the SH itself in a distributed setup

set allow_batch_mode to 1 and batch_search_max_pipeline to 1 (defaults)

etc/system/local/limits.conf[search]allow_batch_mode = 1batch_search_max_pipeline = 1

or disable batch mode with allow_batch_mode to 0

Workaround:
Add

allow_caching=f to the lookup command:

| lookup <name> allow_caching=f ... 

On 7.3+:Add allow_caching=f to the lookup definition on the search head

transforms.conf:[<lookup name>]allow_caching = f

To check if you might be running into this issue, you'll need to enable debug on the search in question by adding:

| noop log_DEBUG=CachedProvider<pre>If you have hits for the cached lookup, like in the sample log below, you can hit this issue.<pre>DEBUG CachedProvider - Cached provider metrics: lookup=<lookup name> hits=67064 misses=321 total=67385

Workaround:
Ensure that the admin role is not configured as "Unset" and is explicitly configured to either no restriction or a restriction in the UI (Navigate to Edit Role > Resources > Role search time window limit), or through conf file authorize.conf under attribute name srchTimeEarliest.

Workaround:
Force a specific time format by using strftime in an eval command.

for example, add

 | convert timeformat="%FT%T.%3Q%z" ctime(_time)

to the end of your search

Workaround:
Set limits.conf back to default, by removing any override of max_searches_per_process.

For example:

[search]max_searches_per_process=1

to

[search]

Workaround:
If you encounter this problem, change the enable_splunkd_kv_lookup_indexing parameter to true in the [lookup] stanza of limits.conf in the $SPLUNK_HOME/etc/system/local directory on your search peers.

Workaround:
Apps may experience this issue if they: implement a custom search command using the Splunk Enterprise SDK for Python between versions 1.6.5 and 1.6.13; are executed by Splunk Enterprise or Splunk Cloud using Python 3; and are sent events with multi-byte characters.

App developers whose apps implement a custom search command using a version of the Splunk Enterprise SDK for Python must update to version 1.6.14 or higher and release new versions of their apps.

Splunk Enterprise and Splunk Cloud administrators who are using apps impacted by this issue must update to app versions that use the Splunk Enterprise SDK for Python version 1.6.14 or higher. If this is not possible, administrators are encouraged to either: allow these apps to be executed using Python 2; or cease usage of impacted apps until updated versions are available.

Workaround:
Dedup values in search before, for example:

instead of

index="field_test" [search index="field_test" globalCallID_callId=1234* | fields globalCallID_callId]

add a stats or dedup in the subsearch:

index="field_test" [search index="field_test" globalCallID_callId=123* | stats values(globalCallID_callId) AS globalCallID_callId | mvexpand globalCallID_callId ]

If that list is still large and you're seeing the slowdown, consider moving the filtering to a | where after the initial search, for example:

index="field_test" globalCallID_callId=* | where [search index="field_test" globalCallID_callId=123* | stats values(globalCallID_callId) AS globalCallID_callId | mvexpand globalCallID_callId ]

Workaround:
- Increase globallimit to the value of "unique values" number mentioned in the warning message:

"The split by field <field> has a large number of unique values <number>. Chart column set will be trimmed to 10. Use globallimit argument to control column count."

- Use very high globallimit in geostats and post process after if needed

- Don't use BY in geostats

- Use lower cardinality BY and/or higher globallimit in geostats

Workaround:
See Troubleshoot referenced real-time searches for workaround details.

Workaround:
Use another browser such as Chrome or Firefox

Workaround:
Two workaround options:

1. Re-upload the image. (Best for non-admins or Splunk Cloud customers)2. Copy/move collections (splunk-dashboard-icons and splunk-dashboard-images) from splunk-dashboard-app to splunk-dashboard-studio to see the custom images/icons in studio dashboards.

Workaround:
Press the download button again

Workaround:
Do not use unicode characters in Dashboards with dark mode.

Workaround:
Use <option name="charting.legend.masterLegend">null</option>

Workaround:
See Troubleshoot referenced real-time searches for workaround details.

Workaround:
Increase server.conf[clustering]streaming_replication_wait_secs to ensure all streaming targets will be completed. streaming_replication_wait_secs=1800 (a maximum wait of 30min) is not unreasonable, but the setting will affect all types of CM-initiated restarts, so make sure not to set it too high.

Workaround:
when only one indexer in a cluster indexed data (has all the searchable copies), rebalance once before adding the new indexer, and then rebalance a second time

Workaround:
Customers can verify whether their environment is affected with following SPL against their SHs:

index=_internal host IN (<CommaSeparatedSHList>) source=*web_service.log* "Splunk appserver version=UNKNOWN_VERSION build=000"

Refreshing the browser tab will temporarily resolve the issue. No root cause/fix has been identified yet.

Workaround:
none

Workaround:
There is no workaround. After upgrading to Splunk Enterprise 8.x, the splunkd process checks and modifies the local/distsearch.conf on each service start. The process will:

• Remove any settings that define default values already set in the /default/distsearch.conf file.
• Removes comments preceded by a hash.
• Reorders the KV pairs alphanumerically within a stanza.
• Reorders stanzas within the file.

Workaround:
As a workaround, disable useAck in outputs.conf on the forwarder. After disabling, indexers start to ingest data.

If customers do need useACK to prevent data loss, disabling autoBatch in outputs.conf can remediate the issue too, but it impacts throughput - no worse than 8.x, but no improvement for 9.0.

Workaround:
whenever changing UF user, pls manually chown SPLUNK_HOME to the new user, including first time install/upgrade, or manually enable boot-start

Workaround:
UF for CentOS7 ARM 64 will be available in the 9.0.1 maintenance release.

Workaround:
1. Do not upgrade past Splunk 8.0.5 on Solaris 11.3

OR2. Upgrade to Solaris 11.4

Workaround:
As a workaround can the customer manually modify the distsearch.conf.

Workaround:
In Splunk Enterprise 8.1.7, 8.2.4, and higher change the job_default_auto_cancel setting in $SPLUNK_HOME/etc/system/local/web.conf from the default value of 30 to 62.

Details
This issue is caused by power saving settings in recent browser versions, where Javascript timers may be throttled. The user typically sees the following message in the search window on foreground searches:

DAG Execution Exception: Search has been cancelled
Search auto-canceled
The search job has failed due to an error. You may be able to view the job in the Job Inspector

Workaround:
Following steps should be followed:

- to reconfigure subscription type to RenderedText:

wecutil ss <subscription-name> /cf:RenderedText

- in order to work around a MS defect on the WindowsEventViewer causing field description resolution failures within the WindowsEventViewer, when configuring RenderedText contentFormat you might want to also change the subscription locale, if not already done, to en-US:

wecutil ss <subscription-name> /l:en-US

and the same also for the datetime format on the WEC server to English (United States), see also here:

https://serverfault.com/questions/606144/win2012r2-eventlog-subscription-dont-display-informationshttps://social.technet.microsoft.com/Forums/ie/en-US/3fd3d1fc-1194-4899-978c-3283085648bc/eventlog-forwarding-issues-either-the-component-that-raises-this-event-is-not-installed-on-your

- please make sure to install the most recent Windows add-on compatible with your Splunk release, following the official installation documentation:

https://docs.splunk.com/Documentation/AddOns/released/Windows/Install

- please configure inputs.conf on the splunk instance running on the WEC server as follows, in order to onboard the ForwardedEvents data in XML format:

[WinEventLog://ForwardedEvents]renderXml = true

then save and restart splunk in order to apply the changes.

- last, but not least, unless renderXml was set to true already before installing/upgrading to a regressed version, you will need to rewrite your searches and reports in order to comply with the new/XML-specific field extractions shipped in the Windows add-on, since the data is now onboarded in XML format.

Workaround:
If you are using addcoltotals to generate a totals data row, renaming the _time field can cause PDF generation issues.

Remove the label and labelfield or change the label to a number to generate the PDF as expected.

Workaround:
There is no workaround. After upgrading to Splunk Enterprise 8.x, the splunkd process checks and modifies the local/distsearch.conf on each service start. The process will:

• Remove any settings that define default values already set in the /default/distsearch.conf file.
• Removes comments preceded by a hash.
• Reorders the KV pairs alphanumerically within a stanza.
• Reorders stanzas within the file.

Workaround:
A timeout or slow performance of the license management page is caused by a build-up of historical license warning messages, which are processed every time the page is accessed. Can be verified by running this search on the License Manager:

| rest splunk_server=local /services/licenser/messages

If a high value is returned for that end point, you are likely affected. Log a support ticket with Splunk to obtain a license reset key, and apply the key to clear out any historical license warning messages. After the reset license is applied, the license management pages should load normally.

Workaround:
Navigate to a visible app, such as the search and reporting app, and access the Splunk settings pages from that app context.

Workaround:
Use REST API to create the federated saved search instead:

curl -k -u <username>:<password> -X POST https://localhost:8089/servicesNS/nobody/search/data/federated/index -d name=federated:index_kathy -d federated.dataset='savedsearch:ss with space' -d federated.provider=remote_deployment_1
See [[Documentation:*:RESTREF:RESTfederated|Federated search endpoint descriptions]] in the REST API Reference Manual.

Workaround:
Turn off useACK

useACK=false

Workaround:
The problem is caused by non-ASCII/UTF-8 characters, that are present in your configuration and are not supported. You can remediate the problem:

1. Either remove non-ASCII/UTF-8 characters from your configuration files.2. Or take a backup of '$SPLUNK_HOME/lib/python3.7/site-packages/splunk/clilib/cli_common.py' and in line 127: Add parameter to "line.decode" - either "errors='replace'", or "errors='ignore'". Eg:line.decode(errors='replace')

Workaround:
Use Verbose and Smart mode specifically for searches with transforming commands like stats, chart, and timechart, and then review the results in the Statistics tab. To review event counts, run non-transforming searches in Fast mode.

Workaround:
To prevent this issue: When you delete a workload pool, please make sure that you delete any disabled workload rules that are associated with that workload pool.

To resolve the issue if you encounter this: Disable or delete the workload rule that is associated with a workload pool that does not exist anymore.

Workaround:
This issue is caused by a single unsearchable bucket that has been frozen while not existing on remote storage. The bucket copy on the peer node's cache remains stuck in the fixup state, resulting in messages to the effect that all data is not searchable, the replication factor is not met, and the search factor is not met.

To resolve, on the peer node, invoke the "/services/cluster/slave/buckets" endpoint, specifying the faulty bucket, setting "search_state=Searchable" to make the bucket searchable. You do not need to restart the peer node afterwards.

Here is the syntax for the required endpoint:

curl -k -u admin https://<peer_node_with_bucket>:<mgmt_port>/services/cluster/slave/buckets/<bucket_id>/change_bucket -d bucket_mask=0 -d search_state=Searchable -d generation_id=0 -d searchable_sources="peer,site,server_name,host_port_pair,replication_port,replication_use_ssl,searchable,bucket_mask

Note that pairs of angle brackets indicate variables that must correspond to your instance and bucket.

Workaround:
Restart the SHC captain

Workaround:
In the volumes using gcp-sse-kms encryption mode, specify "remote.gs.upload_chunk_size = 0" to disable parallel upload.

Workaround:
In the [kvstore] stanza of your server.conf file, set the storageEngine setting to match the storage engine that you're using, either wiredTiger or mmapv1. To learn which storage engine you're using, check whether the file extensions in the var/lib/splunk/kvstore/mongo directory are *.wt for Wired Tiger or *.ns for Memory Mapped.

Workaround:
Before migration, lower the max_concurrent_uploads setting in server.conf to 2.

After migration, revert the setting to the default of 8.

Workaround:
Create a $SPLUNK_HOME/etc/system/local/user-seed.conf and restart Splunk

[user_info]
PASSWORD = <yourpassword>

Workaround:
Splunk believes this was introduced in a Microsoft Windows patch. The workaround is to configure a delayed start of the Splunk service(s) so that it starts after the Windows Event Log service.

Workaround:
If users cannot perform operations on notable event review statuses or have issues viewing "Edit all selected" links on Incident Review, user roles must be provided with the "edit_roles" capability.

Workaround:
Users can do any of the following:

1. Disable certificate verification - the same root certificate is available with every Splunk download so enabling certificate verification while using the default certificates provides very little additional security.

2. Generate new SHA256/2048-bit key certificates using the new 6.6 root certificate and distribute to older versions of Splunk

3. Generate SHA1/1024-bit key certificates using the old root certificate to use with your new 6.6 instance. For convenience, the old root certificate is included in 6.6 in $SPLUNK_HOME/etc/auth/prev_release/

Workaround:
Use bin/splunk createssl server-cert -d etc/auth/ -n server to generate a new certificate.

Workaround:
Upgrade Hadoop to 2.8.2 or higher.

Workaround:
Reenable the bucket copy saved search and let it run, or force the archiving to happen via | archivebuckets force=1 and then rerun the search.

Workaround:
Set minsplits=maxsplits

Workaround:
Add the journal.gz to the input path's blacklist (vix.input.1.ignore = ....)

Workaround:
To mitigate this issue, make sure that vix.splunk.home.hdfs (or Working directory in the UI) is unique on both search heads that are not in a pool. To keep your instances in the same working directory, configure vix.splunk.search.cache.path to be unique on both search heads.