manual for 2024.04 release

main
John DiCarlo 2 years ago
parent 813bd88624
commit 919d174c6f

Binary file not shown.

@ -1,58 +0,0 @@
## Common Issues
### Backends Not Up
The backends need to be up for search to run. When the backends are up, running `npusearch_check` will look something like this:
```bash
lrl_admin@guava:~$ npusearch_check
[
"npusearch:request:guava-0-1710520177",
"npusearch:request:guava-1-1710520177",
"npusearch:request:guava-10-1710520177",
"npusearch:request:guava-11-1710520177",
"npusearch:request:guava-12-1710520177",
"npusearch:request:guava-13-1710520177",
"npusearch:request:guava-14-1710520177",
"npusearch:request:guava-15-1710520177",
"npusearch:request:guava-2-1710520177",
"npusearch:request:guava-3-1710520177",
"npusearch:request:guava-4-1710520177",
"npusearch:request:guava-5-1710520177",
"npusearch:request:guava-6-1710520177",
"npusearch:request:guava-7-1710520177",
"npusearch:request:guava-8-1710520177",
"npusearch:request:guava-9-1710520177"
]
```
When the backend are down, you will see this:
```bash
lrl_admin@guava:~$ npusearch_check
[]
```
Here are some steps to try to bring the backends up when they are down:
#### 1. Restart using systemctl
Run `sudo systemctl restart npusearch.service`. Wait about 15 seconds, then try `npusearch_check` again.
#### 2. Check status using systemctl
Run `sudo systemctl status npusearch.service`.
- If the last thing it prints is that it's satisfying a license, it got stuck during the startup process. If your device has SmartSSDs, try `sudo systemctl restart mpd.service`.
- If it says "Failed to start NPUSearch search backends." and your device has SmartSSDs, try `sudo systemctl restart mpd.service`. If your device has Kuona cards, try `sudo insmod npusearch`. If that errors, try `sudo dpkg-reconfigure npusearch`.
Repeat step 1.
#### 3. Check log messages
In `/opt/lrl/etc/npusearch.conf` the line `export LOGFILE=path/to/logfile` will be where NPUSearch is writing logs. If the line is commented out, un-comment it and set a path for a log file to be written to. Run steps 1 and 2 again and then read the logs to see where the issues may be.
### Search performance is lower than expected
This is commonly caused by the SSDs overheating and throttling.
- Double check to make sure the fan speed is turned up to minimum 100% on iDRAC. Use iDRAC to check that the inlet air temperature into the server is not too hot. If anything is changed at this step, run the tests again.
- Inspect the `ssd_nvme_smart_log_data.ndjson` file to see how hot the SSDs are getting. Each line of that file is a `nvme smart-log` output for each SSD at a given timestamp. The thermal test will fail if any SSD reaches 349 Kelvin, but some SSDs will throttle performance before getting that hot. If desired, send the `ssd_nvme_smart_log_data.ndjson` and `npusearch_install.log` files to [support@lewis-rhodes.com](support@lewis-rhodes.com). LRL can do detailed analysis to help determine if throttling is happening.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.

Binary file not shown.
Loading…
Cancel
Save