Unraid 6/Troubleshooting

From unRAID
Revision as of 10:09, 14 January 2020 by Itimpi (talk | contribs)
Jump to: navigation, search

Official Documentation Contents List

Troubleshooting

THIS SECTION IS STILL UNDER CONSTRUCTION

A lot more detail still needs to be added

Most of the time Unraid systems function with minimal problems. This section is intended to help with resolving issues that are most commonly encountered.

There are some important general guidelines that it is recommended that a user follows that will help with any troubleshooting that may be required:

  • Use the built-in Help: The Unraid GUI has extensive built in Help for most fields in the GUI. This can be accessed at the individual field level by clicking on the field name, or toggled on/off for the whole page by clicking on the Help icon at the top right of the GUI.
  • Enable Notifications: Unraid has a notification system that can be used to keep you informed about the health of your Unraid system. This can be enabled and the level of notifications you receive tuned under Settings->User Preferences->Notification Settings. Since Unraid systems often function for very long times without needing any user oversight it can be important that you are informed problems when they first occur as if left unresolved they can grow into more serious ones.
  • Proactively fix any reported issues:
  • Ask for help in the forums: Unraid has a vibrant user community and many knowledgeable users who are active in the Unraid forums. Any time you encounter a problem and you are not sure how to proceed it is a good idea to ask questions in the forums. There is nothing worse than rushing into trying to fix a problem using a process you do not understand and as a result making a problem that was initially easy to resolve into something more serious
  • Capture diagnostics: If you want to ask a question in the forums about a problem you are encountering you are frequently going to be asked to provide your system diagnostics file. You need to do so, BEFORE YOU REBOOT so that the logs show went wrong BEFORE the reboot (because once you reboot, it's lost)!

Capturing Diagnostic Information

When you encounter any sort of problem it is always recommended that you attempt to capture as much information as possible to help with pin-pointing the cause. If you want to ask questions in the forum such information will typically be requested as it will speed up the process of getting meaningful and accurate feedback.

System Diagnostics

Unraid has a GUI option under Tools->Diagnostics to capture a lot of information about the state of your system that can be helpful when trying to diagnose any issues. Using this tool will result in a zip file being produced that can be downloaded and then attached to forum posts. If the GUI cannot be accessed for any reason then using the diagnostics command from the Linux level will generate the same information and put the resulting zip file into the logs folder on the flash drive. The Diagnostics should if at all possible be captured BEFORE you reboot so that the lops show what happened leading up to the problem occurring. The zip file produced can then be attached to a forum post when asking for help on a problem in the Unraid forums.

Diagnostics

These system diagnostics include configuration information, state information and key system logs. When creating the diagnostics from the GUI then details of the sort of information that will be included is listed. There is an option (set by default) to say that the diagnostics should be anonymized to try and avoid including any information that might be deemed sensitive. All the files in the diagnostics are text files so a user is free to examine them to check for themselves exactly what information is present.

Note on anonymization of the diagnostics
It has been pointed out that the diagnostics are not completely anonymized if you have enabled mover logging under Settings->Mover Settings as the syslog will give details of files that mover is operating on. This is a bit of a catch-22 scenario as when one has enabled mover logging it is normally to investigate a problem where as much detail as terrible is captured so attempting to anonymize such information may well be counter-productive. Since mover logging is disabled by default and recommended practice is to only have it enabled when investigating why mover is not giving the expected results this is probably acceptable?

Persistent Logs

The main system log is the syslog file and it is the contents of this file that is displayed when you click the Log icon at the top right of the Unraid GUI. Note that when posting to the forums extracted fragments of the syslog are rarely helpful as they do not show what lead up to a problem occurring.

Normally the logs are only written to RAM so do not survive the system being rebooted. If you are investigating a system crash then as long as you are running Unraid 6.7.2 or later you can go to Settings->Network Services->Syslog Server and enable the server

Syslog server
  • Tick the Mirror to Flash option to get a syslog that survives a crash written to the flash drive and is appended to on a reboot. You do not want to leave the Mirror to Flash option ticked if you are not investigating a problem where you cannot get the normal system diagnostics as this can cause a lot of additional writes to the flash drive that can shorten it's lifetime.
  • You can also enter the name or IP address of your Unraid server into the Remote syslog server field and enter a local share where such a file can be stored under the Local syslog folder field. In this case a share set to use the cache drive is recommended to avoid spinning up array drives unnecessarily. This is more appropriate if you wand to continue to keep a permanent copy of the syslog but the file will not be as easy to access if the Unraid system is crashing.

Notes:

  • The standard system diagnostics include the RAM copy of the syslog so there is no need to provide this separately. You will need to do so to provide the logs captured by the syslog server as these are not included in the standard system diagnostics.

Docker Containers

The standard system diagnostics do not contain much that will help with diagnosing issues with specific docker containers.

MORE DETAIL NEEDED

VMs

The standard system diagnostics do not contain much that will help with diagnosing issues with specific VMs.

MORE DETAIL NEEDED


Boot Issues

THIS SECTION IS STILL UNDER CONSTRUCTION'

Preparing the flash drive


Boot Process

Most of the time the Unraid boot process runs seamlessly and the user needs no awareness of the various stages involved. However when things go wrong it can be useful to know how far the boot process managed to get as this will be of use in knowing what remedial action to take.

Resolving boot issues will typically need either a locally attached monitor+keyboard or (if the motherboard supports it) an IPMI connection to carry out equivalent functionality. This can then be used to set any required BIOS stings and to monitor the booting process.

The boot process for Unraid proceeds through a number of stages

  1. Bios boot: This is the stage at which the motherboard BIOS recognizes the presence of the Unraid bootable flash drive
    • The way that the Unraid flash drive is set as the default boot device is BIOS dependent so you may need to consult your motherboard's User Manual to determine the correct way to do this.
    • The Unraid flash drive supports booting in Legacy mode (also sometimes known as CSM mode) for older BiOS's and UEFI for more recent ones. Many recent BIOS's support both modes.
    • If you want UEFI boot mode to be used then the EFI folder on the flash drive must not have trailing tilde (~) character.
  2. Syslinux loader:
    Boot Menu
    • The entries that appear on the boot menu are specified by the syslinux/syslinux.cfg file on the flash drive. Although in theory this file can be edited manually as it a text file it is recommended that it is done via the GUI by clicking on the flash drive on the Main tab and going to the Syslinux configuration section
    • The Memtest86+ option only works if booting in Legacy mode. If booting in UEFI mode it will typically simply cause a reboot.
    • If the user does not select a specific option then after a timeout period the default option will be used. If Unraid is running in headless mode this is the option that will be run.
  3. Linux core: This is the stage at which the syslinux boot loader takes over from the BIOS and starts loading the files specified in the syslinux.cfg file.
    • This is when the core Linux system is loaded from the flash drive and unpacked into RAM.
    • There will be messages on the console about the various bz* types being loaded into RAM.
    • If there are any error messages displayed while loading these files then it normally indicates a problem with the flash drive.
    • There will then be messages displayed as Linux start up and detects the hardware environment.
  4. Flash dependent services: At this stage the flash drive is mounted at /boot so that the process can continue
    • If the mount of the flash fails it is still possible to get the login prompt displayed. However this does not necessarily mean the whole boot process completed correctly.
    • If this stage of the boot process has not completed then typical symptoms are that the webGUI and network are not started
    One way to see if this has happened is to login and use the df command. If the flash drive was mounted successfully then you will see it as /boot in the resulting list of mount points.
    The output should have something like following mount points:
    /dev/sdb1 15413232 826976 14586256 6% /boot
    /dev/loop0 9344 9344 0 100% /lib/modules
    /dev/loop1 7424 7424 0 100% /lib/firmware
    • Additional drivers and firmware are now available on the above mount points.
    • Configuration information is read into RAM from the flash drive.
    • Standard Linux services are started. Examples would be networking and (if enabled) WireGuard VPN.
  5. Plugins:
    • If the user has installed plugins then they are normally loaded at this stage.
    • If one of the Safe Boot options was selected from the Unraid Boot menu then the loading of plugins is suppressed.
  6. Web GUI:
    • The Unraid web GUI is started.
    • The webGUI is actually done via an entry in the config/go file on the flash drive so it is possible for user supplied commands to also be run from there either before starting the webGUI or just after doing so.
  7. Array
    If the user has set the array to be auto-mounted then the following will start. If array auto-start is not set then they happen when the user elects to start the array.
    • Drives mounted
    Mount points will now be created of the form /dev/diskX and /mnt/cache (if you have a cache).
    • File Share Services
    Shares will now become available on the network.
    At the Linux level the shares will now appear as paths like /mnt/user/sharename
    • Docker Containers
    If the user has enabled the docker services then the Docker containers will be started using the order on the Docker tab.
    The order of the containers and delays between starting the containers can be set on the Docker tab.
    • VMs
    Any VMs the user has set to auto-start will now be started

By this stage the Unraid server will be fully operational.


Shutdown Issues

THIS SECTION IS STILL UNDER CONSTRUCTION


Crash Issues

THIS SECTION IS STILL UNDER CONSTRUCTION


Data Recovery

THIS SECTION IS STILL UNDER CONSTRUCTION

A lot more detail still needs to be added

This section is about recovering your data when Unraid reports problems with one or more drives.

There are some important points to bear in mind about securing your data(

  • Backup critical data: Unraid will protect you against most types of simple hardware failure, but not catastrophic failure. You should ALWAYS have backups of any critical data that you cannot afford to lose. Ideally one of those copies should be offsite or on the cloud to protect yourself against unforeseen issues such as fire, theft, flood, etc.
    • Each user has to make their own determination of what they deem critical and make an assessment of the level of risk they are prepared to take.
    • Personal data such as photographs & documents tend to always be deemed critical. Luckily these tend to be relatively small so easy to back up.
    • Media files are often deemed non-critical and are relatively large so a user may well decide these do not merit being backed up
    • Personal video that can never be replaced should fall into the critical category regardless of it's size
    • Remember that there are things such as ransomware around so there should be at least one copy of critical files that cannot be accessed online and corrupted if you are unfortunate enough to suffer from such an attack!
  • Be pro-active about resolving any issues that are detected by Unraid. Make sure that notifications are enabled under Settings->Notifications so that you get told as soon as issues are detected. For many users Unraid operates in a fire-and-forget mode so that they will not be actively checking for problems so need such reminders.
  • Ask for Advice:
    • The Unraid forums have lots of knowledgeable users who can help guide you through what needs doing to get your data back into a standard if you are not sure what are the best steps to take.
    • Unraid is very good about protecting your data against typical hardware failures , but it is not immune against users taking inappropriate steps to recover their data after a failure occurs.

Lost Array Configuration

If you have lost the array configuration and do not have a current backup of the flash drive the data will still be intact on the drives.

All configuration information is stored on the flash drive in the config folder. In particular the Unraid array configuration is stored in the config/super.dat file.

  • Do not attempt to use an out-of-date backup that may have incorrect drive assignments.

If you know which drives were the parity drives then you can simply reassign the drives. However if you do not know which were the parity drives then you have to be more careful as incorrectly assigning a drive to parity that is really a data drive will result in you losing its contents.

When you do not know which were your parity drives the following steps can get your array back into operation:

  1. Assign all drives as data drives
  2. Start the array
  3. All the genuine data drives should show as mounted and the parity drive(s) show as unmountable. If this is not the case and too many drives show as unmountable then stop and ask for help in the forums giving details of what happened.
  4. Make a note of the serial numbers of the parity drives.
  5. Stop the array
  6. Go to Tools >>> New Config. Select the option to retain current assignments (as it reduces the chance of error). Click the yes I want to do this and then Apply.
  7. Go back to the Main tab and correct the assignments of the parity drives. Double check you have the right drives now assigned as parity drives as assigning a data drive to parity will lose its contents. You can move any other drives around at this stage as well.
  8. Start the array and the system will start building parity based on the current assignments.

All your User Shares will re-appear (as they are simply the aggregation of the top level folders on each drive) but with default settings so you may need to re-apply any customisation you want.

You can now go any other customisation that is appropriate and add any plugins you normally use.

At this point it is strongly recommended that you click on the flash drive on the Main tab and selecting the option to download a backup of the flash drive. It is always good practice to do this any time you make a significant change.