Bug 4326

Summary: akmods doesn't rebuild nvidia driver on kernel update
Product: Fedora Reporter: MeSat
Component: akmodsAssignee: Richard <hobbes1069>
Status: RESOLVED FIXED    
Severity: major CC: kwizart, leigh123linux
Priority: P1    
Version: 24   
Hardware: x86_64   
OS: GNU/Linux   
namespace:
Attachments: nvidia-bug-report.sh log file from before I rebuilt the module.

Description MeSat 2016-11-03 02:51:58 CET
Created attachment 1675 [details]
nvidia-bug-report.sh log file from before I rebuilt the module.

Second time this has happened in the last few kernel updates.  On reboot, akmods runs but doesn't rebuild the kernel mod properly so install doesn't occur properly.



If I run rpm -qa |grep nvidia, it shows that the latest akmod is installed but there are no kernel modules in the appropriate module directory.  

rpm -qa |grep nvidia
kmod-nvidia-4.7.7-200.fc24.x86_64-367.44-1.fc24.x86_64
xorg-x11-drv-nvidia-367.44-1.fc24.x86_64
kmod-nvidia-4.7.9-200.fc24.x86_64-367.44-1.fc24.x86_64
xorg-x11-drv-nvidia-cuda-367.44-1.fc24.x86_64
xorg-x11-drv-nvidia-kmodsrc-367.44-1.fc24.x86_64
xorg-x11-drv-nvidia-libs-367.44-1.fc24.i686
kmod-nvidia-4.8.4-200.fc24.x86_64-367.44-1.fc24.x86_64
akmod-nvidia-367.44-1.fc24.x86_64
xorg-x11-drv-nvidia-libs-367.44-1.fc24.x86_64





There is no rpm file in the /var/cache/akmods/nvidia either.


ls /var/cache/akmods/nvidia/
358.16-2-for-4.5.5-300.fc24.x86_64.log                      kmod-nvidia-4.5.5-300.fc24.x86_64-367.27-1.fc24.x86_64.rpm
358.16-2-for-4.5.7-300.fc24.x86_64.log                      kmod-nvidia-4.5.7-300.fc24.x86_64-358.16-2.fc24.x86_64.rpm
367.27-1-for-4.5.5-300.fc24.x86_64.log                      kmod-nvidia-4.5.7-300.fc24.x86_64-367.27-1.fc24.x86_64.rpm
367.27-1-for-4.5.7-300.fc24.x86_64.log                      kmod-nvidia-4.5.7-300.fc24.x86_64-367.35-1.fc24.x86_64.rpm
367.27-1-for-4.6.3-300.fc24.x86_64.log                      kmod-nvidia-4.6.3-300.fc24.x86_64-367.27-1.fc24.x86_64.rpm
367.35-1-for-4.5.7-300.fc24.x86_64.log                      kmod-nvidia-4.6.3-300.fc24.x86_64-367.35-1.fc24.x86_64.rpm
367.35-1-for-4.6.3-300.fc24.x86_64.log                      kmod-nvidia-4.6.4-301.fc24.x86_64-367.35-1.fc24.x86_64.rpm
367.35-1-for-4.6.4-301.fc24.x86_64.log                      kmod-nvidia-4.6.5-300.fc24.x86_64-367.35-1.fc24.x86_64.rpm
367.35-1-for-4.6.5-300.fc24.x86_64.log                      kmod-nvidia-4.6.6-300.fc24.x86_64-367.35-1.fc24.x86_64.rpm
367.35-1-for-4.6.6-300.fc24.x86_64.log                      kmod-nvidia-4.6.6-300.fc24.x86_64-367.44-1.fc24.x86_64.rpm
367.35-1-for-4.6.7-300.fc24.x86_64.log                      kmod-nvidia-4.6.7-300.fc24.x86_64-367.35-1.fc24.x86_64.rpm
367.44-1-for-4.6.6-300.fc24.x86_64.log                      kmod-nvidia-4.6.7-300.fc24.x86_64-367.44-1.fc24.x86_64.rpm
367.44-1-for-4.6.7-300.fc24.x86_64.log                      kmod-nvidia-4.7.2-201.fc24.x86_64-367.44-1.fc24.x86_64.rpm
367.44-1-for-4.7.2-201.fc24.x86_64.log                      kmod-nvidia-4.7.3-200.fc24.x86_64-367.44-1.fc24.x86_64.rpm
367.44-1-for-4.7.3-200.fc24.x86_64.log                      kmod-nvidia-4.7.4-200.fc24.x86_64-367.44-1.fc24.x86_64.rpm
367.44-1-for-4.7.4-200.fc24.x86_64.log                      kmod-nvidia-4.7.5-200.fc24.x86_64-367.44-1.fc24.x86_64.rpm
367.44-1-for-4.7.6-200.fc24.x86_64.log                      kmod-nvidia-4.7.6-200.fc24.x86_64-367.44-1.fc24.x86_64.rpm
367.44-1-for-4.7.7-200.fc24.x86_64.log                      kmod-nvidia-4.7.7-200.fc24.x86_64-367.44-1.fc24.x86_64.rpm
367.44-1-for-4.7.9-200.fc24.x86_64.log                      kmod-nvidia-4.7.9-200.fc24.x86_64-367.44-1.fc24.x86_64.rpm
kmod-nvidia-4.5.5-300.fc24.x86_64-358.16-2.fc24.x86_64.rpm  nvidia-kmod-debuginfo-358.16-2.fc24.x86_64.rpm





After removing the appropriate nvidia rpm. 

   rpm -e kmod-nvidia-4.8.4-200.fc24.x86_64 


Run 

  /sbin/akmodsbuild --target x86_64 --kernels 4.8.4-200.fc24.x86_64 /usr/src/akmods/nvidia-kmod.latest 

I get a rpm file that I can then install with dnf.

  kmod-nvidia-4.8.4-200.fc24.x86_64-367.44-1.fc24.x86_64.rpm

sudo dnf install /var/cache/akmods/nvidia/kmod-nvidia-4.8.4-200.fc24.x86_64-367.44-1.fc24.x86_64.rpm 
Last metadata expiration check: 0:39:53 ago on Wed Nov  2 18:39:24 2016.
Dependencies resolved.
========================================================================================================================
 Package                                     Arch             Version                      Repository              Size
========================================================================================================================
Installing:
 kmod-nvidia-4.8.4-200.fc24.x86_64           x86_64           1:367.44-1.fc24              @commandline           5.8 M

Transaction Summary
========================================================================================================================
Install  1 Package

Total size: 5.8 M
Installed size: 17 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Installing  : kmod-nvidia-4.8.4-200.fc24.x86_64-1:367.44-1.fc24.x86_64                                            1/1 
  Verifying   : kmod-nvidia-4.8.4-200.fc24.x86_64-1:367.44-1.fc24.x86_64                                            1/1 

Installed:
  kmod-nvidia-4.8.4-200.fc24.x86_64.x86_64 1:367.44-1.fc24                                                              

Complete!




A reboot and all is well.



This is the same as the last time that this happened at the beginning of October.




When I ran nvidia-bug-report.sh before rebuilding the akmods, I got this.


nvidia-bug-report.sh will now collect information about your
system and create the file 'nvidia-bug-report.log.gz' in the current
directory.  It may take several seconds to run.  In some
cases, it may hang trying to capture data generated dynamically
by the Linux kernel and/or the NVIDIA kernel module.  While
the bug report log file will be incomplete if this happens, it
may still contain enough data to diagnose your problem.

Please include the 'nvidia-bug-report.log.gz' log file when reporting
your bug via the NVIDIA Linux forum (see devtalk.nvidia.com)
or by sending email to 'linux-bugs@nvidia.com'.


Running nvidia-bug-report.sh...ls: cannot access '/proc/driver/nvidia/./gpus/': No such file or directory


If the bug report script hangs after this point consider running with
--safe-mode command line argument.

modprobe: FATAL: Module nvidia not found in directory /lib/modules/4.8.4-200.fc24.x86_64
modprobe: FATAL: Module nvidia not found in directory /lib/modules/4.8.4-200.fc24.x86_64
 complete.


I have attached the nvidia bug report.

Also, according the /var/log/messages, the akmods file is installed when the system was first rebooted after the kernel upgrade which corresponds with the rpm -qa nvidia stated earlier in this bug report.


This is from the dnf.* logs


dnf.log:Nov 02 17:41:57 DEBUG ---> Package kmod-nvidia-4.7.6-200.fc24.x86_64.x86_64 1:367.44-1.fc24 will be erased
dnf.log:Nov 02 17:47:05 DDEBUG Command: dnf -y install --disablerepo=* /tmp/akmods.lNduSWn2/results/kmod-nvidia-4.8.4-200.fc24.x86_64-367.44-1.fc24.x86_64.rpm 
dnf.log:Nov 02 17:47:05 DDEBUG Extra commands: ['/tmp/akmods.lNduSWn2/results/kmod-nvidia-4.8.4-200.fc24.x86_64-367.44-1.fc24.x86_64.rpm']
dnf.log:Nov 02 17:47:05 DEBUG ---> Package kmod-nvidia-4.8.4-200.fc24.x86_64.x86_64 1:367.44-1.fc24 will be installed
dnf.log:Nov 02 19:19:17 DDEBUG Command: dnf install /var/cache/akmods/nvidia/kmod-nvidia-4.8.4-200.fc24.x86_64-367.44-1.fc24.x86_64.rpm 
dnf.log:Nov 02 19:19:17 DDEBUG Extra commands: ['/var/cache/akmods/nvidia/kmod-nvidia-4.8.4-200.fc24.x86_64-367.44-1.fc24.x86_64.rpm']
dnf.log:Nov 02 19:19:18 DEBUG ---> Package kmod-nvidia-4.8.4-200.fc24.x86_64.x86_64 1:367.44-1.fc24 will be installed


dnf.rpm.log:Nov 02 17:45:50 INFO Erased: kmod-nvidia-4.7.6-200.fc24.x86_64-1:367.44-1.fc24.x86_64
dnf.rpm.log:Nov 02 17:47:08 ERROR Non-fatal POSTIN scriptlet failure in rpm package kmod-nvidia-4.8.4-200.fc24.x86_64
dnf.rpm.log:Nov 02 17:47:08 ERROR Non-fatal POSTIN scriptlet failure in rpm package kmod-nvidia-4.8.4-200.fc24.x86_64
dnf.rpm.log:Nov 02 19:19:25 INFO Installed: kmod-nvidia-4.8.4-200.fc24.x86_64-1:367.44-1.fc24.x86_64


19:19:25 was when I re-installed the kmod file.
Comment 1 MeSat 2016-11-03 03:03:51 CET
This is what was in the akmods.log file.

According to the log, the proper modules are built and installed but they are not.

2016/10/26 13:37:07 akmods: Building and installing nvidia-kmod
2016/10/26 13:37:07 akmods: Building RPM using the command '/sbin/akmodsbuild --target x86_64 --kernels 4.7.9-200.fc24.x86_64 /usr/src/akmods/nvidia-kmod.latest'
2016/10/26 13:37:34 akmods: Installing newly built rpms
2016/10/26 13:37:34 akmods: DNF detected
2016/10/26 13:37:39 akmods: Successful.
2016/10/26 16:56:15 akmods: Checking kmods exist for 4.7.6-200.fc24.x86_64
2016/10/26 16:56:15 akmods: Checking kmods exist for 4.7.7-200.fc24.x86_64
2016/10/26 16:56:15 akmods: Checking kmods exist for 4.7.9-200.fc24.x86_64
2016/10/26 16:57:40 akmods: Checking kmods exist for 4.7.9-200.fc24.x86_64
2016/10/26 20:06:27 akmods: Checking kmods exist for 4.7.6-200.fc24.x86_64
2016/10/26 20:06:27 akmods: Checking kmods exist for 4.7.7-200.fc24.x86_64
2016/10/26 20:06:27 akmods: Checking kmods exist for 4.7.9-200.fc24.x86_64
2016/10/26 20:19:40 akmods: Checking kmods exist for 4.7.9-200.fc24.x86_64
2016/10/27 18:35:44 akmods: Checking kmods exist for 4.7.6-200.fc24.x86_64
2016/10/27 18:35:44 akmods: Checking kmods exist for 4.7.7-200.fc24.x86_64
2016/10/27 18:35:44 akmods: Checking kmods exist for 4.7.9-200.fc24.x86_64
2016/10/27 20:30:51 akmods: Checking kmods exist for 4.7.9-200.fc24.x86_64
2016/11/01 19:03:00 akmods: Checking kmods exist for 4.7.6-200.fc24.x86_64
2016/11/01 19:03:00 akmods: Checking kmods exist for 4.7.7-200.fc24.x86_64
2016/11/01 19:03:00 akmods: Checking kmods exist for 4.7.9-200.fc24.x86_64
2016/11/01 20:36:35 akmods: Checking kmods exist for 4.7.9-200.fc24.x86_64
2016/11/02 17:46:38 akmods: Checking kmods exist for 4.8.4-200.fc24.x86_64
2016/11/02 17:46:38 akmods: Building and installing nvidia-kmod
2016/11/02 17:46:38 akmods: Building RPM using the command '/sbin/akmodsbuild --target x86_64 --kernels 4.8.4-200.fc24.x86_64 /usr/src/akmods/nvidia-kmod.latest'
2016/11/02 17:47:04 akmods: Installing newly built rpms
2016/11/02 17:47:04 akmods: DNF detected
2016/11/02 17:47:08 akmods: Checking kmods exist for 4.7.7-200.fc24.x86_64
2016/11/02 17:47:08 akmods: Checking kmods exist for 4.7.9-200.fc24.x86_64
2016/11/02 17:47:08 akmods: Checking kmods exist for 4.8.4-200.fc24.x86_64
2016/11/02 18:25:07 akmods: Checking kmods exist for 4.8.4-200.fc24.x86_64
2016/11/02 18:28:28 akmods: Checking kmods exist for 4.7.7-200.fc24.x86_64
2016/11/02 18:28:28 akmods: Checking kmods exist for 4.7.9-200.fc24.x86_64
2016/11/02 18:28:28 akmods: Checking kmods exist for 4.8.4-200.fc24.x86_64
2016/11/02 18:29:38 akmods: Checking kmods exist for 4.8.4-200.fc24.x86_64
2016/11/02 19:27:17 akmods: Checking kmods exist for 4.7.7-200.fc24.x86_64
2016/11/02 19:27:17 akmods: Checking kmods exist for 4.7.9-200.fc24.x86_64
2016/11/02 19:27:17 akmods: Checking kmods exist for 4.8.4-200.fc24.x86_64
2016/11/02 19:28:54 akmods: Checking kmods exist for 4.8.4-200.fc24.x86_64
Comment 2 leigh scott 2016-11-03 06:53:43 CET
Sorry 367.44 is an old release (not supported). update nvidia and see if the issue still occurs.

P.S The attached report is useless
Comment 3 leigh scott 2016-11-03 07:00:11 CET
Also akmods no longer builds on shutdown

https://koji.rpmfusion.org/koji/buildinfo?buildID=2328

* Fri Oct 14 2016 Richard Shaw <hobbes1069@gmail.com> - 0.5.6-1 - Disable shutdown systemd service file by default. - Remove modprobe line from main service file.

Also post 

rpm -qa akmods kernel\* |sort
Comment 4 MeSat 2016-11-04 05:11:57 CET
(In reply to leigh scott from comment #2)
> Sorry 367.44 is an old release (not supported). update nvidia and see if the
> issue still occurs.
> 
> P.S The attached report is useless

All my machines, after running dnf update tonight still have 367.44 on them.  rpmfusion is enables in all four repo's, including nonfree updates.

rpmfusion-free.repo:enabled=1
rpmfusion-free-updates.repo:enabled=1

rpmfusion-nonfree.repo:enabled=1
rpmfusion-nonfree-updates.repo:enabled=1


If 367.44 is an old release, why isn't it being offered as an update when I run dnf update? 
Both RPM Fusion Free and Nonfree as well as updates are being checked.

Not being offered a new akmod-nvidia from rpmfusion on a F23 or F24 machine.  I guess this is a different bug and I will report it.

Looking at things, I see that akmods is out of date as well.

Now to look at if this is a Fedora, dnf or RPMfusion bug.

This is what happens when I run dnf update.

 dnf list updates

Adobe Systems Incorporated                                                              7.1 kB/s | 1.8 kB     00:00    
Fedora 24 - x86_64 - Updates                                                            1.6 MB/s |  18 MB     00:11    
RPM Fusion for Fedora 24 - Nonfree - Updates                                            323 kB/s | 110 kB     00:00    
RPM Fusion for Fedora 24 - Free - Updates                                               744 kB/s | 320 kB     00:00    
Fedora 24 - x86_64                                                                      2.6 MB/s |  47 MB     00:17    
RPM Fusion for Fedora 24 - Free                                                         287 kB/s | 326 kB     00:01    
RPM Fusion for Fedora 24 - Nonfree                                                      188 kB/s |  64 kB     00:00    
Upgraded Packages
bluez.x86_64                                                    5.43-1.fc24                                      updates
bluez-cups.x86_64                                               5.43-1.fc24                                      updates
bluez-libs.x86_64                                               5.43-1.fc24                                      updates
chrony.x86_64                                                   2.4-4.fc24                                       updates
coreutils.x86_64                                                8.25-7.fc24                                      updates
coreutils-common.x86_64                                         8.25-7.fc24                                      updates
hplip.x86_64                                                    3.16.10-1.fc24                                   updates
hplip-common.x86_64                                             3.16.10-1.fc24                                   updates
hplip-gui.x86_64                                                3.16.10-1.fc24                                   updates
hplip-libs.x86_64                                               3.16.10-1.fc24                                   updates
kdepimlibs.x86_64                                               4.14.10-15.fc24                                  updates
kdepimlibs-akonadi.x86_64                                       4.14.10-15.fc24                                  updates
kdepimlibs-gpgme.x86_64                                         4.14.10-15.fc24                                  updates
kdepimlibs-kxmlrpcclient.x86_64                                 4.14.10-15.fc24                                  updates
libarchive.i686                                                 3.2.2-1.fc24                                     updates
libarchive.x86_64                                               3.2.2-1.fc24                                     updates
libsane-hpaio.x86_64                                            3.16.10-1.fc24                                   updates
libsolv.x86_64                                                  0.6.23-6.fc24                                    updates
libwebp.i686                                                    0.5.1-2.fc24                                     updates
libwebp.x86_64                                                  0.5.1-2.fc24                                     updates
sane-backends.x86_64                                            1.0.25-4.fc24                                    updates
sane-backends-libs.i686                                         1.0.25-4.fc24                                    updates
sane-backends-libs.x86_64                                       1.0.25-4.fc24                                    updates
Comment 5 leigh scott 2016-11-04 07:31:45 CET
There is nothing wrong with the repo

https://paste.fedoraproject.org/469903/14782406/raw/

$ sudo dnf repolist 
Last metadata expiration check: 1:53:42 ago on Fri Nov  4 04:34:54 2016.
repo id                   repo name                                    status
adobe-linux-x86_64        Adobe Systems Incorporated                        2
*fedora                   Fedora 24 - x86_64                           49,722
google-chrome             google-chrome                                     3
google-chrome-beta        google-chrome-beta                                3
rpmfusion-free            RPM Fusion for Fedora 24 - Free                 352
rpmfusion-free-updates    RPM Fusion for Fedora 24 - Free - Updates       347
rpmfusion-nonfree         RPM Fusion for Fedora 24 - Nonfree               89
rpmfusion-nonfree-updates RPM Fusion for Fedora 24 - Nonfree - Updates    144
*updates                  Fedora 24 - x86_64 - Updates                 12,230


post

rpm -qai *\nvidia\* |grep Vendor
Comment 6 MeSat 2016-11-05 18:27:05 CET
Removed the mirrors and ran dnf --refresh update and got latest nvidia rpms.  Now system won't start X on boot.

kmod rpm was created so this bug I guess could be closed unless problem occurs again and then I will create another bug report.