Blank
22 September 2017 20:28
1
for the past three days I have been fighting with /var/logs growing and growing and I can’t figure it out.
Google was no help, I tried everything from rotating logs, .tar logs, delete logs I got so desperate to stop it from getting out of hand I tried everything I could find but still nothing plus I have no idea what I am looking for in the logs
So I stopped with all the commands and just reinstalled. not more then 20 minutes later my root directory was full and off I went again. Still couldn’t find the right answer so I reinstalled again and BOOM 20 minutes later the same thing so I took a shot and reinstalled a 3rd time and here we are but this time I gave it it’s very own partition to play with in hopes it would just stop from running away but No it just took it all.
I can’t install, update or upgrade anything I’m just stuck. I feel like it is a hardware conflict but because I think it happened after and upgrade the first time and the same this last time.
Can anyone pass along some knowledge that might help?
Hi
The version of Ubuntu MATE you’re using
A sample of the logs
which files are growing, syslog, auth? For example if you’re running firewall and log entries, files can grow quite fast. ls -lS (upper S) in /var/log will list files by size, my top 5 are kern, lastlog, dpkg, syslog, and auth (about 1MB each.) logrotate rotates your files weekly but you mentioned 3 days so this does not apply.
Blank
22 September 2017 22:28
4
@ouroumov @pavlos_kairis yeah my OS is 17.04
ouroumov:
A sample of the logs
Which log file? there seems to be many of them
ls -lS is
drwx------ 2 root root 16K Sep 22 13:50 lost+found/
drwxr-xr-x 2 root root 4.0K Sep 22 14:21 backups/
drwxr-xr-x 18 root root 4.0K Sep 22 14:09 cache/
drwxrwsrwt 2 root whoopsie 4.0K Sep 22 15:56 crash/
drwxr-xr-x 64 root root 4.0K Sep 22 15:29 lib/
drwxrwsr-x 2 root staff 4.0K Apr 10 07:29 local/
drwxrwxr-x 11 root syslog 4.0K Sep 22 15:57 log/
drwxrwsr-x 2 root mail 4.0K Apr 11 23:07 mail/
drwxrwsrwt 2 root whoopsie 4.0K Apr 11 23:16 metrics/
drwxr-xr-x 2 root root 4.0K Apr 11 23:07 opt/
drwxr-xr-x 2 root root 4.0K Apr 6 04:32 snap/
drwxr-xr-x 7 root root 4.0K Apr 11 23:12 spool/
drwxrwxrwt 5 root root 4.0K Sep 22 15:58 tmp/
lrwxrwxrwx 1 root root 9 Sep 22 13:52 lock -> /run/lock/
lrwxrwxrwx 1 root root 4 Sep 22 13:52 run -> /run/
but looks like nothing is there when running Disk Usage Analyzer (Can't take screen shot) but it just has
Log 91.6%
installer 19.2%
this was done before all the new installs
ls -lS /var/log | head -10
please mark the text as Blockquote, easier to read.
Blank
22 September 2017 22:58
6
-rw-r----- 1 syslog adm 1.9G Sep 22 18:54 syslog
-rw-r----- 1 syslog adm 1.9G Sep 22 18:54 kern.log
-rw-r--r-- 1 root root 1.5M Sep 22 15:29 dpkg.log
-rw-rw-r-- 1 root utmp 286K Sep 22 13:56 lastlog
-rw-r--r-- 1 root root 67K Sep 22 15:57 Xorg.0.log.old
-rw-r--r-- 1 root root 65K Sep 22 17:21 Xorg.0.log
-rw-r--r-- 1 root root 59K Apr 11 23:08 bootstrap.log
-rw-r--r-- 1 root root 47K Sep 22 15:11 alternatives.log
-rw-r--r-- 1 root root 32K Sep 22 13:56 faillog
Obviously syslog and kern.log at 1.9G
cat /var/log/syslog | tailcat /var/log/kern.log | tail
They will likely be similar so both or either. I cleaned up your last post if you’re wondering.
Blank
22 September 2017 23:27
8
I was wondering because I tried to make it look better but failed! Thanks for the edit..
> Sep 22 15:57:49 DeadR kernel: [ 5.474732] worker_thread+0x4b/0x500
> Sep 22 15:57:49 DeadR kernel: [ 5.474736] kthread+0x109/0x140
> Sep 22 15:57:49 DeadR kernel: [ 5.474738] ? process_one_work+0x4b0/0x4b0
> Sep 22 15:57:49 DeadR kernel: [ 5.474741] ? kthread_create_on_node+0x60/0x60
> Sep 22 15:57:49 DeadR kernel: [ 5.474745] ret_from_fork+0x2c/0x40
> Sep 22 15:57:49 DeadR kernel: [ 5.474747] ---[ end trace 1049abe8e79f7d60 ]---
> Sep 22 15:57:49 DeadR kernel: [ 5.477582] ------------[ cut here ]------------
> Sep 22 15:57:49 DeadR kernel: [ 5.477588] WARNING: CPU: 0 PID: 124 at /build/linux-3PuzWO/linux-4.10.0/drivers/usb/core/urb.c:449 usb_submit_urb.part.8+0x14d/0x530
> Sep 22 15:57:49 DeadR kernel: [ 5.477596] usb 4-3: BOGUS urb xfer, pipe 1 != type 3
> Sep 22 15:57:49 DeadR kernel: [ 5.477605] Modules linked in: edac_mce_amd edac_core kvm_amd kvm irqbypass crct10dif_pclmul crc32_pclmul snd_hda_codec_realtek snd_hda_codec_generic ghash_clmulni_intel pcbc snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_pcm aesni_intel aes_x86_64 crypto_simd snd_seq_midi glue_helper snd_seq_midi_event crywardog@
> Sep 22 15:57:49 DeadR kernel: [ 5.474732] worker_thread+0x4b/0x500
> Sep 22 15:57:49 DeadR kernel: [ 5.474736] kthread+0x109/0x140
> Sep 22 15:57:49 DeadR kernel: [ 5.474738] ? process_one_work+0x4b0/0x4b0
> Sep 22 15:57:49 DeadR kernel: [ 5.474741] ? kthread_create_on_node+0x60/0x60
> Sep 22 15:57:49 DeadR kernel: [ 5.474745] ret_from_fork+0x2c/0x40
> Sep 22 15:57:49 DeadR kernel: [ 5.474747] ---[ end trace 1049abe8e79f7d60 ]---
> Sep 22 15:57:49 DeadR kernel: [ 5.477582] ------------[ cut here ]------------
> Sep 22 15:57:49 DeadR kernel: [ 5.477588] WARNING: CPU: 0 PID: 124 at /build/linux-3PuzWO/linux-4.10.0/drivers/usb/core/urb.c:449 usb_submit_urb.part.8+0x14d/0x530
> Sep 22 15:57:49 DeadR kernel: [ 5.477596] usb 4-3: BOGUS urb xfer, pipe 1 != type 3
> Sep 22 15:57:49 DeadR kernel: [ 5.477605] Modules linked in: edac_mce_amd edac_core kvm_amd kvm irqbypass crct10dif_pclmul crc32_pclmul snd_hda_codec_realtek snd_hda_codec_generic ghash_clmulni_intel pcbc snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_pcm aesni_intel aes_x86_64 crypto_simd snd_seq_midi glue_helper snd_seq_midi_event crywardog@
I was unsure of the commands and didn't have any of my cheat sheets. I don't dig into the logs that often
I don’t much, either. But better get about 50 lines:
cat /var/log/kern.log | tail -50
Blank
22 September 2017 23:47
10
> Sep 22 15:57:49 DeadR kernel: [ 5.470680] ? usb_submit_urb+0x62/0x70
> Sep 22 15:57:49 DeadR kernel: [ 5.470683] ? ath9k_hif_usb_alloc_urbs+0x3f3/0x4f0 [ath9k_htc]
> Sep 22 15:57:49 DeadR kernel: [ 5.470687] ath9k_htc_hw_init+0x11/0x30 [ath9k_htc]
> Sep 22 15:57:49 DeadR kernel: [ 5.470690] ath9k_hif_usb_firmware_cb+0x97/0x1a0 [ath9k_htc]
> Sep 22 15:57:49 DeadR kernel: [ 5.470692] request_firmware_work_func+0x45/0x90
> Sep 22 15:57:49 DeadR kernel: [ 5.470695] process_one_work+0x1fc/0x4b0
> Sep 22 15:57:49 DeadR kernel: [ 5.470697] worker_thread+0x4b/0x500
> Sep 22 15:57:49 DeadR kernel: [ 5.470699] kthread+0x109/0x140
> Sep 22 15:57:49 DeadR kernel: [ 5.470701] ? process_one_work+0x4b0/0x4b0
> Sep 22 15:57:49 DeadR kernel: [ 5.470704] ? kthread_create_on_node+0x60/0x60
> Sep 22 15:57:49 DeadR kernel: [ 5.470706] ret_from_fork+0x2c/0x40
> Sep 22 15:57:49 DeadR kernel: [ 5.470708] ---[ end trace 1049abe8e79f7d5f ]---
> Sep 22 15:57:49 DeadR kernel: [ 5.474575] ------------[ cut here ]------------
> Sep 22 15:57:49 DeadR kernel: [ 5.474581] WARNING: CPU: 0 PID: 124 at /build/linux-3PuzWO/linux-4.10.0/drivers/usb/core/urb.c:449 usb_submit_urb.part.8+0x14d/0x530
> Sep 22 15:57:49 DeadR kernel: [ 5.474583] usb 4-3: BOGUS urb xfer, pipe 1 != type 3
> Sep 22 15:57:49 DeadR kernel: [ 5.474583] Modules linked in: edac_mce_amd edac_core kvm_amd kvm irqbypass crct10dif_pclmul crc32_pclmul snd_hda_codec_realtek snd_hda_codec_generic ghash_clmulni_intel pcbc snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_pcm aesni_intel aes_x86_64 crypto_simd snd_seq_midi glue_helper snd_seq_midi_event cryptd ath9k_htc snd_rawmidi ath9k_common ath9k_hw ath mac80211 serio_raw joydev input_leds cfg80211 snd_seq fam15h_power k10temp snd_seq_device snd_timer snd soundcore shpchp i2c_piix4 tpm_infineon mac_hid parport_pc ppdev lp parport ip_tables x_tables autofs4 hid_logitech_hidpp hid_logitech_dj hid_generic uas usb_storage hid_logitech ff_memless usbhid hid pata_acpi amdkfd amd_iommu_v2 radeon i2c_algo_bit ttm drm_kms_helper syscopyarea sysfillrect psmouse sysimgblt fb_sys_fops
> Sep 22 15:57:49 DeadR kernel: [ 5.474630] pata_atiixp drm r8169 ahci libahci mii wmi fjes
> Sep 22 15:57:49 DeadR kernel: [ 5.474641] CPU: 0 PID: 124 Comm: kworker/0:2 Tainted: G W 4.10.0-35-generic #39-Ubuntu
> Sep 22 15:57:49 DeadR kernel: [ 5.474642] Hardware name: MSI MS-7641/760GM-P34(FX) (MS-7641) , BIOS V17.17 04/22/2015
> Sep 22 15:57:49 DeadR kernel: [ 5.474646] Workqueue: events request_firmware_work_func
> Sep 22 15:57:49 DeadR kernel: [ 5.474648] Call Trace:
> Sep 22 15:57:49 DeadR kernel: [ 5.474652] dump_stack+0x63/0x81
> Sep 22 15:57:49 DeadR kernel: [ 5.474655] __warn+0xcb/0xf0
> Sep 22 15:57:49 DeadR kernel: [ 5.474658] warn_slowpath_fmt+0x5f/0x80
> Sep 22 15:57:49 DeadR kernel: [ 5.474661] usb_submit_urb.part.8+0x14d/0x530
> Sep 22 15:57:49 DeadR kernel: [ 5.474664] usb_submit_urb+0x62/0x70
> Sep 22 15:57:49 DeadR kernel: [ 5.474670] hif_usb_send+0xeb/0x340 [ath9k_htc]
> Sep 22 15:57:49 DeadR kernel: [ 5.474675] htc_issue_send.constprop.2+0x58/0x70 [ath9k_htc]
> Sep 22 15:57:49 DeadR kernel: [ 5.474681] htc_send_epid+0x18/0x20 [ath9k_htc]
> Sep 22 15:57:49 DeadR kernel: [ 5.474686] ath9k_wmi_cmd+0x11a/0x1a0 [ath9k_htc]
> Sep 22 15:57:49 DeadR kernel: [ 5.474691] ath9k_regread+0x4d/0x80 [ath9k_htc]
> Sep 22 15:57:49 DeadR kernel: [ 5.474695] ath_hw_keyreset+0x4a/0x1e0 [ath]
> Sep 22 15:57:49 DeadR kernel: [ 5.474699] ath9k_cmn_init_crypto+0x3f/0x50 [ath9k_common]
> Sep 22 15:57:49 DeadR kernel: [ 5.474704] ath9k_htc_probe_device+0x4d4/0xa00 [ath9k_htc]
> Sep 22 15:57:49 DeadR kernel: [ 5.474707] ? usb_submit_urb+0x62/0x70
> Sep 22 15:57:49 DeadR kernel: [ 5.474712] ? ath9k_hif_usb_alloc_urbs+0x3f3/0x4f0 [ath9k_htc]
> Sep 22 15:57:49 DeadR kernel: [ 5.474717] ath9k_htc_hw_init+0x11/0x30 [ath9k_htc]
> Sep 22 15:57:49 DeadR kernel: [ 5.474723] ath9k_hif_usb_firmware_cb+0x97/0x1a0 [ath9k_htc]
> Sep 22 15:57:49 DeadR kernel: [ 5.474726] request_firmware_work_func+0x45/0x90
> Sep 22 15:57:49 DeadR kernel: [ 5.474730] process_one_work+0x1fc/0x4b0
> Sep 22 15:57:49 DeadR kernel: [ 5.474732] worker_thread+0x4b/0x500
> Sep 22 15:57:49 DeadR kernel: [ 5.474736] kthread+0x109/0x140
> Sep 22 15:57:49 DeadR kernel: [ 5.474738] ? process_one_work+0x4b0/0x4b0
> Sep 22 15:57:49 DeadR kernel: [ 5.474741] ? kthread_create_on_node+0x60/0x60
> Sep 22 15:57:49 DeadR kernel: [ 5.474745] ret_from_fork+0x2c/0x40
> Sep 22 15:57:49 DeadR kernel: [ 5.474747] ---[ end trace 1049abe8e79f7d60 ]---
> Sep 22 15:57:49 DeadR kernel: [ 5.477582] ------------[ cut here ]------------
> Sep 22 15:57:49 DeadR kernel: [ 5.477588] WARNING: CPU: 0 PID: 124 at /build/linux-3PuzWO/linux-4.10.0/drivers/usb/core/urb.c:449 usb_submit_urb.part.8+0x14d/0x530
> Sep 22 15:57:49 DeadR kernel: [ 5.477596] usb 4-3: BOGUS urb xfer, pipe 1 != type 3
> Sep 22 15:57:49 DeadR kernel: [ 5.477605] Modules linked in: edac_mce_amd edac_core kvm_amd kvm irqbypass crct10dif_pclmul crc32_pclmul snd_hda_codec_realtek snd_hda_codec_generic ghash_clmulni_intel pcbc snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_pcm aesni_intel aes_x86_64 crypto_simd snd_seq_midi glue_helper snd_seq_midi_event crywardog@
I’m not sure the source of why it’s dumping the stack. Perhaps someone knows some better log parsing. The last 50 lines is all within the same second 15:57:49 and it might be huge.
From reading the log, I’d say either the usb or ath9k driver is crashing. Not sure what could cause this apart from hardware failure.
Found this:
And this:
What hardware are you using @Blank ? Are you using a wifi USB dongle?
Blank
23 September 2017 00:27
13
TP-Link TL-WN722N
I will locate one of my other dongles and try them...
Blank
23 September 2017 00:55
14
I replaced this... TP-Link
*-network
description: Ethernet interface
product: RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
vendor: Realtek Semiconductor Co., Ltd.
physical id: 0
bus info: pci@0000:02:00.0
logical name: enp2s0
version: 06
serial: d8:cb:8a:33:28:e3
size: 10Mbit/s
capacity: 1Gbit/s
width: 64 bits
clock: 33MHz
capabilities: pm msi pciexpress msix vpd bus_master cap_list ethernet physical tp mii 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=r8169 driverversion=2.3LK-NAPI duplex=half firmware=rtl8168e-3_0.0.4 03/27/12 latency=0 link=no multicast=yes port=MII speed=10Mbit/s
resources: irq:26 ioport:e800(size=256) memory:fdfff000-fdffffff memory:fdff8000-fdffbfff
*-network
description: Wireless interface
physical id: 1
bus info: usb@4:3
logical name: wlx8416f91aa3fd
serial: 84:16:f9:1a:a3:fd
capabilities: ethernet physical wireless
configuration: broadcast=yes driver=ath9k_htc driverversion=4.10.0-35-generic firmware=1.4 ip=192.168.8.181 link=yes multicast=yes wireless=IEEE 802.11
with this... Boise
> *-network
> description: Ethernet interface
> product: RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
> vendor: Realtek Semiconductor Co., Ltd.
> physical id: 0
> bus info: pci@0000:02:00.0
> logical name: enp2s0
> version: 06
> serial: d8:cb:8a:33:28:e3
> size: 10Mbit/s
> capacity: 1Gbit/s
> width: 64 bits
> clock: 33MHz
> capabilities: pm msi pciexpress msix vpd bus_master cap_list ethernet physical tp mii 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation
> configuration: autonegotiation=on broadcast=yes driver=r8169 driverversion=2.3LK-NAPI duplex=half firmware=rtl8168e-3_0.0.4 03/27/12 latency=0 link=no multicast=yes port=MII speed=10Mbit/s
> resources: irq:26 ioport:e800(size=256) memory:fdfff000-fdffffff memory:fdff8000-fdffbfff
> *-network
> description: Wireless interface
> physical id: 1
> bus info: usb@1:6
> logical name: wlxe84e061c9609
> serial: e8:4e:06:1c:96:09
> capabilities: ethernet physical wireless
> configuration: broadcast=yes driver=rtl8192cu driverversion=4.10.0-35-generic firmware=N/A ip=192.168.8.195 link=yes multicast=yes wireless=IEEE 802.11
Let me find my old Alpha card
If with another dongle you don't get a log file explosion like this, I think we'll have found the culprit.
1 Like
Blank
23 September 2017 01:29
16
Valid point, but my Alfa card is hiding so I can't try that one I did find another TP-Link so three cards all toll 3 separate reboots with the same warning on first start stating "0 space left in /var"
so the only way I know how to get the /var back to not exploding size is a reinstall and I wish not to do that... My real question should have been
"How do I clear the log files so I can tell whats what?"
truncate -s 0 /var/log/*.log
Blank
23 September 2017 12:32
18
No Return on that command
Blank
23 September 2017 12:37
19
I take that back it did clear them out!@pavlos_kairis for the command
Blank
24 September 2017 00:07
20
after I entered @pavlos_kairis command It purged my logs down to 66% so I was able to do some trouble shooting.
I unplugged all external hardware and booted up to check the logs to find all was still good. so I then started to add hardware, checking logs until I got to wifi thats when the logs started to move again.
I swapped wifi dongles and checked the logs again only to find it was still growing, so I replaced the usb going to the wifi and low and behold the logs stopped growing. so me being me I tried the “bad” usb again but Nothing! No Log Growth.
so I moved the “bad” usb cable around to see if I could get a reaction in the logs and it started growing again.
I powered down, opened up the case and there it was, a wire from the mother board going to the usb on the case was not properly seated. damn gremlins got it there again!
Thanks for all the help! I never would have thought to look there if it wasn’t for @ouroumov and his links. Thanks @Bill_MI for starting the conversation and the proper cat log commands
2 Likes