#linuxcnc-devel Logs

Apr 14 2023

03:07 AM guest29057 is now known as KimK
05:56 AM JT-Cave: for some reason the forum disk usage went from 61% to 95% yesterday and I can't figure out why
05:58 AM JT-Cave: starting at 8pm yesterday there is a steady rise in disk usage
06:06 AM JT-Cave: and now the forum is down
06:09 AM JT-Cave: and now it's back up...
06:13 AM JT-Cave: and back down again
06:49 AM JT-Cave: seb_kuzminsky, andypugh_ cradek
07:49 AM andypugh_: Can’t get in to look with the admin interface.
07:54 AM andypugh_: Who has physical access?
08:07 AM JT-Shop: I'm not sure, I can ssh in as root but can't get to the admin interface
08:08 AM JT-Shop: root@forum:~# df -H
08:08 AM JT-Shop: Filesystem Size Used Avail Use% Mounted on
08:08 AM JT-Shop: udev 1.1G 0 1.1G 0% /dev
08:08 AM JT-Shop: tmpfs 208M 955k 208M 1% /run
08:08 AM JT-Shop: tmpfs 1.1G 0 1.1G 0% /dev/shm
08:08 AM JT-Shop: tmpfs 5.3M 0 5.3M 0% /run/lock
08:08 AM JT-Shop: tmpfs 1.1G 0 1.1G 0% /sys/fs/cgroup
08:08 AM JT-Shop: tmpfs 208M 0 208M 0% /run/user/0
08:09 AM JT-Shop: I can also log into cloud.dreamhost.com but that's the admin for the droplet
08:11 AM JT-Cave: cloud.digitalocean.com not dreamhost
08:30 AM JT-Cave: just did apt update on the forum and no disk space left
09:16 AM cradek: I don't have any access to the forum
09:17 AM cradek: Was a disk full, and then you ran apt update?
09:17 AM cradek: or was it apt update that caused the disk to be full?
09:20 AM JT-Cave: no, the disk full happened last night starting at 8pm it went from 61% to 95%
09:21 AM JT-Cave: and yes the disk was full when I ran apt update
09:22 AM JT-Cave: root@forum:/# df -H
09:22 AM JT-Cave: Filesystem Size Used Avail Use% Mounted on
09:22 AM JT-Cave: udev 1.1G 0 1.1G 0% /dev
09:22 AM JT-Cave: tmpfs 208M 1.1M 207M 1% /run
09:22 AM JT-Cave: tmpfs 1.1G 0 1.1G 0% /dev/shm
09:22 AM JT-Cave: tmpfs 5.3M 0 5.3M 0% /run/lock
09:22 AM JT-Cave: tmpfs 1.1G 0 1.1G 0% /sys/fs/cgroup
09:22 AM JT-Cave: tmpfs 208M 0 208M 0% /run/user/0
09:22 AM JT-Cave: hmm copy and paste is failing
09:23 AM JT-Cave: /dev/vda1 53G 50G 103M 100% /
09:24 AM cradek: do you have the output from apt? it was probably a mistake to run it with no disk space. I hope it didn't mess anything up.
09:24 AM JT-Cave: apt update failed to do anything
09:24 AM JT-Cave: but no I don't have that output
09:24 AM cradek: I don't have credentials, but I would be looking in /tmp, /var/log, /var/cache/apt
09:24 AM JT-Cave: ok thanks
09:25 AM cradek: ok that's good if it failed fast/early
09:25 AM JT-Cave: I did delete all the old logs in var/log
09:25 AM cradek: you're looking for something **big** or lots and lots of files. don't bother deleting little things
09:26 AM JT-Cave: root@forum:/tmp# ls
09:26 AM JT-Cave: systemd-private-7a6c3f7fe4a746f9aa35c5635a9aefb6-apache2.service-IysByf
09:26 AM JT-Cave: systemd-private-7a6c3f7fe4a746f9aa35c5635a9aefb6-do-agent.service-3bOiej
09:26 AM JT-Cave: systemd-private-7a6c3f7fe4a746f9aa35c5635a9aefb6-systemd-logind.service-5kU1df
09:26 AM JT-Cave: systemd-private-7a6c3f7fe4a746f9aa35c5635a9aefb6-systemd-resolved.service-CnUwDf
09:26 AM JT-Cave: systemd-private-7a6c3f7fe4a746f9aa35c5635a9aefb6-systemd-timesyncd.service-oiAAdj
09:28 AM JT-Cave: 39G in /var
09:28 AM cradek: ok that'll be the problem in there somewhere
09:29 AM cradek: cd /var; du -s *
09:29 AM JT-Cave: root@forum:/var# du -s *
09:29 AM JT-Cave: 9756 backups
09:29 AM JT-Cave: 210400 cache
09:29 AM JT-Cave: 4 crash
09:29 AM JT-Cave: 23043404 lib
09:29 AM JT-Cave: 4 local
09:29 AM JT-Cave: 0 lock
09:29 AM JT-Cave: 4616592 log
09:30 AM JT-Cave: 292 mail
09:30 AM JT-Cave: 8 opt
09:30 AM JT-Cave: 0 run
09:30 AM JT-Cave: 2556 spool
09:30 AM JT-Cave: 44 tmp
09:30 AM JT-Cave: 12471464 www
09:30 AM JT-Cave: lib and www are the big ones
09:30 AM cradek: you might have www logs in /var/www. maybe logrotate isn't working
09:31 AM cradek: or ... it might really be the forum data - be careful
09:31 AM JT-Cave: yup
09:31 AM JT-Cave: root@forum:/var/www/html# du -s *
09:31 AM JT-Cave: 29388 administrator
09:31 AM JT-Cave: 20 bin
09:31 AM JT-Cave: 868 cache
09:31 AM JT-Cave: 44 cli
09:31 AM JT-Cave: 23080 components
09:31 AM JT-Cave: 4 configuration.php
09:31 AM JT-Cave: 4 htaccess.txt
09:31 AM JT-Cave: 94916 images
09:31 AM JT-Cave: 16 includes
09:31 AM JT-Cave: 4 index.php
09:31 AM JT-Cave: 480 language
09:31 AM JT-Cave: 800 layouts
09:31 AM JT-Cave: 24564 libraries
09:31 AM JT-Cave: 20 LICENSE.txt
09:31 AM JT-Cave: 8 logs
09:31 AM JT-Cave: 12166556 media
09:31 AM JT-Cave: 1100 modules
09:31 AM JT-Cave: 3412 plugins
09:31 AM JT-Cave: 8 README.txt
09:32 AM JT-Cave: 4 robots.txt
09:32 AM JT-Cave: 4 robots.txt.dist
09:32 AM JT-Cave: 1864 templates
09:32 AM JT-Cave: 124060 tmp
09:32 AM JT-Cave: 4 web.config.txt
09:32 AM JT-Cave: root@forum:/var/www/html/media# du -s *
09:32 AM JT-Cave: 12150344 kunena
09:32 AM JT-Cave: root@forum:/var/www/html/media/kunena# du -s *
09:32 AM JT-Cave: 12145412 attachments
09:33 AM cradek: ok but that's a normal thing to be big I bet
09:33 AM JT-Cave: yup the attachments is big
09:34 AM cradek: I think you probably haven't found the problem
09:34 AM JT-Cave: I agree
09:42 AM cradek: I sent a note to jeff - not sure how available he is
09:49 AM JT-Woodshop: yhanks
09:53 AM JT-Shop: root@forum:/var# du -sh *
09:53 AM JT-Shop: 22G lib
09:54 AM JT-Shop: root@forum:/var/lib# du -sh *
09:54 AM JT-Shop: 22G mysql
09:55 AM JT-Shop: root@forum:/var/lib/mysql# ls
09:55 AM JT-Shop: about 100 binlog.nnnnnnn files
09:58 AM JT-Shop: and all the binlog files are from yesterday...
10:14 AM roguish[m]: cradek: JT-Shop gentlemen, just curious, but who 'owns' the forum, main website, wiki, and all ?????
10:14 AM roguish[m]: who is the signatory?
10:16 AM JT-Shop: I don' know
10:16 AM cradek: I don't understand what you're asking
10:17 AM cradek: lots of people contribute resources/time for the web stuff
10:18 AM cradek: JT-Shop: again that's a space where big sizes are expected I think - have you looked on the forum itself to see if someone has added a bunch of crazy stuff? big pornspam attachments maybe?
10:19 AM JT-Shop: can't access the forum, and all those binary log files were created yesterday which is not correct and corresponds with the data usage going from 61% to 95%
10:19 AM JT-Shop: a binlog should be created each day and rotated out
10:20 AM cradek: (I'm talking to jeff)
10:20 AM roguish[m]: cradek: well, is there a written list of all the ISP hosts? and who pays the bills? and who all the admins are ????
10:20 AM JT-Shop: I'm an admin
10:20 AM cradek: I pay some bills
10:21 AM JT-Shop: and I pay some bills
10:21 AM roguish[m]: cradek: i've paid a forum bill one year through jepler.... the least I could do to help....
10:21 AM cradek: I help admin some things sometimes
10:24 AM roguish[m]: just seems the 'community' is growing, and the websites, particularly the forum, are growing in size. it's natural. Could it be time for a significant upgrade? and some sort of voluntary financial contribution system ????
10:24 AM cradek: jepler added my key
10:28 AM cradek: lemme try to figure out how to read those binlogs
10:30 AM JT-Shop: ok thanks
10:30 AM JT-Shop: I'm fixing to head out for the day good luck
10:31 AM JT-Shop: roguish[m], digitalocean admins are jeff, chris, seb and myself
10:32 AM JT-Shop: the forum is back up, maybe because I deleted a bunch of old logs in var/log
10:36 AM cradek: I don't understand yet what the unusual traffic is that's causing those large logs
10:36 AM JT-Cave: disk usage is down to 82%
10:36 AM JT-Cave: should only be one log a day...
10:36 AM JT-Cave: gotta run, thanks for helping
03:24 PM JT-Shop: cradek, any luck with the mysql logs?
03:34 PM cradek: I figured out how to read those binary logs, and it appears that something is running the same command over and over. I don't know what's doing it. Database/forum isn't something I know much about.
03:34 PM cradek: so anyway I deleted the oldest logs so there's some time until jeff can look at it
03:34 PM cradek: it needs a real fix
03:36 PM cradek: I'm not sure if I'm reading the logs correctly, but I think the command is SET TRANSACTION ISOLATION LEVEL READ COMMITTED which is some kind of setup thing that shouldn't need to be done over and over
04:38 PM JT-Cave: ok thanks
04:38 PM JT-Cave: disk usage is down to 63%
04:48 PM JT-Shop: I just looked and mysql is still creating a new binlog every few minutes... how did you delete them?
07:35 PM cradek: like a dinosaur I just deleted the oldest ones with rm. jeff says there's a routine cleanup but I couldn't find it.
07:37 PM cradek: I don't know if I have any admin powers on the forum
07:38 PM cradek: I feel pretty unequipped to actually find the problem
07:40 PM cradek: hm it's logging 50MB/minute