#linuxcnc-devel Logs

May 22 2020

12:39 AM rmu|w: mozmck: seems i have some digging to do... https://github.com/LinuxCNC/linuxcnc/commit/b353bd3042fa4a5b97a00d8220fa771b80f52f81
07:55 AM rmu|w: https://dpaste.org/5Hic
08:06 AM cerna: jepler: Thanks. So if I want to know the status of all tests, I would need to query all builders about their cached builds and all cached builds about their branch, SHA and status in loop? No shortcut?
08:14 AM rmu|w: my current theory with this "crazy move" issue is that it has something to do with "lazy close", whatever that is
08:20 AM jepler: I think lazy close has something to do with keeping some g-code file open so that o-words entered in MDI can run subs from it
08:20 AM rmu|w: crazy move goes away when I change _setup.use_lazy_close = 1; to = 0; in src/emc/rs274ngc_pre.cc:932
08:20 AM jepler: cerna: if you need to know whether the package build succeeded, yes. Depending what kind of problem you're chasing (like test failures), no
08:27 AM rmu|w: i think somebody who knows what actually is going on should have a look at that...
08:35 AM pcw_home: There's also something funny about the feed rate, sometimes it gets set to 0, sometimes its unchanged
08:36 AM rmu|w: https://github.com/LinuxCNC/linuxcnc/issues/865
08:37 AM rmu|w: pcw_home: it seems the interpreter is not stopped as it should. can you try changing src/emc/rs274ngc_pre.cc line 932 to = 0; and look if the problem goes away?
08:38 AM rmu|w: this change makes the crazy move disappear with my setup, bit I'm not sure what other consequences that would have or what lazy_close is supposed to accomplish
08:49 AM pcw_home: it makes the crazy move disappear but unless I reload the file every time I re-start, I get a "File not open" error
08:50 AM rmu|w: hmm
08:50 AM pcw_home: emc/task/emctask.cc 69: interp_error: File not open
08:51 AM rmu|w: very strange, i don't get that...
08:58 AM jepler: pi 4 can now boot straight from USB, requires firmware update (beta) : https://www.raspberrypi.org/forums/viewtopic.php?t=274595
09:04 AM rmu|w: another idea would be to revert that lazy_close=0 back to =1 and add emcTaskPlanReset() after emcTaskPlanClose() in emc/task/emctask.cc line 260
10:43 AM pcw_home: yeah that's better (but I have no idea what the side effects are) sorry for the delay, making loquat jam...
11:30 AM jepler: is this new misbehavior? do we know why we're only seeing it now?
11:43 AM rmu|w: I don't know.
11:44 AM rmu|w: It was reported by Horvath Csaba on ems-users@
12:42 PM cerna: Hmm, the buildbot is lying to me. It says that it has only X cached builds, but you can access lot more of them (more than 1000). Wonder if there is some throttling on the endpoint?
12:43 PM jepler: @cerna
12:44 PM jepler: oops, what is this, twitter
12:44 PM jepler: cerna: what are you up to?
12:51 PM cerna: jepler: Automatic synchronization of branches I am interested in and little bit better querying of tests status (I never can make quick sense of waterfall and such).
12:51 PM cerna: Too bad the Github Actions are not authoritative.
01:06 PM seb_kuzminsky: i just released mesaflash 3.4.0~pre1 and put debs on wlo for buster (i386, amd64, and rpi4), stretch, jessie, wheezy, and precise (i386 and amd64)
01:06 PM jepler: seb_kuzminsky: hey do you have time now for talk about debs?
01:06 PM jepler: (you rock, thanks!)
01:06 PM seb_kuzminsky: sure, now's a good time
01:07 PM jepler: seb_kuzminsky: https://media.unpythonic.net/pi4-kernel-pkg/ should be all the artifacts (source & bin) corresponding with what's in the current pi4 image
01:07 PM jepler: can you do the things to make it be in our package repo on wlo?
01:08 PM jepler: or let me know what's missing so I can scrounge around for it
01:08 PM seb_kuzminsky: sure, i'll add them to the buster/rpi4 debs on wlo
01:09 PM jepler: if that is in place then I think I could re-make the image so that all the debs come from apt-get instead of manual download, and add mesaflash
01:09 PM jepler: .. I'll put that on the "might actually get done, and soon" part of my infinite todo list
01:10 PM jepler: seb_kuzminsky: did my search-fu fail or could http://buildbot.linuxcnc.org/ use some info about configuration for pi4s?
01:11 PM jepler: .. are pi4 debs happening on buildbot? I thought they were
01:11 PM seb_kuzminsky: yes, the buildbot is making rpi4 debs
01:11 PM seb_kuzminsky: and yes, there's no info at all about it anywhere :-(
01:12 PM jepler: yay there they are http://buildbot.linuxcnc.org/dists/buster/2.8-rtpreempt/binary-armhf/
01:12 PM Tom_L: are there any updates to the pi img since you posted it?
01:12 PM jepler: seb_kuzminsky: did you see cerna's remarks about scraping buildbot for success/failure info? is there an easy way to make 0000.checkin reflect the status of packaging success/failure?
01:12 PM jepler: Tom_L: no, I haven't built a new one since last time I posted about it, near the start of the month iirc
01:12 PM Tom_L: i'll try an update once it finds a home and see if it works
01:13 PM seb_kuzminsky: i didn't see those remarks
01:13 PM jepler: yesterday at 19:22 my time, cerna> When I want to look if push to master by some SHA passed all tests, all I need to check is the 0000.checkin? Find it in http://buildbot.linuxcnc.org/buildbot/json/builders/0000.checkin/builds/_all?as_text=1 - if it is not there, then it is too old?
01:13 PM jepler: I said that, no, 0000.checkin can be green if the package builds failed (like due to lintian errors)
01:14 PM seb_kuzminsky: the current buildbot has the checkin builder trigger the rip builders and wait for them all to finish, then if they all succeed checkin triggers the packaging builders but doesn't wait for them, so there's no super easy way to get the packaging results into the checkin builder
01:15 PM jepler: is there an efficiency/pipelining reason that it can't wait for the packaging builders?
01:15 PM jepler: (that's my guess)
01:16 PM seb_kuzminsky: err, possibly? it was like 12 years ago i set that all up, i may well have had a reason (good or bad) back then
01:17 PM jepler: hehe
01:17 PM jepler: in a fantasy world where you had infinite time to work on linuxcnc tooling I'd also ask you to look into https://docs.buildbot.net/latest/manual/configuration/reporters.html#githubstatuspush
01:20 PM seb_kuzminsky: we use an ancient version of buildbot (needed to run on the ancient platforms we support), which doesn't have reporters
01:20 PM seb_kuzminsky: but it has something pretty similar called Status, which does have a github plugin
01:20 PM seb_kuzminsky: so it's maybe possible
01:21 PM seb_kuzminsky: i'll look in to it, we can probably at least get rip results into github
01:21 PM jepler: That would be a really nice QOL improvement. Maybe pick one canary deb for a second status, like precise-uspace-amd64 or something.
01:22 PM seb_kuzminsky: the buildbot only tests branches in the linuxcnc org, not random other peoples repos, mostly for security reasons (trolls can put whatever they want into eg our makefile and potentially break the buildslave)
01:24 PM jepler: so it doesn't help pull requests. I keep forgetting.
01:26 PM jepler: Putting a package build into the github actions CI would be the better course of action, if the goal is to give feedback on PRs about whether you've broken packaging entirely
01:26 PM jepler: I can put that on the "won't necessarily happen soon" segment of my infinite task list
01:26 PM seb_kuzminsky: i've locked down sudo on the buildslaves so your joke earlier of pushing a branch that runs 'sudo reboot' on all the buildslaves wouldn't work
01:26 PM jepler: :person pouting:
01:27 PM seb_kuzminsky: but 'cd $HOME; rm -rf *' would sure cause havoc
01:27 PM jepler: "you can't do useful things, but you can still do harmful things" seems to be the result of a lot of "hardening"
01:28 PM seb_kuzminsky: it's sure hard to filter out all the harmful things, but it's very easy to interfere with useful things
01:28 PM jepler: and you can pretend you mean well as you do it :)
01:28 PM seb_kuzminsky: i do back up the buildslaves, but only rarely because they're like 10+ gigs each
01:29 PM jepler: having their creation fully scripted would almost be the better thing, except for the time involved.
01:29 PM seb_kuzminsky: i agree, that would be much better
01:30 PM seb_kuzminsky: if we didn't have the pesky "realtime kernel" requirement they'd all be docker containers
01:30 PM seb_kuzminsky: maybe i could build them with packer
01:30 PM jepler: I was thinking of the time creating the scripting, but the time re-running the creation would also be bad for the speed of buildbot. Going to "fully rebuild VM for every run" doesn't seem productive.
01:30 PM seb_kuzminsky: no no, that would be crazy
01:30 PM jepler: though .. does your virtualization do snapshots? Could you restart the VM from a snapshot everytime?
01:31 PM seb_kuzminsky: as you probably know, docker caches built images so it would only be very slow the first time, then nearly instant after that
01:31 PM seb_kuzminsky: in fact, spinning up docker images is much faster than spinning up pbuilder images, and i keep wishing for a dockerized pbuilder for the buildbot
01:31 PM jepler: re: rtai crashing. I wonder why we unload the kernel modules of RTAI everytime. Why not just leave them loaded? Would that stop the "crash at unload"? Or is it hal-unloading and not RTAI-kernel-module-unloading?
01:32 PM seb_kuzminsky: i use kvm for virtualization, which does support snapshots, that's an interesting idea
01:32 PM jepler: I recently used https://github.com/tsaarni/docker-deb-builder (to build an i386 package) and it seemed to work pretty well
01:32 PM seb_kuzminsky: i think it's rtai unloading that crashes
01:32 PM seb_kuzminsky: neat
01:32 PM jepler: > ./build -i docker-deb-builder:17.04 -o output ~/my-package-source
01:33 PM jepler: needed some tweaking to know about real debian and not just obento but that's life these days
01:34 PM seb_kuzminsky: good old "from debian/eol:wheezy"
01:34 PM seb_kuzminsky: i started a project like that back in ummm april? or something but didn't get all the way before i ran out of oomph
01:35 PM jepler: time has no meaning
01:36 PM seb_kuzminsky: your rpi4 kernel is on wlo, deb & dsc
01:36 PM jepler: > Fri Mar 83 13:27:39 CDT 2020
01:36 PM jepler: thank you !
01:36 PM seb_kuzminsky: thanks for building it :-)
01:37 PM seb_kuzminsky: together we still have enough oomph to be useful
01:38 PM seb_kuzminsky: i'm going to go outside for a bit, talk to you later
01:38 PM seb_kuzminsky: nice chatting with you jepler
01:39 PM jepler: likewise. you heading out?
01:39 PM jepler: :wave:
01:39 PM Tom_L: jepler, andy's test was hal unloading
01:40 PM jepler: if he was invoking "halrun" or similar, it also loads and unloads RTAI kernel modules though
01:40 PM Tom_L: oh
01:41 PM jepler: in my crashes the last-unloaded module was always an RTAI module but that could also be from the previous run. Hard to tell.
01:41 PM jepler: and I don't feel like running the test again right now
01:41 PM jepler: he must have gone back to work, not seeing much andy these days
01:42 PM Tom_L: halcmd loadrt threads
01:42 PM Tom_L: halrun -U
01:45 PM jepler: my crash from running just the rtai load & unload modules scripts over and over again had
01:45 PM jepler: [ 680.021557] Modules linked in: rtai_sched(-) rtai_hal rfkill ... e1000e [last unloaded: rtai_mbx]
01:46 PM jepler: Kernel source says: /* Show a - for module-is-being-unloaded */
01:47 PM jepler: to me this says that it was during the unload of the RTAI modules. rtai_mbx has been unloaded. rtai_sched is in the process of being unloaded. Then the oops occurred.
01:47 PM jepler: If that's all true, then modifying linuxcnc to never unload the RTAI modules could be an effective workaround
01:47 PM jepler: assuming it's the unload of rtai_sched that is the actual problem
01:49 PM jepler: https://gist.github.com/jepler/212d2093f7feb22232f6d2a432e9cacd
01:49 PM Tom_L: his test crashed on my asrock after a couple hundred passes but on the i5 it exceeded 3k before locking up
01:49 PM Tom_L: running the same test with preempt-rt on the same machine ran all 10k iterations fine
01:50 PM jepler: the crashing routine is also within rtai_sched: src/sched/rtai_hal.c:static void rtai_hirq_dispatcher(unsigned int irq)
03:27 PM cerna: The "1902.clang-wheezy-rtai-i386" builder doesn’t seem to like being queried: http://buildbot.linuxcnc.org/buildbot/json/builders/1902.clang-wheezy-rtai-i386/builds/3988/?as_text=1
03:30 PM cerna: The next one too: http://buildbot.linuxcnc.org/buildbot/json/builders/1903.clang-wheezy-amd64/builds/3985/?as_text=1
03:31 PM cerna: Are they even functioning?
03:33 PM seb_kuzminsky: the clang builders are building using clang and reporting the results successfully
03:33 PM seb_kuzminsky: not sure about the json interface...
03:41 PM seb_kuzminsky: it's choking on the ClangResults output of the final build step
03:44 PM seb_kuzminsky: and that link is broken on the waterfall page too
03:49 PM seb_kuzminsky: i have an idea for how to fix that, hang on
03:52 PM cerna: When I skip these two, I can get all the others fine.
04:58 PM rene_dev_: jepler I need to replace minigl/glew with pyopengl and libepoxy in master, how do I add the dependency?
04:59 PM rene_dev_: libepoxy-dev and libepoxy0
04:59 PM rene_dev_: python-opengl / python3-opengl
05:14 PM andypugh: rene_dev_: It’s a file in the debian golder.
05:14 PM andypugh: folder
05:30 PM jepler: rene_dev_: debian dependencies are created by a script with some ".in" files that are substituted similar to what src/configure does
05:31 PM jepler: rene_dev_: I'm guessing that pyopengl and libepoxy don't exist in all supported debian/ubuntu versions, so your next stop is the debian/configure script
05:32 PM jepler: easiest is to go to the stanzas for the modernish distributions and add them to EXTRA_BUILD=
05:34 PM jepler: in theory runtime depends like shared libs and python packages are guessed during the package build process (that's what ${python:Depends}, ${misc:Depends} does) but if you discover the binary package is missing a dependency then you'll have to add that too. There doesn't seem to be a catch-all place for "generic extra stuff" so you might need to create it and add it to the big sed command near the bottom of
05:34 PM jepler: debian/configure
05:34 PM jepler: good luck, let me know if this isn't enough info to get you going
05:35 PM jepler: (if it ends up being a missing dependency on python-opengl or python3-opengl it may be better to pursue that as a question of why ${python:Depends} is not finding it)
06:22 PM Tom_itx is now known as Tom_L
06:37 PM rene_dev_: Jepler I think they do, I want to go from 20 year old gl stuff to 10 year old gl stuff
06:45 PM jepler: hehe if they do all have it, so much the better
06:46 PM rene_dev_: Maybe not on wheezy
06:46 PM rene_dev_: but Jessie has them
06:46 PM rene_dev_: I want to unify and clean up the gl stuff
06:47 PM rene_dev_: https://packages.debian.org/source/jessie/libepoxy
07:20 PM -!- #linuxcnc-devel mode set to +v by ChanServ
07:20 PM -!- #linuxcnc-devel mode set to +v by ChanServ
07:21 PM seb_kuzminsky: linuxcnc-build: force build --branch=master 1902.clang-wheezy-rtai-i386
07:21 PM linuxcnc-build: build #3989 forced
07:21 PM linuxcnc-build: I'll give a shout when the build finishes
07:27 PM rene_dev_: jepler its not available on wheezy...
07:40 PM linuxcnc-build: Hey! build 1902.clang-wheezy-rtai-i386 #3989 is complete: Warnings [8warnings compile]
07:40 PM linuxcnc-build: Build details are at http://buildbot.linuxcnc.org/buildbot/builders/1902.clang-wheezy-rtai-i386/builds/3989
07:41 PM seb_kuzminsky: oops
07:42 PM -!- #linuxcnc-devel mode set to +v by ChanServ
07:43 PM seb_kuzminsky: linuxcnc-build: force build --branch=master 1902.clang-wheezy-rtai-i386
07:43 PM linuxcnc-build: build #3990 forced
07:43 PM linuxcnc-build: I'll give a shout when the build finishes
07:43 PM seb_kuzminsky: linuxcnc-build: force build --branch=master 2000.docs
07:43 PM linuxcnc-build: build #5486 forced
07:43 PM linuxcnc-build: I'll give a shout when the build finishes
07:50 PM linuxcnc-build: Hey! build 2000.docs #5486 is complete: Warnings [8warnings compile]
07:50 PM linuxcnc-build: Build details are at http://buildbot.linuxcnc.org/buildbot/builders/2000.docs/builds/5486
08:01 PM linuxcnc-build: Hey! build 1902.clang-wheezy-rtai-i386 #3990 is complete: Warnings [8warnings compile]
08:01 PM linuxcnc-build: Build details are at http://buildbot.linuxcnc.org/buildbot/builders/1902.clang-wheezy-rtai-i386/builds/3990
08:16 PM jepler: ooh are you moving the doc build out of the individual buildbot runs? should be a good timesaver.
08:19 PM seb_kuzminsky: no, all rip builders build the docs
08:20 PM seb_kuzminsky: we wanna know if they fail on some particular distro
08:20 PM seb_kuzminsky: oh, but the builder that builds the docs that get uploaded, it does only that
08:20 PM seb_kuzminsky: whoa what kind of sentence was that
08:21 PM jepler: oh, okay
08:21 PM jepler: the package builders will build all the pdf docs of course
08:21 PM seb_kuzminsky: cerna: i think all the clang links should work now
08:21 PM jepler: I was wondering because I saw this in configure: scan-build ./configure --disable-simulator --disable-build-documentation;
08:22 PM seb_kuzminsky: oh, the clang builder doesn't build docs
08:22 PM jepler: in http://buildbot.linuxcnc.org/buildbot/builders/1902.clang-wheezy-rtai-i386/builds/3990/steps/configuring/logs/stdio
08:22 PM jepler: oh just the clang builder, gotit
08:22 PM seb_kuzminsky: yeah, clang doesn't run the regular rip "build factory", it's got its own special thing going
08:23 PM seb_kuzminsky: cerna: http://buildbot.linuxcnc.org/buildbot/json/builders/1902.clang-wheezy-rtai-i386/builds/3990/?as_text=1
08:34 PM cerna: Deleting from skip array in my bash script and testing...
08:42 PM cerna: Huh, the older builds like and http://buildbot.linuxcnc.org/buildbot/json/builders/1902.clang-wheezy-rtai-i386/builds/3988/?as_text=1 and mainly http://buildbot.linuxcnc.org/buildbot/json/builders/1902.clang-wheezy-rtai-i386/builds/_all/?as_text=1 are still erroring - with that my script isn't counting. Will have to think it through some more.
08:42 PM cerna: Well, more playing tomorrow.
08:44 PM seb_kuzminsky: yeah, old builds with broken links will stay broken
08:45 PM seb_kuzminsky: new builds from now on should have good links
08:45 PM seb_kuzminsky: if you find any other problems please let me know!