#linuxcnc-devel Logs

Mar 07 2018

07:30 AM jepler: rene-dev: from __future__ import print_function + print(..., file=) would be preferred
07:33 AM rene-dev: Ok, Whats the difference? But I cannot use future currently... will look into it.
07:33 AM jepler: rene-dev: for us to make progress on actually taking parts of your python3 work into master branch, we need pull requests where the system builds and works. Please make a pull request with just a small number of commits that leave a normal python2 build working, because we can take those into master branch immediately. When you make a pull request with 6 commits, and 3 of them aren't working
07:33 AM jepler: commits that compile on travis-ci, we can't accept the pull request.
07:33 AM jepler: rene-dev: print() and file.write() have different semantics when it comes to appending newlines.
07:35 AM rene-dev: Yes I know that. hazzy wanted to test it, and I just pushed it... test fails due to some newline problem. I will clean that up.
07:36 AM rene-dev: Yes, that’s why the test fails.
07:36 AM rene-dev: Im not expecting you to merge that now...
07:44 AM jepler: what I'm saying is, there are parts we CAN merge now
07:44 AM jepler: like, a commit which ONLY fixes the print() syntax in configure.ac, for instance, we can accept
07:45 AM jepler: but if it is in a PR with other things, we can't accept it
07:45 AM jepler: also if the goal is just to share code, you don't have to create a PR at all
07:55 AM rene-dev: Yes, I know... while doing it I didnt know about the newline issue, until the test failed. I think everything that is there can be merged once I fix that. Will do that when I get home.
07:59 AM rene-dev: Pip can install from git, so its easy for other devs to install your yapps fix
08:01 AM jepler: we would prefer not to merge a sequence of 6 commits where 3 are so broken they do not compile
08:01 AM jepler: are you familiar enough with git to use "rebase" so that you can make a pull request with only working commits?
08:02 AM jepler: later on, we frequently want to use "git bisect" to find when a problem was introduced. commits which don't even compile reduce the usefulness of "git bisect", because they are untestable later
08:05 AM jepler: I would like to see a pull request more like this one, in which I believe (because I tested locally) that each intermediate commit compiles. (however, travis-ci will only check the last commit, since I pushed all the commits at the same time)
08:06 AM jepler: afk
08:07 AM sync: hm, I usually like to see the history behind the latest commit which gets lost if you rebase
08:14 AM cradek: that's a valid concern, but unbuildable commits are a problem that lingers forever since you run into them arbitrarily far in the future when bisecting. unbuildable checkpoints aren't usually useful parts of history that you would later want to examine.
08:15 AM cradek: people sure have different feelings about what commits should represent. some think they should be faithful unchangeable history, others think they should show logical/incremental progress with false starts and mistakes gone
08:15 AM cradek: for using git tools like bisect, the latter is much nicer to have
09:14 AM jepler: rene-dev: I am very glad you are undertaking this work related to Python3. Please don't interpret my earlier remarks as intended to be discouraging. I just wanted to be direct and clear about what kind of pull requests are easiest to accept into LinuxCNC.
09:15 AM jepler: rene-dev: It has happened in the past both that we were ultimately unable to accept large amounts of work that were done, because they were combined with changes we did not want (this led to the machinekit fork); and that we carelessly accepted changes with a lot of broken commits in the history (joints-axes). I would very much like to avoid either of those with this python3 work.
09:32 AM hazzy-dev: On a fresh install of the LCNC Stretch r10 ISO I did not have to install yapps for `./configure` to complete successfully.
09:32 AM hazzy-dev: I did have to install the `python3-tk` package.
09:32 AM hazzy-dev: I start to run into compile errors with `taskclass.cc`
09:32 AM hazzy-dev: those look like binding errors
09:58 AM rene-dev: jepler I get what you say, but maybe our opinions differ on how to handle that. I usually commit early and often, as soon as something works before I break it again, so I keep track what Im doing. A feature branch is imho exactly for that, so you can fuck around until everything works, thats one of the big advantages of git. That work in progress doesnt affect anyone.
09:58 AM rene-dev: maybe I made the mistake of creating a pull request...
10:00 AM rene-dev: I know what you are talking about, as I am a software developer myself. But I know of many pepole that do changes to linuxcnc, and they dont understand the procedure to get them upstream, or just cant be bothered.
10:00 AM rene-dev: The tests are really really good, and I even think there should be even more. but there are 2 big problem with them:
10:01 AM rene-dev: they are really really slow, and they cant be run in parallel.
10:01 AM rene-dev: if you cancel them, strange things will happen.
10:01 AM rene-dev: I think machinekti did something that allows multiple instances of linuxcnc running, that would sure speed up the tests...
10:16 AM seb_kuzminsky: rene-dev: i like the "commit early and often" way of developing, and i think most of us do
10:17 AM seb_kuzminsky: i called it the "stream of consciousness development model" in a talk i gave at bsdcon years ago
10:18 AM seb_kuzminsky: i think what jepler is saying is that that history is useful for a developer while the feature/bugfix is in progress, but when the work is finished and the developer wants to offer a PR of it, that history should be cleaned up so that all the false starts and experiments and new bugs are no longer in the history
10:20 AM rene-dev: yes, we can do that now. as of yesterday, nobody knew what we were doing, and tried different approaches.
10:21 AM seb_kuzminsky: sure, that's how it's done :-)
10:21 AM rene-dev: it also doesnt help that Im not a python expert :D
10:21 AM seb_kuzminsky: then rebase, with judicious splitting and squashing and reordering of commits makes it look to everyone else like you're a genius who knew what they were doing all along
10:22 AM rene-dev: I can do it, but its a very stackoverflow driven workflow, and I tend to do things more like you would do them in c :D
10:22 AM rene-dev: so, I create a new PR and cherry pick jepler s commits for halcompile?
10:23 AM seb_kuzminsky: i'm not sure... jepler?
10:24 AM seb_kuzminsky: i would expect jepler's PR to stand on its own, and your work to build on it, but probably you and jeff should work that out and i should shut up
10:25 AM seb_kuzminsky: i think in general cherry-picking from one PR to another isn't the way to go
10:25 AM rene-dev: yeah, but I need his changes to continue my work :D so unless you merge it now into master, I need to find another way
10:26 AM seb_kuzminsky: one option is to base your work on the tip of jepler's PR branch, then rebase it to master when that branch gets merged
10:27 AM rene-dev: yeah, that sounds like my cherrypicking approach
10:29 AM seb_kuzminsky: it's different because it doesn't move his commit to your branch
10:29 AM seb_kuzminsky: if you meant "cherrypick his commit to my branch, then rebase it out after his branch is merged and i rebase onto master", then they act the same
10:31 AM seb_kuzminsky: jepler's commit 6b9c12cf sounds like it's not functional in stretch, is that true?
10:32 AM seb_kuzminsky: that's a bit of a bummer if so...
10:39 AM rene-dev: it is
10:39 AM rene-dev: thats all python2 compatible, otherwise the tests wont pass
10:40 AM rene-dev: ah
10:41 AM rene-dev: only if you run it with python3
10:45 AM rene-dev: https://github.com/LinuxCNC/linuxcnc/pull/283
10:45 AM rene-dev: same stuff. pull request sitting there, and pepole work on it until it works.
10:46 AM rene-dev: there are loads of commits that break the tests.
10:46 AM seb_kuzminsky: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=762804
10:47 AM seb_kuzminsky: i've been in communication with zultron several times about that PR, he has told me he's still working on it and that it's not ready to merge yet
10:48 AM seb_kuzminsky: that PR is pretty confusing, i think partially because the destination branch is wrong ("statetags" instead of "master")
10:49 AM seb_kuzminsky: i think that's why it includes tons of commits that have nothing to do with the statetags work
10:49 AM rene-dev: I was communicating with jepler hazzy and TurBoss yesterday about mine, and we worked it all out together
10:49 AM rene-dev: yes, I see that. looking at the diff now...
10:49 AM seb_kuzminsky: obviously that'll have to be cleaned up before we can review it and merge it
10:51 AM jepler: seb_kuzminsky: so what I was trying to describe in that commit message is: That sequence gets the halcompile itself to all be python3 syntax, but it can't actually be run due to missing python3 support in the yapps package in debian stretch. rene-dev and others have installed yapps with pip, if I understand correctly
10:51 AM seb_kuzminsky: jepler: gotcha
10:51 AM rene-dev: yes, I did. but for your version to work with python3 I need your patched yapps?
10:52 AM seb_kuzminsky: according to the debian bug i linked to above, yapps2 is python3 compatible, but debian (up to and including sid) only has yapps 2.1
10:52 AM jepler: rene-dev: I didn't actually try that, because I don't want to potentially break my development environment by installing things with pip
10:53 AM jepler: seb_kuzminsky: there's some python3 compatible fork of pip on github, but there's an important bug in it that we hit right away that makes halcompile not work. I have a pull request in to that fork, but it seems inactive so it's not likely to be applied right this second.
10:53 AM seb_kuzminsky: jepler: yapps isn't used at runtime, is it? it's just for producing halcompile from halcompile.g at build time?
10:53 AM jepler: yapps has a runtime part too.
10:54 AM jepler: rene-dev: does #417 look good to you? should we just merge it so that you can base your work on top of it?
10:54 AM jepler: it makes travis-ci happy so we think it won't break anything
10:55 AM seb_kuzminsky: jepler: 'git grep yapps' only finds the build-time usage, what am i missing?
10:55 AM seb_kuzminsky: does it embed itself into the program it's producing?
10:55 AM jepler: epler@babs:~/src/linuxcnc$ grep 'from yapps' bin/halcompile
10:55 AM jepler: from yapps import runtime
10:55 AM seb_kuzminsky: ah, yes it does
10:56 AM * seb_kuzminsky grumbles
10:56 AM jepler: $ PYTHONPATH=$HOME/src/yapps/ python3 bin/halcompile --help
10:56 AM jepler: halcompile: Build, compile, and install LinuxCNC HAL components
10:56 AM jepler: well this looks promising though
10:57 AM seb_kuzminsky: is that yapps2 2.2?
10:57 AM seb_kuzminsky: or some frankenversion from github?
10:57 AM jepler: File "bin/halcompile", line 649, in prologue
10:57 AM jepler: if names.has_key(name):
10:57 AM jepler: AttributeError: 'dict' object has no attribute 'has_key'
10:57 AM jepler: hmph
10:58 AM jepler: seb_kuzminsky: it's some frankenversion
10:58 AM jepler: github.com/jepler/yapps branch issue4
11:00 AM hazzy-dev: rene-dev: I did not have to install yapps with pip, and I did not see any errors related to halcompile. I don't think I missed anything. I don't even have pip installed
11:06 AM jepler: $ PYTHONPATH=$HOME/src/yapps/:$PYTHONPATH python3 ../bin/halcompile --install hal/components/and2.comp
11:06 AM jepler: Compiling realtime and2.c
11:06 AM jepler: yay!
11:06 AM jepler: Linking and2.so
11:06 AM jepler: cp and2.so /home/jepler/src/linuxcnc/rtlib/
11:12 AM seb_kuzminsky: is there a p2/p3 well-maintained parser we can use instead of yapps?
11:15 AM jepler: packaging our own yapps has got to be lower impact than making halcompile use a different parser :-/
11:16 AM seb_kuzminsky: :-/
11:21 AM jepler: I don't actually see machinekit running tests in parallel fwiw
11:21 AM jepler: scripts/runtests is still serial, .travis/mk_runtests IS using containers but it still looks serial to me
11:22 AM jepler: I know it's a technology mah or zultron or somebody demoed..
11:26 AM hazzy-dev: I don't understand this stuff very well at all, but is seems like on debian 9 the yapps I installed via `sudo apt-get install yapps2` (which is the dep for master) also works for compiling the python3 branch. I am confused as to what the problem is (but that is nothing new :))
11:27 AM rene-dev: I didnt say they run tests in paralell, but I think they can do it. and I think it would be a good thing to do...
11:34 AM jepler: containerizing the tests also means less chance of damage to the real system, like leftover SHM segments, mutexes, /tmp/linuxcnc.lock, etc
11:37 AM rene-dev: yep
11:40 AM seb_kuzminsky: can you load kernel modules from containers? i thought not
11:40 AM seb_kuzminsky: but it seems like we're straying further from rtai every year
11:41 AM jepler: yeah this wouldn't work for kernel-model realtime
11:41 AM jepler: er it wouldn't work, yeah
11:43 AM rene-dev: the tests dont need to run in kernel... they dont even need realitme
11:43 AM seb_kuzminsky: running linuxcnc on rtai realtime needs to load kernel modules
11:43 AM seb_kuzminsky: all the realtime stuff runs in the kernel on that platform
11:43 AM rene-dev: I know, but test dont need to run in rtai
11:44 AM seb_kuzminsky: you're suggesting we load the realtime part of the test outside the containers, then run the userspace part inside?
11:44 AM seb_kuzminsky: that's a big divergence from hos the tests are set up at the moment, and all the tests one still share one realtime context
11:45 AM jepler: There are some tests which are very unlikely to have different results just due to selecting rtai realtime, like the stuff in interp/
11:45 AM seb_kuzminsky: yeah, maybe the non-realtime tests could all run in parallel in their own container or something
11:45 AM jepler: on the other hand, it's quite valid to say we have to run tests/and.0 on all our realtime platforms
11:45 AM seb_kuzminsky: i gots to go, nice chatting with you all
11:45 AM seb_kuzminsky: bbl
11:46 AM rene-dev: apparently with machinekit you can just run multiple instances side by side. but I never tried, nor do I know how they do it.
11:47 AM rene-dev: (I never touched machinekit=
11:47 AM rene-dev: )
11:48 AM jepler: it's not clear to me how that would work with RTAI kernel realtime
11:51 AM rene-dev: I dont know if they even still support rtai
11:53 AM jepler: based on the block starting AC_DEFUN([_cfg_flavor_params], in their configure.ac, I think they do (also xenomai in a kernel realtime mode)
11:54 AM jepler: huh their rt-preempt flavor doesn't set the FLAVOR_DOES_IO flag. I wonder if that doesn't have the obvious meaning (I took it to mean: can control hardware)
11:54 AM rene-dev: then maybe I was told wrong. Somebody mentioned it at the linuxcnc meeting here in germany
11:55 AM jepler: elsewhere they have
11:55 AM jepler: rtapi/rtapi_compat.h:#define FLAVOR_DOES_IO RTAPI_BIT(0) // userland: whether iopl() needs to be called
11:55 AM jepler: rtapi/rtapi_compat.h:#define RTPREEMPT_FLAVOR_FLAGS (FLAVOR_DOES_IO)
11:55 AM rene-dev: what do the flavour flags do?
11:55 AM jepler: so it might also be that configure.ac is harmlessly out of date? I don't know, I would have to investigate it in detail and I don't really care to
11:56 AM jepler: rene-dev: one major thing in machinekit that is not in linuxcnc was the ability to build all these different realtime ways of working from one source tree in one build. I think this is all related to that.
11:57 AM jepler: (we sort of have that now, a single build can do POISX userspace realtime, RTAI userspace realtime, and Xenomai userspace realtime; but a single build can't support userspace + kernelspace, or multiple kernelspace options)
11:58 AM rene-dev: yeah, there are a few thing that would be good to bring back to linuxcnc
12:01 PM jepler: rene-dev: btw thank you for taking the time to tell me about your viewpoint on patches and pull requests. I also make a lot of intermediate commits that are in various stages of completeness, but prefer to use rebase to hide the evidence before making pull requests.
12:02 PM jepler: rene-dev: you mentioned "here in germany". If you don't mind me asking, what city is home for you?
12:02 PM rene-dev: ok, I think all the confusion started because I made the pull request.
12:02 PM jepler: I'm in Lincoln, Nebraska, USA
12:02 PM rene-dev: I didnt expect you to merge it...
12:03 PM jepler: Yes
12:03 PM rene-dev: I only did that so pepole could work on it, and there was a place to discuss the required changes...
12:04 PM rene-dev: Im from Münster, Germany and there is a annual linuxcnc meeting in stuttgart which I dont host, but sort of organise.
12:04 PM rene-dev: its in a big workshop, and it sometimes pepole bring machines, where pepole then work on
12:04 PM jepler: I would love to come sometime, but it hasn't worked out yet. I have friends and family in Frankfurt and Dresden, and manage to make it to Europe once in every few years.
12:05 PM rene-dev: ok, cool, would be good to have you there. norbert is there every year, and last year andyp as well.
12:08 PM jepler: I think I used to change trains in Stuttgart, when I was travelling between Besancon (in France) and Berlin. I don't think I ever stopped and enjoyed the city, though.
12:08 PM rene-dev: there is a robot I retrofitted with stmbl, a weiler lathe I worked on with andy last year, and a big axa machining centre, and a few routers
12:08 PM rene-dev: there is not much to enjoy, its all a big construction site...
12:09 PM rene-dev: they decided to move their train station underground, and its not going well
12:11 PM jepler: rene-dev: I added a label "for-discussion-only" to that pull request. Maybe using that can help keep everyone on the same page?
12:11 PM jepler: I don't know github well enough to know whether you can add a label to your own issue, or whether a member of the linuxcnc "team" has to do it. In the latter case it's not actually that useful...
12:14 PM jepler: afk
12:31 PM Tom_itx is now known as Tom_L
02:51 PM rene-dev: jepler https://flic.kr/s/aHskP9YBrE
02:56 PM jepler: thanks rene-dev
02:58 PM seb_kuzminsky: that looks like fun
02:58 PM rene-dev: it is :)
02:58 PM rene-dev: http://wiki.linuxcnc.org/cgi-bin/wiki.pl?LinuxCNC_Integrator_Meeting_2014_Germany
02:58 PM rene-dev: but the pictures are lost :(
02:58 PM rene-dev: http://wiki.linuxcnc.org/cgi-bin/wiki.pl?LinuxCNC_Integrator_Meeting_2013_Germany
03:15 PM jepler: rene-dev: I suggest configure ... --disable-check-runtime-deps ... to avoid the pango/cairo check
03:16 PM jepler: happy coincidence, it'll skip that block of code which is causing trouble when you use python3
03:17 PM rene-dev: ah, ok. thats a workaround, but later its better to check
03:39 PM jepler: it seems you can get some useful warnings but a possibly working LinuxCNC by editing Makefile.inc to change "PYTHON=python" to "PYTHON=python -3" before building
03:40 PM jepler: e.g., Syntax checking python script axis
03:40 PM jepler: emc/usr_intf/axis/scripts/axis.py:1744: SyntaxWarning: tuple parameter unpacking has been removed in 3.x
03:40 PM jepler: def dist((x,y,z),(p,q,r)):
03:40 PM rene-dev: ah, to check all the syntax, but keep the bindings on 2?
03:41 PM rene-dev: 2to3 does a good job of doing that automaticly
03:41 PM jepler: it's not "all" the syntax, it's documented as follows: Warn about Python 3.x incompatibilities that 2to3 cannot trivially fix.
03:41 PM jepler: so for instance, no warnings about 'print'
03:41 PM rene-dev: 2to3 fixes print
03:41 PM rene-dev: the tabs you need to do yourself
03:42 PM rene-dev: py2 interprets a tab as 8 spaces
04:02 PM pcw_mesa: Is master currently broken?
04:02 PM pcw_mesa: rtapi_shmem_new failed due to shmget(key=0x00000064): Invalid argument
04:02 PM pcw_mesa: MOTION: rtapi_shmem_new failed, returned -22
04:02 PM pcw_mesa: MOTION: init_comm_buffers() failed
04:02 PM pcw_mesa: motmod: rtapi_app_main: Operation not permitted (-1)
04:06 PM jepler: pcw_mesa: buildbot hasn't alerted us to any problems, and I've been doing runtests locally today too. but only with posix non-realtime
04:06 PM jepler: shmget can return EINVAL in two cases
04:06 PM jepler: EINVAL A new segment was to be created and size is less than SHMMIN or greater than SHMMAX.
04:06 PM jepler: EINVAL A segment for the given key exists, but size is greater than the size of that segment.
04:07 PM jepler: seems like the second might occur if an other-version instance didn't get fully cleaned up. if that was the cause, 'ipcrm -M 0x64' or a reboot might clean it up
04:10 PM rene-dev: working here, with rt preempt
04:20 PM pcw_mesa: Just changed from 2.7 to 2.8 and maybe 2.7 didn't shut down properly (it left a hal show instance running)
04:20 PM pcw_mesa: I closed the halshow pane but maybe something was busted
04:22 PM pcw_mesa: ipcrm fixed it, thanks!
04:38 PM seb_kuzminsky: the easiest way to clean up ipc leftovers is "halrun -U"
04:39 PM rene-dev: hmm, maybe adding that to the tests would be a good idea?
04:46 PM linuxcnc-build: build #1272 of 4017.5.deb-wheezy-armhf is complete: Failure [4failed shell_3] Build details are at http://buildbot.linuxcnc.org/buildbot/builders/4017.5.deb-wheezy-armhf/builds/1272 blamelist: Jeff Epler <jepler@unpythonic.net>, Jeff Epler <jepler@gmail.com>
06:45 PM jepler: /bin/sh: 1: xargs: Bad address
06:45 PM jepler: /usr/bin/ld:objects/multiclick.ver:2: syntax error in VERSION script
06:45 PM jepler: is this the garden variety ARM flakiness or did I break something?
06:46 PM jepler: linuxcnc-build: force build --branch=master 0000.checkin
06:46 PM linuxcnc-build: build forced [ETA 1h26m16s]
06:46 PM linuxcnc-build: I'll give a shout when the build finishes
07:47 PM Chris_Morley: rene-dev: multiple linuxcnc instances:https://www.youtube.com/watch?v=Zk_KiWrhi5Y
07:54 PM jepler: jepler@babs:~/src/linuxcnc$ find tests -name "test.hal" | xargs -P8 -n4 unshare -U -i -p -f --mount-proc ./scripts/rip-environment runtests
07:55 PM jepler: .. with some other local changes
07:56 PM jepler: I briefly had 7 milltasks running, but then one of the tests failed, hm
08:03 PM jepler: ah linuxcncrsh uses tcp/ip
08:04 PM jepler: that's too bad, those are some of the slooooowwwwwweeeeeesssssstttttt tests
08:04 PM rene-dev: exactly :D
08:04 PM rene-dev: why do some take so long?
08:07 PM jepler: adding -n to unshare net doesn't help, so my theory must not be accurate
08:11 PM jepler: weird, *just* running this test in 'unshare' breaks it
08:13 PM jepler: .. because network device lo0 is not set up
08:18 PM linuxcnc-build: Hey! build 0000.checkin #5412 is complete: Success [3build successful]
08:18 PM linuxcnc-build: Build details are at http://buildbot.linuxcnc.org/buildbot/builders/0000.checkin/builds/5412
08:22 PM jepler: yay it's just flaky arm HW that caused the failure earlier :-/
09:00 PM jepler: $ find tests/mdi-* -name "test.sh" | xargs -P4 -n1 unshare -U -i -p -f -n -r --mount-proc env RTAPI_UID=65534 sh -c 'ifconfig lo up && ./scripts/rip-environment runtests "$@" ' --
09:00 PM jepler: well this isn't getting any clearer. thanks linux.
09:16 PM jepler: but it can run those 4 tests in less than half the time
09:39 PM jepler: pushed to github.com/jepler/linuxcnc in branch parallelize-tests-hacks -- the top commit describes in some detail my findings.