#linuxcnc-devel Logs

Jul 04 2018

06:25 AM jepler: interesting, somebody's working on aarch64
07:19 AM linuxcnc-build_: build #3654 of 1405.rip-wheezy-armhf is complete: Failure [4failed compile runtests] Build details are at http://buildbot.linuxcnc.org/buildbot/builders/1405.rip-wheezy-armhf/builds/3654 blamelist: Jeff Epler <jepler@gmail.com>, Anton Midyukov <antohami@altlinux.org>
07:23 AM linuxcnc-build_: build #5502 of 0000.checkin is complete: Failure [4failed] Build details are at http://buildbot.linuxcnc.org/buildbot/builders/0000.checkin/builds/5502 blamelist: Jeff Epler <jepler@gmail.com>, Anton Midyukov <antohami@altlinux.org>
03:14 PM andypugh: Would you expect to be able to use gdb on a hostmot2 driver?
03:14 PM andypugh: I have never really tried to use gdb before, so I can’t really tell if my failure is incompetence or impossibility.
03:15 PM andypugh: (I have still had no luck with valgrind)
03:17 PM andypugh: I wonder if there is anything in the memory map dump that can be used (with addr2line) to pinpoint a line of code?
03:33 PM pcw_home: Andy: did you see this? <jepler> Mind putting up the valgrind output on a pastebin?
03:45 PM andypugh: There isn’t much to paste
03:46 PM andypugh: paste.ubuntu.com//p/kbp8WdkxTq
03:48 PM andypugh: And I wish I had a global clipboard to paste between PCs. Or that the pastebin used a shorter, or not case-sensitive address
03:49 PM jepler: take off the setuid bit manually just for running rtapi_app maybe
03:49 PM andypugh: Where?
03:49 PM jepler: easiest way: rm ../bin/rtapi_app; make; absolutely don't sudo make setuid
03:50 PM andypugh: OK. Watch this space
03:52 PM andypugh: http://paste.ubuntu.com/p/bkWFYpbG5w
03:53 PM andypugh: No, wait
03:53 PM jepler: must be a typo'd link, sadly
03:54 PM andypugh: http://paste.ubuntu.com/p/bkWFYpnG5w/
03:55 PM andypugh: I was hoping to use gdb, but haven’t found any files that gdb can see to set breakpoints in.
03:55 PM andypugh: (I realise that expecting to set a breakpoint in a comp is never going to work, but I have also tried bspi.c)
03:56 PM jepler: ==32556==
03:56 PM jepler: Running with fallback_uid. getuid()=1000 geteuid()=0
03:56 PM jepler: bind: Permission denied
03:57 PM andypugh: The only idea I have left at the moment is to put a debug print before very “free” to see which one is the problem.
03:57 PM jepler: OK I get the same here
03:57 PM jepler: most likely it is not the free that is the problem
03:57 PM jepler: it is an errant store before the start or after the end of some unrelated block
03:58 PM jepler: $ sudo env RTAPI_UID=`id -u` RTAPI_FIFO_PATH=$HOME/.rtapi_fifo valgrind bin/rtapi_app
03:58 PM andypugh: I have tried changing the order of the variables in the comp, because I can’t think of any reason why one particular set of variables should be the issue
03:58 PM jepler: try with this second environment variable specified please
03:59 PM andypugh: OK
03:59 PM andypugh: I have a plan. I will ssh into the LinuxCNC machine...
04:01 PM andypugh: That might be working
04:03 PM andypugh: Well, it crashed in a different way...
04:04 PM andypugh: I think being split between two PCs might have upset things.
04:09 PM andypugh: https://paste.ubuntu.com/p/YHXGdzNy4f/
04:09 PM andypugh: But, this is crashing before the mesa_7i65 driver is even loaded.
04:09 PM jepler: yeah, poop
04:09 PM andypugh: though it is just about possible that it explains _why_ the 7i65 driver fails? Maybe the initial setup is faulty?
04:09 PM jepler: I swear this used to work once upon a time
04:10 PM jepler: I doubt it. that crash probably indicates you just can't do outb() with valgrind
04:10 PM andypugh: I am also now in posix non-realtime. Is that the setuid thing?
04:10 PM jepler: I bet if I tried with hardware it was only ethernet or pci hardware
04:10 PM jepler: > Note: Using POSIX non-realtime
04:10 PM jepler: hm that could also be why it's failing, it's true
04:11 PM jepler: in src/rtapi/uspace_rtapi_app.cc try telling it you REALLY want to use realtime
04:11 PM jepler: static RtapiApp *makeApp()
04:11 PM jepler: {
04:11 PM jepler: - if(euid != 0 || harden_rt() < 0)
04:12 PM jepler: + if(0)
04:12 PM jepler: {
04:12 PM jepler: i.e., change the if condition so that it never operates
04:12 PM andypugh: OK
04:15 PM andypugh: Just the same, except in POSIX realtime
04:16 PM andypugh: pcw_home: Is there a 5i25 firmware with bspi?
04:17 PM jepler: andypugh: sigh I'm feeling worse and worse sending you on this valgrind rabbit trail that's going nowhere.
04:17 PM jepler: andypugh: I suspect that trying variants on 'sudo gdb rtapi_app' will not help either
04:17 PM jepler: possible exception: attaching with gdb AFTER loading the component you're interested in
04:18 PM jepler: like, try adding a sleep(30) to the component startup
04:18 PM jepler: then you have 30 seconds to 'sudo gdb rtapi_app 1234' where 1234 is the pid of rtapi_app
04:18 PM jepler: and maybe that will tell you something
04:18 PM andypugh: will that go straight to a breakpoint?
04:19 PM andypugh: Or am I misunderstanding?
04:19 PM pcw_home: andypugh: I can make one up
04:20 PM andypugh: pcw_home: If it’s not too much trouble. It looks like it might possibly get me further
04:20 PM jepler: andypugh: it should stop it during the sleep(30)
04:21 PM andypugh: OK. And is there are easier way to find the pid than to start (and then exit) top?
04:31 PM jepler: that's sure one way to do it
04:31 PM andypugh: OK, pidof works
04:31 PM andypugh: I think I had it working, maybe, but then HAL got bored of waiting
04:32 PM andypugh: (I don’t know any GDB commands. C to continue?
04:34 PM andypugh: pcw_home: How about I just give both 7i65 owneres their money back?
04:34 PM andypugh: :-(
04:36 PM pcw_home: http://freeby.mesanet.com/5i25_bsp4.bit
04:37 PM jepler: hm2_allocate_tram_regions is sure a place where a fugly memory allocation problem could lurk
04:38 PM pcw_home: It was working, not sure when the breakage happened (and it does work with hm2_eth)
04:38 PM pcw_home: (works to the extent that it does not crash, it didn't try running a 7I65 on Ethernet yet)
04:39 PM pcw_home: on PCI when it works, everything seems OK
04:40 PM jepler: pcw_home: pci crashes sometimes, you mean?
04:41 PM pcw_home: yes but only when exiting in my testing
04:42 PM pcw_home: so maybe different behaviour than hm2_epp
04:43 PM pcw_home: Andy are you using RTAI? (or did you try to fix the compile issues of mesa_7i65 on uspace)
04:45 PM andypugh: I am running uspace
04:45 PM andypugh: ANd it crashes every time, epp or PCI (2 out of 2 times with PCI now)
04:46 PM pcw_home: maybe RTAI/uspace difference
04:46 PM jepler: what would a typical loadrt line for mesa_7i65 look like?
04:46 PM pcw_home: same crash?
04:46 PM andypugh: jepler: With rtapi_set_msg_level(5) you can see the TRAM buffer allocation, and that looks OK and consistent.
04:47 PM jepler: oh I guess I need a fw too
04:47 PM andypugh: loadrt hostmot2 / loadrt hm2_pci / loadrt mesa_7i65 bspi_chans=hm2_5i25.0.bspi.0
04:47 PM andypugh: jepler: Do you have any mesa HW?
04:48 PM jepler: 5i24 pci card is handy
04:48 PM jepler: no 7i65 obviously
04:48 PM jepler: but .. I'm about to be called away for 3 hours
04:48 PM pcw_home: I think there's 7I65 firmware for the 5I24 already
04:49 PM andypugh: You don’t need the 7i65, and there is a 2x7i65 firmware for both sizes of 5i24 in the firmware pack
04:49 PM jepler: 5i24/configs/hostmot2/5i24_16_2x7i65.bit
04:49 PM jepler: OK
04:49 PM pcw_home: Yeah the driver doesn't know if its there or not
04:50 PM pcw_home: analog input will be funny...
04:50 PM andypugh: You don’t even need to start the threads
04:51 PM jepler: hm2_5i24.0: initialized AnyIO board at 0000:06:00.0
04:51 PM jepler: *** Error in `/home/jepler/src/linuxcnc/master/bin/rtapi_app': free(): corrupted unsorted chunks: 0x00005591b9661d60 ***
04:51 PM jepler: yay?
04:52 PM jepler: http://paste.debian.net/1032118/
04:52 PM jepler: ==23129== Invalid write of size 1
04:52 PM jepler: ==23129== at 0x6EBDFFF: hm2_allocate_tram_regions (tram.c:115)
04:52 PM andypugh: If I was doing it now there wouldn’t even be a mesa_7i65 driver, it would be loadrt hm2_pci bspi_0=%fo8%fi8…” like the serial encoders do
04:53 PM jepler: it at least states an exact source code line where the problem occurs
04:53 PM pcw_home: Didn't Cliff Blackburn make some recent changes in either hm2_bspi or mesa_7i65?
04:53 PM andypugh: Why is it so much more helpful to you?
04:53 PM jepler: actually a total of 3 errors were printd
04:53 PM jepler: http://paste.debian.net/1032119/
04:54 PM jepler: andypugh: There must be some step to doing this that I failed to tell you. I'm bad that way
04:54 PM pcw_home: If I was doing it again, Id make a smart interface like sserial thats discoverable and hides all the low level I/O
04:55 PM jepler: tram.c:
04:55 PM jepler: 114 if(hm2->tram_read_size>old_tram_read_size)
04:55 PM jepler: 115 memset(hm2->tram_read_buffer+old_tram_read_size, 0, hm2->tram_read_size-old_tram_read_size);
04:55 PM andypugh: OK, so now I have something to chase. Excellent
04:55 PM jepler: I don't really know what this means...! unallocated block of size 3,346,864 in arena "client"
04:56 PM jepler: have to run, sorry!
04:56 PM pcw_home: Wow thanks!
04:57 PM andypugh: Thanks, you have pushed things usefully forwards
05:35 PM andypugh: pcw_home: I don’t get the uspace crash with epp if I comment-out lines 115 and 123 in tram.c. (and the if-statements on the line above). Can you try that with the 8i20 config? I have a feeling I might be seeing a different bug. (And deleting those lines makes no sense as far as I can see)
06:23 PM pcw_home: 8I20?
06:25 PM andypugh: 5i20.
06:26 PM andypugh: I get those condused the other way earler today.
06:26 PM pcw_home: Yeah I can try that but only at work
06:26 PM pcw_home: (no PCI cards here)
06:27 PM andypugh: At least now Jepler has found two lines, rather than complicated function calls.
06:28 PM pcw_home: yeah that really narrows it down
08:34 PM jepler: I think I have the "aha"
08:43 PM jepler: pcw_home, andypugh: https://github.com/LinuxCNC/linuxcnc/pull/456
08:43 PM jepler: I wrote the bug :-/
09:05 PM skunkworks: I'm going to write me a new minivan this afternoon