#linuxcnc-devel Logs

Mar 15 2023

12:54 AM linuxcnc-build2: Build [#593](http://buildbot2.highlab.com/buildbot/#builders/11/builds/593) of `10-rip.debian-12-bookworm-rtpreempt-amd64` 4failed.
01:52 AM linuxcnc-build2: Build [#590](http://buildbot2.highlab.com/buildbot/#builders/14/builds/590) of `10-rip.debian-10-buster-rtpreempt-amd64` 4failed.
02:06 AM linuxcnc-build2: Build [#590](http://buildbot2.highlab.com/buildbot/#builders/7/builds/590) of `10-rip.debian-11-bullseye-rtpreempt-amd64` 4failed.
02:18 AM linuxcnc-build2: Build [#591](http://buildbot2.highlab.com/buildbot/#builders/14/builds/591) of `10-rip.debian-10-buster-rtpreempt-amd64` 8completed with warnings.
02:18 AM linuxcnc-build2: Build [#591](http://buildbot2.highlab.com/buildbot/#builders/7/builds/591) of `10-rip.debian-11-bullseye-rtpreempt-amd64` 8completed with warnings.
02:24 AM linuxcnc-build2: Build [#594](http://buildbot2.highlab.com/buildbot/#builders/11/builds/594) of `10-rip.debian-12-bookworm-rtpreempt-amd64` 8completed with warnings.
05:07 AM -!- #linuxcnc-devel mode set to +v by ChanServ
05:38 AM -!- #linuxcnc-devel mode set to +v by ChanServ
05:57 AM -!- #linuxcnc-devel mode set to +v by ChanServ
06:08 AM -!- #linuxcnc-devel mode set to +v by ChanServ
06:28 AM -!- #linuxcnc-devel mode set to +v by ChanServ
06:32 AM -!- #linuxcnc-devel mode set to +v by ChanServ
06:43 AM -!- #linuxcnc-devel mode set to +v by ChanServ
07:35 AM -!- #linuxcnc-devel mode set to +v by ChanServ
07:51 AM -!- #linuxcnc-devel mode set to +v by ChanServ
08:27 AM -!- #linuxcnc-devel mode set to +v by ChanServ
08:42 AM -!- #linuxcnc-devel mode set to +v by ChanServ
08:42 AM -!- #linuxcnc-devel mode set to +v by ChanServ
08:44 AM -!- #linuxcnc-devel mode set to +v by ChanServ
09:00 AM -!- #linuxcnc-devel mode set to +v by ChanServ
09:25 AM -!- #linuxcnc-devel mode set to +v by ChanServ
09:39 AM -!- #linuxcnc-devel mode set to +v by ChanServ
09:43 AM -!- #linuxcnc-devel mode set to +v by ChanServ
10:04 AM seb_kuzminsky: pcw-home: it looks to me like USB3 might have good enough latency and throughput to be usable as an interconnect for an hm2 fpga
10:04 AM seb_kuzminsky: folks are talking about sub-100-µs round trip times and multi-Gbps throughput
10:05 AM seb_kuzminsky: USB3 ports are on nearly all computers nowdays
10:05 AM seb_kuzminsky: I havent observed this performance myself, but i'm interested in playing with some hardware and see what it can do
10:08 AM pcw-home: I do think I have narrowed down the location of large network latencies
10:10 AM pcw-home: (limited cable lengths and lack of isolation make USB less interesting for me)
10:11 AM pcw-home: Also I don't know if it applies to USB3 but the need to re-enumerate on faults is a killer for real time
10:17 AM rmu: pcw-home: so you have something to test re. the latencies?
10:18 AM pcw-home: well I have found where the Ethernet latencies have gone bad in recent kernels
10:19 AM pcw-home: its not surprisingly in the network stack (but _not_ the drivers)
10:21 AM rmu: something with send queue scheduling?
10:25 AM pcw-home: Not sure but simply pinging localhost shows the issue
10:26 AM pcw-home: (sudo chrt 99 ping -i .001 -q 127.0.0.1)
10:32 AM JT-Shop: rtt min/avg/max/mdev = 0.001/0.006/0.041/0.001 ms
10:33 AM rmu: on 6.0.0-0.deb11.6-rt-amd64 I get rtt min/avg/max/mdev = 0.011/0.029/1.787/0.014 ms (bullseye backport)
10:34 AM pcw-home: Yep...
10:34 AM rmu: 4.19.0-5-rt-amd64 different hardware has rtt min/avg/max/mdev = 0.013/0.013/0.036/0.003 ms
10:35 AM rmu: non-realtime thinkpad gets rtt min/avg/max/mdev = 0.002/0.004/0.151/0.004 ms with 5.15 ubuntu dist kernel
10:36 AM pcw-home: I think this is a post 4.19 (or so) issue
10:37 AM rmu: i'm going to reboot the 6.0 machine into something earlier... brb
10:37 AM pcw-home: but you need to test for a fairly long time
10:41 AM rmu: 1.787 max localhost ping time was while running for 3 min
10:42 AM rmu: did anybody try user space ip stacks?
10:43 AM pcw-home: you mean netns?
10:43 AM seb_kuzminsky: probably you need a preempt_rt kernel, isolcpus=N in the cmdline, and add `taskset -c N` to the commandline to run the ping on the isolated RT cpu
10:46 AM pcw-home: I can try but I don't think a 2 ms delay is from CPU-CPU
10:47 AM pcw-home: (and why only with newer kernels?)
10:47 AM rmu: i mean something like https://openfastpath.org/
10:48 AM rmu: there are probably a bunch of userspace IP stacks developed for high frequency trading and similar applications
10:48 AM pcw-home: Yeah, unfortunately those tend to work only with specific MACs
10:51 AM rmu: so with 4.19 the machine that had 1.78ms worst case has rtt min/avg/max/mdev = 0.011/0.012/0.057/0.003 ms after a few minutes
10:53 AM pcw-home: That has been my experience (something went wrong in the network stack sometime post 4.19)
10:59 AM JT-Shop: pcw-home, how did you get 2333333333 from 2333 MHz?
11:02 AM pcw-home: You mean the extra 3s?
11:03 AM rmu: i suppose rt preempt has to do only with process scheduling and minimising wakeup-latency, and doesn't have an effect on network packet scheduling
11:03 AM pcw-home: 2333 MHz = 2333000000 Hz
11:03 AM pcw-home: And process priority
11:04 AM pcw-home: (preempting lower priority tasks)
11:05 AM rmu: it can preempt lower priority tasks even while they are in a syscall that would not be preemptible without the preempt-rt patch
11:05 AM pcw-home: Yes
11:07 AM pcw-home: And this all worked on network stacks until sometime after 4.19
11:09 AM pcw-home: The rate of latency excursions is pretty low on the latest kernels so seems acceptable on fast hardware (I'm running 6.2)
11:11 AM pcw-home: (a couple packets dropped a day which don't really cause an issue) but it was so much better before
11:15 AM JT-Shop: thanks
11:17 AM JT-Shop: yippie I got now :)
11:35 AM rmu: would be interesting to look at that latency problem from the other side. i.e. ping the linux kernel from FPGA and look at delay introduced by different kernels
11:35 AM rmu: with ping, it could be something in the networking stack, or it could be scheduling latency affecting ping
11:35 AM rmu: pinging from outside would eliminiate the process scheduling
11:36 AM pcw-home: Sebastians taskset thing does seem to work on 6.2 (so far)
11:36 AM pcw-home: 76319 packets transmitted, 76319 received, 0% packet loss, time 80238ms
11:36 AM pcw-home: rtt min/avg/max/mdev = 0.002/0.004/0.047/0.002 ms
11:37 AM pcw-home: sudo taskset -c 3 chrt 99 ping -i .001 -q 127.0.0.1
11:37 AM pcw-home: (isolcpus=3)
12:00 PM linuxcnc-build2: Build [#598](http://buildbot2.highlab.com/buildbot/#builders/11/builds/598) of `10-rip.debian-12-bookworm-rtpreempt-amd64` 4failed.
12:12 PM rmu: quite odd this ping consumes >90% of isolated CPU on my test
12:16 PM pcw-home: that's marvelously inefficient...
12:17 PM rmu: strange thing linuxcnc with 7i97 runs fine
12:28 PM pcw-home: LinuxCNC doesnt seem to use more than about 10% on this PC (including the ethernet IRQ)
12:33 PM pcw-home: (1 KHz servo thread)
12:34 PM rmu: ping is doing something strange. cpu of ping with 10ms is down in the noise, with 5ms and faster it is ~100%
12:45 PM pcw-home: Yeah, maybe it just busywaits if the time is less than 5 ms
12:47 PM pcw-home: runs fine at 100 KHz :-)
12:48 PM linuxcnc-build2: Build [#595](http://buildbot2.highlab.com/buildbot/#builders/7/builds/595) of `10-rip.debian-11-bullseye-rtpreempt-amd64` 4failed.
12:51 PM linuxcnc-build2: Build [#595](http://buildbot2.highlab.com/buildbot/#builders/14/builds/595) of `10-rip.debian-10-buster-rtpreempt-amd64` 4failed.
01:46 PM linuxcnc-build2: Build [#596](http://buildbot2.highlab.com/buildbot/#builders/14/builds/596) of `10-rip.debian-10-buster-rtpreempt-amd64` 8completed with warnings.
01:46 PM linuxcnc-build2: Build [#596](http://buildbot2.highlab.com/buildbot/#builders/7/builds/596) of `10-rip.debian-11-bullseye-rtpreempt-amd64` 8completed with warnings.
02:06 PM linuxcnc-build2: Build [#599](http://buildbot2.highlab.com/buildbot/#builders/11/builds/599) of `10-rip.debian-12-bookworm-rtpreempt-amd64` 8completed with warnings.
03:30 PM -!- #linuxcnc-devel mode set to +v by ChanServ
04:38 PM -!- #linuxcnc-devel mode set to +v by ChanServ